Techniques for using 3D avatars in augmented reality messaging

Abstract

Techniques are described herein for facilitating transfer of text-based messages between end users that are using messaging applications executing on client-based computing devices with different capabilities. Specifically, a messaging system described herein enables a first end user to add a message element to a text-based message that is received by a message recipient using an augmented reality messaging application, the text-based message will cause a 3D avatar representing the message sender to be animated according to a particular avatar animation associated with the message element. The message element may be an emoticon or a special sequence of characters, and may be a visible or invisible (e.g., metadata) element of a text-based message.

Description

Priority Statement

[0001] This application claims the benefit of priority to U.S. Patent Application Serial No. 18 / 310,282, filed May 1, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to an online or Internet-enabled messaging system that facilitates messaging between end users using messaging applications running on different types of computing devices, including devices with augmented reality capabilities. Background Technology

[0003] Augmented reality (AR) devices, including AR glasses and AR headsets, are becoming increasingly popular due to their ability to provide immersive and interactive experiences to end users. These immersive and interactive experiences can enhance a wide range of activities, including gaming, entertainment, education, training, and productivity. However, because AR devices are fundamentally different from conventional computing devices—specifically, in the mechanisms by which end users provide input and receive output—adapting existing software applications and systems to work with AR devices presents various challenges. Attached Figure Description

[0004] In the accompanying drawings (which are not necessarily drawn to scale), similar reference numerals may describe similar parts in different views. To facilitate identification of any particular element or action being discussed, one or more of the highest-order digits in the reference numerals indicate the drawing number in which the element was first introduced. Some non-limiting examples are shown in the accompanying drawings:

[0005] Figure 1 This is a diagram illustrating an example of two end users exchanging messages via an improved messaging system, according to some examples. The two end users include a first end user using a messaging application executed on a mobile computing device (e.g., a mobile phone), and a second end user wearing an augmented reality (AR) device on which the messaging application is executed.

[0006] Figure 2 This is a diagram illustrating an example of a computer network environment with an interactive server system, consistent with some examples, to which an improved messaging system can be integrated.

[0007] Figure 3 This is another example of an improved messaging system consistent with some of the examples, along with its integrated interactive server system.

[0008] Figure 4This is a diagram illustrating an example of functional components of an improved messaging system consistent with some examples.

[0009] Figure 5 This is a diagram illustrating example data structures for use with an improved messaging system, based on some examples.

[0010] Figure 6 This is a diagram showing the components of a message consistent with some examples.

[0011] Figure 7 This is a data flow diagram illustrating, based on some examples, the various operations that occur during a messaging session between end users using messaging applications executed on different types of computing devices.

[0012] Figure 8 This is an illustration of an end-user wearing an AR device, consistent with some examples, performing actions to "anchor" or "fix" a 3D avatar in AR space.

[0013] Figure 9 This is an example user interface diagram illustrating a user interface based on some examples, through which a first end user can prepare a text-based message with a sequence of characters mapped to an avatar animation, which will be transmitted to a second end user wearing an AR device.

[0014] Figure 10 This is an example user interface diagram illustrating a user interface based on some examples, through which a first end user can prepare a text-based message with emojis mapped to avatar animations.

[0015] Figure 11 This is an illustration of an example of an end user wearing an AR device consistent with some examples, where the AR device presents an AR view to the end user that includes an anchored 3D avatar with chat bubbles, the chat bubbles including a series of dots to indicate that the end user represented by the avatar is typing a message.

[0016] Figure 12 This is an example diagram illustrating an end user of a wearable AR device, which presents the end user with an avatar animation as a result of processing message elements included in the message.

[0017] Figure 13 This is an example diagram showing an AR view and corresponding user interface consistent with some examples, where the user interface includes an icon indicating that an audio recording device has been enabled to capture audio input.

[0018] Figure 14This is an example diagram illustrating an AR view based on some examples, which includes a user interface showing a text-based message derived from audio captured using an audio input device.

[0019] Figure 15 This is a diagram showing two different shape factors for AR devices (including AR glasses and AR headsets) consistent with some examples.

[0020] Figure 16 The diagram illustrates a system including a user system, which in this example is a head-mounted device or AR device with a selector input device.

[0021] Figure 17 It is a block diagram illustrating the software architecture that can be installed on any one or more of the devices described herein.

[0022] Figure 18 It is a graphical representation of a machine in the form of a computer system, based on some examples, within which a set of instructions can be executed to enable the machine to perform any or more of the methods discussed herein. Detailed Implementation

[0023] This document describes techniques (including both systems and methods) for facilitating the exchange of text-based messages between end users using messaging applications running on different types of computing devices with varying capabilities, employing improved messaging systems. More specifically, the improved messaging systems described herein facilitate message exchange between two or more end users, wherein at least one end user is using a messaging application running on a wearable augmented reality (AR) device. Using a messaging application running on an AR device, end users can send and receive messages by interacting with an avatar representing another end user (e.g., a virtual object), thereby creating an immersive and interactive messaging experience. In the following description, numerous specific details and features are set forth for illustrative purposes in order to provide a thorough understanding of various aspects of different examples. However, it will be apparent to those skilled in the art that the invention can be practiced and / or implemented through different combinations of the numerous details and features presented herein.

[0024] Wearable AR devices (including AR glasses and AR headsets) can provide immersive experiences to end users by blending virtual objects with views of the real-world environment. However, adapting existing software applications, systems, and services for use with AR devices can be technically challenging. Consider, for example, messaging applications. Messaging applications developed for conventional computing devices (e.g., mobile phones, desktop computers, and laptops) typically utilize a physical keyboard or touchscreen display, allowing end users to provide text-based input to both the computing device and the messaging application. Thus, the end user can easily type text-based messages using their finger or stylus for transmission to another end user via the messaging system. Furthermore, received messages are presented via the conventional display of the computing device where they were received.

[0025] Unlike conventional computing devices, wearable AR devices do not have the same user input and output mechanisms. Instead, AR devices receive input via simple buttons, audible or voice commands, hand gestures, and, in some cases, custom-designed manual controllers. Typically, these input mechanisms are less well-suited for receiving text-based input. Furthermore, AR devices often use some type of transparent or see-through display to overlay virtual objects or content onto the end-user's view of the real-world environment. Therefore, if the same two-dimensional user interface used with messaging applications on conventional mobile computing devices is simply scaled up for presentation via the AR device's display, the end-user experience will be far from interactive or immersive, as the user interface may obscure significant portions of the view of the real-world environment, thus offering little (if any) benefit to using messaging applications with AR devices.

[0026] To address these and other issues, this paper proposes an improved messaging system and application. The messaging system is backward compatible with existing messaging systems, thus providing end users choosing to use AR devices with the ability to communicate with other end users using conventional computing devices (e.g., mobile phones, desktop computers, or laptops). However, instead of simply presenting a two-dimensional user interface "floating" in AR space as displayed by the AR device's display, the improved messaging application for AR devices enables end users to interact with another end user in the messaging system via a 3D avatar representing that other end user. Using AR devices, the 3D avatar of another end user can be anchored or fixed to a specific location in the real-world environment.

[0027] For example, messaging applications for AR devices allow end users of AR devices to access and view 3D avatars of other end users, where each 3D avatar is a digital representation of another end user in the messaging service, configured by that end user. An end user of an AR device can access a contact list (sometimes called a friends list) to view other end users and then select a specific contact or friend from the list. Once a selection is made, a 3D avatar representing the selected end user is presented via the AR device's display. The end user of an AR device can then anchor or pin the other end user's 3D avatar to a location in the real-world environment, creating an AR space in which the 3D avatar will be presented. Therefore, an end user of an AR device can position one or more avatars in various real-world locations where the end user is most likely to be communicating with another end user represented by the avatar. For example, if an end user tends to communicate with a specific friend via a messaging application when the end user is in his or her kitchen, that end user might want to anchor or pin the friend's 3D avatar to a location in AR space representing a position on a countertop in his or her kitchen. Then, when an end user is in the kitchen and wearing an AR device, they will be able to view and interact with 3D avatars representing their friends. Similarly, an end user can anchor or pin avatars of one or more colleagues to their desktop, making it easy for them to interact with their colleagues via these representative 3D avatars while working at their desk and wearing an AR device.

[0028] In some examples, the 3D avatar's location persists across messaging sessions after another end-user's 3D avatar has been anchored or fixed within AR space. For instance, if an end-user of an AR device's messaging application leaves the real-world environment associated with the AR space where the 3D avatar was anchored, the AR device will perform a process called relocalization when the end-user later returns to the real-world environment. During relocalization, the AR device uses computer vision algorithms and image analysis to identify objects in the real-world environment, enabling it to associate the AR space with the real-world environment and re-render the virtual content associated with the AR space already created for that real-world environment. Specifically, upon determining that the AR device is once again located in the real-world environment where the end-user's avatar was previously anchored or fixed, the AR device will again render the 3D avatar in the AR space at the same location or position as previously anchored and fixed.

[0029] In some examples, the messaging system provides improved presence and activity detection, as well as status indications. For instance, a messaging application running on each client computing device can periodically transmit presence or status data and / or activity data to the server running the messaging system, allowing the system to provide each end user with detailed status information about other end users. Specifically, when an end user of a messaging application running on an AR device wears the AR device and is online relative to the messaging service, other end users can receive updates to their messaging application's user interface to reflect the end user's status. Furthermore, if a first end user wearing an AR device is in a real-world environment associated with an AR space where a second end user's 3D avatar is anchored or fixed, the second end user can receive a status indication via the messaging application specifically indicating that his or her 3D avatar is currently viewable by the first end user. Therefore, the second end user will understand that any messages sent by the second end user to the first end user will be delivered to the first end user via the 3D avatar. This allows the second end user to use the AR device to create content best suited to the message recipient.

[0030] Consistent with some examples, messages delivered to an end user of a messaging application running on an AR device may include specific characters or symbols (e.g., emojis) mapped to avatar animations. Thus, a first end user can send a message to a second end user wearing an AR device, causing the first end user's avatar (as viewed by the second end user) to move according to a specific avatar animation corresponding to a specific character or symbol included in the message sent from the first end user to the second end user. For example, in one example, the message sender may include a special character sequence, such as " / w" or "::w", where a subset of the initial characters indicates that the character sequence is intended as a specific command or instruction for animatening the avatar, and one or more subsequent characters identify the specific avatar animation. In this example, "w" could be an abbreviation for "wave," and therefore, the special character sequence (e.g., " / w" or "::w") would cause the message sender's 3D avatar to perform the avatar animation by waving to the message receiver, i.e., the end user viewing the 3D avatar via the AR device. In another example, a subset of emojis may correspond to or be mapped to avatar animations. As an example, the very popular smiling emoji, when sent in a message to an end user using a messaging application running on an AR device, can make the message sender's 3D avatar smile as a result of executing the avatar animation associated with the smiling emoji. Other innovative aspects and advantages of various embodiments of the invention will become apparent from the following description of the various figures.

[0031] Figure 1 This diagram illustrates an example of two end users exchanging messages via an improved server-based messaging system 100. The two end users include a first end user 102 using a messaging application running on a mobile computing device 104 (e.g., a mobile phone) and a second end user 106 wearing an AR device 108 on which a messaging application runs. In this example, the line with reference numeral 110 is intended to represent the physical separation of the two end users. Specifically, the first end user 102 is shown as existing in a first real-world environment, far removed from the second real-world environment in which the second end user exists. The server-based messaging system 100 facilitates the exchange of messages between the client computing devices of the two end users (e.g., mobile phone 104 and AR device 108), the respective client computing devices being wirelessly connected to the messaging system 100.

[0032] Consistent with some examples, the messaging application running on AR device 108 enables end user 106 to anchor or fix a 3D avatar representing another end user in an AR space associated with a real-world environment selected by the first end user. For example, as Figure 1 As shown, the end user 106 of AR device 108 is looking at the right edge of the desktop of the 3D avatar 114 of end user 102, which the end user has anchored or fixed to. (As shown) Figure 1 As shown, the 3D avatar 114 exists in the view of AR space 112 (e.g., as presented via the display device of AR device 108) because the 3D avatar 114 is a virtual object that does not actually exist as a real-world object in the physical real-world environment. In this example, the two end users can be colleagues, and therefore, when end user 106 is actively engaged in various work tasks at his or her desk, end user 106 of AR device 108 can frequently send messages to end user 102 of mobile phone 104. Thus, by anchoring or fixing the 3D avatar 114 of a colleague (e.g., end user 102) to the desktop, end user 106 will be able to interact with his or her colleague (e.g., end user 102) via messaging applications and the 3D avatar 114 each time end user 106 of AR device 108 is wearing AR device 108 and sitting at his or her desk.

[0033] like Figure 1 As shown, AR device 108 is presenting a realistic augmented view to end user 106. This view, referred to herein as the AR view, is represented by a dashed ellipse with reference numeral 112. For example, the end user is shown looking towards the right edge of the desktop. Figure 1The portion enclosed in the dashed ellipse 112 is an AR view of the desktop generated by AR device 108 and presented to end user 106. In this example AR view 112, a 3D avatar 114 representing a colleague or associate (e.g., end user 102) appears positioned on top of the flat surface provided on the desktop, as if the 3D avatar 114 were standing on the desktop. In this case, the 3D avatar 114 is a digital representation of another end user 102 with whom they exchange messages. For example, consistent with some examples, the 3D avatar 114 is created by end user 102 of mobile phone 104. Before initiating a messaging session with remote end user 102, end user 106 wearing AR device 108 can use a messaging application executed on AR device 108 to place or position the 3D avatar 114 of end user 102 on his or her desktop—an AR operation commonly referred to as “anchoring” or “pinning” a virtual object. Once the 3D avatar is anchored or pinned, a digital representation of the real-world environment is created. This digital representation is called AR space. Therefore, when the end user 106 is in a real-world environment corresponding to the AR space in which the 3D avatar 114 has been anchored, the anchored 3D avatar 114 will be presented and will appear in the AR view 112 of the AR space generated by the AR device 108.

[0034] Consistent with some examples, when end user 102 using mobile phone 104 prepares a text-based message to send to end user 106 wearing AR device 108, end user 102 can add instructions or commands to the message that will animate the 3D avatar 114 presented via AR device 108 in a specific manner. Thus, the text-based message can be presented in a chat bubble displayed next to or near (e.g., close to) the 3D avatar 114, while the 3D avatar 113 performs animated movement. The instruction or command added to the message that ultimately animates the 3D avatar can be a special character or sequence of characters, or a symbol (e.g., an emoji). When messaging system 100 receives an incoming text-based message intended for an end user of the AR device, it will, for example, animate the presentation of avatar 114 according to a specific avatar animation corresponding to the instruction or command, interpreting emojis or special character sequences as instructions or commands that modify the presentation of 3D avatar 114. In one example, the avatar animation can be mapped to a specific character sequence. In another example, avatar animations can be mapped to emojis. In some examples, the command or instruction for an avatar animation can be a visible message element, causing the command to appear in the original message when sent. However, in other cases, the message sender can specify the avatar animation so that the command or instruction is delivered as metadata rather than as a visible message element (e.g., as part of the body of the actual text-based message). For example, in one example, the user interface presented via a messaging application can provide a special set of icons, where each icon represents a specific avatar animation. Selecting an icon or graphic allows the command or instruction to be added as metadata to the message to be sent. Furthermore, the message sender may be able to select an icon to preview the avatar animation before adding the instruction to the message.

[0035] Each avatar animation mapped to an emoji or special character sequence can be represented as a video file, which, when processed by the AR device, will cause the corresponding avatar animation to be displayed via the display device of AR device 108. In some examples, the avatar animation may include an audio component, allowing the 3D avatar to deliver a voice message as part of the avatar animation. In other cases, text-based messages can be converted into audio messages, allowing the 3D avatar to speak the audio message to a message recipient wearing the AR device. For example, the 3D avatar may speak the audio message before, after, or in some cases during the presentation of the avatar animation (i.e., while processing and presenting the media file of the avatar animation).

[0036] Typically, each avatar animation may correspond to human movement or superhuman movement. For example, an avatar animation may be a brief facial expression (e.g., smiling, frowning, expressing pleasure, excitement, doubt, or frustration) or include brief facial expressions. Similarly, in some examples, avatar animation may involve the avatar posing in a specific posture or making a specific body movement or gesture (e.g., jumping up and down, waving a fist, waving one or both arms, and performing a military-style salute). In another example, avatar animation may include actions performed using props, such as swinging a baseball bat, dribbling a basketball, etc.

[0037] Consistent with some examples, certain avatar animations are only selectable by the message sender when the message sender is in a specific geographic area or when the message sender is in the location where a specific event is taking place. For example, a messaging system can utilize location data generated by client computing devices running messaging applications on it. Using this location data, and by querying a database of known events (e.g., live performances, sporting events, etc.), the messaging system can make specific avatar animations available based on time and location data. Therefore, the message sender can select specific avatar animations associated with their current location or the event they are attending to convey the message sender's current location or activity to the message receiver.

[0038] As described in more detail below, a messaging system can be a component of a broader interactive system that facilitates various types of interactions, where text-based messaging is only one type of interaction. When implemented as part of an interactive system, each client-based messaging application can be just one of several different client applications, each utilizing some core functionalities provided by the interactive client. The following section combines... Figure 2 The description provides details of such a system.

[0039] Networked computing environment

[0040] Figure 2This is a block diagram illustrating an example interactive system 200 for facilitating interactions over a network, such as exchanging messages, making audio and video calls, creating and configuring avatars, or playing games. Interactive system 200 includes multiple user systems (e.g., user systems 202-A and 202-B), each hosting multiple applications, including interactive clients 204-A and 204-B, and other applications 206-A and 206-B. Each interactive client is communicatively coupled to other instances of the interactive client (e.g., hosted on corresponding other user systems) via one or more communication networks (e.g., network 210, which may be the Internet or include the Internet), interactive server system 212, and third-party server 214. Interactive client 204-A may also communicate with locally hosted application 206-A, for example, using one or more application programming interface (API) calls.

[0041] Each user system 202-A and 202-B can be one of several different supported device types. Here, device type is a designation that can indicate not only the form factor of the device but also its capabilities. For example, in Figure 2 In this diagram, user system 202-A is shown as a mobile computing device (e.g., a mobile phone or similar digital assistant). User system 202-B is shown as an AR device, specifically, wearable AR glasses. Of course, the interactive server system 212 can also support other device types, including AR headsets, virtual reality headsets, laptop computing devices, and desktop computing devices. The interactive client 204-A and associated application 206-A installed on each device are typically developed and built to be device-specific, such that each interactive client is configured to support the type of device on which it is installed and executed. Therefore, the version of interactive client 204-A and application 206-A installed and executed on a mobile computing device such as user system 202-A will differ to some extent from the version of interactive client 204-B and application 206-B installed and executed on an AR device such as user system 202-B.

[0042] Each interactive client 204-A interacts with other interactive clients 204-B via network 210 and with interactive server system 212. The data exchanged between interactive clients (e.g., interactive 124) and between interactive clients and interactive server system 212 includes functions (e.g., commands to activate functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0043] Interactive server system 212 provides server-side functionality to interactive clients 204-A and 204-B via network 210. While some functions of interactive system 200 are described herein as being performed by interactive clients 204-A and 204-B or by interactive server system 212, the location of certain functions within interactive clients 204-A and 204-B or interactive server system 212 may be a design choice. For example, it may be technically preferred that specific technologies and functions are initially deployed within interactive server system 210, but later migrated to interactive clients 204-A and 204-B, where user systems 202-A and 202-B have sufficient processing power. As an example, a messaging application running on an AR device can process audio recordings captured using audio input (e.g., a microphone) to convert the audio recordings into text-based messages, for example, using a speech-to-text algorithm executed at the AR device. However, due to the potential power and / or processing limitations of some AR devices, in some examples, audio recordings captured by the AR device can be transmitted to a server via network 210, where a speech-to-text translation service performed on the server side translates the captured audio into text. The resulting text-based message can then be forwarded to one or more intended message recipients.

[0044] The interactive server system 212 supports various services and operations provided to interactive clients. These operations include sending data to and receiving data from interactive clients, as well as processing data generated by interactive clients. This data may include message content, client device information, geolocation information, media enhancements and overlays, avatar animation files, message content persistence conditions, entity relationship information, and live event information. Data exchange within the interactive system 200 is activated and controlled through functions available via the user interface (UI) of interactive clients 204-A and 204-B.

[0045] Now, specifically, the interaction server system 212 is used. An application programming interface (API) server 216 is coupled to the interaction server 218 and provides it with a programming interface, making the functionality of the interaction server 218 accessible to interactive clients 204-A and 204-B, other applications 206-A and 206-B, and a third-party server 214. The interaction server 218 is communicatively coupled to a database server 220, thereby facilitating access to a database 222 that stores data associated with the interactions processed by the interaction server 218. Similarly, a web server 224 is coupled to the interaction server 218 and provides a web-based interface to it. For this purpose, the web server 224 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0046] API server 216 receives and sends interactive data (e.g., command and message payloads) between interactive server 218 and user systems 202-A and 202-B (as well as interactive clients 204-A and 204-B and other applications 206-A and 206-B) and third-party server 214. Specifically, API server 216 provides a set of interfaces (e.g., addressable API endpoints) for activating commands, functions, routines, and accessing data sources. Each API call activates a function provided by interactive server 218. API server 216 exposes various functions supported by interaction server 218, including account registration; login functionality; sending interactive data from a specific interactive client 204-A to another interactive client 204-B via interaction server 218; transferring media files (e.g., images or videos) from interactive client 204-A to interaction server 218; setting up a collection of media data (e.g., stories); retrieving friend lists of end users of user systems 202-A and 202-B; retrieving messages and content; adding and deleting entities (e.g., friends) in entity relationship graph 320; locating friends in the entity relationship graph; and opening application events (e.g., related to interactive clients 204-A and 204-B).

[0047] Interactive server 218 hosts multiple systems and subsystems, including an improved messaging system 312, as described below. Figure 3 and Figure 4 Describe it.

[0048] System Architecture

[0049] Figure 3 This is a block diagram illustrating further details of an interactive system 200 based on some examples. Specifically, the interactive system 200 is shown as including an interactive client 204-B and an interactive server 218. The interactive system 200 includes multiple subsystems supported on the client side by the interactive client 204-B and on the server side by the interactive server 218. In some examples, these subsystems are implemented as microservices. A microservice subsystem (e.g., a microservice application) may have components that enable the microservice subsystem to operate independently and communicate with other services. Example components of a microservice subsystem might include:

[0050] Functional logic: Functional logic implements the functions of the microservice subsystem and represents the specific capabilities or functions provided by the microservice.

[0051] API Interface: Microservices can use lightweight protocols such as REST or messaging to communicate with other components through well-defined APIs or interfaces. API interfaces define the inputs and outputs of a microservice subsystem and how it interacts with other microservice subsystems in the interactive system 200.

[0052] Data storage: The microservice subsystem can be responsible for its own data storage, which can be in the form of a database, cache, or other storage mechanism (e.g., using database server 220 and database 222). This allows the microservice subsystem to operate independently of other microservices in the interactive system 200.

[0053] Service discovery: Microservice subsystems can find and communicate with other microservice subsystems in the interactive system 200. The service discovery mechanism enables microservice subsystems to locate and communicate with other microservice subsystems in a scalable and efficient manner.

[0054] Monitoring and logging: Microservice subsystems may need to be monitored and logged to ensure availability and performance. Monitoring and logging mechanisms enable the tracking of the health and performance of microservice subsystems.

[0055] In some examples, the interactive system 200 can adopt a monolithic architecture, a service-oriented architecture (SOA), a function-as-a-service (FaaS) architecture, or a modular architecture:

[0056] Image processing system 302 provides various functions that enable end users to capture and enhance (e.g., annotate or otherwise modify or edit) media content associated with a message. Camera device system 304 includes (e.g., in a camera device application) control software that (e.g., directly or via an operating system) interacts with and controls the camera device hardware of user system 202-B to modify and enhance real-time images captured and displayed via interactive client 204-B.

[0057] Enhancement system 306 provides functionality related to the generation and distribution of enhancements (e.g., media overlays) of images captured in real time by the camera device of user system 202-B or retrieved from the memory of user system 202-B. For example, enhancement system 306 is operable to select, present, and display media overlays (e.g., image filters or image lenses) to interactive client 204-B for enhancing real-time images received via camera device system 304 or stored images retrieved from the memory of user system 202-B. These enhancements are selected by enhancement system 306 based on multiple inputs and data, such as:

[0058] The geographic location of user system 202-B; and

[0059] Entity relationship information of end users in user system 202-B; and

[0060] In the context of user system 202-B as an AR device, virtual objects (including 3D avatars) that have been anchored or fixed to AR space.

[0061] Enhancements may include audio and visual content and visual effects. Examples of audio and visual content include images, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos or videos) at user system 202-B for use in messaging, or to video content such as video content streams or feeds sent from interactive client 204-B. Therefore, image processing system 302 can interact with and support various subsystems of communication system 310, such as messaging system 312, audio communication system 314, and video communication system 316.

[0062] Consistent with some examples, augmentation system 306 can operate in conjunction with camera device system 304 to provide AR tracking capabilities and to render virtual objects in AR space. Therefore, images captured via camera device system 304 can be analyzed to derive a digital model or representation of the real-world environment. Augmentation system 306 can then use the digital representation of the real-world environment to anchor or fix virtual objects in AR space and perform various AR techniques such as repositioning, where the AR device analyzes the real-world environment to determine whether an existing AR space has previously been generated to correspond to the real-world environment.

[0063] Media overlays may include text or image data that can be superimposed on photographs taken by user systems 202A or 202-B, or on video streams produced by user systems 202-A or 202-B. In some examples, media overlays may be location overlays (e.g., Venice Beach), names of live events, or names of businesses (e.g., Beach Cafe). In other examples, image processing system 302 uses the geolocation of user system 202-B to identify media overlays that include the names of businesses located at the geolocation of user system 202-B. Media overlays may include additional tags associated with businesses. Media overlays may be stored in database 222 and accessed through database server 220.

[0064] Image processing system 302 provides a user-based publishing platform that allows end users to select a geographic location on a map and upload content associated with that location. End users can also specify scenarios where specific media overlays should be provided to other end users. Image processing system 302 generates a media overlay that includes the uploaded content and associates it with the selected geographic location.

[0065] The Augmented Creation System 308 supports augmented reality developer platforms and includes applications for content creators (e.g., artists and developers) to create and publish interactive clients 204-B that enhance (e.g., augmented reality experiences). The Augmented Creation System 308 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking technologies, and templates.

[0066] In some examples, the enhancement creation system 308 provides a merchant-based publishing platform that allows merchants to select specific enhancements associated with geolocation through a bidding process. For instance, the enhancement creation system 308 associates the media overlay of the highest-bidder merchant with a corresponding geolocation for a predefined amount of time.

[0067] Communication system 310 is responsible for enabling and processing various forms of communication and interaction within interactive system 200, and includes messaging system 312, audio communication system 314, and video communication system 316. Messaging system 312 is responsible for enabling temporary or time-limited access to content by interactive clients 204-A and 204-B. Messaging system 312 includes (e.g., in a short-lived timer system) multiple timers that selectively enable access to (e.g., for presentation and display) messages and associated content via interactive client 204-B based on duration and display parameters associated with a message or set of messages (e.g., a story). Audio communication system 314 enables and supports audio communication (e.g., real-time audio chat) between multiple interactive clients. Similarly, video communication system 316 enables and supports video communication (e.g., real-time video chat) between multiple interactive clients.

[0068] The user management system 318 is operationally responsible for managing user data and profiles, and maintaining entity information about the end users of the interactive system 200 and the relationships between them (e.g., stored in entity table 506, entity relationship diagram 508, and profile data 516).

[0069] Map system 322 provides various geographic location (e.g., geolocation) functions and supports the presentation of map-based media content and messages by interactive clients 204-A and 204-B. For example, map system 322 enables the display (e.g., stored in profile data 516) of end-user icons or avatars on a map to indicate the current or past locations of the end-user's "friends" within the map context, as well as media content generated by such friends (e.g., a collection of messages including photos and videos). For example, on the map interface of interactive clients 204-A and 204-B, messages posted by the end-user from a specific geographic location to interactive system 200 can be displayed to the end-user's "friends" within the context of that specific location on the map. The end-user can also share his or her location and status information with other end-users of interactive system 200 via interactive clients 204-A and 204-B (e.g., using appropriate status avatars), where the location and status information is similarly displayed to the selected end-users within the context of the map interface of interactive clients 204-A and 204-B.

[0070] Game system 324 provides various game functions within the context of interactive clients 204-A and 204-B. Interactive clients 204-A and 204-B provide a game interface that offers a list of available games that can be launched by an end user within the context of the interactive client and played with other end users of interactive system 200. Interactive system 200 also enables specific end users to invite other end users to participate in a specific game by sending invitations from interactive clients 204-A and 204-B. Interactive clients 204-A and 204-B also support sending and receiving voice, video, and text messages (e.g., chat) within the context of playing the game, provide leaderboards for the game, and also support providing in-game rewards (e.g., game currency and items).

[0071] Artificial intelligence and machine learning system 326 provides various services to different subsystems within interactive system 200. For example, AI and machine learning system 326 operates in conjunction with image processing system 302 and camera device system 304 to analyze images and extract information, such as objects, text, or faces. This information can then be used by image processing system 302 to enhance, filter, or manipulate images. AI and machine learning system 326 can be used by enhancement system 306 to generate enhanced content and augmented reality experiences, such as adding virtual objects or animations to real-world images. Communication system 310 and messaging system 312 can use AI and machine learning system 326 to analyze communication patterns and provide insights into how end users interact with each other, as well as to provide intelligent message classification and tagging, such as classifying messages based on sentiment or topic. AI and machine learning system 326 can also provide chatbot functionality for message interactions between user systems 202-A and 202-B, and between user system 202-A and interactive server system 212. Artificial intelligence and machine learning system 326 can also cooperate with audio communication system 314 and / or messaging system 312 to provide speech recognition and natural language processing capabilities (including speech-to-text and text-to-speech capabilities), enabling end users to interact with interactive system 200 using audio (e.g., voice) commands.

[0072] Message sending and receiving system

[0073] Figure 4 This is a diagram illustrating an example of the functional components of an improved messaging system 312, consistent with some examples. (See diagram for example.) Figure 4 As shown, the messaging system 312 includes an presence and activity detection system 400 and a message routing system 402. The presence and activity detection system 400 receives status and activity data from client computing devices executing instances of interactive clients 204-A and 204-B and / or messaging applications. In one example, a background processing task executed as part of the messaging application actively monitors end-user activity at the client computing device and then generates status and activity data, which is wirelessly transmitted over a network to the presence and activity detection system 400 of the messaging system 312. The status and activity data obtained at the client device and transmitted to the presence and activity detection system 400 may include data indicating any of the following:

[0074] General end-user status information (e.g., end-user login (not logged in) message sending and receiving services).

[0075] Activity status (e.g., the end user is actively using a messaging application, actively typing a message, or actively dictating or speaking an audible message.)

[0076] Device type (e.g., mobile phone, laptop or desktop computer, AR device).

[0077] AR spatial state (e.g., an end user is actively using an AR device and is in a real-world environment associated with an AR space in which avatars have been anchored or fixed).

[0078] Consistent with some examples, the presence and activity detection system 400 updates the corresponding end-user's status record in the user status table of database 412 after receiving status and activity data from the client device. Additionally, the presence and activity detection system 400 can generate instructions and transmit these instructions to other client devices, causing a messaging application executing on the client device to update its user interface to reflect the current status of another end-user. This is particularly advantageous when the end-user status information of the first end-user conveyed to the second end-user reflects the type of client device the first end-user is using. For example, consistent with some examples, the messaging application can indicate to the end-user the specific type of device the other end-user is actively using. Furthermore, in some examples, the status information conveyed to the end-user may include information indicating whether the other end-user is currently viewing an AR space in which the first end-user's avatar is anchored or fixed. Therefore, when an end-user prepares a text-based message, the sending end-user will be able to customize his or her message based on the type of client device used by the message recipient. In the case of AR devices, this means that a message sender may be able to include specific instructions or commands in the message that will cause the avatar corresponding to the message sender to perform a specific avatar animation. For example, if a first end user learns that a second end user is currently using an AR device in a real-world environment or location associated with an AR space where the second end user has anchored or fixed an avatar representing the first end user, the first end user can send a message to the second end user that will cause the avatar to perform a specific avatar animation, such as waving at the second end user.

[0079] like Figure 4 As shown, the message sending and receiving system 312 includes a message routing system 402. Typically, upon receiving a text-based message, the message is made available to any intended message recipient. However, as... Figure 4As shown, the message routing system 402 includes a message content evaluation system 404. The message content evaluation system 404 analyzes received messages and, in some cases, modifies the message or determines whether additional instructions should be transmitted with the message. For example, when the intended recipient of the message is using an AR device, the message content evaluation system 404 analyzes the content of the text-based message to determine whether the message includes message elements that can be associated with an avatar animation. Message elements can be emojis or specific character sequences. In either case, upon detecting the presence of a specific message element in the message, the message content evaluation system 404 can identify a specific avatar animation associated with that message element. The message content evaluation system can then generate metadata or additional instructions to be transmitted to the client device of the message recipient, enabling the client device to process the received metadata or instructions, and in some cases, to cause the avatar to perform an avatar animation.

[0080] like Figure 4 As shown, the avatar command to avatar animation mapping is a table that maps specific message elements to specific avatar animations. For example, a set of emojis can have corresponding avatar animations, so that when a messaging application running on an AR device receives a message containing emojis, the application will process the received emojis by having the avatars perform some animations consistent with the received emojis.

[0081] Consistent with some examples, the message content evaluation system 404 and the avatar command to avatar animation mapping 406 can reside on each client computing device, rather than on the server executing the messaging system or service. Therefore, at least in some examples, the messaging application executing on the AR device will analyze the received message to determine whether it contains any message elements to be processed as special commands to cause the avatar to perform an avatar animation.

[0082] like Figure 4As shown, message routing system 402 includes components for speech-to-text 408 and text-to-speech 410. In some examples, when message routing system 402 receives a text-based message, message routing system 402 can determine that the intended recipient is using an AR device. The text of the received message can be converted into an audio file by text-to-speech component 410. The audio file is then transmitted over the network to a messaging application running at the AR device. When the messaging application running at the AR device receives the audio file, the message can be transmitted to the intended recipient (e.g., the end user of the AR device) by playing back the audio file. In some cases, the avatar can be animated to perform lip-syncing as the audio file is played back. That is, the avatar's mouth and lips can be manipulated to convey to the end user that the avatar is speaking the message played back via the audio file. When the message is transmitted to the messaging application as an audio file, the original text-based message may or may not be transmitted. For example, in some scenarios, this can be a setting configurable by each end user.

[0083] Figure 4 The message routing system 402 of the messaging system 312 shown also includes a speech-to-text component 408. Therefore, consistent with some examples, when an end user records an audio file with a voice message at a messaging application running at the AR device, the audio file can be processed at the AR device and translated into a text-based message. However, in some examples, the audio recording can be sent to the message routing system 402, where the speech-to-text component 408 analyzes the audio and converts it into a text-based message. The text-based message can then be delivered to the end user using a conventional client computing device.

[0084] Consistent with some examples, messaging system 312 supports four specific types or methods of message sending and receiving. These methods include direct text, where the message is text-based. A second messaging method involves text-to-speech messaging, where text input by one end user is converted into an audio message for playback by another end user. A third messaging method involves speech-to-text, where one end user speaks an audio message, which is then converted into text for presentation to another end user. Finally, a fourth messaging method involves direct speech, which can be synchronous or asynchronous. For example, two end users can have a real-time voice conversation, or one end user can record an audio message received by another end user but played back at a later time.

[0085] Data Architecture

[0086] Figure 5This is a schematic diagram illustrating a data structure 500 that can be stored in a database 222 of an interactive server system 200, according to certain examples. Although the contents of the database 222 are shown as including multiple tables, it should be understood that data can be stored in other types of data structures (e.g., object-oriented databases, graph databases, or others).

[0087] Database 500 includes message data stored in message table 502. For any given message, this message data includes at least message sender data, message receiver (or recipient) data, and a payload. See below for reference. Figure 6 Further details describe information that can be included in the message and is contained within the message data stored in message table 502.

[0088] Entity table 506 stores entity data and (e.g., by reference) links to entity relationship diagram 508 and profile data 516. Entities for which records are maintained in entity table 506 can include individuals, company entities, organizations, objects, locations, events, etc. Regardless of entity type, any entity whose data is stored in the interactive server system 212 can be an identified entity. Each entity is assigned a unique identifier and an entity type identifier (not shown).

[0089] Entity relationship graph 508 stores information about the relationships and associations between entities. As an example only, such relationships can be social, professional (e.g., working in a common company or organization), interest-based, or activity-based. Some relationships between entities can be one-way, such as an individual end-user subscribing to digital content from a commercial or publishing end-user (e.g., a newspaper or other digital media channel, or a brand). Other relationships can be two-way, such as the "friendship" relationship between individual end-users of interactive system 200.

[0090] Certain licenses and relationships can be attached to each relationship, and also to each direction of the relationship. For example, a two-way relationship (e.g., a friendship between individual end-users) may include authorization for the posting of digital content items between the individual end-users, but certain restrictions or filters may be imposed on the posting of such digital content items (e.g., based on content characteristics, location data, or time of day data). Similarly, a subscription relationship between an individual end-user and a business end-user may impose varying degrees of restrictions on the posting of digital content from the business end-user to the individual end-user, and may significantly restrict or prevent the posting of digital content from the individual end-user to the business end-user. A specific end-user, as an example of an entity, may (e.g., through privacy settings) record certain restrictions in the record for that entity within entity table 506. Such privacy settings may be applied to all types of relationships in the context of the interaction system 200, or selectively applied to certain types of relationships.

[0091] Profile data 516 stores various types of profile data about a specific entity. Based on privacy settings specified by the specific entity, profile data 516 can be selectively used and presented to other end users of the interaction system 200. In the case of an individual, profile data 516 includes, for example, a username, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a set of such avatar representations) selected or configured by the user, including 3D avatars used in an AR context. The specific end user can then selectively include one or more of these avatar representations within the content of messages transmitted via the interaction system 200 and on map interfaces displayed to other end users by interactive clients 204-A and 204-B. The set of avatar representations can include “status avatars,” which present a graphical representation of a state or activity that the end user can choose to transmit at a specific time.

[0092] In the case that the entity is a group, in addition to the group name, members and various settings for the relevant group (e.g., notifications), the profile data 516 for the group may similarly include one or more avatars associated with the group.

[0093] Database 222 also stores enhancement data, such as overlays or filters, in enhancement table 510. The enhancement data is associated with and applied to videos (video data is stored in video table 504) and images (image data is stored in image table 512).

[0094] In some examples, filters are overlays displayed on top of images or videos during presentation to the receiving end user. Filters can be of various types, including user-selected filters from a set of filters presented to the sending end user by interactive clients 204-A and 204-B while the sending end user is composing a message. Other types of filters include geolocation filters (also known as geographic filters), which can be presented to the sending end user based on geographic location. For example, geolocation filters specific to nearby or particular locations can be presented by the interactive client within the user interface based on geolocation information determined by the user system's Global Positioning System (GPS) unit.

[0095] Other augmented data that can be stored in image table 512 includes augmented reality content items (e.g., corresponding to the application "lens" or augmented reality experience). Augmented reality content items can be real-time special effects and sounds that can be added to images or videos.

[0096] As mentioned above, video table 504 stores video data, which in some examples is associated with messages for which records are maintained within message table 502. Similarly, image table 512 stores image data associated with messages whose message data is stored in entity table 506. Entity table 506 can associate various enhancements from enhancement table 510 with various images and videos stored in image table 512 and video table 504.

[0097] like Figure 4 As mentioned in the description, in some examples, the message routing system 402 may include a message content evaluation system 404 that analyzes the content of the message in transit for the purpose of determining whether a message intended for a recipient using an AR device includes any special message elements that may be mapped to an avatar animation. Therefore, as Figure 5 As shown, consistent with some examples, the avatar command to avatar mapping table 522 is included in the database 22. Consistent with some examples, message content evaluation occurs at the server. However, in other cases, the avatar command to avatar mapping table 522 can be maintained at the server, but copies are distributed to each client computing device, allowing the logic for generating and interpreting messages associated with avatar animations to occur at the client device.

[0098] Data communication architecture

[0099] Figure 6This is a schematic diagram illustrating the structure of message 600 according to some examples, generated by interactive client 204-A for transmission to another interactive client 204-B via interactive server 218. The content of a particular message 600 is used to populate message table 502 within database 222 accessible by interactive server 218. Similarly, the content of message 600 is stored in memory as "in-transit" or "in-flight" data for user system 202-A or 202-B or interactive server 218. Message 600 is shown to include the following example components:

[0100] Message Identifier 602: A unique identifier that identifies message 600.

[0101] Message text payload 604: The text to be generated by the end user via the user interface of the user system and included in message 600.

[0102] Message image payload 606: Image data captured by the camera device component of the user system or retrieved from the memory component of the user system and included in message 600. The image data for the sent or received message 600 can be stored in image table 512.

[0103] Message video payload 608: Video data captured by the camera device component or retrieved from the user system's memory component and included in message 600. The video data for the sent or received message 600 can be stored in image table 512.

[0104] Message audio payload 610: Audio data captured by the microphone or retrieved from the user system's memory components and included in message 600.

[0105] Message enhancement data 612: Enhancement data (e.g., filters, stickers, or other annotations or enhancements) representing enhancements to be applied to the message image payload 606, message video payload 608, or message audio payload 610 of message 600. Enhancement data for the sent or received message 600 can be stored in enhancement table 510.

[0106] Message duration parameter 614: A parameter value, in seconds, indicating the amount of time that the content of the message (e.g., message image payload 606, message video payload 608, message audio payload 610) will be presented to the end user via the interactive client or made accessible to the end user.

[0107] Message geolocation parameter 616: Geolocation data (e.g., latitude and longitude coordinates) associated with the message's content payload. Multiple message geolocation parameter 616 values ​​may be included in the payload, each of which is associated with a content item included in the content (e.g., a specific image within the message image payload 606 or a specific video within the message video payload 608).

[0108] Message Story Identifier 618: An identifier value that identifies one or more sets of content (e.g., "story" identified in a set table) associated with a specific content item in the message image payload 606 of message 600. For example, the identifier value can be used to associate multiple images within the message image payload 606 with multiple sets of content, respectively.

[0109] Message Tag 620: Each message 600 can be labeled with multiple tags, each of which indicates the subject of the content included in the message payload. For example, in the case where a specific image depicts an animal (e.g., a lion) is included in the message image payload 606, a tag value can be included within the message tag 620 indicating the relevant animal. Tag values ​​can be manually generated based on user input, or can be automatically generated using, for example, image recognition.

[0110] Message sender identifier 622: An identifier (e.g., message sending system identifier, email address, or device identifier) ​​indicating the end user of the user system on which message 600 is generated and from which message 600 is sent.

[0111] Message receiver identifier 624: An identifier (e.g., message sending and receiving system identifier, email address, or device identifier) ​​indicating the end user of the user system to which message 600 is addressed.

[0112] The content (e.g., values) of each component of message 600 can be pointers to locations in tables where content data values ​​are stored. For example, image values ​​in message image payload 606 can be pointers to locations (or their addresses) within image table 512. Similarly, values ​​in message video payload 608 can point to data stored in image table 512, values ​​stored in message enhancement data 612 can point to data stored in enhancement table 510, values ​​stored in message story identifier 618 can point to data stored in set table, and values ​​stored in message sender identifier 622 and message receiver identifier 624 can point to user records stored in entity table 506.

[0113] Message sending and receiving between different device types

[0114] Figure 7 This is a flowchart illustrating various operations 700 that occur during a messaging session between two end users using messaging applications executed on different types of computing devices 702 and 704, according to some examples. Figure 7 As shown, a first end user with client device 702 is using a messaging application running on a conventional mobile client device 702 with a touchscreen display. Mobile device 702 can be, for example, a mobile phone. A second end user associated with client device 704 is using a messaging application running on an AR device 704. In this case, AR device 704 is a pair of wearable AR glasses. The exchange of messages between the two devices 702 and 704 is facilitated by a messaging service or messaging system 706, with each device wirelessly connected and communicating with the messaging service or messaging system 706.

[0115] In this example, a text-based messaging session begins when a second end-user wearing AR device 704 performs an anchoring or fixing operation 708 to anchor or fix the 3D avatar representing the first end-user to a location in the real-world environment. Combined with... Figure 8 Anchoring operation 708 is further shown and described.

[0116] like Figure 8As shown, a second end user wearing AR device 704 interacts with the interface of interactive client 204-B or a messaging application running on device 704 to access the 3D avatar of the first end user. While the display device of AR device 704 is presenting the 3D avatar 804, the second end user moves his or her head to look toward a specific location or physical object in the AR space where the second end user wishes to anchor or fix the 3D avatar. In one example, the second end user can press a button on AR device 704 while moving his or her head, thereby signaling to AR device 704 that the end user is attempting to anchor or fix the 3D avatar. As the end user moves his or her head, a computer vision algorithm running on the AR device receives and processes images from the surrounding real-world environment as AR device 704 attempts to identify surfaces (e.g., a horizontal plane) suitable for anchoring or fixing virtual objects (including 3D avatar 804). The AR device 704 uses one or more sensors (e.g., camera devices or image capture devices) to process images frame by frame to identify flat and horizontal surfaces on which the 3D avatar may be anchored or fixed. Figure 8 As shown, the end user is looking to the right of the keyboard 802 on top of the table 800. When the AR device 704 recognizes the flat surface of the tabletop 800, it can provide visual feedback (e.g., an arrow with reference numeral 806) to the end user, indicating that the location in AR space corresponding to the tabletop surface is a candidate location for anchoring or fixing the 3D avatar. This visual feedback can be, for example, some type of graphic or marker displayed at the location on the recognized surface. In one example, a shadow can be generated and projected onto the detected flat surface to support immersion in both the real and digital worlds. Therefore, when the end user brings the appropriate portion of the tabletop surface 800 into his or her line of sight and a visual cue or marker (e.g., arrow 806) is being presented, the end user will release a button on the AR device 704, and the rendering of the 3D avatar 804 via the display device of the AR device 704 will be updated to reflect that the 3D avatar 804 has been anchored or fixed to the tabletop 800. Once the avatar has been anchored or fixed, the end user can move his or her head, and the AR device 704 will continuously track the environment to continuously update the AR view 806, so that the presentation of the 3D avatar remains fixed in a specific position in the AR space and looks on top of the table 800.

[0117] In some examples, during the anchoring or pinning of a 3D avatar, the user interface can provide prompts (e.g., graphics or visual markers) to indicate, for example, that the avatar is currently in a position where it can be anchored or pinned. Similarly, visual prompts or markers can be presented to indicate that the avatar has been successfully anchored or pinned. Additionally, immediately following the anchoring or pinning of the 3D avatar, the user interface can provide ways to scale the avatar to fit the detail of the given environment and the size of the specific object to which the avatar has been anchored, conforming to expectations.

[0118] Refer again Figure 7Once the second end-user wearing AR device 704 has anchored or fixed the 3D avatar representing the first end-user to a table, the messaging application executing on AR device 704 sends a state update 710 to messaging system 706. Specifically, the state update 710 includes data reflecting that the second end-user is using AR device 704, is actively online via the messaging application, and is currently located in the real-world environment associated with the AR space where the 3D avatar representing the first end-user is anchored. Therefore, after updating the second end-user's state (e.g., stored state), messaging system 706 also generates instructions for communicating with the first end-user's mobile client device 702. Specifically, interactive client 204-A or the messaging application executing on client device 702 will receive the state update related to the second end-user's state. Therefore, the user interface will be updated to reflect the state update related to the second end-user. The state update can be presented in any number of different ways. In one example, the visual representation of the second end-user will be updated, for example, by depicting a digital representation of the second end-user wearing AR device. In another example, the graphic or icon may have a specific color to reflect the online or offline status of the second end user. In some cases, the visual representation of the status update may be presented in the context of a contact list or in the user interface of a messaging application. In one example, a location-based map service may display a map including some indication of the current location of other end users. Thus, in one example, the visual representation of the status update may be delivered as part of a location-based map service. For example, the avatar of the second end user may be positioned on the map interface to reflect his or her location as determined by data obtained via AR device 704, and the visual representation of the second end user (e.g., the avatar) may be enhanced to depict the avatar wearing the AR device. Importantly, in some examples, the visual representation of the status update related to the status of the second end user conveys specific information to the first end user. For example, the status update specifically conveys to the first end user that his or her avatar is currently viewable by the second end user. Thus, the status update may provide an incentive for the first end user to send a message to the second end user, where the message is specifically tailored for viewing via AR device 704.

[0119] Next, at the operation indicated by reference numeral 714, the first end user uses a messaging application running on client device 702 to prepare a text-based message for transmission to a second end user wearing AR device 704. In this case, because the first end user knows that the second end user is wearing AR device 704 and viewing an avatar representing the first end user, the first end user may expect to send a text-based message that will cause his or her avatar, as viewed by the second end user, to perform some kind of avatar animation. Figure 9 and Figure 10 An example user interface for preparing text-based messages is shown and described.

[0120] like Figure 9 As shown, the first end user has prepared a text-based message 900. In this example, the text-based message 900 includes the text "Hi, John! / / W". In this example, the text " / / W" will be interpreted by the messaging system 706 or the messaging application executed on the AR device 704 as a special command or instruction mapped to an avatar animation. When presented at the AR device 708, this command or instruction will cause the avatar animation to be presented, such that the 3D avatar representing the first end user will wave to the second end user who is viewing it.

[0121] exist Figure 10 The example presented here is another instance of a user interface. In this example, emojis are mapped to avatar animations, rather than using a combination of specified special characters to identify the avatar animation. Therefore, as... Figure 10 As shown, the first end user has selected a user interface element 1000 (e.g., a symbol representing an emoji), causing the user interface to present a selection of selectable emojis. By selecting a specific emoji 1004, the selected emoji 1004 is inserted into the body of a text-based message in the text input element 1002.

[0122] Consistent with some examples, user interface elements can be presented such that when a user interface element is selected, a preview of an avatar that performs an animation corresponding to the selected emoji or a sequence of special characters already entered is shown to the end user preparing a message. In some examples, instead of combinations of special characters or emojis, a separate set of custom graphics or icons can be presented in the user interface, where the selection of a graphic or icon will trigger a preview of the avatar animation associated with that graphic or icon, and the command or instruction used to present the avatar animation at the receiving AR device is added as metadata to the text-based message.

[0123] like Figure 11As shown, when a first end user uses mobile device 702 to prepare a text-based message intended for a second end user wearing AR device 704, activity data can be transmitted from mobile device 702 to messaging system 706 and ultimately relayed to AR device 704. In this example, the activity data indicates that the first end user is preparing (e.g., typing) a text-based message. Therefore, the messaging application executing at AR device 704 will update AR view 1100 to present chat bubbles 1104 near 3D avatar 1102. In this example, chat bubbles 1102 include three dots (e.g., “…”) 1104 to indicate that the first end user represented by avatar 1102 is typing a message for the second end user wearing AR device 704. Of course, the visual representation of the activity indication—in this case, typing—can be presented in any number of ways. In any case, the second end user wearing AR device 704 is provided with advance notification of a possible incoming message from the first end user represented by 3D avatar 1102.

[0124] Refer again Figure 7 After the first end user has prepared and sent a text-based message 714, at the operation marked 716, the message transceiver system 706 receives the text-based information from the client device 702, wherein, at least in some cases, the information will be analyzed and evaluated. For example, in one example, as reflected by the operation marked 718, upon receiving an incoming message, the message transceiver system 706 may check the status record associated with each expected message recipient to determine whether the expected message recipient is currently online relative to the message transceiver system 706, and if online, determine the specific type of device the end user is using. Figure 7 In the scenario presented, when the intended recipient is actually online and reachable via AR device 704, messaging system 706 will then evaluate the received text-based message to determine whether it includes any message elements mapped to the avatar animation, as indicated by reference numeral 720. If the text-based message includes message elements mapped to a specific avatar animation, messaging system 706 will then prepare instructions and send them to the messaging application executing at AR device 704, thereby instructing the messaging application to present the 3D avatar performing the avatar animation.

[0125] At AR device 704, a possible scenario is that when a message is received at the messaging application running at AR device 702, the end user wearing AR device 704 may not be looking in the direction of the 3D avatar. That is, the 3D avatar may not be in the end user's field of vision. In this case, if the message arrives from the end user operating mobile client device 702, AR device 704 may need to play a sound or activate an on-display visual as a notification or prompt that a new message has arrived from the end user represented by the 3D avatar. The sound could be spatial audio, meaning it could originate from a specific direction of the 3D avatar. The on-display visual could be an arrow inviting the end user to look in a specific direction of the 3D avatar. The avatar animation only begins playing when the end user at the 3D avatar represents the end user of mobile client device 702. This prevents the avatar animation from playing when the wearer of the AR device may not actually be viewing the animation. Additionally, in some cases, this slight delay between receiving a message and playing an avatar animation can advantageously provide additional time for messaging applications running on AR devices to retrieve relevant media files from local storage or a remote server for playing the avatar animation.

[0126] Consistent with some examples, each text-based message, including message elements associated with an avatar animation, is transmitted over the network to AR device 704 along with payload data. For example, payload data (e.g., media files associated with the avatar animation) may be transmitted to AR device 704 along with the text-based message. However, in other examples, text-based messages received at messaging system 706 are simply relayed to AR device 704; no message content evaluation is performed at the server-based messaging system, and no payload data is included. Instead, when a text-based message is received at AR device 704, the messaging application at AR device 704 analyzes and evaluates the message content and determines whether message elements correspond to an avatar animation. If a specific message element corresponding to an avatar animation exists (e.g., an emoji, a special character sequence, or some other metadata), the messaging application accesses the client-side avatar command-to-avatar animation mapping to identify the specific avatar animation and associated media files referenced by the message elements received as text-based messages. If the appropriate media file exists (e.g., is stored) at AR device 704, the messaging application will simply perform the process of reading and processing the media file to render the avatar animation. However, if the media file does not exist at AR device 704, the messaging application can send a request to the messaging system or associated content distribution system to send the appropriate media file to the messaging application at AR device 704. The avatar animation will then be rendered upon receipt of the media file.

[0127] At the point of operation marked with reference numeral 724, the AR device 704 receives messages and instructions to execute or play an avatar animation. In some examples, the avatar animation is presented simultaneously with a text-based message displayed in a corresponding chat bubble near the 3D avatar. In other examples, the media file associated with the avatar animation may include one or more audio files, and therefore, the presentation of the avatar animation may include playback of the audio files—for example, the avatar may speak an audible message as part of the animation, or sound effects may be present to enhance the avatar animation. In some examples, a text-to-speech algorithm is used to convert the text of the text-based message into an audio message, which is then presented before, after, or during the presentation of the avatar animation. Figure 12 An example of presenting avatar animation in an AR view is shown and described.

[0128] like Figure 12 As shown, when the AR device 704 worn by the end user receives a text-based message, the messaging application at the AR device 704 processes the instruction to receive the text-based information, causing the 3D avatar 1202 presented in the AR view 1200 to perform an avatar animation, such as waving its arms. The text of the text-based message is also presented in a chat bubble 1204 that is near (e.g., adjacent to or close to) the 3D avatar 1202. Although Figure 12 It's only a two-dimensional illustration, but one can imagine a 3D avatar waving its hand back and forth according to the avatar animation. In some examples, special commands or emojis mapped to the avatar animation (e.g., ":wave:") will be presented in chat bubble 1204 along with a text-based message. However, in other examples, only portions of the text-based message that are not identified as being associated with the avatar animation will be presented.

[0129] After a second end-user wearing AR device 704 receives a text-based message from a first end-user, the second end-user may expect to prepare and transmit a response message. See again... Figure 7At the point of operation indicated by reference numeral 726, a messaging application running on AR device 704 is continuously receiving and processing sensor data to analyze and track the real-world environment. Therefore, if a second end-user wearing AR device 704 looks in the direction of the 3D avatar, AR device 704 will process the sensor data to determine that AR device 704 is positioned and oriented so that the end-user has a 3D avatar directly in the end-user's line of sight. Based on this determination by AR device 704, the messaging application will update the AR view presenting the 3D avatar by adding some type of visual cue or marker, thereby indicating that the end-user is currently "aiming" at the 3D avatar, and will enable the AR device's audio recording device, allowing the end-user to record messages to be transmitted to the end-user represented by the 3D avatar. Figure 13 The example presented is a second end-user targeting a 3D avatar of a first end-user.

[0130] like Figure 13 As shown, the second end user wearing AR device 704 has turned his head to view the location of the second end user's previously anchored or fixed 3D avatar 1302 on the desktop. Therefore, when AR device 704 detects that 3D avatar 1302 is in the second end user's near-direct line of sight, the messaging application executing at AR device 704 updates the AR view to present an icon representing microphone 1306 above the chat bubble. This presentation of microphone 1306 is a visual indication to the second end user that the AR device 704's audio recording device (e.g., a built-in microphone) has been enabled and is ready to record audio messages.

[0131] Refer again Figure 7 Consistent with some examples, after the message has been recorded, AR device 704 can use a client-side speech-to-text algorithm to convert the audio recording into a text-based message. The text-based message is then transmitted to messaging system 706, where it is processed and relayed to its intended recipient. However, since some client-based AR devices may have limited processing power, consistent with some examples, the audio recording can be transmitted over a network to messaging system 706, where a server-based messaging system processes the audio to create a text-based message that will then be transmitted to the intended recipient.

[0132] like Figure 14 As shown, when a second end user wearing AR device 704 speaks an audible message captured or recorded by AR device 704, the AR view 1400 presented by AR device 704 can be updated to present a text-based version of the audible message. For example, as Figure 12As shown, after the first end user sends a text-based message (“Hi, hello”) 1204 to the second end user, the second end user has responded with a recorded audio message, which has been translated into text and then presented in a separate chat bubble 1404. In some examples, the color of the chat bubble can indicate the source of the text-based message. This can be configured by the end user. Figure 14 As shown, an icon representing microphone 1406 is presented next to chat bubble 1404 to indicate that the microphone is currently enabled and the messaging application is ready to capture audio for sending as a message. In some examples, the graphic or icon (e.g., microphone) may have various versions or be presented in different colors to indicate different states (e.g., disabled or enabled, and ready to record, etc.).

[0133] In some examples, with two or more end users wearing AR devices, audio messages can be transmitted between the devices without any text-to-text conversion. Instead, each AR device can play an audio file, allowing the recipient to hear the audible message, for example, through a built-in speaker or similar device. In some examples, each messaging application provides a user interface with access to various configuration settings for the messaging system. Therefore, end users can establish configuration settings that determine how messages are processed in a particular scenario. For example, end users might be able to configure specific settings to ensure that all messages are converted and served as text-based messages, or alternatively, specific settings to ensure that all messages are converted and served as audible messages.

[0134] Systems with wearable AR devices

[0135] Figure 15 Two examples of AR devices are shown, each with a different shape factor. As those skilled in the art will understand, AR devices can have a variety of different shape factors. In some examples, the various components of the AR device are built into a pair of glasses 1500, commonly referred to as smart glasses or AR glasses. In other examples, the AR device can be more robust than a pair of glasses and is therefore referred to as an AR head-mounted device 1502. Although in Figure 15 Not shown in the image, but in other examples, various components that enable AR experiences can be built into hats, helmets, or protective face shields.

[0136] Figure 16 The system 1600, which includes a user system according to some examples, is shown. In this example, the user system is a head-mounted device or AR device with a selector input device. Figure 16This is a high-level functional block diagram of an example AR device 202-B that is communicatively coupled to mobile devices 1602 and various server systems 1604 (e.g., interactive server system 212) via various networks 1616.

[0137] AR device 202-B includes one or more camera devices, each of which may be, for example, a visible light camera device 1606, an infrared emitter 1608, and an infrared camera device 1610.

[0138] Mobile device 1602 connects to AR device 202-B using both low-power wireless connection 1612 and high-speed wireless connection 1614. Mobile device 1602 also connects to server system 1604 and network 1616.

[0139] AR device 202-B also includes two display devices or image displays in the image display 1618 of the optical components. The image display 1618 of the two optical components includes an image display associated with the left lateral side of AR device 202-B and an image display associated with the right lateral side of AR device 202-B. AR device 202-B also includes an image display driver 1620, an image processor 1622, low-power circuitry 1624, and high-speed circuitry 1626. The image display 1618 of the optical components is used to present images and videos (e.g., avatars and animated avatars) to the user of AR device 202-B, including images that may include a graphical user interface.

[0140] The image display driver 1620 commands and controls the image display 1618 of the optical components. The image display driver 1620 can directly deliver image data to the image display 1618 of the optical components for presentation, or it can convert image data into a signal or data format suitable for delivery to the image display device. For example, the image data can be video data formatted according to compression formats such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc., while still image data can be formatted according to compression formats such as Portable Network Group (PNG), Joint Photo Experts Group (JPEG), Tagged Image File Format (TIFF), or Exchangeable Image File Format (EXIF), etc.

[0141] AR device 202-B includes a frame and a handle (or temple) extending laterally from the frame. AR device 202-B also includes a user input device 1628 (e.g., a touch sensor or button) including an input surface on AR device 202B. User input device 1628 (e.g., a touch sensor or a press button) is used to receive input selections from a user for manipulating a graphical user interface of the presented image.

[0142] Figure 16 The components shown for AR device 202-B are located on one or more circuit boards (e.g., PCBs or flexible PCBs) in the frame or temples. Alternatively or additionally, the depicted components may be located in chunks, frames, hinges, or nose bridges of AR device 202-B. The left and right visible light camera devices 506 may include digital camera elements, such as complementary metal-oxide-semiconductor (CMOS) image sensors, charge-coupled devices, camera lenses, or any other corresponding visible light or light-capturing elements that can be used to capture data, including images of scenes with unknown objects.

[0143] AR device 202-B includes memory 1602 that stores instructions for performing a subset or all of the functions described herein. Memory 1602 may also include storage devices.

[0144] like Figure 16As shown, the high-speed circuit 1626 includes a high-speed processor 1630, a memory 1602, and a high-speed wireless circuit 1632. In some examples, an image display driver 1620 is coupled to the high-speed circuit 1626 and operated by the high-speed processor 1630 to drive the left and right image displays in the image display 1628 of the optical components. The high-speed processor 1630 can be any processor capable of managing the high-speed communication and operation of any general-purpose computing system required by the AR device 202-B. The high-speed processor 1630 includes the processing resources required to manage high-speed data transmission to a wireless local area network (WLAN) over the high-speed wireless connection 1614 using the high-speed wireless circuit 1632. In some examples, the high-speed processor 1630 executes the operating system of the AR device 202-B (e.g., a LINUX operating system) or other such operating system, and this operating system is stored in the memory 1602 for execution. Among other duties, the high-speed processor 1620, which executes the software architecture of the AR device 202-B, manages data transmission with the high-speed wireless circuit 1632. In some examples, the high-speed wireless circuit 1632 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi®. In other examples, the high-speed wireless circuit 1632 can implement other high-speed communication standards.

[0145] The low-power wireless circuitry 1634 and high-speed wireless circuitry 1632 of the AR device 202-B may include a short-range transceiver (Bluetooth™) and a wireless wide area network transceiver, a wireless local area network transceiver, or a wide area network transceiver (e.g., cellular or Wi-Fi®). The mobile device 202-B, including transceivers communicating via low-power wireless connection 1612 and high-speed wireless connection 1614, can be implemented using the architectural details of the AR device 202-B, and other components of the network 1616 can also be implemented in this way.

[0146] Memory 1602 includes any storage device capable of storing various data and applications, including camera data generated by the left and right visible light cameras 1606, the infrared camera 1610, and the image processor 1622, as well as images generated for display on an image display in an optical component image display 1618 via an image display driver 1620. While memory 1602 is shown as integrated with high-speed circuitry 1626, in some examples, memory 1602 may be a separate, independent component of the AR device 202-B. In some such examples, electrical wiring may provide a connection from the image processor 1622 or the low-power processor 1636 to memory 1602 via a chip including high-speed processor 530. In some examples, high-speed processor 1630 may manage addressing of memory 1602 such that low-power processor 1636 will activate high-speed processor 1630 whenever a read or write operation involving memory 1602 is required.

[0147] like Figure 16 As shown, the low-power processor 1636 or high-speed processor 1630 of the AR device 202-B can be coupled to a camera device (visible light camera 1636, infrared emitter 1608 or infrared camera 1620), an image display driver 1620, a user input device 1628 (e.g., a touch sensor or a press button), and a memory 1602.

[0148] AR device 202-B is connected to a host computer. For example, AR device 202-B is paired with mobile device 1601 via high-speed wireless connection 1614 or connected to server system 1604 via network 1616. Server system 1604 may be one or more computing devices as part of a service or network computing system, for example, it includes a processor, memory, and network communication interface to communicate with mobile device 1601 and AR device 202-B via network 1616.

[0149] Mobile device 1601 includes a processor and a network communication interface coupled to the processor. The network communication interface enables communication via network 516, low-power wireless connection 512, or high-speed wireless connection 514. Mobile device 1601 may also store at least a portion of the instructions for generating binaural audio content in the memory of mobile device 1601 to implement the functions described herein.

[0150] The output components of AR device 202-B include visual components, such as displays (e.g., liquid crystal displays (LCDs), plasma display panels (PDPs), light-emitting diode (LED) displays, projectors, or waveguides). The image display of the optical components is driven by an image display driver 520. The output components of AR device 202-B also include acoustic components (e.g., speakers), haptic components (e.g., vibration motors), other signal generators, etc. The input components (e.g., user input device 528) of AR device 202-B, mobile device 1601, and server system 504 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), pointing-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide position and force for touch or touch gestures), audio input components (e.g., microphones), etc.

[0151] AR device 202-B may also include additional peripheral device elements. Such peripheral device elements may include biometric sensors, additional sensors, or display elements integrated with AR device 202-B. For example, peripheral device elements may include any I / O components, including output components, motion components, positioning components, or any other such components described herein.

[0152] For example, biometric components include those for detecting expressions (e.g., hand gestures, facial expressions, vocal expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Biometric components may include brain-computer interface (BMI) systems that allow communication between the brain and external devices or machines. This can be achieved by recording brain activity data, converting that data into a format that can be understood by a computer, and then using the resulting signals to control the device or machine.

[0153] Examples of BMI technology types include:

[0154] Brain-based brain-computer interfaces (BMIs) use electrodes placed on the scalp to record electrical activity in the brain.

[0155] Invasive BMI uses electrodes surgically implanted into the brain.

[0156] Optogenetics BMI uses light to control the activity of specific nerve cells in the brain.

[0157] Any biometric data collected by the biometric component is captured and stored only with user approval and is deleted upon user request. Furthermore, such biometric data may be used for very limited purposes (e.g., authentication). To ensure restricted and authorized use of biometric information and other personally identifiable information (PII), access to this data is limited to authorized personnel (if applicable). Any use of biometric data may be strictly limited to authentication purposes, and such biometric data may not be shared or sold to any third party without the user's explicit consent. In addition, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of this sensitive information.

[0158] Moving components include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Positioning components include position sensor components (e.g., GPS receiver components) for generating position coordinates, Wi-Fi or Bluetooth™ transceivers for generating positioning system coordinates, altitude sensor components (e.g., altimeters or barometers that detect air pressure, from which altitude can be obtained), orientation sensor components (e.g., magnetometers), etc. Such positioning system coordinates can also be received from mobile device 1601 via low-power wireless circuit 534 or high-speed wireless circuit 532, through low-power wireless connection 512 and high-speed wireless connection 514.

[0159] Software Architecture

[0160] Figure 17 This is a block diagram 1700 illustrating a software architecture 1702 that can be installed on any one or more of the devices described herein. The software architecture 1702 is supported by hardware such as a machine 1704 including a processor 1706, memory 1708, and I / O components 1710. In this example, the software architecture 1702 can be conceptualized as a stack of layers, where each layer provides a specific function. The software architecture 1702 includes layers such as an operating system 1712, libraries 1714, frameworks 1716, and applications 1718. Operationally, application 1718 activates API calls 1720 via the software stack and receives messages 1722 in response to API calls 1720.

[0161] Operating system 1712 manages hardware resources and provides public services. Operating system 1712 includes, for example, a kernel 1724, services 1726, and drivers 1728. Kernel 1724 serves as an abstraction layer between the hardware layer and other software layers. For example, kernel 1724 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1726 can provide other public services to other software layers. Drivers 1728 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1728 may include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® low-power drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), Wi-Fi® drivers, audio drivers, power management drivers, etc.

[0162] Library 1714 provides common low-level infrastructure used by application 1718. Library 1714 may include system library 1730 (e.g., the C standard library), which provides functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 1714 may include API library 1732, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering graphic content on a display in two-dimensional (2D) and three-dimensional (3D) formats), database libraries (e.g., SQLite, which provides various relational database functions), web libraries (e.g., WebKit, which provides web browsing capabilities), and so on. Library 1714 may also include various other libraries 734 to provide many other APIs to application 1718.

[0163] Framework 1716 provides common high-level infrastructure for use by application 1718. For example, Framework 1716 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. Framework 1716 can provide a wide range of other APIs that can be used by application 1718, some of which may be specific to a particular operating system or platform.

[0164] In the example, application 1718 may include home application 1736, contact application 1738, browser application 1740, book reader application 1742, location application 1744, media application 1746, messaging application 1748, game application 1750, and a wide variety of other applications such as third-party application 1752. Application 1718 is a program that performs the functions defined in the program. One or more applications 1718 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 1752 (e.g., an application developed by an entity other than a platform vendor using the Android™ or iOS™ Software Development Kit (SDK)) may be mobile software that runs on mobile operating systems such as iOS™, Android™, Windows® Phone, or other mobile operating systems. In this example, a third-party application 1752 can activate API call 1720 provided by the operating system 1712 to facilitate the functionality described herein.

[0165] Machine structure

[0166] Figure 18This is a schematic representation of machine 1800, within which instructions 1802 (e.g., software, program, application, app, or other executable code) can be executed to cause machine 1800 to perform any or more of the methods discussed herein. For example, instructions 1802 can cause machine 1800 to perform any or more of the methods described herein. Instructions 1802 transform the general, unprogrammed machine 1800 into a specific machine 1800 programmed to perform the described and illustrated functions in the described manner. Machine 1800 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 1800 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1800 may include, but is not limited to, server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 1802 specifying actions to be taken by machine 1800. Furthermore, although only a single machine 1800 is shown, the term "machine" should also be considered as a collection of machines that individually or jointly execute instructions 1802 to perform any one or more of the methods discussed herein. For example, machine 1800 may include user system 102 or any of a plurality of server devices forming part of interactive server system 110. In some examples, machine 1800 may also include both client and server systems, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of that particular method or algorithm are performed on the client side.

[0167] Machine 1800 may include a processor 1804, a memory 1806, and an input / output (I / O) unit 1808 that can be configured to communicate with each other via a bus 1810. In the example, processor 1804 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processors 1812 and 1814 that execute instruction 1802. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 18 Multiple processors 1804 are shown, but machine 1800 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0168] Memory 1806 includes main memory 1816, static memory 1818, and memory cell 1820, all of which are accessible to processor 1804 via bus 1810. Main memory 1806, static memory 1818, and memory cell 1820 store instructions 1802 embodying any one or more of the methods or functions described herein. Instructions 1802 may also reside wholly or partially within main memory 1816, static memory 1818, machine-readable medium 1822 within memory cell 1820, at least one processor of processor 1804 (e.g., within the processor's cache memory), or any suitable combination thereof during execution by machine 1800.

[0169] I / O component 1808 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O component 1808 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine is unlikely to include such a touch input device. It will be appreciated that I / O component 1808 may include... Figure 18Many other components are not shown. In various examples, I / O component 1808 may include user output component 1824 and user input component 1826. User output component 1824 may include visual components (e.g., displays such as plasma display panels (PDP), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. User input component 1826 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), pointing-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens that provide the position and force or gesture of a touch, or other haptic input components), audio input components (e.g., microphones), etc.

[0170] In other examples, I / O component 1808 may include biometric component 1828, motion component 1830, environmental component 1832, or position component 1834, as well as a wide range of other components. For example, biometric component 1828 includes components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and recognizing people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Biometric component may include a brain-computer interface (BMI) system that allows communication between the brain and external devices or machines. This can be achieved by recording brain activity data, converting that data into a format that can be understood by a computer, and then using the resulting signals to control devices or machines.

[0171] Examples of BMI technology types include:

[0172] Brain-based brain-computer interfaces (BMIs) use electrodes placed on the scalp to record electrical activity in the brain.

[0173] Invasive BMI uses electrodes surgically implanted into the brain.

[0174] Optogenetics BMI uses light to control the activity of specific nerve cells in the brain.

[0175] Any biometric data collected by the biometric component is captured and stored only with user approval and is deleted upon user request. Furthermore, such biometric data may be used for very limited purposes (e.g., authentication). To ensure restricted and authorized use of biometric information and other personally identifiable information (PII), access to this data is limited to authorized personnel (if applicable). Any use of biometric data may be strictly limited to authentication purposes, and such biometric data may not be shared or sold to any third party without the user's explicit consent. In addition, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of this sensitive information.

[0176] The moving part 1830 includes an acceleration sensor part (e.g., an accelerometer), a gravity sensor part, and a rotation sensor part (e.g., a gyroscope).

[0177] Environmental component 1832 includes, for example, one or more camera devices (with still image / photograph and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting the concentration of hazardous gases or measuring pollutants in the atmosphere for safety purposes), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0178] Regarding the camera device, user system 102 may have a camera device system including, for example, a front-facing camera on the front surface of user system 102 and a rear-facing camera on the rear surface of user system 102. The front-facing camera may be used, for example, to capture still images and videos (e.g., “selfies”) of the user of user system 102, which can then be enhanced with the enhancement data (e.g., filters) described above. The rear-facing camera may be used, for example, to capture still images and videos in a more conventional camera device mode, wherein these images are similarly enhanced with the enhancement data. In addition to the front-facing and rear-facing cameras, user system 102 may also include a 360° camera for capturing 360° photos and videos.

[0179] Furthermore, the camera system of user system 102 may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even triple, quadruple, or quintuple rear camera configurations on the front and rear sides of user system 102. For example, these multi-camera systems may include wide-angle cameras, ultra-wide-angle cameras, telephoto cameras, macro cameras, and depth sensors.

[0180] The positioning component 1834 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure and from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.

[0181] A wide variety of technologies can be used to implement communication. I / O component 1808 also includes a communication component 1836 operable to couple machine 1800 to network 1838 or device 1840 via a suitable coupling or connection. For example, communication component 1836 may include a network interface component or other suitable device that interfaces with network 1838. In other examples, communication component 1836 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components for providing communication via other modalities. Device 1840 may be another machine or any peripheral device from a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0182] Furthermore, communication component 1836 can detect identifiers, or includes components operable to detect identifiers. For example, communication component 1836 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, data matrices, and data symbols). TM The device can be equipped with optical sensors for MaxiCode, PDF417, UltraCode, UCC RSS-2D barcodes, and other optical codes, or acoustic detection components (e.g., microphones for identifying audio signals of the tags). Additionally, various information can be obtained via communication component 1836, such as location obtained via Internet Protocol (IP) geolocation, location obtained via Wi-Fi® signal triangulation, or location obtained by detecting NFC beacon signals that indicate a specific location.

[0183] Various memories (e.g., main memory 1816, static memory 1818, and the memory of processor 1804) and storage unit 1820 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instruction 1802) cause various operations to implement the disclosed examples when executed by processor 1804.

[0184] Instructions 1802 can be sent or received over network 1838 via a network interface device (e.g., a network interface component included in communication component 1836) using a transmission medium and employing any of the known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 1802 can be sent or received via a transmission medium through coupling with device 1840 (e.g., peer-to-peer coupling).

[0185] Example

[0186] Example 1 is a system for providing a messaging service, the system comprising: at least one processor; a memory device storing instructions that, when executed by the processor, cause the system to perform operations including: receiving a message from a computing device of a first end user, the message being addressed to a second end user; processing the message by: determining that the second end user has a state indicating that the second end user is online for a messaging service via an augmented reality (AR) device located in a real-world space associated with an AR space, in which the second end user has anchored a 3D avatar representing the first end user; determining that a message element received with the message is associated with an avatar animation; and sending data to the second end user's AR device, the data, when processed by a messaging application executed at the AR device, causing the messaging application to render the 3D avatar performing the avatar animation via a user interface on the AR device's display.

[0187] In Example 2, the subject of Example 1 includes rendering the 3D avatar in an appearance consistent with the avatar configuration data specified by the first end user when the 3D avatar representing the first end user is presented in the user interface on the display of an AR device.

[0188] In Example 3, the subject matter of Examples 1 and 2 includes, wherein, avatar animation is humanoid movement, including: poses, facial expressions, gestures, actions performed using props, or any combination thereof.

[0189] In Example 4, the subject of Examples 1 to 3 includes, wherein the message element received by the message is an emoji, and determining that the message element received by the message is associated with an avatar animation includes: using an emoji query to map each of a plurality of emojis to a corresponding avatar animation data structure, wherein an indication of the avatar animation is returned as a result of the query.

[0190] In Example 5, the subject of Examples 1 to 4 includes the following: wherein the message element received by the message is a combination of two or more characters, and determining that the message element received by the message is associated with an avatar animation includes: querying a data structure that maps each of two or more combinations of two or more characters to a corresponding avatar animation using the combination of two or more characters, wherein an indication of the avatar animation is returned as a result of the query.

[0191] In Example 6, the subject of Examples 1 to 5 includes an instruction that, when executed by a processor, causes the system to perform additional operations, said additional operations including: processing a message by sending data to a second end-user's AR device, which, when processed by a messaging application executed at the AR device, causes the messaging application to display the text contained in the message in a chat bubble positioned close to the 3D avatar performing the avatar animation.

[0192] In Example 7, the subject matter of Examples 1 through 6 includes an instruction that, when executed by a processor, causes the system to perform additional operations, said additional operations including: processing a message by: generating an audio clip based on the text included in the message using a text-to-speech algorithm; and sending the data and audio clip to a second end-user's AR device, which, when processed by a messaging application executed at the AR device, will cause the messaging application to play the audio clip output via the AR device's speaker during the presentation of a 3D avatar performing an avatar animation.

[0193] In Example 8, the subject matter of Examples 1 through 7 includes an instruction that, when executed by a processor, causes the system to perform additional operations, said additional operations including: before receiving a message from a first end-user's computing device: receiving data from a messaging application executed at an AR device, the data indicating that a second end-user is online for a messaging service via an augmented reality (AR) device located in a real-world space associated with an AR space, in which the second end-user has anchored a 3D avatar representing the first end-user; updating the state of the second end-user to indicate that the second end-user is online for a messaging service via an augmented reality (AR) device located in a real-world space associated with an AR space, in which the second end-user has anchored a 3D avatar representing the first end-user; and transmitting data to the first end-user's computing device, the data, when processed by the computing device, causing a presence indication to appear on a user interface on the display of the computing device, the presence indication indicating the state of the second end-user.

[0194] In Example 9, the subject of Example 8 includes an instruction that, when executed by a processor, causes the system to perform additional operations, said additional operations including: before receiving a message from a first end-user's computing device: transmitting data to the first end-user's computing device, which, when processed by the computing device, will cause a user interface to be displayed via the display of the computing device, the user interface displaying a plurality of icons, wherein each icon represents an avatar animation, wherein the first end-user's selection of an icon will activate a 3D avatar to perform a preview presentation of the avatar animation corresponding to the selected icon.

[0195] Example 10 illustrates a computer-implemented method performed by one or more server computers providing a messaging service. The method includes: receiving a message from a first end-user's computing device, the message being addressed to a second end-user; processing the message by: determining that the second end-user has a state indicating that the second end-user is online for a messaging service via an augmented reality (AR) device located in a real-world space associated with an AR space in which the second end-user has anchored a 3D avatar representing the first end-user; determining that a message element received with the message is associated with an avatar animation; and sending data to the second end-user's AR device, which, when processed by a messaging application executed at the AR device, will cause the messaging application to render the 3D avatar performing the avatar animation via a user interface on the AR device's display.

[0196] In Example 11, the subject of Example 10 includes rendering the 3D avatar in an appearance consistent with the avatar configuration data specified by the first end user when the 3D avatar representing the first end user is presented in the user interface on the display of an AR device.

[0197] In Example 12, the subject matter of Examples 10 and 11 includes, wherein, avatar animation is humanoid movement, including: poses, facial expressions, gestures, actions performed using props, or any combination thereof.

[0198] In Example 13, the subject of Examples 10 to 12 includes, wherein the message element received by the message is an emoji, and determining that the message element received by the message is associated with an avatar animation includes: using an emoji query to map each of a plurality of emojis to a corresponding avatar animation data structure, wherein an indication of the avatar animation is returned as a result of the query.

[0199] In Example 14, the subject of Examples 10 to 13 includes the following: the message element received as a message is a combination of two or more characters, and determining that the message element received as a message is associated with an avatar animation includes: querying a data structure that maps each of two or more combinations of two or more characters to a corresponding avatar animation using a combination of two or more characters, wherein an indication of the avatar animation is returned as a result of the query.

[0200] In Example 15, the subject of Examples 10 to 14 includes an instruction that, when executed by a processor, causes the system to perform additional operations, said additional operations including: processing a message by sending data to a second end-user's AR device, which, when processed by a messaging application executed at the AR device, causes the messaging application to display the text contained in the message in a chat bubble positioned close to the 3D avatar performing the avatar animation.

[0201] In Example 16, the subject matter of Examples 10 to 15 includes an instruction that, when executed by a processor, causes the system to perform additional operations, said additional operations including: processing a message by: generating an audio clip based on the text included in the message using a text-to-speech algorithm; and sending the data and audio clip to a second end-user's AR device, which, when processed by a messaging application executed at the AR device, will cause the messaging application to play the audio clip output via the AR device's speaker during the presentation of a 3D avatar performing an avatar animation.

[0202] In Example 17, the subject matter of Examples 10 through 16 includes an instruction that, when executed by a processor, causes the system to perform additional operations, said additional operations including: before receiving a message from a first end-user's computing device: receiving data from a messaging application executed at an AR device, the data indicating that a second end-user is online for a messaging service via an augmented reality (AR) device located in a real-world space associated with an AR space, in which the second end-user has anchored a 3D avatar representing the first end-user; updating the state of the second end-user to indicate that the second end-user is online for a messaging service via an augmented reality (AR) device located in a real-world space associated with an AR space, in which the second end-user has anchored a 3D avatar representing the first end-user; and transmitting data to the first end-user's computing device, which, when processed by the computing device, will cause a presence indication to appear on a user interface on the display of the computing device, the presence indication indicating the state of the second end-user.

[0203] In Example 18, the subject of Example 17 includes an instruction that, when executed by a processor, causes the system to perform additional operations, said additional operations including: before receiving a message from a first end-user's computing device: transmitting data to the first end-user's computing device, which, when processed by the computing device, will cause a user interface to be displayed via the display of the computing device, the user interface displaying a plurality of icons, wherein each icon represents an avatar animation, wherein the first end-user's selection of an icon will activate a 3D avatar to perform a preview presentation of the avatar animation corresponding to the selected icon.

[0204] Example 19 is a system comprising: a method for receiving a message from a computing device of a first end user, the message being addressed to a second end user; and a method for processing the message by: determining that the second end user has a state indicating that the second end user is online for a messaging service via an augmented reality (AR) device located in a real-world space associated with an AR space in which the second end user has anchored a 3D avatar representing the first end user; determining that a message element received with the message is associated with an avatar animation; and sending data to the second end user's AR device that, when processed by a messaging application executed at the AR device, will cause the messaging application to render the 3D avatar performing the avatar animation via a user interface on the display of the AR device.

[0205] In Example 20, the subject of Example 19 includes, wherein the message element received as a message is an emoji, and the method for processing the message by determining that the message element received as a message is associated with an avatar animation further includes: a method for using an emoji query to map each of a plurality of emojis to a corresponding avatar animation to a data structure, wherein an indication of the avatar animation is returned as a result of the query.

[0206] Example 21 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any one of Examples 1 to 20.

[0207] Example 22 is an apparatus that includes a method for implementing any one of Examples 1 through 20.

[0208] Example 23 is a system for implementing any one of Examples 1 through 20.

[0209] Example 24 is one method for implementing any of Examples 1 through 20.

[0210] Glossary

[0211] "Carrier signal" refers to any intangible medium, such as a medium capable of storing, encoding, or carrying machine-executable instructions and including digital or analog communication signals, or other intangible medium that facilitates the transmission of such instructions. Instructions can be transmitted or received over a network via a transmission medium using network interface devices.

[0212] "Client device" means any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0213] "Communications network" refers to one or more parts of a network, such as an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or a part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, coupling can implement any data transmission technology of various types, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standard, other data transmission technologies defined by various standards-setting organizations, other long-distance protocols, or other data transmission technologies.

[0214] A “component” refers to a logical or physical entity having boundaries defined by functional or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularity for a particular processing or control function. A component can be combined with other components via its interface to perform machine processing. A component can be an encapsulated functional hardware unit designed for use with other components, and part of a program that typically performs a related function. A component can constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various examples, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuitry temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a specific part of a machine) uniquely tailored to perform the configured function and is no longer a general-purpose processor. It will be appreciated that a decision can be made, for cost and time considerations, whether to implement a hardware component mechanically in a dedicated and permanently configured circuit or in a temporarily configured (e.g., software-configured) circuit. Therefore, the phrase “hardware component” (or “hardware-implemented component”) should be understood to include tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Consider the example of a hardware component being temporarily configured (e.g., programmed), without needing to configure or instantiate each hardware component at any given time. For example, in cases where the hardware components include a general-purpose processor that is configured as a dedicated processor via software, this general-purpose processor can be configured at different times as its respective dedicated processor (e.g., including different hardware components). For instance, the software accordingly configures one or more specific processors to constitute a specific hardware component at one time and different hardware components at different times. Hardware components can provide information to and receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from the memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Another hardware component can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed, at least in part, by one or more processors configured, either temporarily (e.g., by software) or permanently, to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by processors, where a particular processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or processor-implemented components. Furthermore, one or more processors can also operate to support the execution of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS) operations. For example, at least some operations can be performed by a group of computers (as an example of machines including processors), where these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of some operations can be distributed among processors, not residing within a single machine, but deployed across multiple machines. In some examples, the processors or processor-implemented components can reside in a single geographic location (e.g., within a home environment, office environment, or server cluster). In other examples, the processors or processor-implemented components can be distributed across multiple geographic locations.

[0215] "Computer-readable storage medium" refers to both, for example, machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" refer to the same thing and can be used interchangeably in this disclosure.

[0216] A "brief message" is a message that is accessible for a limited time, such as a short period of time. Brief messages can be text, images, videos, etc. The access time for a brief message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting method, the message is transient.

[0217] "Machine storage medium" refers to one or more storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Therefore, this term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium."

[0218] "Non-transitory computer-readable storage medium" means, for example, a tangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine.

[0219] "Signal medium" means, for example, any intangible medium capable of storing, encoding, or carrying machine-executable instructions and including digital or analog communication signals, or other intangible media that facilitates the transmission of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0220] "User equipment" means, for example, a device that is accessed, controlled, or owned by a user and that the user interacts with to perform actions or interactions (including interactions with other users or computer systems).

Claims

1. A system for providing message sending and receiving services, the system comprising: At least one processor; A memory device that stores instructions, which, when executed by a processor, cause the system to perform operations, including: A message is received from the computing device of a first terminal user, and the message is addressed to a second terminal user; The message is processed using the following operations: Determine that the second terminal user has a status indicating that the second terminal user is online for the messaging service via an augmented reality (AR) device located in a real-world space associated with an AR space, in which the second terminal user has anchored a 3D avatar representing the first terminal user; Determine that the message element received by the message is associated with the avatar animation; and Data is sent to the AR device of the second terminal user. When the data is processed by a messaging application running on the AR device, the messaging application will display the 3D avatar performing the avatar animation via a user interface on the AR device's display.

2. The system according to claim 1, wherein, When the 3D avatar representing the first end user is presented in the user interface on the display of the AR device, the 3D avatar is rendered with an appearance consistent with the avatar configuration data specified by the first end user.

3. The system according to claim 1, wherein, The avatar animation is humanoid movement, including: Posture, facial expressions, gestures, actions performed using props, or any combination thereof.

4. The system according to claim 1, wherein, The message element received in the message is an emoji, and determining that the message element received in the message is associated with an avatar animation includes: The emoji query maps each of a plurality of emojis to a corresponding avatar animation in a data structure, wherein the indication of the avatar animation is returned as the result of the query.

5. The system according to claim 1, wherein, The message element received in the message is a combination of two or more characters, and determining that the message element received in the message is associated with the avatar animation includes: A combination query using the two or more characters maps each of two or more combinations of two or more characters to a corresponding avatar animation data structure, wherein the indication of the avatar animation is returned as a result of the query.

6. The system according to claim 1, wherein, When executed by the processor, the instruction causes the system to perform additional operations, including: The message is processed using the following operations: Data is sent to the AR device of the second terminal user, and when the data is processed by the messaging application running on the AR device, the messaging application will display the text contained in the message in a chat bubble positioned close to the 3D avatar performing the avatar animation.

7. The system according to claim 1, wherein, When executed by the processor, the instruction causes the system to perform additional operations, including: The message is processed using the following operations: Using a text-to-speech algorithm, an audio clip is generated based on the text included in the message; and Data and the audio clip are sent to the AR device of the second terminal user. When the data and the audio clip are processed by the messaging application executed at the AR device, the messaging application will play the audio clip, which is output through the speaker of the AR device, during the presentation of the 3D avatar performing the avatar animation.

8. The system according to claim 1, wherein, When executed by the processor, the instruction causes the system to perform additional operations, including: Before receiving the message from the computing device of the first terminal user: Data is received from the messaging application running on the AR device, the data indicating that the second terminal user is online for the messaging service via the augmented reality (AR) device, the AR device being located in a real-world space associated with an AR space, in which the second terminal user has anchored a 3D avatar representing the first terminal user; Update the status of the second terminal user to indicate that the second terminal user is online for the messaging service via an augmented reality (AR) device located in a real-world space associated with the AR space, in which the second terminal user has anchored a 3D avatar representing the first terminal user; and Data is transmitted to the computing device of the first terminal user, and when the data is processed by the computing device, an presence indication is displayed on the user interface of the display of the computing device, the presence indication indicating the state of the second terminal user.

9. The system according to claim 8, wherein, When executed by the processor, the instruction causes the system to perform additional operations, including: Before receiving the message from the computing device of the first terminal user: Data is transmitted to the computing device of the first terminal user. When the data is processed by the computing device, a user interface is displayed on the display of the computing device. The user interface displays multiple icons, each of which represents an avatar animation. The first terminal user's selection of an icon will activate the 3D avatar to perform a preview of the avatar animation corresponding to the selected icon.

10. A computer-implemented method, executed by one or more server computers providing message sending and receiving services, the method comprising: A message is received from the computing device of a first terminal user, and the message is addressed to a second terminal user; The message is processed using the following operations: Determine that the second terminal user has a status indicating that the second terminal user is online for the messaging service via an augmented reality (AR) device located in a real-world space associated with an AR space, in which the second terminal user has anchored a 3D avatar representing the first terminal user; The message element received by the message is determined to be associated with the avatar animation; as well as Data is sent to the AR device of the second terminal user. When the data is processed by a messaging application running on the AR device, the messaging application will display the 3D avatar performing the avatar animation via a user interface on the AR device's display.

11. The computer-implemented method according to claim 10, wherein, When the 3D avatar representing the first end user is presented in the user interface on the display of the AR device, the 3D avatar is rendered with an appearance consistent with the avatar configuration data specified by the first end user.

12. The computer-implemented method according to claim 10, wherein, The avatar animation is humanoid movement, including: Posture, facial expressions, gestures, actions performed using props, or any combination thereof.

13. The computer-implemented method according to claim 10, wherein, The message element received in the message is an emoji, and determining that the message element received in the message is associated with an avatar animation includes: The emoji query maps each of a plurality of emojis to a corresponding avatar animation in a data structure, wherein the indication of the avatar animation is returned as the result of the query.

14. The computer-implemented method according to claim 10, wherein, The message element received in the message is a combination of two or more characters, and determining that the message element received in the message is associated with the avatar animation includes: A combination query using the two or more characters maps each of two or more combinations of two or more characters to a corresponding avatar animation data structure, wherein the indication of the avatar animation is returned as a result of the query.

15. The computer-implemented method according to claim 10, wherein, When executed by the processor, the instruction causes the system to perform additional operations, including: The message is processed using the following operations: Data is sent to the AR device of the second terminal user, and when the data is processed by the messaging application running on the AR device, the messaging application will display the text contained in the message in a chat bubble positioned close to the 3D avatar performing the avatar animation.

16. The computer-implemented method according to claim 10, wherein, When executed by the processor, the instruction causes the system to perform additional operations, including: The message is processed using the following operations: Using a text-to-speech algorithm, an audio clip is generated based on the text included in the message; and Data and the audio clip are sent to the AR device of the second terminal user. When the data and the audio clip are processed by the messaging application executed at the AR device, the messaging application will play the audio clip, which is output through the speaker of the AR device, during the presentation of the 3D avatar performing the avatar animation.

17. The computer-implemented method according to claim 10, wherein, When executed by the processor, the instruction causes the system to perform additional operations, including: Before receiving the message from the computing device of the first terminal user: Data is received from the messaging application running on the AR device, the data indicating that the second terminal user is online for the messaging service via the augmented reality (AR) device, the AR device being located in a real-world space associated with an AR space, in which the second terminal user has anchored a 3D avatar representing the first terminal user; Update the status of the second terminal user to indicate that the second terminal user is online for the messaging service via an augmented reality (AR) device located in a real-world space associated with the AR space, in which the second terminal user has anchored a 3D avatar representing the first terminal user; and Data is transmitted to the computing device of the first terminal user, and when the data is processed by the computing device, an presence indication is displayed on the user interface of the display of the computing device, the presence indication indicating the state of the second terminal user.

18. The system according to claim 17, wherein, When executed by the processor, the instruction causes the system to perform additional operations, including: Before receiving the message from the computing device of the first terminal user: Data is transmitted to the computing device of the first terminal user. When the data is processed by the computing device, a user interface is displayed on the display of the computing device. The user interface displays multiple icons, each of which represents an avatar animation. The first terminal user's selection of an icon will activate the 3D avatar to perform a preview of the avatar animation corresponding to the selected icon.

19. A memory device having instructions stored thereon, the instructions causing the system to perform operations when executed by a processor, the operations including: A message is received from the computing device of a first terminal user, and the message is addressed to a second terminal user; The message is processed using the following operations: Determine that the second terminal user has a status indicating that the second terminal user is online for messaging services via an augmented reality (AR) device located in a real-world space associated with an AR space, in which the second terminal user has anchored a 3D avatar representing the first terminal user; The message element received by the message is determined to be associated with the avatar animation; as well as Data is sent to the AR device of the second terminal user. When the data is processed by a messaging application running on the AR device, the messaging application will display the 3D avatar performing the avatar animation via a user interface on the AR device's display.

20. The memory device of claim 19, wherein, The message element received as the message is an emoji, and the method for processing the message by determining that the message element received as the message is associated with an avatar animation further includes: The emoji query maps each of a plurality of emojis to a corresponding avatar animation in a data structure, wherein the indication of the avatar animation is returned as the result of the query.

Images