XR experience based on generative model output

Abstract

Methods and systems are disclosed for operating an augmented reality (XR) experience using one or more machine learning models. The method and system access the XR application through the interactive application and receive a query defining one or more attributes of the XR application through the interactive application. The methods and systems generate prompts for a generative machine learning model using a query, and process the prompts using the generative machine learning model to generate one or more data objects that match one or more attributes defined by the query. The methods and systems generate one or more XR objects using an XR application based on one or more data objects generated by a generative machine learning model in response to a prompt.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of priority of U.S. Provisional Application Serial No. 63 / 460,190, filed April 18, 2023, and U.S. Patent Application Serial No. 18 / 481,932, filed October 5, 2023, each of which is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates generally to operating extended reality (XR) experiences / applications. BACKGROUND

[0004] Augmented reality (AR) is a modification of a virtual environment. For example, in virtual reality (VR), a user is fully immersed in a virtual world, whereas in AR, a user is immersed in a world that combines or superimposes virtual objects on a real world. AR systems are designed to generate and present virtual objects that interact realistically with the real-world environment and with each other. Examples of AR applications can include single- or multi-player video games, instant messaging systems, and the like. Generally, these systems are referred to as XR systems. BRIEF DESCRIPTION OF DRAWINGS

[0005] In the drawings, which are not necessarily drawn to scale, like numerals can describe similar components in different views. To easily identify elements or acts in any particular embodiment, one or more of the elements or acts can be designated with one or more of the following: a prime (′) symbol, a double prime (″) symbol, and / or a superscript letter. Some non-limiting examples are illustrated in the drawings, in which:

[0006] Figure 1 is a diagrammatic representation of a networked environment in which the present disclosure can be deployed according to some examples.

[0007] Figure 2 is a diagrammatic representation of a messaging system having both client-side functionality and server-side functionality according to some examples.

[0008] Figure 3 is a diagrammatic representation of a data structure as maintained in a database according to some examples.

[0009] Figure 4 is a diagrammatic representation of a message according to some examples.

[0010] Figure 5 shows an example architecture for applying personalized personal AI agents to identify relevant characteristics of a user according to some examples.

[0011] Figure 6 is a block diagram of a personal AI system according to some examples.

[0012] Figure 7 、 Figure 8-1 、 Figure 8-2 is a schematic representation of example input and output of an XR application generation system according to some examples.

[0013] Figure 9 is a flowchart illustrating example operations and methods of an XR application generation system according to some examples.

[0014] Figure 10 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions can be executed to cause the machine to perform any one or more of the methods discussed herein, according to some examples.

[0015] Figure 11 is a block diagram illustrating a software architecture, wherein an example can be implemented.

[0016] Figure 12 illustrates a system in which a head-mounted device can be implemented, according to some examples. DETAILED DESCRIPTION

[0017] The description that follows includes systems, methods, techniques, instruction sequences, and computing machine program products embodying illustrative examples of the present disclosure. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various examples. It will be apparent, however, to one skilled in the art that examples can be practiced without some or all of these specific details. In general, well-known instruction instances, protocols, structures, and techniques have not been shown in detail.

[0018] Generally, various communication platforms allow users to share content and create images for transmission to other users. These images can be used to promote products or services and / or simply to represent different real-world objects in simulated or real environments. However, these systems require users to use expensive equipment and techniques to create high-quality, engaging images. Moreover, users can spend significant effort carefully placing objects in different environments and manually adjusting lighting and other image properties to enhance the presentation of the objects in the images. All of these factors can add up to make the creation of high-quality images (e.g., for advertising) a significant expense and detract from the overall use and enjoyment of the system. Moreover, as users can not have the resources required to create high-quality images, opportunities to share and present objects in ideal scenarios are missed. Furthermore, presenting low-quality images of such objects can cause other users to overlook the value of the objects, wasting resources used to create and display the objects.

[0019] Typical users of applications often apply XR applications to modify images depicting real-world objects with virtual objects. These XR applications create more engaging experiences and enable users to connect with each other in unique and interesting ways. However, since the same XR application is shared among groups of users, the diversity of the types of content that users can create is limited. For example, a user of an XR application can create one content item that is very similar to a content item created by another user of the same XR application. This can detract from the overall use and enjoyment of the XR application. Programming an XR application to perform unique and interesting tasks is very time consuming and not easily done by novices and average users. Modifying an XR application to perform a new task involves analyzing the code of the XR application and programming new code to perform the given new task. Since reprogramming an XR application to perform a new task is challenging, users avoid doing so and stop using the XR application after a few times. This results in wasted effort and resources of the device and the XR application.

[0020] As generative machine learning models become ubiquitous, the ability to program new tasks using generative machine learning models becomes easier. That is, generative machine learning models (e.g., generative content artificial intelligence (AI)) are currently popular methods for programming certain operations and actions and can generate content from two-dimensional (2D) models, three-dimensional (3D) models, code, etc. However, due to the instability of their nature, these methods often have limited success in production. That is, in many cases, human agents are still needed to filter and “clean up” the results. For example, a large language model (LLM) that aims to write code can have “spelling mistakes” that hinder its operation. A generative machine learning model that generates 3D models can have too many polygons to work effectively in a game. Attempting to create XR applications based solely on generative machine learning models complicates these challenges because using unoptimized 3D models as content can cause the application to crash.

[0021] The disclosed technology seeks to improve the efficiency of using electronic devices by intelligently and automatically generating code segments for XR applications in a simple and intuitive way to create more engaging and unique experiences that can be shared with other users. This can reduce the total time and expense required to develop high-quality XR applications. Additionally, the disclosed technology utilizes information learned about the user to dynamically generate context-sensitive hints for generating code segments for XR experiences. XR experiences and XR applications are phrases that can be used interchangeably and should be understood to have the same meaning.

[0022] For example, the disclosed technology accesses an XR application by an interaction application and receives a query defining one or more attributes of the XR application by the interaction application. The disclosed technology generates a prompt for a generative machine learning model using the query and processes the prompt using the generative machine learning model to generate one or more data objects (e.g., code segments) matching the one or more attributes defined by the query. The disclosed technology generates one or more XR objects using the XR application based on the one or more data objects generated by the generative machine learning model in response to the prompt. The disclosed technology modifies an image or video depicting a real-world object using the XR application.

[0023] In this way, the disclosed technology improves the overall experience of a user in using an electronic device and reduces the total amount of resources required to complete the task of making a high-quality and unique XR application. As used herein, “article of apparel,” “fashion item,” and “apparel” can be used interchangeably and should be understood to have the same meaning. An article of apparel, apparel, or fashion item can include a shirt, a skirt, a dress, a shoe, a purse, a furniture item, a home item, an eyewear article, eyeglasses, an AR badge, an AR patch, pants, shorts, a jacket, a t-shirt, a vest, eyeglasses, jewelry, earrings, rabbit ears, a hat, an ear warmer, a cosmetic, or any other suitable item or object.

[0024] Networked computing environment

[0025] Figure 1 is a block diagram illustrating an example interaction system 100 for facilitating interactions (e.g., exchanging text messages, making text, audio, and video calls, or playing games) over a network. The interaction system 100 includes a plurality of user systems 102, each of which hosts a plurality of applications including an interaction client 104 and other applications 106. Each interaction client 104 is communicatively coupled to other instances of the interaction client 104 (e.g., hosted on respective other user systems 102), an interaction server system 110, and third-party servers 112 via one or more communication networks including a network 108 (e.g., the Internet). The interaction client 104 can also communicate with locally-hosted applications 106 using an application program interface (API).

[0026] Each user system 102 can include a plurality of user devices, such as a mobile device 114, a head-mounted device 116, and a computer client device 118, which are communicatively connected to exchange data and messages.

[0027] The interaction clients 104 interact with other interaction clients 104 and with the interaction server system 110 via the network 108. Data exchanged between interaction clients 104 (e.g., interactions 120) and between interaction clients 104 and the interaction server system 110 includes functionality (e.g., commands to activate functionality) and payload data (e.g., text, audio, video, or other multimedia data).

[0028] The interaction server system 110 provides server-side functionality to the interaction clients 104 via the network 108. While certain functionality of the interaction system 100 is described herein as being performed by the interaction clients 104 or by the interaction server system 110, whether certain functionality is located within the interaction clients 104 or within the interaction server system 110 can be a design choice. For example, it can be technically preferable to initially deploy a particular technology or functionality within the interaction server system 110, but later migrate that technology or functionality to an interaction client 104 that has sufficient processing power of the user system 102.

[0029] The interaction server system 110 supports various services and operations provided to the interaction clients 104. Such operations include sending data to the interaction clients 104, receiving data from the interaction clients 104, and processing data generated by the interaction clients 104. This data can include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, entity relationship information, and live event information. Data exchange within the interaction system 100 is activated and controlled through functionality available via the user interfaces of the interaction clients 104.

[0030] Turning now to the interaction server system 110 in particular, the API server 122 is coupled to, and provides a programmatic interface to, the interaction server 124 that exposes the functionality of the interaction server 124 to the interaction clients 104, other applications 106, and third-party servers 112. The interaction server 124 is communicatively coupled to a database server 126 that facilitates access to a database 128 that stores data associated with interactions processed by the interaction server 124. Similarly, the web server 130 is coupled to the interaction server 124 and provides a web-based interface to the interaction server 124. In support of the web interface, the web server 130 processes incoming network requests over the Hypertext Transfer Protocol (HTTP) and several related protocols.

[0031] The API server 122 receives and sends interaction data (e.g., command and message payloads) between the interaction server 124 and the user systems 102 (and, e.g., the interaction clients 104 and other applications 106) and the third-party servers 112. Specifically, the API server 122 provides a set of interfaces (e.g., routines and protocols) that the interaction clients 104 and other applications 106 can call or query to activate functionality of the interaction server 124. The API server 122 exposes various functions supported by the interaction server 124, including account registration; login functionality; sending interaction data from a particular interaction client 104 to another interaction client 104 via the interaction server 124; transferring media files (e.g., images or videos) from an interaction client 104 to the interaction server 124; setting a collection of media data (e.g., a story); retrieving a list of friends of a user of a user system 102; retrieving information and content; adding and deleting entities (e.g., friends) to an entity relationship graph (e.g., the entity graph 310); pinning friends within an entity relationship graph; and opening an application event (e.g., related to an interaction client 104).

[0032] The interaction server 124 hosts a number of systems and subsystems, described below with reference to Figure 2

[0033] Linked applications

[0034] Returning to the interaction clients 104, features and functionality of external resources (e.g., linked applications 106 or widgets) are made available to users via the interface of the interaction clients 104. In this context, “external” refers to the fact that the applications 106 or widgets are external to the interaction clients 104. External resources are typically provided by third parties, but can also be provided by the creator or provider of the interaction clients 104. The interaction clients 104 receive user selections of options to launch or access features of such external resources. The external resources can be applications 106 installed on the user systems 102 (e.g., “native apps”), or small-scale versions of applications (e.g., “widgets”) hosted on the user systems 102 or located remotely from the user systems 102 (e.g., on the third-party servers 112). The small-scale versions of applications include a subset of the features and functionality of the applications (e.g., full-scale native versions of the applications), and are implemented using markup language documents. In some examples, the small-scale versions of applications (e.g., “widgets”) are web-based markup language versions of the applications, and are embedded in the interaction clients 104. In addition to using markup language documents (e.g.,.html files), the widgets can include scripting languages (e.g.,.js files or.json files) and style sheets (e.g.,.ss files).

[0035] ​In response to receiving a user selection of an option to launch or access a feature for an external resource, the interaction client 104 determines whether the selected external resource is a web-based external resource or a locally installed application 106. In some cases, an application 106 that is locally installed on the user system 102 can be launched independently of and separately from the interaction client 104, such as by selecting an icon corresponding to the application 106 on a home screen of the user system 102. A small-scale version of such an application can be launched or accessed via the interaction client 104, and in some examples, no part of the small-scale application can be accessed outside of the interaction client 104 or only limited portions of the small-scale application can be accessed outside of the interaction client 104. The small-scale application can be launched by the interaction client 104 receiving a markup language document associated with the small-scale application from the third-party server 112 and processing such a document.

[0036] In response to determining that the external resource is a locally installed application 106, the interaction client 104 instructs the user system 102 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interaction client 104 communicates with the third-party server 112, for example, to obtain a markup language document corresponding to the selected external resource. The interaction client 104 then processes the obtained markup language document to present the web-based external resource within the user interface of the interaction client 104.

[0037] The interaction client 104 can notify a user of the user system 102 or other users (e.g., "friends") associated with such a user of activity occurring in one or more external resources. For example, the interaction client 104 can provide participants in a conversation (e.g., a chat session) in the interaction client 104 with notifications related to external resources currently or recently used by one or more members of a group of users. One or more users can be invited to join an active external resource or launch a recently used but not currently active external resource (of the group of friends). An external resource can provide participants in a conversation each using a respective interaction client 104 with the ability to share an item, condition, status, or location in the external resource with one or more members of a group of users in a chat session. The shared item can be an interactive chat card that members of the chat can utilize to interact with, such as to launch a corresponding external resource, view particular information within the external resource, or bring the member of the chat to a particular location or status within the external resource. Within a given external resource, a response message can be sent to the user on the interaction client 104. The external resource can selectively include different media items in the response based on a current context of the external resource.

[0038] The interaction client 104 can present a list of available external resources (e.g., applications 106 or widgets) to the user to launch or access a given external resource. The list can be presented in the manner of a context-sensitive menu. For example, icons representing different applications (or widgets) of the applications 106 (or widgets) can vary based on how the user launches the menu (e.g., from a conversational interface or from a non-conversational interface).

[0039] System Architecture

[0040] Figure 2 is a block diagram illustrating additional details regarding the interaction system 100, according to some examples. In particular, the interaction system 100 is shown to include the interaction client 104 and the interaction server 124. The interaction system 100 contains a number of subsystems that are supported on the client side by the interaction client 104 and on the server side by the interaction server 124.

[0041] In some examples, these subsystems are implemented as microservices. A microservice subsystem (e.g., a microservice application) can have components that enable it to operate independently and communicate with other services. Example components of a microservice subsystem can include:

[0042] • Functional logic: The functional logic implements the functionality of the microservice subsystem, representing a particular capability or function that the microservice provides.

[0043] • API interface: The microservice can communicate with other components using a well-defined API or interface, such as REST or messaging, using a lightweight protocol. The API interface defines the inputs and outputs of the microservice subsystem and how it interacts with other microservice subsystems of the interaction system 100.

[0044] • Data storage: The microservice subsystem can be responsible for its own data storage, which can be in the form of a database, cache, or other storage mechanism (e.g., using the database server 126 and the database 128). This enables the microservice subsystem to operate independently of other microservices of the interaction system 100.

[0045] • Service discovery: The microservice subsystem can find and communicate with other microservice subsystems of the interaction system 100. The service discovery mechanism enables the microservice subsystem to locate and communicate with other microservice subsystems in a scalable and efficient manner.

[0046] • Monitoring and logging: The microservice subsystem can need to be monitored and logged to ensure availability and performance. The monitoring and logging mechanism enables tracking of the health and performance of the microservice subsystem.

[0047] In some examples, the interaction system 100 can employ a monolithic architecture, a service-oriented architecture (SOA), a function-as-a-service (FaaS) architecture, or a modular architecture:

[0048] The image processing system 202 provides various functionality that enables a user to capture and enhance (e.g., annotate or otherwise modify or edit) media content associated with a message.

[0049] The camera system 204 includes control software (e.g., in a camera application) that interacts with and controls the camera hardware of the user system 102 (e.g., directly or via an operating system) to modify and enhance live images captured and displayed via the interaction client 104.

[0050] The augmentation system 206 provides functionality related to the generation and publication of augmentations (e.g., media overlays) for images captured in real-time by the camera of the user system 102 or images retrieved from the memory of the user system 102. For example, the augmentation system 206 is operable to select, present, and display media overlays (e.g., image filters or image lenses) for the interaction client 104 for use in augmenting live images received via the camera system 204 or stored images retrieved from the memory 1202 (shown in FIG. 1) of the user system 102. These augmentations are selected by the augmentation system 206 based on some inputs and data, such as: Figure 12

[0051] • the geolocation of the user system 102; and

[0052] • entity relationship information of the user of the user system 102.

[0053] Augmentations can include audio and visual content as well as visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. The audio and visual content or visual effects can be applied to a media content item (e.g., a photo or video) at the user system 102 for communication in a message or to video content, such as a video content stream or feed sent from the interaction client 104. Thus, the image processing system 202 can interact with and support various subsystems of the communication system 208, such as the messaging system 210 and the video communication system 212.

[0054] ​Media overlays can include text or image data that can be overlaid on a photo taken by a user system 102 or a video stream made by a user system 102. In some examples, a media overlay can be a location overlay (e.g., Venice Beach), a name of a live event, or a business name overlay (e.g., Beachside Café). In other examples, the image processing system 202 uses the geolocation of a user system 102 to identify a media overlay that includes the name of a business at the geolocation of the user system 102. The media overlay can include other indicia associated with the business. The media overlays can be stored in the database 128 and accessed by the database server 126.

[0055] The image processing system 202 provides a user-based publishing platform that enables a user to select a geolocation on a map and upload content associated with the selected geolocation. The user can also specify circumstances under which a particular media overlay should be provided to other users. The image processing system 202 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geolocation.

[0056] The augmentation creation system 214 supports an AR developer platform and includes an application for content creators (e.g., artists and developers) to create and publish augmentations (e.g., AR experiences) for the interactive client 104. The augmentation creation system 214 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking techniques, and templates.

[0057] In some examples, the augmentation creation system 214 provides a business-based publishing platform that enables businesses to select particular augmentations associated with a geolocation via a bidding process. For example, the augmentation creation system 214 associates a media overlay of the highest bidding business with a corresponding geolocation for a predefined amount of time.

[0058] The communication system 208 is responsible for enabling and handling various forms of communication and interaction within the interactive system 100 and includes a messaging system 210, an audio communication system 216, and a video communication system 212. The messaging system 210 is responsible for enforcing temporary or time-limited access to content by the interactive client 104. The messaging system 210 includes a plurality of timers (within the user management system 218) that selectively enable access (e.g., for presentation and display) of messages and associated content via the interactive client 104 based on a duration and display parameters associated with a message or collection of messages (e.g., a story). The audio communication system 216 enables and supports audio communication (e.g., real-time audio chat) between a plurality of interactive clients 104. Similarly, the video communication system 212 enables and supports video communication (e.g., real-time video chat) between a plurality of interactive clients 104.

[0059] The user management system 218 is operationally responsible for managing user data and profiles, and maintaining entity information regarding the relationships between users and users of the interaction system 100 (e.g., stored in...). Figure 3 (In entity table 308, entity diagram 310, and profile data 302).

[0060] The collection management system 220 is operationally responsible for managing collections or sets of media (e.g., collections of text, images, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be made available for a specified time period (e.g., the duration of the event to which the content relates). For example, content related to a concert can be available as a "story" for the duration of the concert. The collection management system 220 can also be responsible for publishing icons that notify the user interface of the interactive client 104 of the availability of specific collections. The collection management system 220 includes curation functions that enable collection managers to manage and curate specific content collections. For example, a curation interface enables event organizers to curate collections of content related to a specific event (e.g., removing inappropriate content or redundant messages). Additionally, the collection management system 220 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, users may be compensated for including user-generated content in a collection. In such cases, the collection management system 220 operates to automatically pay such users for using their content.

[0061] Map system 222 provides various geolocation (geographic location) functions and supports the presentation of map-based media content and messages by interactive client 104. For example, map system 222 enables the display (e.g., stored on) maps. Figure 3 The user's profile data 302 (in which user icons or avatars are used) indicates the current or past location of the user's "friends" within the context of the map, as well as media content generated by these friends (e.g., a collection of messages including photos and videos). For example, on the map interface of interactive client 104, messages posted by the user from a specific geographic location to interactive system 100 can be displayed to the specific user's "friends" within the context of that specific location on the map. The user can also share his or her location and status information with other users of interactive system 100 via interactive client 104 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to selected users within the context of the map interface of interactive client 104.

[0062] The game system 224 provides various game functionality within the context of the interaction client 104. The interaction client 104 provides a game interface that provides a list of available games that can be launched by a user within the context of the interaction client 104 and played with other users of the interaction system 100. The interaction system 100 also enables a particular user to invite such other users to participate in playing a particular game by issuing invitations from the interaction client 104 to the other users. The interaction client 104 also supports audio, video, and text messaging (e.g., chat) within the context of playing games, provides leaderboards for games, and also supports providing in-game rewards (e.g., game currency and items).

[0063] The external resource system 226 provides an interface for the interaction client 104 to communicate with remote servers (e.g., third-party servers 112) to launch or access external resources (i.e., applications or applets). Each third-party server 112 hosts an application or a small-scale version of an application (e.g., a game application, a utility application, a payment application, or a ride-sharing application) based on a markup language (e.g., HTML5), for example. The interaction client 104 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from the third-party server 112 associated with the web-based resource. The application hosted by the third-party server 112 is programmed in JavaScript with a software development kit (SDK) provided by the interaction server 124. The SDK includes APIs with functionality that can be called or activated by the web-based application. The interaction server 124 hosts a JavaScript library that provides given external resource access to certain user data of the interaction client 104. HTML5 is an example of a technology used to program games, but applications and resources programmed based on other technologies can be used.

[0064] To integrate the functionality of the SDK into the web-based resource, the SDK is downloaded by the third-party server 112 from the interaction server 124 or otherwise received by the third-party server 112. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functionality of the SDK to integrate features of the interaction client 104 into the web-based resource.

[0065] The SDK stored on the interaction server system 110 effectively provides a bridge between external resources (e.g., applications 106 or widgets) and the interaction client 104. This gives the user a seamless experience of communicating with other users on the interaction client 104 while also preserving the look and feel of the interaction client 104. To bridge the communication between external resources and the interaction client 104, the SDK facilitates the communication between the third party servers 112 and the interaction client 104. The BridgeScript running on the user system 102 establishes two one-way communication channels between the external resources and the interaction client 104. Messages are sent asynchronously between the external resources and the interaction client 104 via these communication channels. Each SDK function activation is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with the callback identifier.

[0066] By using the SDK, not all information from the interaction client 104 is shared with the third party servers 112. The SDK limits which information is shared based on the needs of the external resources. Each third party server 112 provides the interaction server 124 with an HTML5 file corresponding to the web-based external resource. The interaction server 124 can add a visual representation (e.g., a box design or other graphic) of the web-based external resource in the interaction client 104. Once the user selects the visual representation or indicates through the graphical user interface (GUI) of the interaction client 104 that the interaction client 104 is to access a feature of the web-based external resource, the interaction client 104 obtains the HTML5 file and instantiates the resource for accessing the feature of the web-based external resource.

[0067] The interaction client 104 presents a GUI (e.g., a landing page or title screen) for the external resource. During, before, or after presenting the landing page or title screen, the interaction client 104 determines whether the launched external resource has been previously authorized to access user data of the interaction client 104. In response to determining that the launched external resource has been previously authorized to access user data of the interaction client 104, the interaction client 104 presents another GUI of the external resource that includes functionality and features of the external resource. In response to determining that the launched external resource has not been previously authorized to access user data of the interaction client 104, after displaying the landing page or title screen of the external resource for a threshold period of time (e.g., 3 seconds), the interaction client 104 slides up a menu (e.g., animates the menu to float up from the bottom of the screen to the middle or other portion of the screen) for authorizing the external resource to access user data. The menu identifies types of user data that the external resource will be authorized to use. In response to receiving a user selection of an accept option, the interaction client 104 adds the external resource to a list of authorized external resources and allows the external resource to access user data from the interaction client 104. The external resource is authorized by the interaction client 104 to access user data under an OAuth 2 framework.

[0068] The interaction client 104 controls types of user data shared with external resources based on types of the external resources that are authorized. For example, external resources including full-scale applications (e.g., the application 106) are provided access to a first type of user data (e.g., 2D avatars of users with or without different avatar characteristics). As another example, external resources including small-scale versions of applications (e.g., web-based versions of the application) are provided access to a second type of user data (e.g., payment information, 2D avatars of users, 3D avatars of users, and avatars with various avatar characteristics). Avatar characteristics include different ways of customizing the appearance of an avatar (e.g., different poses, facial features, clothing, etc.).

[0069] The advertising system 228 is operable to enable third parties to purchase advertisements for presentation to end users via the interaction client 104 and also handles the delivery and presentation of these advertisements.

[0070] The artificial intelligence and machine learning system 230 provides various services to different subsystems within the interaction system 100. For example, the artificial intelligence and machine learning system 230 operates with the image processing system 202 and the camera system 204 to analyze images and extract information such as objects, text, or faces. This information can then be used by the image processing system 202 to augment, filter, or manipulate images. The artificial intelligence and machine learning system 230 can be used by the augmentation system 206 to generate augmented content, XR experiences, and AR experiences, such as adding virtual objects or animations to real-world images. The communication system 208 and the messaging system 210 can use the artificial intelligence and machine learning system 230 to analyze communication patterns and provide insights into how users interact with each other and provide intelligent message classification and tagging, such as classifying messages based on sentiment or topic. The artificial intelligence and machine learning system 230 can also provide chatbot functionality for the message interactions 120 between the user systems 102 and between the user systems 102 and the interaction server system 110. The artificial intelligence and machine learning system 230 can also work with the audio communication system 216 to provide speech recognition and natural language processing capabilities, allowing users to interact with the interaction system 100 using voice commands. The artificial intelligence and machine learning system 230 can also provide personalized AI agent system 232 functionality for the message interactions 120 between the user systems 102 and between the user systems 102 and the interaction server system 110.

[0071] In some cases, the artificial intelligence and machine learning system 230 can implement one or more machine learning models that generate code snippets for XR applications or other applications or features of the interaction client 104. The code snippets can be generated based on prompts that are defined or generated based at least in part on user queries that define attributes or desired content for the XR application and one or more templates associated with the XR application. Once the code snippets are received from the one or more machine learning models, the templates for the XR application or other application are populated and used to generate and provide unique and engaging XR experiences to end users of the interaction client 104.

[0072] Broadly, machine learning can involve using computer algorithms to automatically learn patterns and relationships in data, potentially without explicit programming to do so after the algorithms are trained. Examples of machine learning algorithms can be divided into three broad categories: supervised learning, unsupervised learning, and reinforcement learning.

[0073] • Supervised learning involves training a model using labeled data to predict an output for new, unseen inputs. Examples of supervised learning algorithms include linear regression, decision trees, and neural networks.

[0074] • Unsupervised learning involves training a model on unlabeled data to find hidden patterns and relationships in the data. Examples of unsupervised learning algorithms include clustering, principal component analysis, and generative models such as autoencoders.

[0075] • Reinforcement learning involves training a model to make decisions in a dynamic environment by receiving feedback in the form of rewards or penalties. Examples of reinforcement learning algorithms include Q-learning and policy gradient methods.

[0076] According to some examples, examples of particular machine learning algorithms that can be deployed include logistic regression, which is a type of supervised learning algorithm for binary classification tasks. Logistic regression models the probability of a binary response variable based on one or more predictor variables. Another example type of machine learning algorithm is Naive Bayes, which is another supervised learning algorithm for classification tasks. Naive Bayes is based on Bayes’ theorem and assumes that the predictor variables are independent of each other. Random forests are another type of supervised learning algorithm for classification, regression, and other tasks. Random forests construct a collection of decision trees and combine their outputs for prediction. Further examples include neural networks, which are composed of interconnected layers of nodes (or neurons) that process information and make predictions based on input data. Matrix factorization is another type of machine learning algorithm for recommendation systems and other tasks. Matrix factorization decomposes a matrix into two or more matrices to discover hidden patterns or relationships in the data. Support vector machines (SVM) are a type of supervised learning algorithm for classification, regression, and other tasks. SVMs find a hyperplane that separates different classes in the data. Other types of machine learning algorithms include decision trees, k-nearest neighbors, clustering algorithms, and deep learning algorithms such as convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformer models. The choice of algorithm depends on the nature of the data, the complexity of the problem, and the performance requirements of the application.

[0077] Although several particular examples of machine learning algorithms are discussed herein, the principles discussed herein can also be applied to other machine learning algorithms. Deep learning algorithms such as convolutional neural networks, recurrent neural networks, and Transformers can be used in various machine learning applications, as well as more traditional machine learning algorithms such as decision trees, random forests, and gradient boosting.

[0078] Generating trained machine learning programs, e.g., artificial intelligence and machine learning systems 230 can include multiple types of stages that form part of a machine learning pipeline, for example including the following stages:

[0079] • Data collection and pre-processing: This can include acquiring and cleaning data to ensure it is suitable for use in machine learning models. Data can be collected from user content creation and labeled using machine learning algorithms trained to label data. Data can be generated by applying machine learning algorithms to identify or generate similar data. This can also include removing duplicates, handling missing values, and converting data into a suitable format.

[0080] • Feature engineering: This can include selecting and transforming training data to create features useful for predicting the target variable. Feature engineering can include (1) receiving features (e.g., as structured data or labeled data in supervised learning) and / or (2) identifying features in training data (e.g., unstructured data or unlabeled data for unsupervised learning).

[0081] • Model selection and training: This can include specifying the particular problem or desired response from the input data, selecting an appropriate machine learning algorithm and training it on the pre-processed data. This can also include splitting data into training and testing sets, using cross-validation to evaluate the model, and adjusting hyperparameters to improve performance. Model selection can be based on factors such as data type, problem complexity, computational resources, or desired performance.

[0082] • Model evaluation: This can include evaluating the performance of a trained model (e.g., a trained machine learning program) on a separate test data set. This can help determine whether the model is overfitting or underfitting and whether the model is ready for deployment.

[0083] • Prediction: This involves using a trained model (e.g., a trained machine learning program) to generate predictions for new, unseen data.

[0084] • Validation, refinement, or retraining: This can include updating the model based on feedback generated from the prediction stage, such as new data or user feedback.

[0085] • Deployment: This can include integrating a trained model (e.g., a trained machine learning program) into a larger system or application, such as a web service, mobile application, or IoT device. This can involve installing APIs, setting up user interfaces, and ensuring the model is scalable and can handle large amounts of data.

[0086] Prior to the training phase, feature engineering is used to identify features. This can include identifying informative, discriminative, and independent features for effective operation of the trained machine learning program in pattern recognition, classification, and regression. In some examples, the training data includes labeled data of known data for pre-identified features and one or more outcomes.

[0087] Each of the features can be a variable or attribute, such as an individual measurable property of a process, an item, a system, or a phenomenon being observed, for example, represented by a dataset (e.g., training data). The features can also be of different types, such as numeric features, strings, vectors, matrices, encodings, and graphs, and can include data obtained from multi-modal memory 508. Conceptual features can include abstract relationships or patterns in the data, such as determining a topic of a document or a discussion in a chat window between users. Content features include determining a context based on input information, such as determining a user’s context based on user interactions or ambient environmental factors. Contextual features can include textual features, such as frequency or preference of words or phrases, image features, such as pixel, texture, or pattern recognition, audio classification, such as spectrograms, etc. Attribute features include intrinsic attributes (directly observable) or extrinsic features (derived), such as identifying square footage, location, or age of a property identified in a camera feed. User data features include data related to a particular individual or a group of individuals, for example, in a geographic location or sharing demographic characteristics. User data can include demographic data (e.g., age, gender, location, or occupation), user behavior (e.g., browsing history, purchase history, conversion rate, click-through rate, or engagement metrics), or user preferences (e.g., preference for certain video, text, or digital content items). Historical data includes past events or trends that can help identify patterns or relationships over time.

[0088] In the training phase, the machine learning pipeline uses the training data to find correlations between the features that affect the predicted outcome or the predicted / inferred data. With the training data and the identified features, the trained machine learning program is trained during the training phase during the machine learning program training. The machine learning program training evaluates the values of the features as they relate to the training data. The result of the training is the trained machine learning program (e.g., a trained or learned model).

[0089] Further, the training phase can involve machine learning in which the training data is structured (e.g., labeled during pre-processing operations) and the trained machine learning program implements a relatively simple neural network capable of performing, for example, classification and clustering operations. In other examples, the training phase can involve deep learning in which the training data is unstructured and the trained machine learning program implements a deep neural network capable of performing both feature extraction and classification / clustering operations. The neural network includes a hierarchical (e.g., layered) organization of neurons, where each layer includes multiple neurons or nodes. The neurons in the input layer receive input data, while the neurons in the output layer produce the final output of the network. Between the input and output layers, there can be one or more hidden layers, each including multiple neurons.

[0090] Each neuron in a neural network operatively computes a small function, such as an activation function, that takes as input the weighted sum of the outputs of the neurons in the previous layer and a bias term. The output of this function is then passed as input to the neurons in the next layer. If the output of the activation function exceeds a certain threshold, the output is transmitted from the neuron (e.g., sending neuron) to the connected neuron (e.g., receiving neuron) in the successive layer. The connections between neurons have associated weights that define the influence of the input from the sending neuron to the receiving neuron. During a training phase, these weights are adjusted by a learning algorithm to optimize the performance of the network. Different types of neural networks can use different activation functions and learning algorithms, which can affect their performance on different tasks. In summary, the hierarchical organization of neurons, along with the use of activation functions and weights, enables neural networks to model complex relationships between inputs and outputs, and generalize to new inputs not seen during training.

[0091] In some examples, the neural network can also be one of several different types of neural networks or combinations thereof, such as, by way of example only, a single-layer feedforward network, a multilayer perceptron (MLP), an artificial neural network (ANN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a bidirectional neural network, a connectionist temporal classification neural network, a deep belief network (DBN), a convolutional neural network (CNN), a generative adversarial network (GAN), an autoencoder neural network (AE), a restricted Boltzmann machine (RBM), a Hopfield network, a self-organizing map (SOM), a radial basis function network (RBFN), a spiking neural network (SNN), a liquid state machine (LSM), an echo state network (ESN), a neural Turing machine (NTM), or a Transformer network.

[0092] A neural network can be iteratively trained by adjusting model parameters to minimize a particular loss function or maximize some objective. The system can continue training the neural network by adjusting the parameters based on the refined output of the validation or retraining the block and re-running the prediction against new or already run training data. The system can employ optimization techniques for these adjustments, such as gradient descent algorithms, momentum algorithms, Nesterov accelerated gradient (NAG) algorithms, and the like. The system can continue to iteratively train the neural network even after deploying the neural network. The neural network can be continually trained, for example, based on user-created or system-generated training data as new data emerges.

[0093] In some examples, the trained machine learning programs, such as the personalized Al agent system 232, can be generative Al models. Generative Al is a term that can refer to any type of artificial intelligence that can create new content from training data. For example, generative Al can produce text, images, videos, audio, code, or synthetic data that is similar to, but not identical to, the original data.

[0094] Some techniques that can be used in generative Al are:

[0095] • Convolutional Neural Networks (CNNs): CNNs are commonly used for image recognition and computer vision tasks. They are designed to extract features from images by using filters or kernels that scan the input image and highlight important patterns. CNNs can be used for applications such as object detection, facial recognition, and autonomous driving.

[0096] • Recurrent Neural Networks (RNNs): RNNs are designed to handle sequential data such as speech, text, and time series data. They have a feedback loop that allows them to capture temporal dependencies and remember past inputs. RNNs can be used for applications such as speech recognition, machine translation, and sentiment analysis.

[0097] • Generative Adversarial Networks (GANs): These are models composed of two neural networks: a generator and a discriminator. The generator tries to create realistic content that can “fool” the discriminator, while the discriminator tries to distinguish between real and fake content. The two networks compete with each other and improve over time. GANs can be used for applications such as image synthesis, video prediction, and style transfer.

[0098] • Variational Autoencoders (VAEs): These are models that encode input data into a latent space (compressed representation) and then decode it back to output data. The latent space can be manipulated to generate new variations of the output data. They can use self-attention mechanisms to process input data, allowing them to handle long sequences of text and capture complex dependencies.

[0099] • Transformer Models: These are models that use attention mechanisms to learn the relationships between different parts of input data (e.g., words or pixels) and generate output data based on these relationships. Transformer models can handle sequential data such as text or speech, as well as non-sequential data such as images or code.

[0100] In generative Al examples, the outputted predictive / inferential data includes trend assessment and prediction, translation, summarization, image or video recognition and classification, natural language processing, facial recognition, user sentiment assessment, ad targeting and optimization, speech recognition or media content generation, recommendations and personalization.

[0101] The personalized AI agent system 232 provides personalized features to users of the interactive client 104 by analyzing user data and behavior to understand their preferences and interests. By utilizing machine learning algorithms and data analysis, the personalized AI agent system 232 can learn and adapt to a user’s inferences and then generatively suggest content that is relevant, specific, and customized to the user. The personalized AI agent system 232 can analyze data from multiple sources such as various user systems 102, messages, profile information, external data sources, image data captured in real-time by a camera of the user system 102, and / or any combination thereof to generate content items (e.g., codes and / or prompts) in real-time and provide such content items to the user.

[0102] The personalized AI agent system 232 tracks user activity, such as posts that the user likes, shares, or comments on, topics that the user follows, people that the user connects with, and the time that the user spends on the platform. The tracking performed by the personalized AI agent system 232 is only enabled when the user opts in to receive the experience of real-time generated content. The personalized AI agent system 232 can present the user with a complete list of all activities and information that will be tracked and used to generate real-time content recommendations. Only after receiving confirmation from the user that the user consents to such activities and information being tracked, does the personalized AI agent system 232 begin collecting such data and using such data to provide and generate real-time and instant content to present to the user.

[0103] The personalized AI agent system 232 can retrieve data from multiple data sources, such as activity on the user’s phone, AR / VR device, smartwatch, laptop, or other user devices. Based on this information, the personalized AI agent system 232 can identify patterns and predict the user’s interests to generate a multi-modal memory for a particular user. The personalized AI agent system 232 analyzes the user’s profile information, such as their age, gender, location, exchanged messages, and / or interactions performed on the user system 102 to provide personalized features. In some examples, the personalized AI agent system 232 suggests nearby events and groups, or recommends job opportunities that match the user’s qualifications. The personalized AI agent system 232 can generate real-time AR experiences and / or message content that is relevant to the current situation and / or real-world environment perceived by the user.

[0104] Further, the personalized Al agent system 232 analyzes user-created content and suggests the best time to post, the best tags to use, and the type of content that gets the most engagement. By doing so, the personalized Al agent system 232 helps users increase their visibility and reach a wider audience. In this way, the personalized Al agent system 232 can evaluate data from different devices to provide personalized features through a variety of different devices. The personalized Al agent system 232 provides such personalized features based on the multi-modal memory associated with the user and in real-time automatically in the communication channel of the specific content preferred by the user. Analyzing user data and behavior to understand their preferences and interests and then suggesting and generating content relevant to the user not only enhances the user experience but also improves engagement and retention on the platform.

[0105] Data Architecture

[0106] Figure 3 FIG. 3 is a schematic diagram illustrating a data structure 300 that can be stored in the database 304 of the interaction server system 110, in accordance with certain examples. While the contents of the database 304 are illustrated as including a plurality of tables, it will be appreciated that data can be stored in other types of data structures (e.g., as an object-oriented database).

[0107] The database 304 includes message data stored within a message table 306. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and a payload. Reference is made below to FIG. 4 for a more detailed description of the message data. Figure 4 Additional details are described below regarding information that can be included in a message and included within the message data stored in the message table 306.

[0108] An entity table 308 stores entity data and is linked (e.g., referentially) to the entity graph 310 and the profile data 302. Entities for which records are maintained within the entity table 308 can include individuals, corporate entities, organizations, objects, locations, events, and the like. Regardless of the entity type, any entity for which the interaction server system 110 stores data can be an identified entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0109] The entity graph 310 stores information regarding relationships and associations between entities. Such relationships can be, by way of example only, social relationships based on interests or based on activities, professional relationships (e.g., working at a common corporation or organization). Certain relationships between entities can be unidirectional, such as a personal user's subscription to a commercial or publication user's (e.g., a newspaper or other digital media outlet or brand) digital content. Other relationships can be bidirectional, such as a "friend" relationship between various users of the interaction system 100.

[0110] Certain permissions and relationships can be attached to each relationship, and also to each direction of the relationship. For example, a bidirectional relationship (e.g., a friendship relationship between respective users) can include authorization for publication of digital content items between the respective users, but can impose certain restrictions or filters on the publication of such digital content items (e.g., based on content properties, location data, or time-of-day data). Similarly, a subscription relationship between a personal user and a business user can impose different degrees of restriction on publication of digital content from the business user to the personal user, and can significantly restrict or prevent publication of digital content from the personal user to the business user. A particular user, as an example of an entity, can record certain restrictions in the record for that entity within the entity table 308 (e.g., by way of privacy settings). Such privacy settings can apply to all types of relationships in the context of the interaction system 100, or can selectively apply to certain types of relationships.

[0111] The profile data 302 stores various types of profile data about a particular entity. Based on privacy settings specified by the particular entity, the profile data 302 can be selectively used and presented to other users of the interaction system 100. In the case of an entity being a person, the profile data 302 includes, for example, a username, a phone number, an address, settings (e.g., notification and privacy settings), and a user-selected avatar representation (or a collection of such avatar representations). The particular user can then selectively include one or more of these avatar representations within the content of messages transmitted via the interaction system 100 and on map interfaces displayed by the interaction client 104 to other users. The collection of avatar representations can include a "status avatar" that presents a graphical representation of a status or activity that the user can select to transmit at a particular time.

[0112] In the case of an entity being a group, the profile data 302 for the group can similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings (e.g., notifications) for the relevant group.

[0113] The database 304 also stores augmentation data, such as overlays or filters, in an augmentation table 312. The augmentation data is associated with and applied to videos (data for which is stored in a video table 314) and images (data for which is stored in an image table 316).

[0114] In some examples, filters are overlays that are displayed as overlays on images or videos during presentation to recipient users. Filters can be of various types, including a user-selected filter from a set of filters presented to a sender user by the interactive client 104 while the sender user is composing a message. Other types of filters include geolocation filters (also referred to as geo filters), which can be presented to a sender user based on geolocation. For example, a geolocation filter specific to a nearby or special location can be presented by the interactive client 104 within a user interface based on geolocation information determined by a global positioning system (GPS) unit of the user system 102.

[0115] Another type of filter is a data filter, which can be selectively presented to a sender user by the interactive client 104 based on other inputs or information collected by the user system 102 during the message creation process. Examples of data filters include a current temperature at a particular location, a current speed at which the sender user is traveling, a battery life of the user system 102, or a current time.

[0116] Other augmentation data that can be stored within the image table 316 includes AR content items (e.g., corresponding to an application “lens” or XR experience). XR content items (e.g., XR objects) can be real-time special effects and sounds that can be added to images or videos. Any discussion regarding XR content and / or XR experiences and XR applications can similarly apply to AR and VR content, experiences, and / or applications.

[0117] The collection table 318 stores data regarding collections of messages and associated image, video, or audio data that are compiled into a collection (e.g., a story or gallery). Creation of a particular collection can be initiated by a particular user (e.g., each user for which a record is maintained in the entity table 308). A user can create a “personal story” in the form of a collection of content that has been created and sent / broadcast by that user. To this end, a user interface of the interactive client 104 can include a user-selectable icon to enable a sender user to add particular content to his or her personal story.

[0118] A collection can also constitute a “live story,” which is a collection of content from multiple users created manually, automatically, or using a combination of manual and automatic techniques. For example, a “live story” can constitute a curated stream of user-submitted content from different locations and events. Users whose client devices have location services enabled and are at a common location event at a particular time can be presented with an option to contribute content to a particular live story, e.g., via a user interface of the interactive client 104. A live story can be identified to a user by the interactive client 104 based on the user’s location. The end result is a “live story” told from a group perspective.

[0119] Another type of content collection is referred to as a “location story,” which enables users whose user systems 102 are located within a particular geographic location (e.g., at a college or university campus) to contribute to a particular collection. In some examples, contributions to a location story can employ secondary authentication to verify that an end user belongs to a particular organization or other entity (e.g., is a student in a university campus).

[0120] As mentioned above, the video table 314 stores video data that, in some examples, is associated with messages for which records are maintained within the message table 306. Similarly, the image table 316 stores image data that is associated with messages for which message data is stored in the entity table 308. The entity table 308 can associate various augmentations from the augmentation table 312 with various images and videos stored in the image table 316 and the video table 314.

[0121] The database 304 also includes trained machine learning techniques 307 that store parameters of one or more machine learning models that have been trained during training of the XR application generation system 590. For example, the trained machine learning techniques 307 store training parameters of one or more artificial neural network machine learning models or techniques.

[0122] Data communication architecture

[0123] Figure 4 is a schematic diagram showing the structure of a message 400 generated by an interaction client 104 for transmission to another interaction client 104 via an interaction server 124, in accordance with some examples. The content of a particular message 400 is used to populate the message table 306 stored within the database 304 that is accessible by the interaction server 124. Similarly, the content of the message 400 is stored in memory as “in-transit” or “in-flight” data for the user system 102 or the interaction server 124. The message 400 is shown to include the following example components:

[0124] • message identifier 402: a unique identifier that identifies the message 400.

[0125] • message text payload 404: text to be generated by a user via a user interface of the user system 102 and included in the message 400.

[0126] • message image payload 406: image data captured by a camera component of the user system 102 or retrieved from a memory component of the user system 102 and included in the message 400. Image data for a sent or received message 400 can be stored in the image table 316.

[0127] • message video payload 408: video data captured by a camera component or retrieved from a memory component of the user system 102 and included in the message 400. Video data for a sent or received message 400 can be stored in the image table 316.

[0128] • message audio payload 410: audio data captured by a microphone or retrieved from a memory component of the user system 102 and included in the message 400.

[0129] • message augmentation data 412: augmentation data (e.g., filters, stickers, or other annotations or augmentations) that represent augmentations to be applied to the message image payload 406, message video payload 408, or message audio payload 410 of the message 400. Augmentation data for a sent or received message 400 can be stored in the augmentation table 312.

[0130] • message duration parameter 414: a parameter value indicating, in seconds, an amount of time for content (e.g., message image payload 406, message video payload 408, message audio payload 410) of the message to be presented to or made accessible to a user via the interaction client 104.

[0131] • message geolocation parameter 416: geolocation data (e.g., latitude and longitude coordinates) associated with a content payload of the message. Multiple message geolocation parameter 416 values can be included in the payload, each of which is associated with a content item included in the content (e.g., a particular image within the message image payload 406 or a particular video in the message video payload 408).

[0132] • message story identifier 418: an identifier value that identifies one or more content collections (e.g.,“stories” identified in the collection table 318) associated with a particular content item in the message image payload 406 of the message 400. For example, multiple images within the message image payload 406 can each be associated with multiple content collections using the identifier value.

[0133] • message tags 420: each message 400 can be tagged with a plurality of tags, each of which indicates a subject matter of content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value can be included within the message tags 420 that indicates the relevant animal. Tag values can be generated manually based on user input, or can be generated automatically using, for example, image recognition.

[0134] • message sender identifier 422: an identifier (e.g., messaging system identifier, email address, or device identifier) that indicates a user of a user system 102 on which the message 400 was generated and from which the message 400 was sent.

[0135] • message receiver identifier 424: an identifier (e.g., messaging system identifier, email address, or device identifier) that indicates a user of a user system 102 to which the message 400 is addressed.

[0136] The content (e.g., values) of the various components of a message 400 can be pointers to locations in tables where the content data values are stored. For example, the image value in the message image payload 406 can be a pointer (or address) to a location within the image table 316. Similarly, the value within the message video payload 408 can point to data stored within the image table 316, the value stored within the message augmentation data 412 can point to data stored in the augmentation table 312, the value stored within the message story identifier 418 can point to data stored in the collection table 318, and the values stored within the message sender identifier 422 and the message receiver identifier 424 can point to user records stored within the entity table 308.

[0137] Personal AI agent system

[0138] Figure 5 An example architecture 500 for applying a personal AI agent 502 to identify relevant features for personalization to a user is shown. The example architecture 500 can include a personal AI agent 502, a user database 504, a tool component 512, an XR application generation system 590, and a UI component 520. The personal AI agent 502 implements one or more machine learning models that communicate with the user database 504, the UI component 520, and the tool component 512 to selectively and intelligently generate content for a user.

[0139] The user database 504 includes a user custom database 506, a multi-modal memory 508, and a user-specific model 510. In some cases, the personal AI agent 502 collects data from various sources and generates a multi-modal memory 508 specific to a particular user. The personal AI agent 502 then provides personalized features to the user based on the identity model captured in the multi-modal memory 508.

[0140] The multi-modal memory 508 stores information related to the user. Any of the data collected and stored in the multi-modal memory 508 is collected and stored on an opt-in basis with explicit permission from the user. Non-limiting examples of multi-modal memory types include:

[0141] • Demographic data: information such as age, gender, location, income, education, and occupation.

[0142] • Behavioral data: information about individual behavior and interactions with websites, applications, VR devices, or other digital touchpoints. This data can include website visits, clicks, downloads, purchases, and user interactions with other users.

[0143] • Psychological data: information about an individual’s personality.

[0144] • Contextual data: information about the time, location, and device used by an individual when interacting with a digital touchpoint. This data can help the personal AI agent 502 understand the context of the interaction and personalize the experience accordingly.

[0145] • Purchase history data: information about an individual’s past purchases, such as products purchased, purchase frequency, and purchase amounts. The personal AI agent 502 can use this data to create personalized recommendations and offers. The data can also include advertising interaction data, which includes user interactions with advertisements such as clicks, impressions, and conversions.

[0146] • Interest data: information about an individual’s hobbies, interests, and passions, which can be collected through online posts and activities.

[0147] • Communication data: information about how an individual prefers to be contacted and their communication history with other users. This data can be used to personalize communication channels and messaging (e.g., SMS messages on a phone or pop-up messages on an AR device).

[0148] • Data from enhanced devices (e.g., AR / VR devices): information from camera feeds that users use to capture images or video of their surroundings; selection of digital content items (e.g., augmentations or overlays used on camera feeds); biometric data such as heart rate, body temperature, facial expressions, etc.; the user’s preferred method of interaction on the AR device; the type / duration of AR interactions; eye tracking, eye focus, and gaze direction and gaze time of what the user is looking at; body movements (e.g., head or body movements), gestures, or interactions with the virtual environment; hand and finger movements using controllers or tracking devices; audio data from microphones; user emotional responses such as heart rate, skin conductance, or facial expressions; user cognitive performance such as attention, memory, or problem-solving abilities; etc.

[0149] • Contact and connection data: data about the user’s contacts and connections, including their friends, followers, and groups.

[0150] • Device data: information about the user’s devices, including the type of device, operating system, and browser, which can be used to optimize the performance of the platform and provide a seamless experience across different devices.

[0151] The personal AI agent 502 links the different aspects of the user profile into the multi-modal memory 508. The multi-modal memory 508 stores various aspects of the user (e.g., the user’s preferences, lifestyle, interests, friends), and this contextual information can be used later to create tailored content. As the personal AI agent 502 collects more data related to the user, the value of such tailored content increases over time and across other forms of factors. Over time, as more and more digital touchpoints from the user are collected, the personal AI agent 502 gains a more in-depth understanding of the user and can use historical context to find relevance in any current user activity.

[0152] The multi-modal memory 508 includes entity-based memory, which refers to memory that focuses on specific things, such as objects, people, places, events, and experiences. This aspect of the multi-modal memory 508 stores information about the attributes, characteristics, and traits of these specific objects or people, such as remembering the name of a person, their appearance, their profession, or their interests. The multi-modal memory 508 also includes a knowledge graph memory, which refers to memory that focuses on how things are related or connected to each other. This aspect of the multi-modal memory 508 organizes information in a structured way, where entities are linked together in a weighted manner based on their relationships and attributes. The knowledge graph memory helps the personal AI agent 502 understand the context and meaning of information by showing how different entities and concepts are related.

[0153] Each entity in the multi-modal memory 508 can be linked to each other entity. The links between these entities can be weighted in different ways to represent how closely related the entities are to each other. For example, an entity representing a dog name can be linked to an entity for the user and an entity representing an animal, while another entity representing a person with the same name can be linked to an entity for the user and another entity representing a user contact. The entity representing the dog name can be linked to the entity for the animal with a greater weight than the link between the entity representing the person with the same name and the entity for the animal. Similarly, the entity representing the person with the same name as the dog can be linked to the entity for the contact with a greater weight than the link between the entity representing the dog name and the entity for the person. In this way, various information about a given user or individual can be collected and associated to establish links between the information.

[0154] In some cases, entity-based memories focus on specific things, while knowledge graph memories focus on how to relate things. Entity-based memories store information about individual entities, while knowledge graph memories organize information based on relationships and connections between entities. Both types of memories can be used to learn and understand a current context associated with one or more users.

[0155] The personal AI agent 502 uses embeddings of the multi-modal memory 508, which refers to techniques used in machine learning to represent and store data from multiple modalities (e.g., images, text, and audio) in a common vector space. The purpose of embeddings is to capture semantic meaning and relationships between different modalities, enabling more efficient and accurate processing of multi-modal data. In some examples, the personal AI agent 502 feeds a knowledge graph to an external or internal process to generate latent embeddings. The personal AI agent 502 stores the latent embeddings in the multi-modal memory 508 for use in generating dynamic content recommendations and analysis. In some cases, content recommendations are provided to the user system 102 without requiring a specific request for content recommendations from the user. For example, the user system 102 is used to capture or access an image, and the personal AI agent 502 detects the image that the user system 102 is viewing. The personal AI agent 502 generates a specific AR experience using the user database 504, and can automatically apply the generated AR experience to the image that the user system 102 is currently accessing or viewing. In some cases, the personal AI agent 502 uses the tool component 512 (discussed below) to generate a unique AR experience, and provides and activates the AR experience on the user system 102.

[0156] The personal AI agent 502 uses these embeddings for cross-modal comparison and analysis. In some examples, the embeddings of an image are compared to the embeddings of a corresponding textual description to identify semantic relationships between the two. In the context of the multi-modal memory 508, the embeddings are used to store and retrieve information from different modalities in a more efficient and effective manner. In some examples, if a user has stored an entity (e.g., a memory object) that includes an image, a textual description, and an audio recording, the embeddings can be used to represent each of these modalities in a common vector space. This enables efficient retrieval and integration of information from multiple modalities when accessing the memory object.

[0157] The personal AI agent 502 generates the embeddings using one or more techniques, such as neural networks. These embeddings can be fine-tuned and optimized for specific applications and tasks. The personal AI agent creates embeddings for the multi-modal memory that include information about individuals and their relationships to other individuals, entities, and devices, and are generated using various data sources as described herein by identifying patterns and connections between entities. Thus, the personal AI agent 502 can apply the multi-modal memory to the user in a variety of different ways to provide personalized features to the user.

[0158] A user can have selected certain customizations in the past, such as content augmentations, graphics, or features within an application or AR device. The interaction system stores such user customizations in the user customizations database 506. The personal AI agent 502 uses such customizations when providing relevant content, such as by identifying preferences that the user has customized. In some examples, the user can have used a certain type of augmentation (e.g., adding pizza to a camera feed) or a sticker that the user sent to a friend.

[0159] The user customizations database 506 includes customizations that the user makes within the interaction system. The user customizations database 506 stores profile customizations (e.g., profile pictures, cover photos, and bios), news feed preferences (e.g., determining the priority of certain friends, pages, and groups), and privacy settings (e.g., who can see posts, profile information, and activity). The user customizations database 506 stores personalized avatar and sticker selections, custom content augmentations, and how these apply to camera feeds, sound and music preferences for content creation, channel and creator subscription preferences, and other types of user customizations based on the user’s viewing history and engagement.

[0160] User-specific models 510 include generative models for generating graphics, text, and images for users. For example, user-specific models 510 can generate stickers that include photos, graphics, or animations. User-specific models 510 generate avatars that represent users on the interactive system platform. Users can customize avatars to reflect the user’s appearance, personality, and interests. User-specific models 510 generate filters and content augmentations, such as XR effects that can be applied in real-time, to enhance photos and videos. User-specific models 510 generate memes to share humorous images, videos, and captions that convey a particular cultural idea or trend. User-specific models 510 generate topic tags to categorize and organize user posts based on a particular theme or topic, enabling users to connect with other users who share similar interests or participate in a trending conversation. User-specific models 510 generate short-lived photos and videos that their followers can view for a limited time, which include filters, stickers, and text overlays to make them more engaging.

[0161] Tool components 512 include one or more neural network engines 514, one or more external data sources 516, and one or more feature APIs 518. Neural network engines 514 include one or more generative machine learning models. Generative machine learning models can be trained to generate a variety of different content. For example, generative machine learning models are trained to receive a prompt as input, which can include any combination of text, images, audio, and / or video, and generate an output that responds to the prompt. In some cases, generative machine learning models generate artificial images / videos, code snippets, and / or text that respond to the prompt. In some cases, generative machine learning models generate content augmentations, such as filters that can be overlaid, modified, or enhanced a real-world camera feed with digital content items.

[0162] In some cases, personal AI agent 502 provides a prompt as input to neural network engines 514, which is generated by personal AI agent 502 based on information collected from UI components 520 (representing the current real-world environment) and user database 504. That is, personal AI agent 502 generates a prompt that includes images captured by user system 102 and one or more vectors derived from multi-modal memory 508. The prompt can be provided as input to neural network engines 514. Neural network engines 514 then access additional data sources, such as external data sources 516 and / or feature APIs 518, to generate content that matches the input of the prompt. In some examples, personal AI agent 502 collects contextual data for a conversation, hears audio from a user, and / or receives some other input from a user or a user’s environment and generates a request to neural network engines 514.

[0163] The external data sources 516 can include various search engines, chatbots, email applications, calendar applications, messaging applications, social network applications, news feeds, real-time media sources, and / or any combination thereof. The personal AI agent 502 accesses external data from the external data sources 516 to generate and / or apply the multi-modal memory 508 for the user. In some examples, the personal AI agent 502 collects user data in different media types and creates an embedding for the user’s multi-modal memory 508. The personal AI agent 502 can retrieve data from the external data sources 516 to apply to the multi-modal memory 508 and / or to generate input for the neural network engine. The external data sources 516 can include any combination of scientific paper repositories including content information about papers, titles, authors, abstracts, keywords, and citations; data from emails such as email addresses, contact lists, email content, attachments, and metadata such as timestamps and IP addresses; search engine data such as search queries, search history, location data, device information, and web activity; and / or communication data such as messages, voice and video calls, roles in group messages, posts, comments, votes, user subscriptions, likes, followers, and topic tags.

[0164] The neural network engine 514 can intelligently select one or more of the external data sources 516 to populate data in response to a prompt received from the personal AI agent 502. The feature API 518 can provide access to a variety of different additional tools, which can be proprietary. Using one of the APIs given in the feature API 518, the neural network engine 514 can access additional machine learning tools to generate additional content. For example, one of the proprietary tools can include an AR experience generation tool. The neural network engine 514 can access the API of this tool to prompt the tool to generate an AR experience with specific features generated by the neural network engine 514 based on the prompt received from the personal AI agent 502. The neural network engine 514 can receive the specific AR experience and can further apply various effects and modifications to the AR experience to facilitate generating a response to the personal AI agent 502 that includes the AR experience matching the prompt. The personal AI agent 502 can then automatically augment and / or modify the images currently being presented on the user system 102 using the unique AR experience generated by the neural network engine 514. Figure 4 One example of one of the many tool components 512 that can be accessed by the personal AI agent 502 to generate content is shown. Figure 4 The components shown can be part of the neural network engine 514.

[0165] The personal AI agent 502 retrieves personalized content for the user and applies the content through various feature APIs 518. The personal AI agent 502 applies personalized and / or recommended content to interactive client features via the feature APIs 518, including photos, videos, or podcasts that the user captures and shares with friends. The personal AI agent 502 recommends filters, stickers, text overlays, or content enhancement effects to apply to camera feeds that can then be recorded and sent to individual users. The personal AI agent 502 applies personalized and / or recommended content to interactive client features via the feature APIs 518, including messages, photos, and videos to individual friends or to groups; collections of photos or videos, such as collections that can be viewed by friends over a certain time period (e.g., 24 hours); content from media partners, such as news articles, videos, and curated selections of programs tailored to the user; map-related features, such as when and with whom to share a location, how the system shares, and what information to display on a map user interface; filters and / or XR experiences that include visual overlays applied to photos or videos, such as adding location-based information, temperature, time, and other graphics; and custom avatars that can be used in photos / videos, chat messages, and other features.

[0166] In many cases, the personal AI agent 502 can access multiple input devices to generate prompts for the neural network engine 514. These input devices or combinations of input devices can include a wearable device 522, such as an AR / VR device 526, a device with a monitor 524, such as the user system 102, and / or a smartwatch 528. The output generated by the personal AI agent 502 can be provided to any of the same or different combinations of the AR / VR device 526, the user system 102, and / or the smartwatch 528. The personal AI agent 502 identifies a preferred method of communication using the multi-modal memory 508. In some examples, the personal AI agent 502 identifies that the user prefers to be reminded via a smartwatch, gives an indication on an AR device, and receives a text message on the user’s mobile phone.

[0167] The process for training the neural network engine 514 can be the same as discussed previously. The neural network engine 514 can receive a set of training data that includes various different prompts and real responses. These real responses can include identification of data sources that can be used to respond to the prompts and / or generated image content or textual content that responds to the prompts. The neural network engine 514 can iterate through the training data until a loss function meets a stopping criterion, where at each iteration the parameters of the neural network engine 514 are updated. In a similar manner, the personal AI agent 502 can be trained based on training data to generate prompts to provide to the neural network engine 514. The training data can include various potential vectors, images, and information obtained from the user database 504 and corresponding real prompts. The personal AI agent 502 can iterate through the training data until a loss function meets a stopping criterion, where at each iteration the parameters of the personal AI agent 502 are updated. In this manner, the personal AI agent 502 can generate real-time prompts based on current information accessed or obtained by the UI component 520 to provide to the tool component 512 for generating content to provide back to the UI component 520.

[0168] The XR application generation system 590 can continuously analyze images captured by one or more interaction clients 104, such as the UI component 520. The XR application generation system 590 can generate one or more machine learning models applied to detect fashion items or objects (e.g., real-world objects) depicted in the images. In response to detecting a fashion item or object, the XR application generation system 590 can generate a prompt to modify a texture of the fashion item and / or object depicted in the image. In some cases, the prompt is generated based on voice input continuously captured and received by the interaction client 104. In some cases, the prompt is dynamically generated based on contextual cues gathered from the images and / or information retrieved from the multi-modal store 508.

[0169] The XR application generation system 590 accesses the XR application through the interaction client 104. The XR application generation system 590 can receive a query through the interaction client 104 that defines one or more attributes of an XR application. The XR application generation system 590 can use the query, together with and / or separate from the personal AI agent 502, to generate a prompt for a generative machine learning model. In some cases, the XR application generation system 590, together with and / or separate from the personal AI agent 502, can provide the prompt to the tool component 512 to process the prompt using a generative machine learning model (e.g., the neural network engine 514) to generate one or more data objects that match the one or more attributes defined by the query. The XR application generation system 590 can receive the one or more data objects and, in response, can use the XR application to generate one or more XR objects based on the one or more data objects generated by the generative machine learning model. The XR application generation system 590 can then launch the XR application and overlay the one or more XR objects on real-world objects depicted in real-time images or video and / or previously captured images or video. In some cases, the XR application generation system 590 can be part of the personal AI agent 502 and perform similar functions as the personal AI agent 502.

[0170] Figure 6 is a block diagram of an example personal AI agent system 600 according to some examples. The personal AI agent system 600 includes a client system 606, an intent component 614, a neural network 636, a content response component 612, a response filter component 604, additional services 628, a user profile 616, and a personal AI configuration component 634. The user profile 616 can include some or all of the same components as the user database 504. For example, the user profile 616 can include or provide access to a multi-modal memory 626, which can include some or all of the components of the multi-modal memory 508. The client system 606 can include the same or similar components as the user system 102.

[0171] In some examples, the personal AI agent system 600 is a software platform designed to simulate human decision making through the use of a multi-modal memory 508. The personal AI agent system 600 can employ embeddings to generate a multi-modal memory 626 in order to generate a user profile 616 and apply machine learning (ML) / artificial intelligence methods to generate recommended content for a user.

[0172] In some examples, the personal AI agent architecture includes one or more intent components 614 that can include the same features and functionality as the intent component 612. The intent component 612 can be configured to understand the intent 622 of the user 638 and extract relevant information from the user input 602, responses to recommended content 618, or from collected data used to generate embeddings that can be stored in the multi-modal memory 626. This can be achieved by analyzing user data to generate the multi-modal memory 626 and mapping user data to intents and / or receiving intents from the multi-modal memory 626. The intent component 612 can use various methods such as rule-based systems, statistical models, large language models (LLMs), neural networks, etc. to understand the intent of the user.

[0173] The content response component 612 generates one or more content items 608 to present to the user 638. The content response component 612 uses the intent 622 received from the intent component 614 and any extracted information from the intent component 612 to determine appropriate content items 610. This can be done using rule-based systems, decision trees, statistical models, LLMs, neural networks, etc. In some examples, the intent component 614 and the content response component 612 are a single component, and in some other examples, they are part of different devices / systems. The content response component 612 generates content items in a human-like manner by using methods such as, but not limited to, text generation and machine learning models. The content items can take the form of indications, stickers, text, content augmentations, other features using the feature API 518, etc.

[0174] In some examples, the intent component 612 and / or the content response component 612 can use a neural network 636 to improve their intent understanding and content management capabilities. For example, the intent component 614 uses the neural network 636 to analyze the user’s input, the multi-modal memory 626 of the user 638, and extract relevant information such as the intent of the user and entities. If the response content is text, the content response component 612 uses the neural network 636 to identify and extract important information from unstructured text such as the user’s 638 question or request. This can be done using methods such as named entity recognition, part-of-speech tagging, sentiment analysis, etc. The content response component 612 identifies other types of content that are relevant to the user’s intent as further described herein. Figure 1

[0175] ​In some examples, the intent component 614 passes information 640 extracted or obtained from the multi-modal memory 626 and / or user input 602 and / or interactions directly to the neural network 636. The neural network 636 receives the extracted information and generates one or more content items 610. In a similar manner, the content response component 612 can receive the intent 622 and generate a prompt based on the intent passed to the neural network 636. The neural network 636 receives the prompt and generates one or more content items 610.

[0176] The personal AI agent system 600 receives user data from the client system 606 that can be used to determine the intent of the user 638 during an interactive session. For example, the user 638 can access an interactive server hosting the personal AI agent system 600 using the client system 606. The user 638 interacts with the device, for example by entering a prompt into the client system 606 as user input 602, and the client system 606 passes the user input 602 to the personal AI agent system 600. In some examples, the user input 602 can include other types of data in addition to text, such as but not limited to: image data, video data, audio data, electronic documents, links to data stored on the internet or the client system 606, and the like. Additionally, the user input 602 can include media, such as but not limited to: audio media, image media, video media, text media, and the like. Regardless of the type of data of the user input 602, keyword attributes and extensions can be used to automatically generate clusters of keywords or attributes associated with the received user input 602. For example, image recognition can be deployed in an automated manner to identify objects and locations associated with visual media and image data, and generate keyword clusters or clouds that are then associated with the image-based prompt.

[0177] User input 602 or interactions can also be received through any number of interfaces and I / O components of the interactive client 104. These include gesture-based input obtained from biometric components and input received via a brain-computer interface (BCI). In some examples, the personal AI agent system 600 is integrated into various platforms, such as but not limited to websites, messaging apps, and mobile apps, enabling users to interact with it through text or voice commands.

[0178] The personal AI agent system 600 determines the intent 622 of the user 638 using information obtained from the multi-modal memory 626 and user input 602 and / or interactions. For example, the intent component 612 receives information obtained from the multi-modal memory 626 of the user 638, the user input 602, and any user feedback including follow-up messages or interactions from the user 638 and uses an intent processing pipeline to determine the intent 622. The pipeline uses machine learning models, natural language processing (NLP) methods, or other models or methods to map the messages or interactions to intents (e.g., map a conversation set to a bag of intent keywords and concepts). Stemmed keywords and expanded concepts are generated in addition to the keywords used in the original user input 602. The platform also assigns a weight to each concept based on who is in the conversation, the importance of the type of interaction, and the characteristics of the interactions and messages, among other things. These keywords, interaction types, and concepts are aggregated and mapped to the user 638 as part of an intent profile or intent vector that has weighted keywords and concepts based on their importance in the conversation.

[0179] In some examples, the personal AI agent system 600 associates a time factor to keywords, interactions, or concepts, where the time factor decays over time. For example, the personal AI agent system 600 attaches a time factor to keywords, interactions, and concepts that indicates how much that concept should be used for targeting and bidding. For example, “hotels in Cancun for spring vacation” versus “planning for a wedding next year” would have two different decay factors. In some examples, the personal AI agent system 600 applies a time factor to the conversation state.

[0180] In some examples, the overall intent of the user is established using various signals or data such as user demographics, location, device, engagement with the organic surface as a user interface, and service features and consumption patterns provided by the interactive platform application, and overall friend graph proximity signals or data that can be discernible from the user’s connection with the interactive platform.

[0181] In some examples, the intent vector is an additional dimension that can be used to refresh and update the intent profile stored in the user profile 616. In some examples, the personal AI agent system 600 uses the user profile 616 including the multi-modal memory 624 of the user 638 to determine the intent of the user. In some examples, the intent component 614 uses artificial intelligence methods including machine learning (ML) models to generate the intent 622. The personal AI agent system 600 uses data about the intent of the user 638 to generate one or more content items and receives feedback to improve its responsiveness and optimization capabilities over time by ingesting the feedback. The personal AI agent system 600 collects data about the user’s 638 engagement with ads, organic features of the interactive platform system (user interfaces and services provided to the user by the interactive platform), and also in response to the personal AI agent system 600 itself and feeds it into the intent component 614. In this way, the personal AI agent system 600 can fine-tune or further pre-train the ML models of the intent component 614, taking into account not only the intent of the user, but also fine-tuning the intent of the user’s reasoning model using subsequent actions.

[0182] In some examples, the personal AI agent system 600 collects a set of interactions (e.g., prompts or clicks) during an interactive session. The personal AI agent system 600 maps the set of interactions to an intent vector. The personal AI agent system 600 assigns weights to the interaction features based on the importance scores of the interactive conversation and determines the intent of the user based on the intent vector including the weighted interaction features. In some examples, the personal AI agent system 600 stores the interaction state in the user profile 616 as part of the multi-modal memory 626 of a series of interactive sessions so that the personal AI agent system 600 can have the interaction context that occurred over multiple interactive sessions.

[0183] In some examples, the knowledge base of the personal AI agent system 600 includes a set of information that the personal AI agent can use to understand and respond to inputs or interactions of the user 638. This includes, but is not limited to, a set of predefined intents, entities, and responses, as well as external sources of information such as databases or APIs. In some examples, intent information of one or more friends of the user 638 is used to understand, inform, and / or respond to the intent of the user.

[0184] The personal AI agent system 600 generates one or more content items 610 using the content response component 612 using the intent 622. For example, the content response component 612 receives the intent 622 and communicates the intent 622 to the neural network 636 as a neural network prompt. The neural network 636 receives the neural network prompt and generates the content item 610. The neural network 636 communicates the content item 610 to the content response component 612. The content response component 612 receives the content item 610 and communicates the content item 610 to the response filter component 604 for additional processing. In some examples, the content response component 612 generates the content item 610 using a set of additional services 628, such as but not limited to an image generation or content augmentation system. The content response component 612 generates a service request 630 using the intent 622 and communicates the service request 630 to the additional services 628. The additional services 628 generate a request response 632 using the service request 630 and communicate the request response 632 to the content response component 612. The content response component 612 generates the content item 610 using the request response 632.

[0185] In some examples, the personal AI agent system 600 uses the response filter component 604 to filter the raw content response 620 generated by the content response component 612. For example, the content response component 612 communicates the raw content response 620 to the response filter component 604 and the response filter component 604 filters the raw content response 620 based on a set of filtering criteria to eliminate certain content from the raw content response 620, such as profane words, images, or concepts, or content that some people can find objectionable. In some examples, the response filter component 604 generates an adjusted intent 622 based on filtering the raw content response 620 and the content response component 612 generates an additional raw content response 620 using the adjusted intent 622.

[0186] In some examples, the neural network 636 and the additional services 628 are hosted by the same system that hosts the other components of the personal AI agent system 600. In some examples, the neural network 636 and the additional services 628 are hosted by a server system that is separate from the system that hosts the personal AI agent system 600 and the personal AI agent system 600 communicates with the neural network 636 and the additional services 628 over a network. For example, the personal AI agent system 600 receives the user input 602 and the multi-modal memory 626 and communicates the user input 602 and the multi-modal memory 626 to the neural network 636 located on a separate server system. The neural network 636 receives the user input 602 and the multi-modal memory 626 and generates the raw content response 620. The neural network 636 then communicates the raw content response 620 to the personal AI agent system 600. The personal AI agent system 600 receives the response for subsequent processing as described herein.

[0187] In some examples, as part of the content item 610, the personal AI agent system 600 generates a set of interaction options that prompt the user 638 to interact with the personal AI agent system 600, which are displayed to the user 638 by the client system 606. The interaction options generated by the personal AI agent system 600 can include chat information such as, but not limited to, contextually sensitive material, instructions to the user 638, possible conversation topics, interactive digital content items, etc. In some examples, the personal AI agent system 600 uses the interaction options to suggest a chat topic to the user 638 or to guide the user 638 through certain interaction steps that the user can take (e.g., providing instructive material about various topics or digital content items that the user can click on). In some examples, the interaction options include suggestions for a conversation, questions, or interaction steps that are designed to request input from the user 638 or to solicit a particular interaction with the system. The interaction options are designed to help the user 638 get the information they need, but provide the beneficial side effect of generating additional user interactions with the personal AI agent system 600. These additional user interactions improve the ability of the personal AI agent system 600 to determine the user’s intent by providing additional context and information to the personal AI agent system 600.

[0188] In some examples, the personal AI agent system 600 determines a personality or tone for responses or content generated by the personal AI agent system 600. For example, the personal AI agent system 600 stores the conversation or interaction state of the user 638 as part of the user’s profile stored in the user profile 616. The personal AI agent system 600 uses the conversation or interaction state of the user 638 along with demographic or other information to determine a personality or tone for the user 638, such as by adopting a formal tone for older users and a more informal tone for younger users.

[0189] In some examples, the user 638 explicitly changes the “personality” of the personal AI agent system 600 by interacting with the personal AI configuration component 634 and explicitly requests that the personal AI agent system 600 respond or act in a particular way (e.g., “be more fun,” “answer in riddles,” etc.) or provide certain types of content more frequently than other ways. In addition to modifying the personality or tone of the responses, the visual interface presented by the chat system 600 can be changed to reflect the particular “personality” of the personal AI agent system 600. In some examples, a slide switch key can be presented to enable the user 638 to select between a menu of personality traits (e.g., “cheeky,” “wry,” “adorable,” etc.).

[0190] In some examples, the system hosting the personal AI agent system 600 can not be a component of an interactive platform, but rather another interactive system that provides services and information to a group of users, such as but not limited to: a platform that provides enterprise-wide connectivity to a group of users such as employees of a company, customers of a business that provides professional services, educational institutions, etc. In some of such examples, the content provided to the users can not be advertisements, but rather other types of useful information such as company policies, status messages for projects, newsworthy events, etc.

[0191] In some examples, end-to-end encryption is used to secure communications, ensuring that only the sender and the intended recipient can read the messages being exchanged. By implementing end-to-end encryption, the personal AI agent system 600 can provide a secure and private messaging experience for the user while still extracting intent for advertisement experience enhancement. In the context of user intent extraction, this means that once a conversation is end-to-end encrypted, the personal AI agent system 600 as an end of the conversation can decrypt the messages and pass them to the intent extraction pipeline of the intent component 614. In some examples, the personal AI agent system 600 is operably connected to internet search engines using additional services 628, and the user can use the personal AI agent system 600 as a smart search engine to search the internet. In some examples, the personal AI agent system 600 is operably connected to proprietary databases via additional services 628, and the user can use the personal AI agent system 600 as a smart search assistant for searching the proprietary databases.

[0192] In some examples, the intent component 614, the content response component 612, and the machine learning component of the neural network 636 are continually retrained or fine-tuned based on user interactions with content items 610. For example, user interactions with content items 610 are stored in an analytics database (not shown). Then, when a sequence of responses by the content response component 612 results in a successful intent 622 determination and thus an appropriate target content item 610, the interaction metrics are used to provide reinforcement to the content response component 612.

[0193] In some examples, the personal AI agent system 600 uses interactions between the user and generated content from other personal AI agents as input into the intent component 614 and / or the content response component 612. The other personal AI agents can be sponsored personal AI agents, custom personal AI agents established by the user from a template / tool of their own, etc.

[0194] In some examples, the information extracted by the personal AI agent system 600 from the conversation or interaction focuses on the user intent. Additionally, the personal AI agent system 600 further dialogues or interacts with the user by understanding the user intent (e.g., if the user asks for a good hotel in Cancun, the personal AI agent system 600 responds with, “Here is the list of hotels. I also know about the great promotion activities of the hotels, do you want to see?” or provides 5D models of various hotels in Cancun on the AR device that are of a certain type of luxury that the user is accustomed to). This provides the user intent extracted from the user 638, matches the intent with other content, and embeds the content into the content item 610. In some examples, the text component of the additional content is used to fine-tune the intent 622 determined by the intent component 614 and / or the response of the content item 610 generated by the content response component 612. This improves the suggestions provided by the personal AI agent system 600 and increases user engagement.

[0195] In some examples, the personal AI agent system 600 can use the ad content placement and bidding component to provide advertisers with the opportunity to bid for certain actions by selecting keywords or a set of expanded concepts (automatically expanded) to select their potential target audience and display the ad content to be placed to users that match the criteria within their intent vector. Overall, by mapping the conversation to a set of keywords, interactions, and concepts, and mapping the conversation to the user intent and profile, the ad content placement and bidding component of the personal AI agent system 600 enables advertisers to find their target audience more precisely. In some examples, the AI-driven ad creative generation system of the personal AI agent system 600 assists advertisers by simplifying the process of creating and / or generating ad creatives for the advertisers. By providing inputs such as website, target application, additional assets, target keywords, the personal AI agent system 600 can automatically generate ad creatives. This can save the advertisers’ time and resources, enabling them to focus on other aspects of the advertising campaign.

[0196] The personal AI agent system 600 provides users with opt-in / opt-out options to opt-out of the use of user information. The personal AI agent system 600 can operate as an opt-in system by default. The opt-in and opt-out options are mechanisms used by the system to give users control over the use of their personal data. These options are particularly important in the era of data privacy regulations such as the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA). Below are some examples of the opt-in mechanism:

[0197] 1. Opt-in: The opt-in approach requires users to actively give consent before an application or website collects, processes, or shares their personal data. This method is considered more privacy-friendly because it ensures that users are fully aware of the data practices and intentionally choose to participate. Such opt-in options can appear as banners or pop-ups. These opt-in options appear when a user first accesses a website or launches an application, requesting permission to collect and process personal data for specific purposes (e.g., targeted advertising, analytics, or personalization). Other opt-in options can be displayed in the form of checkboxes or toggle switches, enabling users to individually enable or disable data collection for specific purposes. In some examples, opt-in options are shown as contextualized prompts, which users can encounter when accessing specific features or functionalities within the application or website that rely on data collection (e.g., location-based services).

[0198] 2. Opt-out: In the opt-out approach, the system assumes that users consent to data collection and processing by default. However, users are provided with the option to withdraw their consent at any time. The system applies the opt-out approach for a limited number of times.

[0199] The system provides a privacy policy or settings, where users can access the privacy policy of the application or website, which includes information on how to opt-out of data collection and processing, or preference options that enable users to manage their privacy settings and disable specific data collection and sharing practices. To improve transparency and user control, the system clearly communicates data collection and processing practices and provides easy-to-use opt-in and opt-out options.

[0200] XR application generation system

[0201] Referring to Figure 5 The XR application generation system 590 generates an XR application of one or more XR objects using a predefined template implementation. For example, the template can define placeholders for images, text, video, graphics, behaviors, segmentation models, conditions, etc. These placeholders can define: the position of a single XR object, how the XR object looks, the behavior (e.g., animation) of the XR object, and / or the conditions under which the XR object is displayed and / or animated. The XR application is configured to receive the template and generate an XR experience for display in which the XR objects (e.g., in real-time according to the XR object definitions defined by the template) are superimposed on real-world objects depicted in the images or videos.

[0202] For example, a template can define a sticker or graphical object that displays a message with a particular theme description (e.g., which Disney princess are you? or which race car driver are you?). That is, the template can include a theme description field and can populate a message overlaid on a real-world object, where at least a portion of the theme description is stored in the theme description field. In some cases, the theme description field is populated based on input received orally from a user or based on text received from a user filling out a form.

[0203] The template can also define, within the message, a region for displaying a content item associated with the theme description. For example, the content item can present a character or theme (e.g., the name of a Disney princess or the name of a race car driver), as well as an image or graphical element associated with the character or theme and some color or other visual property modification to the image or graphical element. In such cases, the template can include multiple regions, each associated with a different character or theme. The XR application can randomly cycle through the multiple regions to select a particular character or theme defined by one of the regions. Once the particular character or theme is selected, the XR application retrieves, from the template, a unique description of the character or theme (e.g., the name) and the image or graphical element associated with the character or theme and some color or other visual property modification to the image or graphical element stored in that portion of the template. The XR application can then populate the displayed message with the retrieved content to overlay the populated message on the real-world object depicted in the image.

[0204] In some examples, content items of a template can be populated based on information and / or data objects received from tool component 512. For example, XR application generation system 590 can generate a prompt that includes a theme description received from user input and requests a list of content items that match a portion of the template. That is, the prompt can specify a theme description and can request a list of characters or subjects and images or graphical elements associated with the characters or subjects and some color or other visual property modifications to the images or graphical elements. Tool component 512 can generate one or more data objects that match or respond to the prompt. For example, tool component 512 can generate a first data object that includes a first character or subject associated with the theme description and a first image or graphical element associated with the first character or subject and some color or other visual property modifications to the first image or graphical element. Tool component 512 can generate a second data object that includes a second character or subject associated with the theme description and a second image or graphical element associated with the second character or subject and some color or other visual property modifications to the second image or graphical element. XR application generation system 590 can receive the first and second data objects and can populate the portion of the template for each of the first and second data objects to generate a populated template. The XR application can then be launched to present a message with an XR object generated based on a randomly selected template portion that includes the data objects generated by tool component 512.

[0205] In this way, novice users can easily define and modify XR applications with minimal user input and in a way that avoids runtime errors. This enables users to create new XR application experiences to share with other users by providing only minimal input, such as a theme or theme description.

[0206] In some examples, the template can define a segmentation model associated with a given data object. For example, the theme description can specify different hair colors. In such a case, the template can include generated messages for representing different hair colors. The template can include multiple sections, each associated with a different hair color and a particular segmentation model. The XR application generation system 590 can determine that the theme indicates a certain body part, such as hair. In such a case, the XR application generation system 590 can store an indication of the hair segmentation model in the template. This causes the hair segmentation model to be activated when the XR application is operated. As mentioned above, the XR application generation system 590 can generate a prompt to be provided to the tool component 512 based on the theme description, requesting different color portions of hair on a person’s head. The tool component 512 can provide a data object (e.g., a JavaScript object) including a list of different hair colors that can be used to populate different sections of the template.

[0207] The hair segmentation model can be a specially trained machine learning model that processes or analyzes an input image and identifies portions of hair on a person’s head depicted in the image. The output of the hair segmentation model generates a boundary that defines pixels in the image that correspond to portions of hair on a person’s head. Segmentation can be applied to the image, extracting only those pixels that correspond to portions of hair on a person’s head. After the pixels are extracted, the XR application generation system 590 can use the segmentation of the hair to replace pixels on portions of hair on a person’s head depicted in the image with pixels of a color defined by the template, which is defined based on the data object received from the tool component 512.

[0208] When the XR application is launched, the XR application can randomly cycle through multiple regions of the template to select a particular hair color and hair segmentation model. Once a particular hair color is selected, the XR application modifies the image based on the application of the segmentation model to the image. That is, the XR application can modify pixel values corresponding to the hair segmentation with pixel values associated with the selected particular hair color.

[0209] In a similar manner, a person or other portion of the real world can be modified by selecting an appropriate segmentation model. For example, a theme description can specify a different nose style. In such a case, the template can include a generated message for representing the different nose style. The template can include multiple portions, each associated with a different nose style and a particular nose segmentation model. The XR application generation system 590 can determine that the theme indicates a certain body part, such as a nose. In such a case, the XR application generation system 590 can store an indication of the nose segmentation model in the template. This causes the nose segmentation model to be activated when the XR application is operated. As mentioned above, the XR application generation system 590 can generate a prompt to be provided to the tool component 512 based on the theme description, requesting a different nose style for the person. The tool component 512 can provide a data object (e.g., a JavaScript object) including a list of different nose styles for populating different portions of the template.

[0210] The nose segmentation model can be a specially trained machine learning model that processes or analyzes an input image and identifies a nose portion on a head of a person depicted in the image. The output of the nose segmentation model generates a boundary that defines pixels in the image that correspond to the nose portion on the head of the person. Segmentation can be applied to the image, extracting only those pixels that correspond to the nose portion on the head of the person. After the pixels are extracted, the XR application generation system 590 can use the segmentation of the nose to replace the pixels on the nose portion on the head of the person depicted in the image with pixels associated with the nose style defined by the template, which is defined based on the data object received from the tool component 512.

[0211] When the XR application is launched, the XR application can randomly cycle through the multiple regions of the template to select a particular nose style and nose segmentation model. Once a particular nose style is selected, the XR application modifies the image based on the application of the segmentation model to the image. That is, the XR application can modify the pixel values corresponding to the nose segmentation with pixel values associated with the particular nose style selected.

[0212] In some examples, the user system 102 can share the populated template of the XR application with another user system 102. For example, based on a theme description received from a user using a first user system 102, the first user system 102 can share the populated template with a second user system 102 of another user after populating the template using the tool component 512. A unique icon can be associated with the template of the XR application. The unique icon, when selected by a user of the second user system 102, can cause the XR application to launch using the populated template received from the first user system 102. For example, the populated template can be used to present a message with various different Disney princesses that are randomly selected and displayed according to portions of the template that define their appearance and behavior. The second user system 102 can receive input selecting the icon associated with such a template and, in response, the second user system 102 can cycle through different portions of the template to select a data object defining one particular Disney princess to generate an XR object corresponding to the particular Disney princess for display. Additional input can be received to randomly select another Disney princess from the template to generate an XR object corresponding to the other Disney princess for display.

[0213] The second user system 102 can receive input from a user defining a new XR experience. For example, the input can define a new theme description. The XR application generation system 590 can generate a prompt including the new theme description (e.g., race car drivers) and a request for a list of race car drivers and a corresponding visual graphical element representing each race car driver. The tool component 512 can generate a list of data objects representing the list of race car drivers and provide the list of data objects to the XR application generation system 590. In some cases, the tool component 512 can obtain the multi-modal memory 508 from the personal AI agent 502 to customize the selection of data objects based on the preferences of a given user.

[0214] The XR application generation system 590 can then use the new theme description, and populate the new template with the corresponding respective portions of the visual graphical elements including the racecar driver identifier (e.g., name) received from the tool component 512. The XR application, when launched by the second user system 102, can then loop through the various portions of the template to display a message (e.g., Who are you?) representing the theme description, and the message includes the XR objects corresponding to the randomly selected portions of the template (e.g., the name of the particular racecar driver and the visual graphical elements). The XR application generation system 590 can receive a request from the second user system 102 to store the populated template associated with the XR application. In some cases, the XR application generation system 590 can receive a request to lock the populated template under certain conditions. The XR application generation system 590 can generate a new icon associated with the new populated template, and associate the new icon with the XR application. The new icon can be locked and unlocked by another user when the conditions are met (e.g., receiving a password matching the password associated with the icon or receiving a payment).

[0215] In some cases, the XR application generation system 590 of the second user system 102 can not know that the template received from the first user system 102 was generated using the tool component 512. The XR application generation system 590 of the second user system 102 has the data generated by the tool component 512 (e.g., machine learning model). This enables an end user to quickly and efficiently generate many different XR applications with different themes, and publish such XR applications for access by other users.

[0216] In some cases, the XR application generation system 590 of the second user system 102 can receive information indicating that the template received from the first user system 102 associated with the XR application was generated using the tool component 512. In such cases, the XR application can include an object for changing the prompt used to generate the XR application. In response to input selecting the option to change the prompt, text can be received from the second user system 102 defining the attributes of the new prompt. The modified prompt can then be sent to the tool component 512 to populate a new template to generate a new XR experience.

[0217] The new XR experience created by the second user system 102 can be connected to the XR experience received from the first user system 102. That is, after selecting an option to launch the XR experience associated with the presentation of the Disney princess, other users can access and / or launch the new XR experience associated with the presentation of the list of racecar drivers. In some cases, the new XR experience can be published and launched independently of the XR experience received from the first user system 102, and can exist independently.

[0218] A third user of the third user system 102 can receive an XR application with multiple icons, said multiple icons (e.g., a first icon associated with a Disney princess template generated by a first user of the first user system 102 and a second icon associated with a race car driver template generated by a second user of the second user system 102) associated with multiple filled templates. The third user can use the third user system 102 to select one of said icons to launch the XR application using the filled template corresponding to that icon.

[0219] For example, such as Figure 7 As shown, a first graphical user interface 700 can be presented to a first user on a first user system 102. The first graphical user interface 700 can be presented through a specific XR application accessed on the first user system 102. The first graphical user interface 700 may include images or videos 710 depicting one or more real-world objects. Input to select configuration options 722 can be received from the user (shown in a second user interface 701, but not in the first graphical user interface 700). In response, the XR application may present a notification 712 allowing the user to input one or more attributes of the XR application. The notification 712 may include a message requesting the user to input a subject description. For example, the XR application may receive a subject description (e.g., Disney princesses) as one or more attributes, either verbally or via text received from the user through a keyboard (e.g., a touchscreen keyboard).

[0220] In response to receiving an XR application generation system 590, a prompt can be populated or generated, which includes one or more attributes of a template associated with the XR application, as well as element or data definitions. Data definitions may specify characters, graphic elements, and / or one or more visual modifications to graphic elements. A prompt can be generated to request tool component 512 to generate a list of characters, graphic elements, and / or one or more visual modifications matching one or more attributes. For example, a prompt may request tool component 512 to generate a list of Disney princesses along with their names, graphic elements, and / or visual modifications or elements. The XR application generation system 590 can receive data objects from tool component 512 and can populate a portion of the template based on the data objects. The XR application generation system 590 can then launch the XR application and randomly select one of the data objects to generate an XR object 714 corresponding to the randomly selected data object for display. For example, XR object 714 may include a character name (e.g., Disney princess), one or more visual elements and / or graphic elements associated with and representing the character, or modifications to graphic elements generated by tool component 512.

[0221] After the threshold period of time and / or in response to receiving input from the user, the XR application can present a second user interface 701. That is, the XR application can generate a message 720 including one or more attributes (e.g., specifying “Which Disney Princess are you?”). The XR application can then randomly cycle through the populated templates to select a portion of a template corresponding to a given one of the data objects received from the tool component 512. The XR application can then generate a new XR object 730 and present the new XR object 730 in a third user interface 702. The new XR object 730 presents a name of the second character (e.g., a Disney Princess), one or more visual elements, and / or a graphical element associated with and representative of the second character or a modification to a graphical element generated by the tool component 512.

[0222] In some examples, the populated template can be provided or shared with the second user system 102. The second user system 102 can present a fourth user interface 703. The fourth user interface 703 can present an icon 740 associated with the populated template received from the first user system 102 (e.g., corresponding to the Disney Princess theme description). In a similar manner, the second user system 102 can receive input from a second user to define a new template (e.g., corresponding to a theme description for race car drivers). After receiving data objects generated by the tool component 512 that match the theme description for race car drivers, the XR application generation system 590 populates the new template using the received data objects for the theme description for race car drivers, which include a name of a person (e.g., a race car driver), one or more visual elements, and / or a graphical element associated with and representative of the person or a modification to a graphical element generated by the tool component 512. The XR application can generate a new icon 742 to enable any user with access to the populated template to view an XR object generated based on the data objects received from the tool component 512. For example, the second user system 102 can present an XR object 744 that includes a name of a person (e.g., a race car driver), one or more visual elements, and / or a graphical element associated with and representative of the person or a modification to a graphical element generated by the tool component 512.

[0223] In some examples, in generating the data objects, the tool component 512 can include or utilize, or be in communication with, one or more other systems to generate 2D and / or 3D models that match the descriptions of one or more of the data objects. For example, a template associated with the XR application can include fields for inputting 3D models and / or 2D models. In such a case, the tool component 512 can first generate descriptions of one or more data objects that correspond to the received prompts. For example, the tool component 512 can receive prompts indicating a dinosaur and a city name. The tool component 512 can generate a list of dinosaur types and a list of buildings that match the city. The tool component 512 can then provide the list of dinosaur types and buildings to the XR application generation system 590 for inclusion in the relevant portions of the template.

[0224] For example, the template can include a first portion for defining a first object (e.g., a dinosaur type) and a second portion for defining a second object (e.g., a building). The XR application generation system 590 can determine that the first portion and the second portion also require data defining 2D models and / or 3D models of the respective objects. For example, the first portion can include a field for inputting a 2D model or a 3D model of a dinosaur, and the second portion can include a field for inputting a 2D model or a 3D model of a building. In such a case, the XR application generation system 590 can transmit the dinosaur type and the building received from the tool component 512 to a 2D object model generator and / or a 3D object model generator that can also be implemented by the tool component 512. The 2D object model generator and / or the 3D object model generator can access a pre-defined list of 2D models or 3D models to find and retrieve a 2D model or a 3D model that matches the descriptions of the dinosaur type and the building received from the XR application generation system 590.

[0225] The XR application generation system 590 can receive the 2D model or the 3D model from the 2D / 3D model generator and store the respective model in the respective portion of the template. The XR application generation system 590 can also receive input from the user defining behaviors of the dinosaurs and / or buildings included in the template. For example, the input can define a display position of each dinosaur and / or each building. The input can define conditions that animate the dinosaurs and / or buildings. For example, the conditions can specify a time (e.g., a time of day) and / or a virtual distance between the respective object and a camera of the user system 102. The XR application generation system 590 can trigger the animation of the object when the conditions are met.

[0226] For example, as Figure 8-1As shown, a graphical user interface 800 is presented on the user system 102. The graphical user interface 800 can be presented by the XR application generation system 590 based on a template that is partially based on user input and partially filled in based on data objects received from the tool components 512. That is, the graphical user interface 800 includes one or more real-world objects 812 depicted in an image 810, a first virtual object 830 (e.g., an XR object), and a second virtual object 820 (e.g., another XR object). The first virtual object 830 can correspond to, or be generated based on, a template portion that defines the appearance and behavior of the first virtual object 830 received by the one or more tool components 512. The second virtual object 820 can correspond to, or be generated based on, a template portion that defines the appearance and behavior of the second virtual object 820 received by the one or more tool components 512. In some cases, the first virtual object 830 can be animated based on animation patterns received from the 3D model generator (e.g., a dinosaur can be made to fly) when one or more conditions are met. The user can select the second virtual object 820 to cause the 3D model corresponding to that virtual object to be displayed and animated. In this way, the user can easily design an XR game using output from the one or more tool components 512 based on prompts generated based on input from the user and generated by the XR application generation system 590.

[0227] In some examples, the XR application generation system 590 can receive only a query or input from the user (e.g., verbally and / or through a text input interface) and can process the query to retrieve one or more keywords, rather than generating an XR experience for an XR application using a predefined template. Using the keywords, the XR application generation system 590 can automatically generate a list of relevant objects. The relevant objects can be selected based on information stored in the multi-modal store 508. The XR application generation system 590 can create corresponding 2D / 3D models for the relevant objects using the 2D / 3D model generator. These 2D / 3D objects can then be placed within a real-world environment depicted in one or more images and can be controlled based on additional input from the user.

[0228] For example, Figure 8-2 A backend system 801 (e.g., which can be part of the XR application generation system 590) is shown for generating the XR experience shown in the graphical user interface 800 is shown. As an example, user input including the phrase 880 (e.g., “dinosaurs walk through prehistoric New York”) can be received. This phrase 880 can be processed by the XR application generation system 590 to understand that the user is looking for “characters” in a “location.” This can be done without causing the user to fill in any portion of a template or form.

[0229] The XR application generation system 590 can parse the phrase 880 into a plurality of components, including keywords 881, 882, 883, and 884. In some cases, in processing the keywords, the XR application generation system 590 can derive an intent 890 of the user, e.g., based on the output of the personal AI agent 502. In some examples, the XR application generation system 590 can determine that the keywords 881 and 882 characterize or define an object (e.g., a dinosaur) and a behavior of the object (e.g., walking). In response, the XR application generation system 590 can apply a character scripting or database to retrieve attributes of a specified character. The XR application generation system 590 can also pass the keyword 881 to the tool component 512 to retrieve a list of related objects 893 (e.g., types of dinosaurs). This list of related objects 893 can be customized according to the interests of the user, which can be retrieved from the multi-modal memory 508, including an indication of interests 892 (e.g., for a particular type of dinosaur). In this case, the tool component 512 can include a particular type of dinosaur that the user is interested in in the list of related objects 893.

[0230] The XR application generation system 590 can then apply the 3D model generator 894 to each object in the list of related objects 893. This can generate a 3D model of each object in the list of related objects 893. Similarly, the XR application generation system 590 can determine that the keywords 883 and 884 characterize or define a location (e.g., New York) and an attribute of the location (e.g., prehistoric). In response, the XR application generation system 590 can apply a location scripting or database to retrieve attributes of a specified location. The XR application generation system 590 can also pass the keyword 883 to the tool component 512 to retrieve a list of related objects (e.g., types of buildings in New York). This list of related objects can be customized according to the interests of the user, which can be retrieved from the multi-modal memory 508. The XR application generation system 590 can then apply the 3D model generator 894 to each object in the list of related objects of the location. This can generate a 3D model of each object in the list of related objects of the location.

[0231] The XR application generation system 590 can then place each of the 3D models in the real-world environment using the XR application 896. The XR application 896 can also receive inputs that define conditions for each of the objects (e.g., when to apply a particular animation and where to place each of the objects in the real-world environment).

[0232] Figure 9is a flowchart of a process or method 900 performed by the XR application generation system 590 according to some examples. While the flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be re-arranged. A process is terminated when its operations are completed. A process can correspond to a method, a procedure, etc. Steps of a method can be performed entirely or partially by one or more systems or any portion thereof (e.g., a processor in any of the systems included in the system).

[0233] At operation 901, the XR application generation system 590 (e.g., the user system 102 or a server) accesses an XR application by an interactive application as discussed above.

[0234] At operation 902, the XR application generation system 590 receives a query defining one or more attributes of the XR application by the interactive application as discussed above.

[0235] At operation 903, the XR application generation system 590 generates a prompt for a generative machine learning model using the query as discussed above.

[0236] At operation 904, the XR application generation system 590 processes the prompt using the generative machine learning model to generate one or more data objects matching the one or more attributes defined by the query as discussed above.

[0237] At operation 905, the XR application generation system 590 generates one or more XR objects using the XR application based on the one or more data objects generated by the generative machine learning model in response to the prompt as discussed above.

[0238] For example, the XR application generation system 590 can obtain a data definition from the XR application. The XR application generation system 590 can add the data definition and the one or more attributes of the query to the prompt. In some cases, the data definition includes a theme, a graphical element, and a visual attribute of the graphical element.

[0239] In some examples, the one or more attributes include a theme definition. In such cases, the XR application generation system 590 can receive a first data object from the generative machine learning model. The first data object can include a first theme matching the theme definition, a first graphical element associated with the first theme and having a first visual attribute. The XR application generation system 590 can receive a second data object from the generative machine learning model. The second data object can include a second theme matching the theme definition, a second graphical element associated with the second theme and having a second visual attribute.

[0240] In some examples, the XR application generation system 590 generates a first XR object including a first data object and a second XR object including a second data object. The XR application generation system 590 can superimpose the first XR object on a real-world object depicted in the image. In some cases, the XR application generation system 590 superimposes the second XR object instead of the first XR object on the real-world object depicted in the image after a threshold period of time. In some examples, the XR application generation system 590 randomly selects between presenting the first XR object and presenting the second XR object in response to a user request to launch the XR application.

[0241] In some examples, the XR application generation system 590 populates a data template associated with the XR application with one or more data objects generated by the generative machine learning model, the data template used by the XR application to present one or more XR objects. In some cases, the query is received from the first user system 102 and the prompt is generated by the first user system 102. The XR application generation system 590 can store the populated data template associated with the XR application and transmit the XR application with the populated data template to the second user system 102. The XR application generation system 590 can present one or more XR objects on the second user system 102 based on the populated data template.

[0242] In some examples, the one or more attributes include a first set of attributes and the populated data template is a first populated data template. The XR application generation system 590 can receive, by the second user system 102, an additional query defining a second set of attributes for the XR application. The XR application generation system 590 generates an additional prompt for the generative machine learning model using the additional query and processes the additional prompt using the generative machine learning model to generate a second set of data objects matching the second set of attributes. The XR application generation system 590 populates the data template using the second set of data objects to generate a second populated data template and generates a second set of XR objects using the XR application based on the second populated data template.

[0243] In some examples, the XR application generation system 590 presents a first icon associated with the XR application for launching the XR application using the first populated data template. The XR application generation system 590 presents a second icon associated with the XR application for launching the XR application using the second populated data template. In some cases, the one or more attributes include a body part of a person. The one or more data objects include a description of the body part and an attribute of the body part. The XR application generation system 590 can select, by the XR application, a segmentation machine learning model from a plurality of segmentation machine learning models based on the description of the body part. The XR application generation system 590 segments, based on the segmentation machine learning model, a portion of a real-world object depicted in an image, and applies one or more XR objects in real-time to the portion of the real-world object to modify the portion of the real-world object to match the attribute of the body part. In some examples, the description of the body part includes hair on a head of a person. The segmentation machine learning model outputs a segmentation of hair on a head of a real-world object depicted in an image, and the attribute of the body part includes a hair color.

[0244] In some examples, the one or more attributes include a description of a scene and one or more characters in the scene. The one or more data objects include an object type associated with the scene and a character type that matches the one or more characters. The XR application generation system 590 processes the object type associated with the scene and the character type that matches the one or more characters using a visual object model generator. The XR application generation system 590 receives, from the visual object model generator, a visual object definition corresponding to the character type and the object type associated with the scene, and generates one or more XR objects using the visual object definition.

[0245] In some examples, the visual object model generator includes a 3D model generator, and the visual object definition includes a 3D model of the character type and the object type associated with the scene. In some cases, the XR application generation system 590 animates the one or more XR objects based on the visual object definition.

[0246] In some examples, the XR application generation system 590 receives input defining one or more conditions that trigger animation of the one or more XR objects based on the visual object definition, and defining a location within the real-world scene to place the one or more XR objects. In some cases, the one or more conditions include at least one of a time or a virtual distance between a user system and a location of a given one of the XR objects.

[0247] Examples

[0248] Example 1. A method comprising: storing, in a multi-modal memory, interaction data representing usage of one or more interactive features, the interaction data comprising data of different modalities; detecting an object depicted in an image captured by an interactive client; generating, by a machine learning model, a hint based on the object depicted in the image and the interaction data in the multi-modal memory; generating an artificial texture based on the hint; and modifying a texture of the object depicted in the image using the artificial texture generated based on the hint.

[0249] Example 2. The method of example 1, wherein the hint comprises a textual description of an item of fashion worn by the object depicted in the image.

[0250] Example 3. The method of any one of examples 1-2, further comprising: processing the hint using a generative machine learning model to generate the artificial texture.

[0251] Example 4. The method of any one of examples 1-3, further comprising: receiving, from the interactive client, input specifying one or more parameters of the hint.

[0252] Example 5. The method of any one of examples 1-4, further comprising: processing the image to generate a segmentation of an item of fashion worn by the object depicted in the image.

[0253] Example 6. The method of example 5, wherein the item of fashion worn by the object is replaced with the artificial texture based on the segmentation of the item of fashion.

[0254] Example 7. The method of any one of examples 1-6, further comprising: continuously monitoring speech input received from a user of the interactive client; and generating the hint in response to detecting one or more keywords in the speech input.

[0255] Example 8. The method of example 7, wherein the one or more keywords correspond to an item of fashion depicted in the image and one or more attributes.

[0256] Example 9. The method of example 8, wherein the one or more attributes comprise an item of fashion style, the method further comprising: adding, to the hint, an identification of the item of fashion and a target style corresponding to the item of fashion style, wherein the artificial texture comprises an artificial item of fashion of the target style.

[0257] Example 10. The method of example 9, further comprising: identifying one or more preferences of the user based on the interaction data stored in the multi-modal memory; adding an indication of the one or more preferences to the prompt, wherein the artificial texture includes a representation of the one or more preferences.

[0258] Example 11. The method of example 10, wherein the one or more preferences correspond to a sporting event associated with the user, wherein the fashion item style corresponds to a jersey associated with the sporting event, and wherein the one or more preferences include a team associated with the user, the representation including a logo of the team.

[0259] Example 12. The method of any one of examples 1-11, further comprising: generating, by the machine learning model, an annotation representing the prompt; and superimposing the annotation on the image including the object with the modified texture.

[0260] Example 13. The method of any one of examples 1-12, further comprising: detecting an ambiguity in speech input received from a user of the interaction client; and disambiguating the ambiguity based on the multi-modal memory, wherein the prompt is generated in response to disambiguating the ambiguity in the speech input.

[0261] Example 14. A system comprising: at least one processor; and at least one memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: storing, in a multi-modal memory, interaction data representing usage of one or more interactive features, the interaction data including data of different modalities; detecting an object depicted in an image captured by an interaction client; generating, by a machine learning model, a prompt based on the object depicted in the image and the interaction data in the multi-modal memory; generating an artificial texture based on the prompt; and modifying a texture of the object depicted in the image using the artificial texture generated based on the prompt.

[0262] Example 15. The system of example 14, the operations further comprising: processing the prompt using a generative machine learning model to generate the artificial texture.

[0263] Example 16. The system of any one of examples 14-15, the operations further comprising: receiving input from the interaction client specifying one or more parameters of the prompt.

[0264] Example 17. The system of any of examples 14-16, the operations further comprising: processing the image to generate a segmentation of a fashion item worn by the subject depicted in the image.

[0265] Example 18. The system of example 17, wherein the fashion item worn by the subject is replaced with the artificial texture based on the segmentation of the fashion item.

[0266] Example 19. The system of any of examples 14-18, the operations further comprising: continuously monitoring voice input received from a user of the interaction client; and generating the prompt in response to detecting one or more keywords in the voice input.

[0267] Example 20. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising: storing, in a multi-modal memory, interaction data representing usage of one or more interactive functions, the interaction data including data of different modalities; detecting a subject depicted in an image captured by an interaction client; generating, by a machine learning model, a prompt based on the subject depicted in the image and the interaction data in the multi-modal memory; generating an artificial texture based on the prompt; and modifying a texture of the subject depicted in the image using the artificial texture generated based on the prompt.

[0268] Machine Architecture

[0269] Figure 10is a diagrammatic representation of a machine 1000 within which instructions 1002 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine 1000 to perform any one or more of the methodologies discussed herein can be executed. For example, the instructions 1002 can cause the machine 1000 to execute any one or more of the methods described herein. The instructions 1002 transform the general, non-programmed machine 1000 into a particular machine 1000 programmed to carry out the described and illustrated functions in the manner described. The machine 1000 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1000 can operate in the capacity of a server machine or a client machine in server-client network environments, or as a peer machine in peer-to-peer (or distributed) network environments. The machine 1000 can comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions 1002, sequentially or otherwise, that specify actions to be taken by machine 1000. Further, while a single machine 1000 is illustrated, the term “machine” shall also be taken to include a collection of machines that individually or jointly execute the instructions 1002 to perform any one or more of the methodologies discussed herein. For example, the machine 1000 can include the user system 102 or any of the multiple server devices that form part of the interaction server system 110. In some examples, the machine 1000 can further include both the user system and the server system, where some operations of a particular method or algorithm are performed on the server side and where some operations of the particular method or algorithm are performed on the client side.

[0270] The machine 1000 can include processors 1004, memory 1006, and input / output (I / O) components 1008, which can be configured to communicate with each other via a bus 1010. In an example, the processors 1004 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 1012 and a processor 1014 that execute instructions 1002. The term “processor” is intended to include multiple cores of a single processor, which can execute instructions contemporaneously. For instance, a multiple-core processor can enable a single computing device to perform multiple tasks contemporaneously. Also, the term “processor” can refer to more than one processor that are adapted to execute instructions. In this Figure 10 Multiple processors 1004 are illustrated, but the machine 1000 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0271] The memory 1006 includes a main memory 1016, a static memory 1018, and a storage unit 1020 each of which can be accessed via the bus 1010 by the processors 1004. The main memory 1016, static memory 1018, and storage unit 1020 store the instructions 1002 embodying any one or more of the methodologies or functions described herein. The instructions 1002 can also reside, completely or

[0272] The I / O components 1008 can include various components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so forth. The specific I / O components 1008 that are included in the machine 1000 will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I / O components 1008 can include Figure 10Many other components not shown in FIG. 10 can also be included. In various examples, the I / O components 1008 can include user output components 1024 and user input components 1026. The user output components 1024 can include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The user input components 1026 can include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., a physical button, a touch screen that provides location and force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like. Any biometric collected by biometric components is captured and stored with user consent and deleted upon user request.

[0273] Furthermore, such biometric data can be used for very limited purposes, such as identity verification. To ensure limited and authorized use of biometric information and other personally identifiable information (PII), access to such data is limited to authorized personnel, if permitted at all. Any use of biometric data can be strictly limited to identity verification purposes, and the data cannot be shared or sold to any third party without the explicit consent of the user. Furthermore, appropriate technical and organizational measures are also implemented to ensure the security and confidentiality of such sensitive information.

[0274] In further examples, the I / O components 1008 can include biometric components 1028, motion components 1030, environmental components 1032, or position components 1034, among a wide array of other components. Biometric components 1028, as well as the motion, environment, and position components, can include inputs, outputs, or provisions for

[0275] Example types of BMI technology include:

[0276] • Electroencephalography (EEG)-based BMIs, which use electrodes placed on the scalp to record electrical activity in the brain.

[0277] • Invasive BMIs, which use electrodes implanted in the brain through surgery.

[0278] • Optogenetic BMIs, which use light to control the activity of specific neural cells in the brain.

[0279] Motion components 1030 include acceleration sensor components (e.g., accelerometers), gravitation sensor components, rotation sensor components (e.g., gyroscopes).

[0280] Environmental components 1032 include, for example, one or more cameras (with still and / or video capabilities), illumination sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors for detecting concentrations of hazardous gases for safety or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements or signals corresponding to a surrounding physical environment.

[0281] With respect to cameras, user system 102 can have a camera system that includes, for example, a front-facing camera on a front surface of user system 102 and a rear-facing camera on a rear surface of user system 102. The front-facing camera can be used, for example, to capture still and video images of a user of user system 102 (e.g., a “selfie”), which can then be enhanced with the enhancement data described above (e.g., filters). The rear-facing camera can be used, for example, to capture still and video images in a more traditional camera mode, where the images are similarly enhanced with the enhancement data. In addition to the front-facing and rear-facing cameras, user system 102 can include a 360° camera for capturing 360° photos and videos.

[0282] In addition, the camera system of user system 102 can include dual rear-facing cameras (e.g., a primary camera and a depth-sensing camera), or even triple, quad, or penta rear-facing camera configurations on the front and rear sides of user system 102. These multiple camera systems can include, for example, a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0283] Position component 1034 includes location sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect atmospheric pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and the like.

[0284] Communication can be implemented using a wide variety of technologies. The I / O component 1008 also includes communication component 1036, which can be operable to couple the machine 1000 to a network 1038 or to devices 1040 via respective coupling or connection. For example, the communication component 1036 can include a network interface component or another suitable device to interface with the network 1038. In further examples, the communication component 1036 can include a wired communication component, a wireless communication component, a cellular communication component, a Near Field Communication (NFC) component, a Bluetooth® component (e.g., Bluetooth® Low Energy), a Wi-Fi® component, and other communication components to provide communication via other modalities. The devices 1040 can be another machine or any of a wide array of devices (e.g., a peripheral device coupled via a universal serial bus (USB)).

[0285] Moreover, the communication component 1036 can detect identifiers or include components operable to detect identifiers. For example, the communication component 1036 can include radio frequency identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., an optical sensor to detect one-dimensional bar codes such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph TM codes, and other optical codes), or acoustic detection components (e.g., microphones to identify acoustic signatures of identified speakers). In addition, a variety of information can be derived via the communication component 1036, such as location via internet Protocol (IP) geo-location, location via Wi-Fi® signal tri angulation, location via detecting an NFC beacon signal that can indicate a particular location, and so forth.

[0286] The various memories (e.g., the main memory 1016, the static memory 1018, and the memory of the processor 1004) and the storage unit 1020 can store one or more sets of instructions and data structures (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. These instructions (e.g., instructions 1002), when executed by the processor 1004, cause various operations to implement the disclosed examples.

[0287] The instructions 1002 can be transmitted or received over the network 1038 by a network interface device (e.g., a network interface component included in the communication components 1036) using a transmission medium and any one of a number of well-known transfer protocols (e.g., HTTP). Similarly, the instructions 1002 can be transmitted or received using a transmission medium via the coupling (e.g., a peer-to-peer coupling) to the devices 1040.

[0288] Software Architecture

[0289] Figure 11 is a block diagram 1100 illustrating software architectures for an apparatus 1102, which can be any one or more of the apparatus described herein. The apparatus 1102 is illustrated as including hardware 1104, which can include a processor 1106, memory 1108, and an input / output (I / O) interface 1110. In this example, the software architecture 1102 can be conceptualized as a stack of layers, where each layer provides particular functionality. The software architecture 1102 includes layers such as an operating system 1112, libraries 1114, frameworks 1116, and applications 1118. Operationally, the applications 1118 and the

[0290] The operating system 1112 manages hardware resources and provides common services. The operating system 1112 includes, for example, a kernel 1124, services 1126, and drivers 1128. The kernel 1124 acts as an abstraction layer between the hardware and the other software layers. For example, the kernel 1124 provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functionality. The services 1126 can provide other common services for the other software layers. The drivers 1128 are responsible for controlling or interfacing with the underlying hardware. For instance, the drivers 1128 can include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), WI-FI® drivers, audio drivers, power management drivers, and so forth.

[0291] The libraries 1114 provide a common low-level infrastructure used by the applications 1118. The libraries 1114 can include system libraries 1130 (e.g., C standard library) providing functions such as memory allocation functions, string manipulation functions, mathematic functions, and the like. In addition, the libraries 1114 can include API libraries 1132 such as media libraries (e.g., libraries to support presentation and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., an OpenGL framework used to render 2D and 3D in 2D and 3D graphics content on a display), database libraries (e.g., SQLite to provide various relational database functions), web libraries (e.g., WebKit to provide web browsing functionality), and the like. The libraries 1114 also include a wide variety of other libraries 1134 to provide many other APIs to the applications 1118.

[0292] The frameworks 1116 provide a common high-level infrastructure used by the applications 1118. For example, the frameworks 1116 provide various GUI functionality, high-level resource management, and high-level positioning services. The frameworks 1116 can provide a broad spectrum of other APIs that can be used by the applications 1118, some of which can be specific to a particular operating system or platform.

[0293] In an example, the applications 1118 include a home application 1136, a contacts application 1138, a browser application 1140, a book reader application 1142, a location application 1144, a media application 1146, a messaging application 1148, a game application 1150, and a broad assortment of other applications such as a third-party application 1152. The applications 1118 are programs that execute functions defined in the programs. Programs

[0294] System with head-mounted device

[0295] Figure 12 A system 1200 including a head-mounted device 116 having a selector input device is shown in accordance with some examples. Figure 12 is a high-level functional block diagram of an example head-mounted device 116 communicatively coupled to a mobile device 114 and various server systems 1204 (e.g., an interaction server system 110) via various networks 1216.

[0296] The head-mounted device 116 includes one or more cameras, each of which can be, for example, a visible light camera 1206, an infrared emitter 1208, and an infrared camera 1210.

[0297] The mobile device 114 connects with the head-mounted device 116 using both a low-power wireless connection 1212 and a high-speed wireless connection 1214. The mobile device 114 also connects to the server systems 1204 and the networks 1216.

[0298] The head-mounted device 116 also includes two image displays of an optical assembly 1218. The two image displays of the optical assembly 1218 include an image display associated with a left lateral side of the head-mounted device 116 and an image display associated with a right lateral side of the head-mounted device 116. The head-mounted device 116 also includes an image display driver 1220, an image processor 1222, low-power circuitry 1224, and high-speed circuitry 1226. The image displays of the optical assembly 1218 are used to present images and videos including images, which can include GUIs, to a user of the head-mounted device 116.

[0299] The image display driver 1220 commands and controls the image displays of the optical assembly 1218. The image display driver 1220 can pass image data directly to the image displays of the optical assembly 1218 for presentation or can convert the image data to a signal or data format suitable for passing to the image display devices. For example, the image data can be video data formatted according to a compression format such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, or the like, and still image data can be formatted according to a compression format such as PNG, JPEG, Tagged Image File Format (TIFF), or Exchangeable image file format (EXIF), among others.

[0300] The head-mounted device 116 includes a frame and poles (or legs) extending from the sides of the frame. The head-mounted device 116 also includes a user input device 1228 (e.g., a touch sensor or button) that includes an input surface on the head-mounted device 116. The user input device 1228 (e.g., a touch sensor or button) receives input selections from the user to manipulate a GUI of presented images.

[0301] Figure 12 The components shown for the head-mounted device 116 are located on one or more circuit boards (e.g., PCBs or flexible PCBs) on the edges or legs. Alternatively or additionally, the depicted components may be located in blocks, frames, hinges, or bridges of the head-mounted device 116. The left and right visible light imaging devices 1206 may include digital imaging elements, such as complementary metal-oxide-semiconductor (CMOS) image sensors, charge-coupled devices, imaging lenses, or any other corresponding visible or light-capturing elements that can be used to capture data including images of a scene with unknown objects.

[0302] The head-mounted device 116 includes a memory 1202 that stores instructions for performing a subset or all of the functions described herein. The memory 1202 may also include a storage device.

[0303] like Figure 12 As shown, the high-speed circuit system 1226 includes a high-speed processor 1230, a memory 1202, and a high-speed wireless circuit system 1232. In some examples, an image display driver 1220 is coupled to the high-speed circuit system 1226 and operated by the high-speed processor 1230 to drive the left and right image displays of the image display of the optical assembly 1218. The high-speed processor 1230 can be any processor capable of managing high-speed communication and operation of any general-purpose computing system required for the head-mounted device 116. The high-speed processor 1230 includes the processing resources required to manage high-speed data transmission over a high-speed wireless connection 1214 to a wireless local area network (WLAN) using the high-speed wireless circuit system 1232. In some examples, the high-speed processor 1230 executes an operating system such as the LINUX operating system or another such operating system for the head-mounted device 116, and the operating system is stored in the memory 1202 for execution. Among other duties, the high-speed processor 1230, which executes the software architecture for the head-mounted device 116, also manages data transmission with the high-speed wireless circuit system 1232. In some examples, the high-speed wireless circuit system 1232 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as WiFi. In some examples, other high-speed communication standards may be implemented by the high-speed wireless circuit system 1232.

[0304] The low-power wireless circuitry system 1234 and high-speed wireless circuitry system 1232 of the head-mounted device 136 may include a short-range transceiver (Bluetooth™) and a wireless wide-area network transceiver, a local area network transceiver, or a wide-area network transceiver (e.g., cellular or WiFi). The mobile device 114 (including transceivers communicating via low-power wireless connection 1212 and high-speed wireless connection 1214) can be implemented using the architectural details of the head-mounted device 116, as can other components of the network 1216.

[0305] Memory 1202 includes any storage device capable of storing various data and applications, including camera data generated by the left and right visible light cameras 1206, the infrared camera 1210, and the image processor 1222, as well as images generated by the image display driver 1220 on the image display of the optical assembly 1218 for display. While memory 1202 is shown as integrated with the high-speed circuitry 1226, in some examples, memory 1202 may be a separate, independent component of the head-mounted device 116. In some such examples, electrical wiring may provide a connection from the image processor 1220 or the low-power processor 1236 to memory 1202 via a chip including the high-speed processor 1230. In some examples, the high-speed processor 1230 may manage addressing of memory 1202 such that the low-power processor 1236 will activate the high-speed processor 1230 whenever a read or write operation involving memory 1202 is required.

[0306] like Figure 12 As shown, the low-power processor 1236 or high-speed processor 1230 of the head-mounted device 136 may be coupled to a camera device (visible light camera 1206, infrared emitter 1208 or infrared camera 1210), an image display driver 1220, a user input device 1228 (e.g., a touch sensor or button) and a memory 1202.

[0307] The head-mounted device 116 is connected to a host computer. For example, the head-mounted device 116 is paired with the mobile device 114 via a high-speed wireless connection 1214, or connected to the server system 1204 via a network 1216. The server system 1204 may be one or more computing devices as part of a service or network computing system, including, for example, a processor, memory, and a network communication interface for communicating with the mobile device 114 and the head-mounted device 116 via the network 1216.

[0308] The mobile device 114 includes a processor and a network communication interface coupled to the processor. The network communication interface allows for communication over the network 1216, the low power wireless connection 1212, or the high speed wireless connection 1214. The mobile device 114 can also store, in memory of the mobile device 114, at least a portion of instructions for generating binaural audio content to implement the functionality described herein.

[0309] The output components of the head-mounted device 116 include visual components, such as displays (e.g., LCD, PDP, LED displays), projectors, or waveguides. The image displays of the optical assembly are driven by an image display driver 1220. The output components of the head-mounted device 116 also include acoustic components (e.g., speakers), haptic components (e.g., a vibrating motor), other signal generators, and the like. The input components (e.g., user input devices 1228) of the head-mounted device 116, the mobile device 114, and the server system 1204 can include alphanumeric input components (e.g., a keyboard, a touch screen, a photo-optical keyboard, or other alphanumeric input components configured to receive alphanumeric input), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., physical buttons, a touch screen that provides the location and force of touch or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0310] The head-mounted device 116 can also include additional peripheral elements. Such peripheral elements can include biometric sensors, additional sensors, or display elements integrated with the head-mounted device 116. For example, the peripheral elements can include any I / O components including output components, motion components, location components, or any other such elements described herein.

[0311] For example, the biometric components include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like. The biometric components can include BMI systems that allow communication between a brain and an external device or machine. This can be achieved, for example, by recording brain activity data, converting the data into a format understandable by a computer, and then using the resulting signals to control a device or machine.

[0312] Motion components include acceleration sensor components (e.g., accelerometers), gravitation sensor components, rotation sensor components (e.g., gyroscopes), and so forth. Position components include location sensor components (e.g., GPS receiver components) for generating location coordinates, Wi-Fi or Bluetooth™ transceivers for generating positioning system coordinates, altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and so forth. Such positioning system coordinates can also be received from mobile device 114 via low power wireless circuitry 1234 or high speed wireless circuitry 1232 over low power wireless connection 1212 and high speed wireless connection 1214.

[0313] Glossary

[0314] "Carrier signal" refers to any intangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions can be transmitted or received over a network via a network interface device using a transmission medium.

[0315] "Client device" refers to any machine that interfaces to a communications network to obtain resources from one or more server systems or other client devices. A client device can be, but is not limited to, a mobile phone, desktop computer, laptop computer, portable digital assistants (PDAs), smartphones, tablets, ultrabooks, netbooks, notebooks, multi-processor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access a network.

[0316] A “communication network” refers to, for example, one or more portions of a network that can be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a WLAN, a wide area network (WAN), a wireless WAN (WW AN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or a portion of a network can include a wireless or cellular network, and coupling can be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other type of cellular or wireless coupling. In this example, coupling can enable any of a variety of types of data transfer technology including Single Carrier Radio Transmission Technology (lxRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, third Generation Partnership Project (3GPP) including 3G, fourth generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standard, others defined by various standards setting organizations, other long range protocols, or other data transfer technology.

[0317] A “component” refers to, for example, a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, APIs, or other technologies that provide for the partitioning or modularization of a particular processing or control function. Components can be combined via their interfaces to execute machine processes. A component can be a packaged functional hardware unit designed for use with other components, and can be a part of a program, or a piece of code that usually performs a particular function. Components can constitute either software components (e.g., code embodied on a machine-readable medium) or hardware components.

[0318] A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or application portion) as a hardware component that operates to perform certain operations as described herein.

[0319] Hardware components can also be implemented mechanically, electronically, or in any suitable combination of the two. For example, a hardware component can include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component can be implemented in a number of different ways. For example, a hardware component can include a special-purpose processor, e.g., a field-programmable gate array (FPGA) or an ASIC. A hardware component can also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, hardware components become specific machines (or specific components of a machine) uniquely tailored to perform the configured functions and are no longer general-purpose processors. It will be recognized that a hardware component can be a general-purpose processor, e.g., an Intel Pentium® processor or AMD Athlon® processor, that is programmed to behave as an ASIC. It will also be recognized that a hardware component can be part of a larger physical entity, e.g., a computer having a number of processors, that is configured by the software to behave as a particular machine. In general, a hardware component can include any physical or tangible hardware component.

[0320] Consider a scenario in which a hardware component is temporarily configured (e.g., programmed) to perform particular operations. The hardware component can subsequently be reconfigured (e.g., programmed) to perform different operations. In some scenarios, a hardware component can be configured to perform a first operation and then reconfigured to perform a second operation, even within a very short period of time. Since more recent configuration of a hardware component can override previous configurations of that hardware component, a hardware component can concurrently perform multiple tasks, e.g., by virtue of its multiple configurations.

[0321] As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, methods described herein can be implemented, at least in part, by a processor, wherein a particular one or more processors are examples of hardware. For example, at least some of the operations of a method can be performed by one or more processors or processor-implemented components. Also, one or more processors can operate to support performance of the relevant operations in a "cloud computing" environment or as a "Software as a Service" (SaaS). For example, at least some of the operations can be performed by a group of computers (as examples of machines including processors), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., an API). The performance of certain of the operations can be distributed among the processors, not only residing within a single machine, but deployed across a number of machines. In some examples, processors or processor-implemented components can be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other examples, processors or processor-implemented components can be distributed across multiple geographic locations.

[0322] "Computer-readable storage media" refers to both machine-storage media and transmission media. Thus, the terms include both storage devices / media and carrier waves / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure. A "transient message" refers to a message that is accessible for a limited duration of time, for example. A transient message can be text, an image, a video, etc. The access time for a transient message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is temporary.

[0323] "Machine-storage media" refers to machine-storage media and means for example, single or multiple storage devices and media (e.g., a centralized or distributed database, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be taken to include, but not be limited to, solid state storage, as well as optical and magnetic media, including memory internal or external to processors. Specific examples of machine-storage media, computer-storage media, and device-storage media include non-volatile memory, including, for example, semiconductor memory devices, e.g., Erasable Programmable Read-Only Memories (EPROM), Electrically Erasable Programmable Read-Only Memories (EEPROM), FPGAs, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine-storage media," "device-storage media," "computer-storage media," mean the same thing and can be used interchangeably in this disclosure.

[0324] The terms “machine-storage medium,” “computer-storage medium,” and “device-storage medium” exclude, among other things, a carrier wave, modulated data signal, and other such media, at least some of which are covered under the term “signal medium.” A “non-transitory computer-readable storage medium” refers to a tangible medium that is capable of storing, encoding, or carrying instructions for execution by a machine. A “signal medium” refers to any intangible medium that is capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communications signals or other intangible media to facilitate communication of software or data. The term “signal medium” shall be taken to include any form of a modulated data signal, carrier wave, and so on. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms “transmission medium” and “signal medium” mean the same thing and can be used interchangeably in this disclosure.

[0325] “User device” refers to a device accessed by, controlled by, or owned by a user, and with which a user interacts to perform actions on the user device or to interact, including interacting with other users or computer systems. “Carrier signal” refers to any intangible medium that is capable of storing, encoding, or carrying instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions can be transmitted or received via a network interface device using a transmission medium. “Client device” refers to any machine that interfaces to a communications network to access resources from a server or other client device. A client device can be, but is not limited to, a mobile telephone, a desktop computer, a laptop computer, a PDA, a smart phone, a tablet, an ultrabook, a netbook, a notebook, a multi-processor system, a microprocessor-based or programmable consumer electronics, a game console, a STB, or any other communication device that a user can use to access a network.

[0326] A "communication network" refers to one or more portions of a network that can be an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, the Internet, a portion of the Internet, a portion of the PSTN, the Public Switched Telephone Network (PSTN), a POTS network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or a portion of a network can include a wireless or cellular network, and a coupling can be a CDMA connection, a GSM connection, or other type of cellular or wireless coupling. In this example, the coupling can enable any of a variety of types of data transfer technology, such as 1xRTT, EVDO technology, GPRS technology, EDGE technology, 3GPP including 3G, 4G networks, UMTS, HSPA, WiMAX, LTE standards, other data transfer technology defined by various standards-setting organizations, other long-range protocol, or other data transfer technology.

[0327] A component can constitute either software components (e.g., code embodied on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit capable of performing certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or application portion) as a hardware component that operates to perform certain operations as described herein.

[0328] A hardware component can also be implemented mechanically, electronically, or in any suitable combination of the two. For example, a hardware component can include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component can be a special-purpose processor, such as a FPGA or an ASIC. A hardware component can also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, hardware components become specific machines (or specific components of a machine) uniquely tailored to perform the configured functions and are no longer general-purpose processors. It will be appreciated that the

[0329] The various operations of example methods described herein can be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that conduct operations to execute one or more operations or functions described herein.

[0330] Changes and modifications can be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure, as expressed in the appended claims.

Claims

1. A method comprising: accessing an extended reality (XR) application through an interactive application; receiving, through the interactive application, a query defining one or more attributes of the XR application; generating, using the query, a prompt for a generative machine learning model; processing, using the generative machine learning model, the prompt to generate one or more data objects matching the one or more attributes defined by the query; and generating, using the XR application, one or more XR objects based on the one or more data objects generated by the generative machine learning model in response to the prompt.

2. The method of claim 1, further comprising: obtaining, from the XR application, a data definition; and adding the data definition and the one or more attributes of the query to the prompt. The data definition comprises a theme, a graphical element, and a visual attribute of the graphical element. The one or more attributes comprise a theme definition, the method further comprising:

3. The method of any one of claims 1-2, wherein, receiving, from the generative machine learning model, a first data object comprising a first theme matching the theme definition, a first graphical element associated with the first theme and having a first visual attribute; and 4. The method of any one of claims 1 to 3, wherein, receiving, from the generative machine learning model, a second data object comprising a second theme matching the theme definition, a second graphical element associated with the second theme and having a second visual attribute.

5. The method of claim 4, further comprising: generating a first XR object comprising the first data object and a second XR object comprising the second data object; and superimposing the first XR object on a real-world object depicted in an image.

6. The method of any one of claims 1-5, further comprising: after a threshold period of time, superimposing the second XR object instead of the first XR object on the real-world object depicted in the image.

7. The method of any one of claims 1-6, further comprising, in response to a user request to launch the XR application, randomly selecting between presenting the first XR object and presenting the second XR object.

8. The method of any one of claims 1-7, further comprising: populating a data template associated with the XR application using the one or more data objects generated by the generative machine learning model, the data template used by the XR application to present the one or more XR objects.

9. The method of claim 8, the query received from a first user system and the prompt generated by the first user system, the method further comprising: storing the populated data template associated with the XR application; sending the XR application with the populated data template to a second user system; and presenting the one or more XR objects on the second user system based on the populated data template. ​ ​ ​ 10. The method of any one of claims 1 to 9, wherein, The one or more attributes include a first set of attributes, and wherein the populated data template is a first populated data template, the method further comprising: receiving, by the second user system, an additional query defining a second set of attributes for the XR application; generating, using the additional query, an additional prompt for the generative machine learning model; processing, using the generative machine learning model, the additional prompt to generate a second set of data objects matching the second set of attributes; populating, using the second set of data objects, the data template to generate a second populated data template; and generating, using the XR application, a second set of XR objects based on the second populated data template.

11. The method of claim 10, further comprising: presenting a first icon associated with the XR application for launching the XR application using the first populated data template; and presenting a second icon associated with the XR application for launching the XR application using the second populated data template. The one or more attributes include a body part of a person, wherein the one or more data objects include a description of the body part and an attribute of the body part, the method further comprising:

12. The method of any one of claims 1 to 11, wherein, selecting, by the XR application, a segmentation machine learning model from a plurality of segmentation machine learning models based on the description of the body part; segmenting, based on the segmentation machine learning model, a portion of a real-world object depicted in an image; and applying, in real-time, the one or more XR objects to the portion of the real-world object to modify the portion of the real-world object to match the attribute of the body part. The description of the body part includes hair on a head of a person, wherein the segmentation machine learning model outputs a segmentation of hair on a head of the real-world object depicted in the image, and wherein the attribute of the body part includes a hair color.

13. The method of claim 12, wherein, The one or more attributes include a description of a scene and one or more characters in the scene, wherein the one or more data objects include an object type associated with the scene and a character type matching the one or more characters, the method further comprising:

14. The method of any one of claims 1 to 13, wherein, processing, using a visual object model generator, the object type associated with the scene and the character type matching the one or more characters; receiving, from the visual object model generator, a visual object definition corresponding to the character type and the object type associated with the scene; and generating, using the visual object definition, the one or more XR objects. The visual object model generator includes a three-dimensional (3D) model generator, and wherein the visual object definition includes a 3D model of the character type and the object type associated with the scene.

15. The method of claim 14, wherein, 16. The method of any one of claims 1-15, further comprising: animating, based on the visual object definition, the one or more XR objects.

17. The method of claim 16, further comprising: ​ receiving input defining one or more conditions that, based on the visual object, define triggering animation of the one or more XR objects, and defining a location within a real-world scene at which to place the one or more XR objects.

18. The method of any one of claims 1 to 17, wherein, The one or more conditions include at least one of a time or a virtual distance between a user system and a location of a given one of the XR objects.

19. A system comprising: at least one processor; and at least one memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: accessing, by an interactive application, an extended reality (XR) application; receiving, by the interactive application, a query defining one or more attributes of the XR application; generating, using the query, a prompt for a generative machine learning model; processing, using the generative machine learning model, the prompt to generate one or more data objects matching the one or more attributes defined by the query; and generating, using the XR application, one or more XR objects based on the one or more data objects generated by the generative machine learning model in response to the prompt.

20. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising: accessing, by an interactive application, an extended reality (XR) application; receiving, by the interactive application, a query defining one or more attributes of the XR application; generating, using the query, a prompt for a generative machine learning model; processing, using the generative machine learning model, the prompt to generate one or more data objects matching the one or more attributes defined by the query; and generating, using the XR application, one or more XR objects based on the one or more data objects generated by the generative machine learning model in response to the prompt. ​

Images