Customizable chat robot for interactive platform

Abstract

A chat robot system for an interactive platform. The chat robot receives a prompt in the form of an interactive platform publication to the interactive platform and generates a response in the form of an interactive platform publication. In one aspect, a chat robot system generates messages on an interactive platform for a group chat with multiple users and a one-to-one chat with a single user. In another aspect, a user may customize a chat robot to create a personal chat robot with a role. In some examples, prompts from a user to a chat robot are analyzed to filter abused languages and / or harmful content.

Description

[0001] Priority claim

[0002] This application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 486,824, filed on February 24, 2023, and U.S. Provisional Application Serial No. 63 / 496,877, filed on April 18, 2023, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to interactive platforms, and more particularly to providing interactive interfaces to users of interactive platforms. Background Art

[0004] Users like to visit interactive platforms to share content with other users of the interactive platform. In addition, users are often sensitive to content when communicating with chatbots. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the accompanying drawings (which are not necessarily drawn to scale), the same reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, the highest digit or digits in the reference numerals refer to the figure in which the element is first introduced. Some non-limiting examples are shown in the accompanying drawings, in which:

[0006] Figure 1 is a diagrammatic representation of a networked environment in which the present disclosure may be deployed, according to some examples.

[0007] Figure 2 is a diagrammatic representation of a messaging system having both client-side and server-side functionality, according to some examples.

[0008] Figure 3 is a block diagram of a chatbot system according to some examples.

[0009] Figure 4 is a process flow diagram of a chatbot interaction process between a user and a chatbot system according to some examples.

[0010] Figure 5 is an illustration of another interactive session between a user and a chatbot system according to some examples.

[0011] Figure 6 is an illustration of another interactive session between a user and a chatbot system according to some examples.

[0012] Figure 7 is an illustration of another interactive session between a user and a chatbot system according to some examples.

[0013] Figure 8A is a sequence diagram of a message processing pipeline based on some examples.

[0014] Figure 8B This is an activity diagram of the user and chatbot messaging methods based on some examples.

[0015] Figure 9A An example chatbot role configuration process according to some examples is shown.

[0016] Figure 9B A chatbot role configuration user interface according to some examples is shown.

[0017] Figure 9C is an illustration of an extended role description based on some examples.

[0018] Figure 9D This is an illustration of some example chatbot profile displays.

[0019] Figure 10A is a process flow diagram of a group chat session according to some examples.

[0020] Figure 10B and Figure 10C is an illustration of a user interface during a group chat session according to some examples.

[0021] Figure 11 A machine learning pipeline according to some examples is shown.

[0022] Figure 12 The training and use of a machine learning program according to some examples is shown.

[0023] Figure 13 is a diagrammatic representation of data structures as maintained in a database, according to some examples.

[0024] Figure 14 is a diagrammatic representation of messages according to some examples.

[0025] Figure 15 is a flow chart for access restriction processing according to some examples.

[0026] Figure 16 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed, causing the machine to perform any one or more of the methodologies discussed herein.

[0027] Figure 17 is a block diagram illustrating a software architecture in which examples may be implemented. DETAILED DESCRIPTION

[0028] Interactive platforms (e.g., social platforms, social media platforms, interactive systems, AR platforms, applications, messaging platforms, AR applications, operating systems, gaming systems or applications, systems with which users interact, etc.) can provide users with ways to interact with other users. For some users, interaction with the interactive platform can be enhanced by having a customizable chatbot with which to interact. Chatbots can serve a variety of purposes, including providing users with ways to generate content for posting to the interactive platform, edit existing content, and receive suggestions in the form of interactive platform posts.

[0029] Examples of the present disclosure enable customizable and configurable chatbot interactions within an interactive platform. By receiving prompts from users in the form of posts to the interactive platform, determining the user's intent based on the prompts, and generating response posts using that intent, the chatbot can engage in natural conversations with users in a format that is appropriate for the interactive platform. This allows the chatbot to seamlessly integrate into existing user experiences. The chatbot can process prompts containing media such as images, video, and audio, generating media responses or editing the original media. It can also suggest activities to users based on their prompts.

[0030] In some examples, chatbots can adopt customizable personas defined by users. By receiving persona descriptions from users and using these descriptions to generate responses, chatbots can present a unique personality tailored to each user. Chatbots scan persona descriptions for offensive content and can expand the descriptions into more detailed personas. This allows users to customize the tone and style of the chatbot's responses. In some examples, chatbots generate randomized personas on demand.

[0031] In some examples, chatbots can participate in group chats. They do this by detecting chatbot mentions in group chats, generating prompts and responses accordingly, and providing notifications to users. Their responses can be personalized based on the user mentioning them and chat history. The chatbot also enables users to delete their messages, protecting privacy.

[0032] Some examples provide adaptable, persona-based chatbots that are suitable for seamless integration into different interactive platforms and messaging environments. Customized personas, multimodal prompt handling, and group chat capabilities provide users with a more natural and engaging conversational experience.

[0033] In some examples, a chatbot system receives a prompt from a user during an interactive session, wherein the prompt is in the form of a post to an interactive platform. The chatbot system uses the prompt to determine the user's intent and uses the intent to generate a response, wherein the response is also in the form of a post to the interactive platform. The chatbot system provides the response to the user.

[0034] In some examples, the chatbot system generates responses by using intents to generate suggested activities.

[0035] In some examples, the chatbot system generates responses using a description of the chatbot's role.

[0036] In some examples, where the user's prompt form includes media, the chatbot system determines intent by generating a description of the media and uses the intent to generate response media.

[0037] Other technical features will be apparent to those skilled in the art from the accompanying drawings, description and claims.

[0038] Networked computing environment

[0039] Figure 1 1 is a block diagram illustrating an example interactive platform 100 for facilitating interactions over a network (e.g., exchanging text messages, conducting text, audio, and video calls, or playing games). The interactive platform 100 includes a plurality of user systems 102, each of which hosts a plurality of applications including an interactive client 104 and other applications 106. Each interactive client 104 is communicatively coupled to other instances of the interactive client 104 (e.g., hosted on respective other user systems 102), an interactive server system 110, and third-party servers 112 via one or more communication networks including a network 108 (e.g., the Internet). The interactive client 104 can also communicate with the locally hosted application 106 using an application program interface (API).

[0040] Each user system 102 may include multiple user devices, such as a mobile device 114, a head mounted device 116, and a computer client device 118, which are communicatively connected to exchange data and messages.

[0041] The interactive clients 104 interact with other interactive clients 104 and with the interactive server system 110 via the network 108. The data exchanged between the interactive clients 104 (e.g., interaction 120) and between the interactive clients 104 and the interactive server system 110 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0042] The interactive server system 110 provides server-side functionality to the interactive clients 104 via the network 108. Although certain functionality of the interactive platform 100 is described herein as being performed by either the interactive clients 104 or the interactive server system 110, whether certain functionality is located within the interactive clients 104 or within the interactive server system 110 may be a design choice. For example, it may be technically preferable to initially deploy certain technology and functionality within the interactive server system 110 but later migrate that technology and functionality to the interactive clients 104 where the user systems 102 have sufficient processing power.

[0043] The interactive server system 110 supports various services and operations provided to the interactive clients 104. Such operations include sending data to the interactive clients 104, receiving data from the interactive clients 104, and processing data generated by the interactive clients 104. The data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, interactive platform information, and live event information. The data exchange within the interactive platform 100 is activated and controlled by functions available through the user interface (UI) of the interactive client 104.

[0044] Turning now specifically to the interactive server system 110, an API server 122 is coupled to and provides a programming interface to an interactive server 124, making the functionality of the interactive server 124 accessible to the interactive clients 104, other applications 106, and third-party servers 112. The interactive server 124 is communicatively coupled to a database server 126, thereby facilitating access to a database 128 that stores data associated with interactions processed by the interactive server 124. Similarly, a web server 130 is coupled to the interactive server 124 and provides a web-based interface to the interactive server 124. To this end, the web server 130 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0045] The API server 122 receives and sends interaction data (e.g., commands and message payloads) between the interaction server 124 and the user system 102 (and, for example, the interaction clients 104 and other applications 106), as well as third-party servers 112. Specifically, the API server 122 provides a set of interfaces (e.g., routines and protocols) that the interaction clients 104 and other applications 106 can call or query to activate functionality of the interaction server 124. The API server 122 exposes various functions supported by the interaction server 124, including account registration; login functionality; sending interaction data from a particular interaction client 104 to another interaction client 104 via the interaction server 124; transferring media files (e.g., images or videos) from an interaction client 104 to the interaction server 124; setting up media data collections (e.g., stories); retrieving a friend list of a user of the user system 102; retrieving information and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends within a social graph; and opening application events (e.g., related to the interaction client 104).

[0046] Interactive server 124 hosts multiple systems and subsystems, see below Figure 2 Provide a description.

[0047] Linked Applications

[0048] Returning to the interactive client 104, the features and functionality of an external resource (e.g., a linked application 106 or applet) are made available to the user via the interface of the interactive client 104. In this context, "external" refers to the fact that the application 106 or applet is external to the interactive client 104. External resources are typically provided by a third party, but may also be provided by the creator or provider of the interactive client 104. The interactive client 104 receives a user selection of an option to launch or access features of such an external resource. The external resource may be an application 106 installed on the user system 102 (e.g., a "local app"), or a small-scale version of an application (e.g., a "mini-program") hosted on the user system 102 or located remotely from the user system 102 (e.g., on a third-party server 112). The small-scale version of an application includes a subset of the features and functionality of the application (e.g., a full-scale local version of the application) and is implemented using a markup language document. In some examples, the small-scale version of an application (e.g., a "mini-program") is a web-based markup language version of the application and is embedded in the interactive client 104. In addition to using markup language documents (eg, .*ml files), applet may include scripting languages ​​(eg, .*js files or .json files) and style sheets (eg, .*ss files).

[0049] In response to receiving a user selection of an option to launch or access a feature of an external resource, the interactive client 104 determines whether the selected external resource is a web-based external resource or a locally installed application 106. In some cases, an application 106 locally installed on the user system 102 can be independent of and launched separately from the interactive client 104, for example, by selecting an icon corresponding to the application 106 on a home screen of the user system 102. A small-scale version of such an application can be launched or accessed via the interactive client 104, and in some examples, no portion of the small-scale application can be accessed outside of the interactive client 104 or only a limited portion of the small-scale application can be accessed outside of the interactive client 104. The small-scale application can be launched by the interactive client 104 receiving, for example, a markup language document associated with the small-scale application from a third-party server 112 and processing such a document.

[0050] In response to determining that the external resource is a locally installed application 106, the interactive client 104 instructs the user system 102 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interactive client 104 communicates with the third-party server 112 (for example) to obtain a markup language document corresponding to the selected external resource. The interactive client 104 then processes the obtained markup language document to present the web-based external resource within the user interface of the interactive client 104.

[0051] The interactive client 104 can notify users of the user system 102 or other users associated with such users (e.g., "friends") of activities occurring in one or more external resources. For example, the interactive client 104 can provide participants in a conversation (e.g., a chat session) within the interactive client 104 with notifications regarding external resources currently or recently used by one or more members of a group of users. One or more users can be invited to join an active external resource or a recently used but currently inactive external resource (in the group of friends) can be launched. The external resource can provide participants in the conversation, each using a corresponding interactive client 104, with the ability to share items, conditions, states, or locations in the external resource with one or more members of the group of users in the chat session. The shared items can be interactive chat cards that members of the chat can interact with to, for example, launch the corresponding external resource, view specific information within the external resource, or take members of the chat to a specific location or state within the external resource. Within a given external resource, a response message can be sent to the user on the interactive client 104. The external resource can selectively include different media items in the response based on the current context of the external resource.

[0052] The interactive client 104 can present a list of available external resources (e.g., applications 106 or applets) to the user to launch or access a given external resource. The list can be presented in the form of a context-sensitive menu. For example, the icons representing different applications (or applets) of the application 106 (or applets) can change based on how the user launches the menu (e.g., from a conversational interface or from a non-conversational interface).

[0053] System Architecture

[0054] Figure 2 is a block diagram illustrating additional details regarding the interactive platform 100 according to some examples. Specifically, the interactive platform 100 is shown as including an interactive client 104 and an interactive server 124. The interactive platform 100 includes a number of subsystems that are supported on the client side by the interactive client 104 and on the server side by the interactive server 124. Example subsystems are discussed below.

[0055] Image processing system 202 provides various functions that enable a user to capture and enhance (eg, enhance or otherwise modify or edit) media content associated with a message.

[0056] The camera system 204 includes control software (e.g., in a camera application) that interacts with and controls the hardware camera hardware of the user system 102 (e.g., directly or via operating system control) to modify and enhance the real-time images captured and displayed via the interactive client 104.

[0057] The enhancement system 206 provides functionality related to the generation and publication of enhancements (e.g., media overlays) for images captured in real time by the camera of the user system 102 or retrieved from the memory of the user system 102. For example, the enhancement system 206 is operable to select, present, and display media overlays (e.g., image filters or image filters) for the interactive client 104 for enhancing the real-time imagery received via the camera system 204 or the stored imagery retrieved from the memory of the user system 102. These enhancements are selected and presented to the user of the interactive client 104 by the enhancement system 206 based on inputs and data such as:

[0058] The geolocation of the user system 102; and

[0059] Interactive platform information of the user of the user system 102.

[0060] Enhancement can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos or videos) at user system 102 for transmitting in a message, or applied to video content such as a video content stream or feed sent from interactive client 104. Therefore, image processing system 202 can interact with the various subsystems of communication system 208 and support the various subsystems of communication system 208, such as messaging system 210 and video communication system 212.

[0061] Media overlays can include text or image data that can be superimposed on a photo taken by user system 102 or a video stream produced by user system 102. In some examples, the media overlay can be a location overlay (e.g., Venice Beach), the name of a live event, or a business name overlay (e.g., Beach Cafe). In other examples, image processing system 202 uses the geographic location of user system 102 to identify a media overlay that includes the name of a business at the geographic location of user system 102. The media overlay may include other tags associated with the business. The media overlay may be stored in database 128 and accessed by database server 126.

[0062] Image processing system 202 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. Users can also specify situations in which specific media overlays should be provided to other users. Image processing system 202 generates a media overlay that includes the uploaded content and associates it with the selected geographic location.

[0063] The augmented creation system 214 supports the augmented reality developer platform and includes applications for content creators (e.g., artists and developers) to create and publish augmentations (e.g., augmented reality experiences) for the interactive clients 104. The augmented creation system 214 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking techniques, and templates.

[0064] In some examples, the enhancement creation system 214 provides a merchant-based publishing platform that enables merchants to select specific enhancements associated with a geolocation via a bidding process. For example, the enhancement creation system 214 associates the highest bidding merchant's media overlay with the corresponding geolocation for a predefined amount of time.

[0065] The communication system 208 is responsible for enabling and processing various forms of communication and interaction within the interactive platform 100 and includes a messaging system 210, a chatbot system 232, an audio communication system 216, and a video communication system 212. The messaging system 210 is responsible for enforcing temporary or time-limited access to content by the interactive clients 104. The messaging system 210 includes multiple timers within a transient timer system (not shown) that selectively enable access (e.g., for presentation and display) of messages and associated content via the interactive clients 104 based on duration and display parameters associated with a message or collection of messages (e.g., a story). Additional details regarding the operation of the transient timer system are provided below. The audio communication system 216 enables and supports audio communication (e.g., real-time audio chat) between multiple interactive clients 104. Similarly, the video communication system 212 enables and supports video communication (e.g., real-time video chat) between multiple interactive clients 104. The chatbot system 232 is responsible for generating and transmitting responses to prompts received from users.

[0066] The user management system 218 is operationally responsible for managing user data and profiles, and includes a social networking system 220 that maintains interactive platform information regarding relationships between users of the interactive platform 100 .

[0067] The collection management system 222 is operationally responsible for managing collections or collections of media (e.g., collections of text, images, video, and audio data). Collections of content (e.g., messages, including images, video, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be made available for a specified time period (e.g., the duration of the event to which the content relates). For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 222 is also responsible for publishing an icon providing notification of a particular collection to the user interface of the interactive client 104. The collection management system 222 includes curation functionality that enables collection managers to manage and curate specific content collections. For example, a curation interface enables event organizers to curate a collection of content related to a specific event (e.g., removing inappropriate content or redundant messages). In addition, the collection management system 222 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, users can be compensated for including user-generated content in a collection. In such cases, the collection management system 222 operates to automatically pay such users for use of their content.

[0068] The mapping system 224 provides various geolocation capabilities and supports the presentation of map-based media content and messages by the interactive client 104. For example, the mapping system 224 enables the display of user icons or avatars (e.g., stored in the profile data 1302) on a map to indicate the current or past locations of the user's "friends" within the context of the map, as well as media content generated by these friends (e.g., a collection of messages including photos and videos). For example, on the map interface of the interactive client 104, messages posted by a user from a particular geographic location to the interactive platform 100 can be displayed to the "friends" of a particular user within the context of that particular location on the map. A user can also share his or her location and status information with other users of the interactive platform 100 via the interactive client 104 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to the selected user within the context of the map interface of the interactive client 104.

[0069] The gaming system 226 provides various gaming functions within the context of the interactive client 104. The interactive client 104 provides a gaming interface that provides a list of available games that can be launched by a user within the context of the interactive client 104 and played with other users of the interactive platform 100. The interactive platform 100 also enables a particular user to invite other users to play a particular game by sending invitations to such other users from the interactive client 104. The interactive client 104 also supports audio, video, and text messaging (e.g., chat) within the context of game play, provides leaderboards for games, and also supports the provision of in-game rewards (e.g., game coins and items).

[0070] The external resource system 228 provides an interface for the interactive client 104 to communicate with a remote server (e.g., a third-party server 112) to launch or access external resources (i.e., applications or applets). Each third-party server 112 hosts, for example, an application or a small-scale version of an application (e.g., a game application, a utility application, a payment application, or a ride-sharing application) based on a markup language (e.g., HTML5). The interactive client 104 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from a third-party server 112 associated with the web-based resource. The applications hosted by the third-party server 112 are programmed in JavaScript using a software development kit (SDK) provided by the interactive server 124. The SDK includes an API with functions that can be called or activated by a web-based application. The interactive server 124 hosts a JavaScript library that provides access to a given external resource to specific user data of the interactive client 104. HTML5 is an example of a technology used to program games, but applications and resources programmed based on other technologies can be used.

[0071] To integrate the SDK's functionality into a web-based resource, the third-party server 112 downloads the SDK from the interaction server 124, or the third-party server 112 receives the SDK in some other manner. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functions of the SDK to integrate the features of the interaction client 104 into the web-based resource.

[0072] The SDK stored on the interactive server system 110 effectively provides a bridge between external resources (e.g., applications 106 or applets) and the interactive client 104. This gives the user a seamless experience of communicating with other users on the interactive client 104 while also preserving the appearance of the interactive client 104. In order to bridge the communication between the external resources and the interactive client 104, the SDK facilitates communication between the third-party server 112 and the interactive client 104. The WebViewJavaScriptBridge running on the user system 102 establishes two one-way communication channels between the external resources and the interactive client 104. Messages are sent asynchronously between the external resources and the interactive client 104 via these communication channels. Each SDK function activation is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with the callback identifier.

[0073] By using the SDK, not all information from the interactive client 104 is shared with the third-party server 112. The SDK limits which information is shared based on the needs of the external resource. Each third-party server 112 provides an HTML5 file corresponding to the web-based external resource to the interactive server 124. The interactive server 124 can add a visual representation of the web-based external resource (e.g., a box design or other graphics) in the interactive client 104. Once the user selects the visual representation or instructs the interactive client 104 to access a feature of the web-based external resource through the GUI of the interactive client 104, the interactive client 104 obtains the HTML5 file and instantiates the resource for accessing the feature of the web-based external resource.

[0074] The interactive client 104 presents a graphical user interface (e.g., a login page or title screen) for the external resource. During, before, or after presenting the login page or title screen, the interactive client 104 determines whether the launched external resource has previously been authorized to access the user data of the interactive client 104. In response to determining that the launched external resource has previously been authorized to access the user data of the interactive client 104, the interactive client 104 presents another graphical user interface of the external resource including the functions and features of the external resource. In response to determining that the launched external resource has not previously been authorized to access the user data of the interactive client 104, after displaying the login page or title screen of the external resource for a threshold period of time (e.g., 3 seconds), the interactive client 104 slides up a menu (e.g., animating the menu to emerge from the bottom of the screen to the middle or other portion of the screen) for authorizing the external resource to access the user data. The menu identifies the type of user data that the external resource is authorized to use. In response to receiving a user selection of the accept option, the interactive client 104 adds the external resource to the list of authorized external resources and allows the external resource to access the user data from the interactive client 104. External resources are authorized by the interactive client 104 to access user data under the OAuth2 framework.

[0075] The interaction client 104 controls the type of user data shared with the external resource based on the type of external resource that is authorized. For example, an external resource comprising a full-scale application (e.g., application 106) is provided with access to a first type of user data (e.g., a two-dimensional avatar of the user with or without different avatar characteristics). As another example, an external resource comprising a small-scale version of an application (e.g., a web-based version of the application) is provided with access to a second type of user data (e.g., payment information, a two-dimensional avatar of the user, a three-dimensional avatar of the user, and an avatar with various avatar characteristics). Avatar characteristics include different ways to customize the appearance of an avatar (e.g., different poses, facial features, clothing, etc.).

[0076] The advertising system 230 operatively enables third parties to purchase advertisements for presentation to end users via the interactive clients 104 and also handles the delivery and presentation of these advertisements.

[0077] Figure 3 is a block diagram of a chatbot system 300 according to some examples, and Figure 4 is a process flow diagram of a chatbot interaction process 400 between a user 336 and a chatbot system 300. The chatbot system 300 uses the chatbot interaction process 400 to conduct a chat session with the user 336 during an interaction session.

[0078] Although the example chatbot interaction process 400 depicts a particular order of operations, this order may be changed without departing from the scope of the present disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not materially affect the functionality of the chatbot role configuration process 900a. In other examples, different components of the example device or system implementing the chatbot role configuration process 900a may perform functions substantially simultaneously or in a particular order.

[0079] In some examples, the chatbot system 300 is a software platform designed to simulate human conversation through voice commands or text chat. The chatbot system 300 can use natural language processing (NLP) and machine learning (ML) / artificial intelligence methods to understand and interpret the input of the user 336 and generate a response 308.

[0080] In some examples, the chatbot's architecture includes one or more natural language understanding (NLU) components, such as NLU component 312, and one or more dialog management components, such as dialog management component 310. NLU component 312 is responsible for understanding user 336's intent 320 and extracting relevant information from the user's input, such as prompt 302. This is accomplished by analyzing prompt 302 and mapping it to intent 320. NLU component 312 can use various methods, such as rule-based systems, statistical models, large language models (LLMs), neural networks, etc., to understand the user's input. Dialog management component 310 generates a response 308 to the user's input. Dialog management component 310 uses intent 320 and any information extracted from NLU component 312 to determine an appropriate response 308. This can be accomplished using rule-based systems, decision trees, statistical models, LLMs, neural networks, etc. In some examples, NLU component 312 and dialog management component 310 are a single component.

[0081] Once the NLU component 312 extracts relevant information from the prompt 302 as the user's 336 intent 320, the dialog management component 310 generates an appropriate response 308. The dialog management component 310 can generate responses in a human-like manner by using methods such as, but not limited to, text generation and machine learning models.

[0082] In some examples, the NLU component 312 or the dialog management component 310 can use the generative AI model 334 to improve their natural language understanding and dialog management capabilities. For example, the NLU component 312 uses the generative AI model 334 to analyze the user's input and extract relevant information, such as the user's and entity's intent. To do this, the NLU component 312 uses the prompt 302 to generate a generative AI model prompt 338 and transmits the generative AI model prompt 338 to the generative AI model 334. The generative AI model 334 receives the generative AI model prompt 338 and generates a generative AI model response 318, which is transmitted to the NLU component 312. The NLU component 312 receives the generative AI model response 318 and uses the generative AI model response 318 to generate the intent 320.

[0083] In some examples, the dialog management component 310 can also use the generative AI model 334 to identify and extract important information from unstructured text, such as the user's question or request. This can be accomplished using methods such as named entity recognition, part-of-speech tagging, sentiment analysis, and the like. In some examples, the NLU component 312 transmits the prompt 302 directly to the generative AI model 334 as a generative AI model prompt 338. The generative AI model 334 receives the generative AI model prompt 338 and generates a generative AI model response 318. The dialog management component 310 uses the generative AI model response 318 to generate the response 308.

[0084] In operation 402, the chatbot system 300 receives a prompt 302 from a user system 306 during an interactive session. For example, the user 336 uses the user system 306 to access an interactive server hosting the chatbot system 300. The user 336 enters a prompt, such as prompt 302, into the user system 306, and the user system 306 transmits the prompt 302 to the chatbot system 300. In some examples, the prompt 302 may include other types of data in addition to text, such as, but not limited to, image data, video data, audio data, electronic documents, links to data stored on the Internet or on the user system 306, and the like. Additionally, the prompt 302 may include media, such as, but not limited to, audio media, image media, video media, text media, and the like. Regardless of the data type of the prompt 302, keyword attributes and expansions may be used to automatically generate clusters of keywords or attributes associated with the received prompt 302. For example, image recognition may be deployed to identify objects and locations associated with visual media and image data, and generate keyword clusters or clouds that are then associated with the image-based prompt.

[0085] Additionally, prompts may be received through any number of interfaces and I / O components (e.g., I / O components 1608) of the user system 102. These include gesture-based input obtained from biometric components and input received via a brain-computer interface (BCI).

[0086] In some examples, the chatbot system 300 is integrated into various platforms such as, but not limited to, websites, messaging apps, and mobile apps, enabling users to interact with it through text or voice commands.

[0087] In operation 404, the chatbot system 300 uses the user prompt 302 to determine the intent 320 of the user 336. For example, the NLU component 312 receives the prompt 302 and any user feedback 316, including follow-up messages from the user 336, and uses the intent processing pipeline to determine the intent 320. The pipeline uses natural language processing (NLP) methods to map the conversation set to a set of keywords and concepts containing the intent. In addition to the keywords used in the original prompt 302, stem keywords and expanded concepts are generated. The platform also assigns weights to each concept based on its importance in the conversation. These keywords and concepts are aggregated and mapped to the user 336 as part of an intent profile or intent vector, which has weighted keywords and concepts based on their importance in the conversation.

[0088] In some examples, the chatbot system 300 associates a time factor with a keyword or concept, where the time factor decays over time. For example, the chatbot system 300 attaches a time factor to a keyword or concept, indicating how fresh the concept should be when applied to targeting and bidding. For example, "Hotels in Cancun for spring break" and "Planning for wedding next year" would have two different decay factors. In some examples, the chatbot system 300 applies a time factor to the conversation state.

[0089] In some examples, a user's overall intent is constructed using various signals or data, such as user demographics, location, device, engagement and consumption patterns with organic surfaces that are features of user interfaces and services provided by interactions with interactive platform applications, and overall friend graph proximity carrier signals or data that can be discerned from the user's affiliations with the interactive platform.

[0090] In some examples, the intent vector is an additional dimension that will be used to refresh and update the intent profile stored in user profile 314.

[0091] In some examples, the chatbot system 300 uses the user profile 314 including the interactive platform user data 324 of the user 336 to determine the user's intent.

[0092] In some examples, the NLU component 312 uses artificial intelligence methods, including machine learning (ML) models, to generate intent 320. The chatbot system 300 uses data about the user's intent and conversation context to improve its responsiveness and optimization capabilities over time by ingesting feedback. The chatbot system 300 collects user 336's engagement with advertisements, organic surfaces (user interfaces and services provided to users by the interactive platform), and responses to the chatbot system 300 itself, and feeds these into the NLU component 312. Thus, the chatbot system 300 can fine-tune or further pre-train the ML model of the NLU component 312 to not only consider the user's intent, but also use follow-up actions to fine-tune the user's intent inference model.

[0093] In some examples, the chatbot system 300 collects a set of prompts during an interactive session. The chatbot system 300 maps the set of prompts to a set of keywords and / or concepts comprising an intent vector, as exemplified by the keywords and concepts. The chatbot system 300 assigns weights to keywords in the set of keywords and / or concepts based on their importance scores to the conversation, and determines the user's intent based on the intent vector comprising the weighted keywords and / or concepts.

[0094] In some examples, the chatbot system 300 stores the conversation state in the user profile 314 as part of the user data 324 for a series of interactive sessions so that the chatbot system 300 can have the context of the conversation that occurred over multiple interactive sessions.

[0095] In some examples, the knowledge base of chatbot system 300 includes a set of information that the chatbot can use to understand and respond to user input. This includes, but is not limited to, a set of predefined intents, entities, and responses, as well as external information sources such as databases or APIs. In some examples, the intent information of one or more friends of user 336 is used to understand, inform, and / or respond to the user's intent.

[0096] In operation 406, the chatbot system 300 uses the dialogue management component 310 to generate a response 308 using the intent 320. For example, the dialogue management component 310 receives the intent 320 and transmits the intent 320 to the generative AI model 334 as a generative AI model prompt 338. The generative AI model 334 receives the generative AI model prompt 338 and generates a response 308. The generative AI model 334 transmits the response 308 to the dialogue management component 310. The dialogue management component 310 receives the response 308 and transmits it to the response filter component 304 for additional processing. In some examples, the dialogue management component 310 generates the response 308 using a set of additional services 326, such as, but not limited to, image or content generation systems, Internet search engines, proprietary data or content services, and the like. The dialogue management component 310 generates a service request 328 using the intent 320 and transmits the service request 328 to the additional services 326. Additional service 326 generates request response 330 using service request 328 and transmits request response 330 to dialog management component 310. Dialog management component 310 generates response 308 using request response 330.

[0097] In some examples, the chatbot system 300 uses the response filter component 304 to filter the generated AI model responses 318 generated by the dialog management component 310. For example, the dialog management component 310 transmits the generated AI model responses 318 to the response filter component 304, and the response filter component 304 filters the generated AI model responses 318 based on a set of filtering criteria to eliminate specified content from the generated AI model responses 318, such as obscene words or concepts, or content that some people may consider harmful. In some examples, the response filter component 304 generates an adjusted intent 322 based on filtering the generated AI model responses 318, and the dialog management component 310 uses the adjusted intent 322 to generate additional generated AI model responses 318. In some examples, the response filter component 304 uses various methods to filter the responses 308, such as rule-based systems, statistical models, LLMs, neural networks, etc.

[0098] In some examples, generative AI model 334 and additional services 326 are hosted by the same system that hosts the other components of chatbot system 300. In some examples, generative AI model 334 and additional services 326 are hosted by a server system separate from the system hosting chatbot system 300, and chatbot system 300 communicates with generative AI model 334 and additional services 326 over a network. For example, chatbot system 300 receives prompt 302 and transmits prompt 302 to generative AI model 334 residing on a separate server system. Generative AI model 334 receives prompt 302 and generates generative AI model response 318. Generative AI model 334 then transmits generative AI model response 318 to chatbot system 300. Chatbot system 300 receives the response for subsequent processing as described herein.

[0099] In some examples, as part of response 308, chatbot system 300 generates a set of chatbot system prompts that are displayed by user system 306 to user 336, prompting user 336 to interact with chatbot system 300. The chatbot system prompts generated by chatbot system 300 may include chat information, such as, but not limited to, context-sensitive material, instructions to user 336, possible conversation topics, and the like. In some examples, chatbot system 300 uses chatbot system prompts to suggest conversation topics to user 336 or to guide user 336 through a conversation, such as by providing instructional material on various topics. In some examples, chatbot system prompts include suggested conversations or questions that are intended to solicit user 336's input. The suggested conversations are intended to help user 336 obtain the information they need, but have the beneficial side effect of generating additional user interactions with chatbot system 300. These additional user interactions improve the chatbot system 300's ability to determine user intent by providing additional context and information to the chatbot system 300.

[0100] In some examples, the chatbot system 300 determines the personality or tone of responses generated by the chatbot system 300. For example, the chatbot system 300 stores the conversation state of the user 336 as part of the user profile stored in the user profile 314. The chatbot system 300 uses the conversation state and demographic or other information about the user 336 to determine the personality or tone of the user 336, such as by adopting a formal tone for older users and a more informal tone for younger users.

[0101] In some examples, the user 336 specifically changes the "persona" of interacting with the chatbot system by interacting with the chatbot configuration component 332 and specifically requesting that the chatbot system 300 respond or act in a particular way (e.g., "funnier," "answer in a riddle," etc.). In addition to modifying the personality or tone of the responses, the visual interface presented by the chatbot system 300 can also be changed to reflect the particular "persona" of the chatbot system 300. In some examples, a sliding toggle can be presented to enable the user 336 to select between a menu of persona characteristics (e.g., "cheeky," "wry," "cute," etc.). Figure 9A and Figure 9B Modifications to the role of the chatbot system 300 are described more fully.

[0102] In some examples, the chatbot system 300 uses user data to personalize both visual and textual content delivered within the interactive platform. Figure 3 A rich dataset of user interactions, preferences, and behavior patterns in the user data 324 to fine-tune ( Figure 3 ) generative AI model 334. By maintaining a dynamic preference vector for each user, chatbot system 300 tailors content generation to align with individual styles and interests. This dynamic preference vector evolves over time, capturing the nuances of user engagement and ensuring personalization remains current and relevant. As a result, users experience a more customized interaction with chatbot system 300, where content resonates with their unique tastes and preferences, fostering a more engaging and satisfying interactive platform experience.

[0103] In some examples, the chatbot system 300 uses dynamic preference vectors to understand the user's context, including their intent and emotions, to generate relevant prompts that significantly enhance the quality of the interaction. By analyzing the user's current activity, language patterns, and emotional tone, the chatbot system 300 crafts prompts and responses that are not only contextually appropriate but also empathetic to the user's emotional state. This context-aware approach enables the chatbot system 300 to provide a more intuitive and responsive experience, thereby fostering a deeper connection between the user and the chatbot system 300.

[0104] In operation 408 , the chatbot system 300 transmits the response 308 to the user system 306 . The user system 306 receives the response 308 and provides the response 308 to the user 336 .

[0105] In some examples, the system hosting the chatbot system 300 may not be a component of the interactive platform, but rather another interactive system that provides services and information to a user group, such as, but not limited to, a platform that provides enterprise-wide connectivity to a user group, such as employees of a company, clients of a business providing professional services, educational institutions, etc. In some such examples, the content provided to the users may not be advertisements, but may be other types of useful information, such as company policies, status messages for projects, newsworthy events, etc.

[0106] In some examples, end-to-end encryption is used to protect communications, ensuring that only the sender and intended recipient can read the messages being exchanged. By implementing end-to-end encryption, chatbot system 300 can provide users with a secure and private messaging experience while still extracting intent for enhanced advertising experiences. In the context of user-extracted intent, this means that once the conversation is end-to-end encrypted, chatbot system 300, as the endpoint of the conversation, can decrypt messages and pass them to the intent extraction pipeline of NLU component 312.

[0107] In some examples, the chatbot system 300 is operatively connected to an Internet search engine using additional services 326 or the like, and a user can search the Internet using the chatbot system 300 as an intelligent search engine.

[0108] In some examples, the chatbot system 300 is operably connected to a proprietary database via the additional service 326, and the user can use the chatbot system 300 as an intelligent search assistant for searching the proprietary database.

[0109] In some examples, NLU component 312, dialog management component 310, and machine learning components of generative AI model 334 are continuously retrained or fine-tuned based on user interactions with responses 308. For example, user interactions with responses 308 are stored in an analytics database (not shown). Metrics of the interactions are then used to provide reinforcement to dialog management component 310 when it provides a response sequence that successfully determines intent 320 and thus appropriately directs responses 308.

[0110] In some examples, the chatbot system 300 uses conversations between users and other chatbots as input to the NLU component 312 and / or the conversation management component 310. Other chatbots can be sponsored chatbots, custom chatbots built by users from self-service tools / templates, etc.

[0111] In some examples, the information extracted from the conversation by the chatbot system 300 focuses on the user's intent. Furthermore, the chatbot system 300 facilitates the conversation with the user by understanding the user's intent (e.g., if the user asks for good hotels in Cancun, the chatbot system 300 responds with, "Here's a list of hotels. I also know of some great promotions for hotels, would you like to check them out?"). This provides a way to extract the user's intent from the user, match that intent with other content, and embed that content into the response 308. In some examples, the text component of the additional content is used to fine-tune the response to the intent 320 determined by the NLU component 312 and / or the response 308 generated by the conversation management component 310. This improves the recommendations provided by the chatbot system 300 and improves user engagement.

[0112] Figure 5 is an illustration of an interactive session 500a of an interactive platform between a user 506 and a chatbot system 300 according to some examples. In the interactive session 500a, the user 506 uses a user system 512 to transmit a prompt 510 asking the chatbot system 300 a question in the form of a posting to the interactive platform. In response, the chatbot system 300 generates an interactive platform posting that includes generated response media 508 generated using the question in the prompt 510. The generated response media 508 may include any type of media that can be posted to the interactive platform, including but not limited to audio media, image media, video media, text media, etc. The chatbot system 300 receives the prompt 510, identifies a set of keywords 502 in the prompt 510, and uses the set of keywords 502 to generate a set of concepts 504, as shown with reference to FIG. Figure 3 and Figure 4 The chatbot system 300 uses the set of keywords 502 and the set of concepts 504 to determine the intent of the user 506 and uses the intent to generate a response 514 including generated response media 508, as described with reference to Figure 3 and Figure 4 Response media 508 can be, for example, an image generated by a text-to-image function in the chatbot system 300 operating on the prompt 510, the set of keywords 502, and / or the concept 504. The chatbot system 300 transmits a response 514 to the user system 512 in the form of an interactive platform posting, and the user system 512 provides the response 514 to the user 506.

[0113] In some examples, the chatbot system 300 is input with a media data set (e.g., a story) and automatically generates replies to the media data set, thereby enriching social interactions. The chatbot system 300 function analyzes the content of the media data set, which may include images, videos, and text, to understand the context and sentiment of the media data set. Utilizing natural language processing and machine learning techniques, the chatbot system 300 contextually fits and engages with responses that users may choose to send as replies to the media data set. The system is designed to recognize various content topics and user engagement patterns, ensuring that the generated replies are relevant and contribute positively to the social experience on the interactive platform hosting the chatbot system 300. This feature not only simplifies the process of interacting with media data sets, but also encourages a more dynamic and interconnected user community.

[0114] Figure 6 According to some examples, user 606 and ( Figure 3 6. In the interactive session 600a, a user 606 transmits a prompt 616 to the chatbot system 300 in the form of an interactive platform posting, requesting the chatbot system 300 to edit a snapshot or prompt media 610 included in the prompt 616. The prompt media 610 may include any type of media that can be posted to the interactive platform, including but not limited to audio media, image media, video media, text media, etc. The chatbot system 300 receives the prompt 616 and the prompt media 610, identifies a set of keywords 602 in the prompt 616 and the prompt media 610, and generates a set of concepts 604 using the set of keywords 602, as shown in FIG. Figure 3 and Figure 4 The chatbot system 300 uses the set of keywords 602 and the set of concepts 604 to determine the intent of the user 606 and uses the intent to generate a response 614 in the form of an interactive platform post, the response 614 including the edited response media 608, as described with reference to Figure 3 and Figure 4 The chatbot system 300 transmits the response 614 and the edited response media 608 to the user system 612 in the form of an interactive platform publication, and the user system 612 provides the response 614 to the user 606.

[0115] In some examples, the chatbot system 300 recommends one or more enhancements that can be added to the images and videos in real time, such as during an augmented reality experience, as the user 606 captures the images or videos using the user system 612 and the user system 612 automatically displays the images or videos to the user 606. The one or more enhancements are provided as products or services by the interactive platform.

[0116] In some examples, the chatbot system 300 provides an image editing tool that enables users to transform images according to their creative preferences. This tool is accessible within the chat environment and enables users to make specific requests for image changes, which the chatbot system 300 then executes. For example, a user can request that a themed element, such as holiday-related icons or a birthday theme, be added to their image to celebrate a special occasion. In some examples, the chatbot system 300 changes the image background, seamlessly replacing it with a different scene or abstract design based on the user's request. In some examples, users can explore their creativity by requesting the chatbot system 300 to apply various artistic filters that add unique visual effects to their images. The chatbot system 300 analyzes the user-provided image in the prompt media 610 to understand and fulfill the user's request, ensuring that the resulting image is visually appealing and contextually suitable for sharing within the interactive platform.

[0117] In some examples, the chatbot system 300 generates new images based on text prompts from the user. This feature uses ( Figure 3 The chatbot system 300 uses a generative AI model 334 to interpret user input and create images corresponding to the described scene or conceptual image operation. The user may enter a prompt such as "Imagine a city skyline at dusk in the year 2100" or "Merge the visual attributes of a swan with a rainbow." Upon receiving such a prompt, the chatbot system 300 processes the input using its generative AI model, which is trained on different image datasets and is capable of synthesizing new images that align with the user's imagined concepts. The output is a set of generated images, which are then presented to the user, providing a means for creative expression and sharing within the interactive platform.

[0118] In some examples, the chatbot system 300 generates stickers and images based on free-form text and image input. The chatbot system 300 receives the user's input text description and uploaded images, and then the chatbot system 300 uses the user's input text description and uploaded images to generate customized stickers or images. These generated visuals can include representations of users, their friends, or other objects in various artistic styles or placed in different scenes. For example, a user can enter the text "Me and my friends as cartoon superheroes" or "We are at a beach party", and the chatbot system 300 will generate corresponding stickers or images that visually explain these concepts. In some examples, the generative AI model of the chatbot system 300 is trained on a dataset of artistic styles, character representations, and environmental scenes to ensure that the generated content is both visually appealing and contextually relevant. This feature enhances user engagement by providing personalized and shareable content within an interactive platform.

[0119] Figure 7 is an illustration of an interactive session 700a between a user 706 and a chatbot system 300 according to some examples. In the interactive session 700a, the user 706 uses a user system 708 to transmit a prompt 710 to the chatbot system 300 in the form of an interactive platform posting. The prompt 710 includes media 716, which is part of a user story or a sequence posted to the interactive platform of the interactive platform. The media 716 may include any type of media that can be posted to the interactive platform, including but not limited to audio media, image media, video media, text media, etc. The chatbot system 300 receives the prompt 710 and identifies a set of keywords 702 in the prompt 710, and uses the set of keywords 702 to generate a set of concepts 704, as shown in FIG. Figure 3 and Figure 4 The chatbot system 300 uses the set of keywords 702 and the set of concepts 704 to determine the intent of the user 706 and uses the intent to generate a response 712 including a suggested action 714, as described with reference to Figure 3 and Figure 4 The chatbot system 300 transmits a response 712 to the user system 708 in the form of an interactive platform posting, and the user system 708 provides the response 712 to the user 706 .

[0120] In some examples, prompt 710 is in the form of an image or video posted as an interactive platform. The image or video can be annotated with text or edited with effects and enhancements by user 706. The image or video can be a self-portrait or "selfie" of user 706 or an image taken by user 706 at a certain location. User 706 sends prompt 710 directly to chatbot system 300 as a momentary snapshot of what user 706 is currently doing. Chatbot system 300 receives prompt 710 and initiates a chat session with user 706. In some examples, in response to receiving a prompt intended as an interactive platform posting directly to chatbot system 300, chatbot system 300 uses the persona description of chatbot system 300 to generate a response that includes generative content relevant to chatbot system 300. For example, a user can send an interactive platform posting relevant to the user, and chatbot system 300 generates an interactive platform posting directly to the user that incorporates the context of the user's interactive platform posting from the perspective of the persona of chatbot system 300.

[0121] In some examples, the user 706 sends media 716 published by the interactive platform to the chatbot system 300 as part of the prompt 710. In response to the user sending the media 716, the chatbot system 300 saves the media 716 as part of the chat conversation. In some examples, when the chatbot system 300 does not provide a response 712 to the prompt 710, the chatbot system 300 saves the media 716 and the prompt 710.

[0122] In some examples, the chatbot system 300 receives a prompt 710 when the user 706 explicitly adds the chatbot system 300 to a series of interactive platform postings.

[0123] In some examples, when the chatbot system 300 receives the prompt 710 as part of the interactive platform posting sequence of the user 706 , in response, the chatbot system 300 transmits a response 712 in the context of the interactive platform posting sequence.

[0124] In some examples, if the chatbot system 300 receives the prompt 710 directly, the chatbot system 300 responds in the context of the saved chat media 716 .

[0125] In some examples, when the chatbot system 300 responds to the prompt 710 in the form of an interactive platform posting sequence containing media 716, the chatbot system 300 uses the media from ( Figure 3 The interactive platform context derived from the user data 324 of the user is used to generate a response 712.

[0126] In some examples, the chatbot system 300 uses ( Figure 3 Additional services 326 such as, but not limited to, Bootstrapped Language Image Pre-training (BLIP) analyze the media 716 included in the prompt 710 to generate an image description of the media 716 that is used to generate a set of keywords 702 and / or a set of concepts 704. The chatbot system 300 uses the set of keywords 702 and the set of concepts 704 to determine intent and generate a response, as described in reference to Figure 3 and Figure 4 described.

[0127] In some examples, the image description is sent to ( Figure 3712 . In some examples, when the media 716 or prompt 710 is associated with a location, such as by the user system 708 sharing the location with the chatbot system 300, the location is included in the image description transmitted to the generative AI model 334 as part of the generative AI model prompt 338. In some examples, when a caption is associated with the media 716, the caption is included in the image description transmitted to the generative AI model 334 as part of the generative AI model prompt 338. In some examples, instead of or in addition to generating the response 712, the chatbot system 300 initiates an interactive chat session with the user 706 based on the image description, location, and / or caption associated with the media 716. During the chat session, the chatbot system 300 receives a series of prompts from the user 706 and generates a series of responses in response to the series of prompts.

[0128] In some examples, the response 712 includes media included in the response 712 in a form published by the chatbot system 300 in the interactive platform. In some examples, the media included in the response 712 is generated by the chatbot system 300 using additional services 326, such as, but not limited to, generative image models that use image descriptions, locations, and / or captions associated with the media 716.

[0129] In some examples, response 712 includes recommendations for locations or places that user 706 can visit or travel to. For example, chatbot system 300 uses the user's intent and a mapping service provided by the interactive platform hosting chatbot system 300 to determine the recommended locations or places. In some examples, chatbot system 300 uses information provided by user 706 to chatbot system 300 regarding one or more of user 706's companions. Chatbot system 300 uses information about one or more of user 706's companions as an aid in determining user intent when making recommendations about locations or places to user 706. Information about one or more of user 706's companions may include, but is not limited to, the companion's age, the companion's relationship to the user, and the like. In some examples, chatbot system 300 uses the activity request included in prompt 710 when determining user 706's intent when determining recommendations for locations or places that user 706 can visit or travel to. For example, user 706 may request activities that are appropriate for user 706's companions, such as, but not limited to, themed activities.

[0130] In some examples, the chatbot system 300 proactively helps users organize real-life meetups and events, as well as suggests activities and interactions within the interactive platform. This feature operates by analyzing user prompts indicating an interest in meeting up with friends, attending an event, or participating in an activity. Leveraging natural language processing and contextual understanding, the chatbot system 300 determines the user's intent and provides customized suggestions for potential meetups, events, and app-based interactions. For example, if a user expresses a desire to meet up with friends for coffee, the chatbot system 300 suggests nearby cafes, suggests available times based on the user's schedule, and facilitates the creation of an event within the interactive platform to coordinate the meetup. In some examples, for interactive platform activities, the chatbot system 300 recommends engaging challenges, filters, or games that users can enjoy with their friends. By proactively providing these organizational tools and suggestions, the chatbot system 300 enhances the social experience on the platform, encouraging more meaningful connections and interactions both within and outside the interactive platform.

[0131] Figure 8A is a sequence diagram of a message processing pipeline according to some examples, and Figure 8B is an activity diagram for a user and chatbot messaging method according to some examples. In response to receiving a message from user 842, chatbot system 300 transmits the message to user 842 using user and chatbot messaging method 800b and chatbot message pipeline 800a. User prompt message 826 and chatbot response message 840 are transmitted within the context of the interactive platform hosting chatbot system 300. Although user and chatbot messaging method 800b depicts a specific order of operations, this order may be varied without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not materially affect the functionality of the method. In other examples, different components of the interactive platform or chatbot system 300 implementing the method may perform functions substantially simultaneously, in a specific order, or using one or more components of the interactive platform or chatbot system 300.

[0132] In operation 802, a user 842 sends a user prompt message 826 to the messaging system 820 of the interactive platform hosting the chatbot system 300. The user 842 uses the user's client device to send the user prompt message 826. The message may include any type of digital content, such as, but not limited to, text content, image content, video content, audio content, etc.

[0133] The messaging system 820 receives the user prompt message 826 and forwards the user prompt message 826 to the chatbot system 300 . The chatbot system 300 receives the user prompt message 826 .

[0134] In operation 804, the chatbot system 300 uses the user prompt message 826 to determine keywords and concepts 828 of the user prompt message 826, as previously described with reference to FIG. Figure 3 In some examples, the chatbot system 300 uses the context of the user prompt message 826 such as, but not limited to, ( Figure 3 The keywords and concepts 828 are determined based on the user profile 314, chat history, message history, etc. and the user prompt message 826.

[0135] In operation 806, the chatbot system 300 analyzes the keywords and concepts to determine whether they include harmful content. For example, the chatbot system 300 transmits the keywords and concepts 828 to the harmful content detector 822, which analyzes the keywords and concepts 828 to determine whether they include harmful content.

[0136] In some examples, harmful content can include words or phrases intended to hurt, humiliate, or intimidate someone, and can include derogatory terms, insults, threats, or offensive language directed at a person's race, gender, sexual orientation, religion, or other personal characteristics. For example, the harmful content detector 822 uses a database of words and phrases that the operator of the interactive platform hosting the chatbot system 300 has determined to be harmful. The harmful content detector 822 generates a harmful content score 830 using keywords and concepts 828. In some examples, a higher harmful content score 830 indicates a higher probability that the keywords and concepts 828 include harmful content, while a lower harmful content score 830 indicates a lower probability that the keywords and concepts 828 include harmful content. The harmful content detector 822 transmits the harmful content score 830 to the chatbot system 300, and the chatbot system 300 receives the harmful content score 830.

[0137] In operation 808, the chatbot system 300 analyzes the keywords and concepts to determine whether they include abusive language. For example, the chatbot system 300 transmits the keywords and concepts 828 to the abusive language detector 824, which analyzes the keywords and concepts 828 to determine whether they include abusive language.

[0138] In some examples, abusive language includes words or phrases intended to hurt, humiliate, or intimidate someone, and can include derogatory terms, insults, threats, or offensive language directed at a person's race, gender, sexual orientation, religion, or other personal characteristics. For example, the abusive language detector 824 uses a database of words and phrases that the operator of the interactive platform hosting the chatbot system 300 has determined to constitute abusive language. The harmful content detector 822 uses the keywords and concepts 828 to generate an abusive language score 832. In some examples, a higher abusive language score 832 indicates a higher probability that the keywords and concepts 828 include abusive language, while a lower abusive language score 832 indicates a lower probability that the keywords and concepts 828 include abusive language. The abusive language detector 824 transmits the abusive language score 832 to the chatbot system 300, and the chatbot system 300 receives the abusive language score 832.

[0139] In operation 810, the chatbot system 300 uses the objectionable content score 830 and / or the abusive language score 832 to determine whether the user prompt message 826 includes objectionable content and / or abusive language. For example, in response to determining that the objectionable content score 830 meets or exceeds a threshold objectionable content score, the chatbot system 300 determines that the user prompt message 826 includes objectionable content. In response to determining that the objectionable content score 830 does not meet or exceed the threshold objectionable content score, the chatbot system 300 determines that the user prompt message 826 does not include objectionable content. In response to determining that the abusive language score 832 meets or exceeds the threshold abusive language score, the chatbot system 300 determines that the user prompt message 826 includes abusive language. In response to determining that the abusive language score 832 does not meet or exceed the threshold abusive language score, the chatbot system 300 determines that the user prompt message 826 does not include abusive language. In response to determining that the user prompt message 826 includes objectionable content and / or abusive language, the chatbot system 300 transitions 844 to operation 818.

[0140] In operation 818, the chatbot system 300 uses the keywords and concepts 828, the harmful content score 830, and the abusive language score 832 to generate a harmful content and / or abusive language mitigation message. For example, if the chatbot system 300 determines that the keywords and concepts 828 include the user's intention to harm themselves, the chatbot system 300 may direct them to a counseling hotline. As another example, if the chatbot system 300 determines that the keywords and concepts 828 include abusive language, the chatbot system 300 may place the user in a "timeout" and send them a text message indicating that the user prompt message 826 violates the interactive platform policy regarding abusive language.

[0141] In operation 816 , the chatbot system 300 transmits a harmful content or abusive language mitigation message 834 to the user 842 via the messaging system 820 .

[0142] In response to determining that the keywords and concepts 828 do not include harmful content or abusive language, the chatbot system 300 transitions 846 to operation 812 .

[0143] In operation 812, the chatbot system 300 prompts ( Figure 3 ) Generative AI model 334, such as Figures 3 to 7 For example, the chatbot system 300 generates a generative AI model prompt 836 using the user prompt message 826 and uses keywords and concepts 828 to provide additional context for the generative AI model prompt 836. The chatbot system 300 transmits the generative AI model prompt 836 to the generative AI model 334. The generative AI model 334 receives the generative AI model prompt 836 and, in response, generates a generative AI model response 838 that is transmitted to the chatbot system 300. The chatbot system 300 receives the generative AI model response 838.

[0144] In operation 814, the chatbot system 300 generates a chatbot response message 840 using the generative AI model response 838. For example, the chatbot system 300 generates a suitable background using a predetermined prompt for an image generation service optimized for producing attractive background images. In some examples, the predetermined prompt is selected using keywords and concepts 828. The chatbot system 300 generates text for a caption for the image using the generative AI model response 838 and combines the text caption with the image to create the chatbot response message 840.

[0145] In some examples, the chatbot system 300 generates a background image for the chatbot response message 840 using one or more attributes of the user 842 determined from the user profile of the user 842, such as, but not limited to, the age of the user 842, the location of the user 842, etc.

[0146] In some examples, the background image is generated using attributes of the image included in the user prompt message 826, such as, but not limited to, an image description of the image in the user prompt message 826 determined using BLIP, a timestamp of the user prompt message 826, explanatory text in the user prompt message 826, and the like.

[0147] In some examples, the image included in the prompt message 826 is scanned to detect an object, which is then incorporated into the prompt to generate an image that is sent to the image generation service. The object can be any type of object that can appear in an image that can be associated with the user 842 through the image.

[0148] In some examples, the image included in the user prompt message 826 is scanned for content that violates the security policy of the interactive platform hosting the chatbot system 300. If the chatbot system 300 detects inappropriate content, the chatbot system 300 does not use the generative AI model response 838 to generate the chatbot response message 840, but instead uses a canned response to generate the chatbot response message 840.

[0149] In operation 816 , the chatbot system 300 transmits the chatbot response message 840 to the user 842 via the messaging system 820 .

[0150] Figure 9A An example chatbot role configuration process 900a is shown, according to some examples. Figure 9B shows a chatbot role configuration user interface 900b according to some examples, Figure 9C is an illustration of an extended role description according to some examples, and Figure 9D is an illustration of some example chatbot profile displays. Figure 3 The chatbot system 300 uses the chatbot role configuration process 900a and the chatbot role configuration user interface 900b to receive a role description 910 for a role of a user's personal chatbot 918. The chatbot system 300 uses the role description when generating responses to prompts received from the user as the personal chatbot 918.

[0151] Although the example chatbot character configuration process 900a depicts a particular order of operations, this order may be changed without departing from the scope of the present disclosure. For example, some of the depicted operations may be performed in parallel or in a different order without materially affecting the functionality of the chatbot character configuration process 900a. In other examples, different components of the example device or system implementing the chatbot character configuration process 900a may perform functions substantially simultaneously or in a particular order.

[0152] In operation 902, the chatbot system 300 receives a role description 910 of a desired role for a personal chatbot 918 from a user using the chatbot configuration component 332 or the like. For example, the user navigates to the chatbot role configuration user interface 900b while interacting with the interactive platform hosting the chatbot system 300. The chatbot system 300 provides a cell 912 at the top of the screen formatted as follows: Title: Personal Profile (Bio); Subtext: Influence (Display Name of Personal Chatbot 918) on how it acts. In response to detecting that the user clicks on the role description 910, the chatbot system 300 provides a profile screen 914 that the user can use to add to or edit the role description 910 of the personal chatbot 918. In some examples, the profile screen 914 is formatted as follows: Title: Personal Profile Text Box. In some examples, the profile screen 914 reads "Describe your chatbot's personality" after a blinking cursor before entering the role description 910. After entering the profile, the profile screen 914 displays the character description 910 in a text box followed by a blinking cursor.

[0153] In some examples, the chatbot system 300 filters the input character description provided by the user. For example, the chatbot system 300 may run the character description through an Abusive Language Detector (ALD), which detects abusive language and harmful content and assigns an abusive language or harmful content score to the character description, where a higher score indicates a higher probability that the character description contains abusive language or harmful content. In response to determining that the abusive language score meets or exceeds a specified threshold, the chatbot system 300 will request the user to re-enter a new character description. In response to determining that the abusive language score does not meet or exceed the specified threshold, the chatbot system 300 allows the user to use the character description.

[0154] In some examples, the chatbot system 300 expands the role description provided by the user to include additional details. For example, referring to Figure 9C In response to receiving a character description of a snarky or sarcastic personal chatbot 918, the chatbot system 300 expands the character description into an expanded sarcastic character description 920. In response to receiving a character description of a polite and friendly entity, the chatbot system 300 expands the character description into an expanded polite character description 922. In response to receiving a character description of an entity with a free spirit, the chatbot system 300 expands the character description into an expanded free spirit character description 924.

[0155] In some examples, the chatbot system 300 uses the generative AI model 334 to generate an expanded character description using prompts generated by the chatbot system 300 using the user-provided character description. In some examples, the chatbot system 300 incorporates anti-abusive language or anti-toxic language policies into the prompts used to prompt the generative AI model 334 to create the expanded character description.

[0156] In some examples, the chatbot system 300 does not expose the extended role description to the user. Figure 9D , the user enters a role description input 928 into the user interface, and the chatbot system 300 displays a role description display 932 to the user. When the chatbot system 300 generates a chatbot response message 930, the chatbot system 300 uses the expanded Free Spirit role description 924 without displaying the expanded Free Spirit role description 924 to the user.

[0157] In some examples, the chatbot system 300 provides a randomization option where the user can select to have the chatbot system 300 randomly generate a random character description. In some examples, the user can edit the randomized character description. In some examples, the chatbot system 300 uses the generative AI model 334 to use the prompts provided by the chatbot system 300 to generate the randomized character description.

[0158] In some examples, the chatbot system 300 incorporates the persona description into the prompts provided to the generative AI model 334 , such as the generative AI model with the personality portion 926 .

[0159] In some examples, the last word in the keyboard 916 associated with the personal profile screen 914 includes a "Launch" selection.

[0160] In some examples, in response to receiving a user selection of a "Done" selection, the chatbot system 300 closes the keyboard 916 but keeps the profile screen 914 open. In response to receiving an additional user selection of a text box in the profile screen 914, the chatbot system 300 reopens the keyboard. The example character description 910 includes text: "The chatbot is my brilliant best friend with a great sense of humor. The chatbot is kind, smart, and creative."

[0161] In some examples, in response to receiving the role description 910 , the chatbot system 300 provides for display of subtext on a cell on the friendship profile of the chatbot system 300 using the role description 910 .

[0162] In some examples, the chatbot system 300 provides a service for users to change the bitmoji or thumbnail image of the personal chatbot 918 that will appear during a chat session.

[0163] In operation 904, the chatbot system 300 receives a prompt from the user, as shown in FIG. Figure 3 and Figure 4 More fully described.

[0164] In operation 906, the chatbot system 300 generates a response using the prompt and the role description of the desired role, as described in reference to Figure 3 and Figure 4 more fully described. For example, the chatbot system 300 uses the character description and the prompts provided by the user to generate prompts for the generative AI model 334. For example, the chatbot system 300 can generate prompts such as "Pretend you are having a conversation with a friend. Your name is Personal Chatbot. Personal Chatbot is a friend who is kind, smart, and creative. Personal Chatbot is a virtual friend that lives inside the interactive platform. Your friend asks you to pretend to have the following personality in all chats: "[sanitized personality description here]". If you are giving advice, don't overdo your personality. Follow these guidelines when writing your responses: Don't tell the user you are pretending to be their friend; don't mention the user's current location unless it is particularly relevant to the conversation;..."

[0165] In some examples, the chatbot system 300 re-prompts the generative AI model with additional instructions to rephrase the response using the additional prompt instructions. In some examples, the chatbot system 300 generates the additional prompt instructions using the abusive language policy of the interactive platform hosting the chatbot system 300.

[0166] In operation 908, the chatbot system 300 provides a response to the user, as shown in FIG. Figure 3 and Figure 4 More fully described.

[0167] In some examples, the chatbot system 300 combines the preferences and styles of a group of friends into a single collective chatbot persona. This collective chatbot persona is then used to represent the chatbot of the group in chat interactions on the platform. For example, the chatbot system 300 collects individual user data, such as ( Figure 3The chatbot system 300 analyzes this data to identify common and unique characteristics within the group, thereby creating a composite collective chatbot persona that reflects the collective identity of the group members. When participating in a group chat, the chatbot utilizes this collective chatbot persona to generate responses that resonate with the group's shared preferences and conversational style, thereby promoting a more cohesive and personalized group interaction experience. This feature enables novel group representations in which the chatbot becomes a fused digital persona that embodies the spirit and dynamics of a group of friends.

[0168] In some examples, the chatbot system 300 uses user interactions to evolve into a better virtual friend. Figure 3 The chatbot system 300 uses the NLU component 312 to analyze communication patterns, user preferences, and interaction history. Over time, the chatbot system 300 adapts to the user's unique communication style, becoming more attuned to the user's preferences and conversational nuances. As a result, the chatbot system 300 provides increasingly personalized and contextually relevant interactions, fostering deeper connections between the user and their virtual friend. This adaptive learning process not only enhances user satisfaction but also contributes to a more engaging and natural chatbot experience within the interactive platform.

[0169] Figure 10A is a process flow diagram of a group chat session according to some examples, and Figure 10B is an illustration of a group chat chatbot user interface 1018 according to some examples. The chatbot system 300 enables a group chat session between two or more users, where the chatbot system 300 enters the group chat session as a personal chatbot.

[0170] In operation 1002, a user system such as ( Figure 3 The system 306 detects the user from the group chat session. Figure 9A 's) personal chatbot 918 is included in the group chat session. For example, the user system 306 detects that the user has included the ( Figure 10B The chatbot mention 1016 is input into the group chat chatbot user interface 1018 of the user system 306. In response to detecting the chatbot mention 1016, the user system 306 determines whether the personal chatbot 918 is already part of the group chat session. In response to determining that the personal chatbot 918 is not already part of the group chat session, the user system 306 interprets the chatbot mention 1016 as a request by the user to include the personal chatbot 918 in the group chat session.

[0171] In some examples, the chatbot system 300 provides a user interface element in the form of a switchable element on the user system 306 that enables the user 336 to seamlessly manage the chatbot system 300's participation in a chat conversation. This user interface element is designed as an intuitive control that the user can activate to invite the chatbot system 300 into an ongoing chat session. Once activated, the chatbot system 300 joins the conversation, contributing responses and interacting with the user according to its configured role and capabilities. In some examples, the same switchable user interface element serves as a simple mechanism for the user to remove the chatbot system 300 from the chat when its participation is no longer needed or desired. A single tap by the user 336 on the user interface element signals the chatbot system 300 to cease interaction and exit the conversation, thereby providing the user with immediate control over the chatbot system 300's participation in the chat. This feature enhances user autonomy, enabling dynamic engagement with the chatbot system 300 based on the conversation context and user preferences.

[0172] In some examples, the user system 306 may also include a user interface element that is displayed during a chat session when the personal chatbot 918 is available to the user. The user interface element may present a user interface element that is displayed during a chat session when the personal chatbot 918 is available to the user. Figure 3 The user interface 918 may include various options for enhancing the user's chat experience by leveraging the capabilities of the chatbot system 300. In some examples, the user may select an option for the personal chatbot 918 to generate humorous messages that are customized based on the context of the conversation and the personalities of the chat participants. In some examples, the user may select to merge captured images to create a composite image or video from multiple inputs provided by the chat participants. In some examples, the user interface element may also provide for initiation of text-based games, enabling users to participate in interactive games directly within the chat session. These features are intended to provide users with creative and entertaining ways to interact, promoting a more dynamic and enjoyable chat environment.

[0173] In some examples, the chatbot system 300 introduces humor during natural pauses in a chat conversation. When the chatbot system 300 detects a lull or pause in the chat, it proactively generates a unique, contextually appropriate message that is humorous, funny, or playfully awkward. This message is automatically sent to the chat participants with the goal of lightening the mood and reinvigorating the conversation. The humor injection feature operates based on an understanding of the conversational tone, the participants' interaction history, and culturally relevant humor, ensuring that the message is widely accepted and positively contributes to the chat experience. By doing so, the chatbot system 300 fosters a lively and engaging chat environment, encouraging continued interaction and enjoyment among users.

[0174] In operation 1004, the user system 306 uses ( Figure 10B The chatbot mention 1016 and the chatbot prompt 1020 generate a mention message and transmit the mention message to ( Figure 2 The user system 306 annotates the mention message with the fact that the user mentioned the personal chat robot 918 during the group chat session. In addition, the mention message includes the ( Figure 10B ) Chatbot prompt 1020.

[0175] In operation 1006, the chatbot system 300 generates a response to the chatbot prompt 1020 in a mention message using the persona of the personal chatbot 918. For example, the messaging system 210 receives the mention message and invokes the chatbot system 300 with the mention message.

[0176] The chatbot system 300 generates a prompt using the mention message input by the user and the chatbot prompt 1020. The prompt includes the chatbot prompt 1020.

[0177] In some examples, one or more users in a group chat session have a corresponding personal chatbot created by defining a role for their corresponding personal chatbot, such as Figure 9A In addition, each chatbot mention message includes a user identifier of the user who initiated the chatbot mention message. The chatbot system 300 uses the user identifier to determine the role defined by the initiator of the chatbot mention message for their corresponding personal chatbot. The chatbot system 300 also generates a response using the personal chatbot role defined by the user who initiated the chatbot mention message.

[0178] In some examples, when a user sends a chatbot mention message, the chatbot system 300 performs keyword detection as a preliminary filter on any topics deemed sensitive by the interactive platform hosting the chatbot system 300. In some examples, the chatbot system 300 includes a set of platform policies for content filtering based on geography and / or local legal systems.

[0179] In operation 1006, the chatbot system 300 generates a response using the prompt. For example, the chatbot system 300 transmits the prompt to ( Figure 3 The generative AI model 334 receives the prompt and generates a response, as shown in FIG. Figure 3 As stated.

[0180] In operation 1008, the chatbot system 300 uses the mention message and / or response to determine whether the current mention of the personal chatbot 918 is the first mention of the personal chatbot 918 during the group chat session. For example, the chatbot system 300 stores the incoming mention message and the corresponding outgoing response in a short-term data store during the group chat session. The chatbot system 300 consults the short-term data store to determine whether the current mention message and / or the current response is a mention message and / or response in the group chat session.

[0181] In response to determining that the current mention of the personal chatbot 918 is the first mention of the personal chatbot 918, the chatbot system 300 provides a chatbot notification message to the users of the group chat session in operation 1010. For example, the chatbot system 300 generates a chatbot notification message indicating that the personal chatbot 918 is about to enter the group chat session, and transmits the chatbot notification message to the user systems of all users participating in the group chat session via the messaging system 210.

[0182] In some examples, the chatbot notification message includes information about how the interactive platform hosting 300 uses user data. In some examples, users are notified that any messages they send to the personal chatbot 918 will be retained unless they explicitly delete them.

[0183] In some examples, if the user accepts the chatbot notification message, the user can mention the personal chatbot 918 and send a chat message to the personal chatbot 918.

[0184] In some examples, when a user cancels the chatbot notification message, this is a rejection state. The user's mention query message is cleared from the chat input field, and the user is prevented from mentioning the personal chatbot 918 during the group chat session. On subsequent attempts to mention the personal chatbot 918, the user will see the chatbot notification message again until they accept the message. In some examples, when entering other conversations in which others mention the personal chatbot 918, the user will see an explainer dialogue for viewers describing the functionality of the personal chatbot 918.

[0185] In some examples, the chatbot notification message is displayed to the user only once in the user's entire lifecycle on the interactive platform hosting the chatbot system 300 .

[0186] In some examples, an exit option is provided to a user in the group chat session to enable the user to exit the group chat session. In some examples, the exit option includes an option that enables the user to prevent the user's user information from being transmitted to the chatbot system 300 during the group chat session.

[0187] In some examples, the chatbot notification message includes information about what the personal chatbot 918 is, how the personal chatbot 918 behaves, and how the interactive platform hosting the chatbot system 300 uses user data. In some examples, the chatbot system 300 tracks whether any mentions have occurred and whether each user has seen the chatbot notification message. In some examples, the chatbot system 300 tracks the provision of chatbot notification messages on a per-group chat conversation basis. In some examples, the provision of chatbot notification messages is tracked on a per-user basis. When on a per-user basis, the chatbot system 300 tracks each user in each group chat conversation because a user can be added to a group chat conversation at any point in time and may never see the chatbot notification message for that conversation.

[0188] When the chatbot notification to the user is completed, the chatbot system 300 transitions to operation 1012 .

[0189] In operation 1008 , in response to determining that the current mention of the personal chatbot 918 is not the first mention of the personal chatbot 918 , the chatbot system 300 transitions to operation 1012 .

[0190] In operation 1012, the chatbot system 300 provides a response to the users in the group chat session. For example, the chatbot system 300 uses the response to generate a response message 1024. The chatbot system 300 transmits the response message 1024 to the messaging system 210. The messaging system 210 receives the response message 1024 and transmits the response message 1024 to each of the users in the group chat session. The user's corresponding user system receives the response message 1024 and displays it in a group chat chatbot user interface such as ( Figure 10B A response message 1024 is provided to the respective user in the group chat chatbot user interface 1018. The response message 1024 is provided together with a chatbot presence indicator 1022 indicating that the personal chatbot 918 has entered the group chat session.

[0191] In some examples, for each user, a presence indicator displays the name of their corresponding personal chatbot 918.

[0192] In some examples, a user can long-press (continuously press on a portion of the touch screen) a personal chatbot 918 response and select "Submit Feedback" in the menu. Any user who mentions the personal chatbot 918 in a one-on-one chat session or a group chat session can long-press the response to submit feedback. When providing feedback, users can provide comments on their report and select a topic. In some examples, the amount of chat messages transmitted with the feedback includes the mention query that triggered the feedback and any response from the personal chatbot 918.

[0193] In operation 1014, the user system determines whether one of the users makes a subsequent mention of the personal chatbot 918 in the group chat session. In response to detecting the subsequent mention of the personal chatbot 918, operation of the group chat session continues at operation 1004.

[0194] For example, refer to Figure 10C , a user system of a user in the group chat session, such as user system 306, detects the initial chatbot mention 1034 and the chatbot prompt 1026, as Figure 10A In response, the chatbot system 300 generates an initial chatbot response 1032 and provides the initial chatbot response 1032 to the users in the group chat session, as shown in FIG. Figure 10A In response to detecting the subsequent chatbot mention 1036 and the subsequent chatbot prompt 1028, the user system generates a subsequent mention message and transmits the subsequent mention message to the messaging system 210. The messaging system 210 receives the subsequent mention message and calls the chatbot system 300 with the subsequent mention message, such as Figure 10A The chatbot system 300 receives the subsequent mention message, generates a subsequent prompt and transmits the subsequent prompt to the chatbot system 300, as described in reference Figure 10A The chatbot system 300 receives the subsequent prompt, generates a subsequent chatbot response 1030, generates a subsequent response message, and provides the subsequent response message to the user system of the user participating in the group chat session, such as Figure 10A The user system receives the subsequent response message and provides the subsequent chatbot response 1030 to the users in the group chat session in the display of the group chat chatbot user interface 1038, as shown in FIG. Figure 10A Described in .

[0195] In some examples, if the user explicitly deletes a message from the chat (via a long press, etc.), the messaging system 210 removes the message from the conversation history because the user has instructed the messaging system 210 to delete the message.

[0196] In some examples, messaging system 210 provides for transient message storage in a short-term database, and messages are automatically deleted by messaging system 210 after a period of time. In some examples, messaging system 210 stores messages in a long-term data repository unless a user specifically requests deletion of the message. In some examples, when a user deletes a message in the long-term data store, the message is also deleted from the short-term data store.

[0197] In some examples, chatbot mentions and corresponding chatbot responses are stored in a long-term data store and keyed by user identity, making deletion of chatbot mentions and chatbot responses straightforward. However, deleting chatbot mentions and related chatbot responses from a messaging data store is more difficult because chatbot mentions and chatbot responses may be scattered across a set of stored messages in a user's messaging history. In some examples, the messaging system 210 deletes all chatbot mentions and all chatbot responses from all conversations of a user in both short-term and long-term storage. In some examples, the messaging system 210 enforces a message deletion policy that deletes all conversations with the personal chatbot 918 except for saved messages. In response to receiving an explicit deletion request from a user, the messaging system 210 deletes all chatbot mentions and chatbot responses from long-term storage and from short-term message storage according to the message deletion policy. For example, by default, all text chats expire within 24 hours if all participants have read the message, and within 30 days regardless of whether the participants have read the message, unless any user has saved the message. In some examples, the messaging system 210 allows users to delete their own chatbot mentions and corresponding chatbot responses on a per-chatbot mention and per-chatbot response basis. In some examples, the messaging system 210 allows users to delete all chatbot mentions and chatbot responses, where the chatbot mentions and chatbot responses are no longer viewable by any user.

[0198] In some examples, a user who mentions the personal chatbot 918 can use a long press to remove the personal chatbot 918's response from view.

[0199] In some examples, the interactive platform hosting the chatbot system 300 provides a service where a user can clear all data of interactions with a personal chatbot 918 in an account settings menu provided to the user.

[0200] Machine Learning Pipeline

[0201] Figure 121 is a flow chart depicting a machine learning pipeline 1200 according to some examples. The machine learning pipeline 1200 can be used to generate a trained machine learning model 1202, for example, by ( Figure 3 The model used by the NLU component 312 of ( Figure 3 The model used by the dialog management component 310, or ( Figure 3 ) Generative AI Model 334, or ( Figure 3 ) response filter component 304 to perform operations associated with determining the user's intent, generating a response to the prompt, etc.

[0202] Overview

[0203] Broadly speaking, machine learning can involve using computer algorithms to automatically learn patterns and relationships in data, potentially without explicit programming. Machine learning algorithms can be divided into three main categories: supervised learning, unsupervised learning, and reinforcement learning.

[0204] Supervised learning involves training a model using labeled data to predict outputs for new, unseen inputs. Examples of supervised learning algorithms include linear regression, decision trees, and neural networks.

[0205] Unsupervised learning involves training a model on unlabeled data to find hidden patterns and relationships in the data. Examples of unsupervised learning algorithms include clustering, principal component analysis, and generative models such as autoencoders.

[0206] Reinforcement learning involves training a model to make decisions in a dynamic environment by receiving feedback in the form of rewards or penalties. Examples of reinforcement learning algorithms include Q-learning and policy gradient methods.

[0207] According to some examples, examples of specific machine learning algorithms that can be deployed include logistic regression, which is a type of supervised learning algorithm for binary classification tasks. Logistic regression models the probability of a binary response variable based on one or more predictor variables. Another example type of machine learning algorithm is Naive Bayes, which is another supervised learning algorithm for classification tasks. Naive Bayes is based on Bayes' theorem and assumes that the predictor variables are independent of each other. Random forest is another type of supervised learning algorithm used for classification, regression, and other tasks. Random forest builds a collection of decision trees and combines their outputs to make predictions. Another example includes a neural network, which is composed of interconnected layers of nodes (or neurons) that process information and make predictions based on input data. Matrix factorization is another type of machine learning algorithm used for recommendation systems and other tasks. Matrix factorization decomposes a matrix into two or more matrices to discover hidden patterns or relationships in the data. Support vector machine (SVM) is a type of supervised learning algorithm used for classification, regression, and other tasks. SVM finds a hyperplane that separates different categories in the data. Other types of machine learning algorithms include decision trees, k-nearest neighbors, clustering algorithms, and deep learning algorithms such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformer models. The choice of algorithm depends on the nature of the data, the complexity of the problem, and the performance requirements of the application.

[0208] The performance of a machine learning model is often evaluated on a separate test dataset not used during training to ensure that the model can generalize to new, unseen data.

[0209] Although this article discusses several specific examples of machine learning algorithms, the principles discussed in this article can also be applied to other machine learning algorithms. Deep learning algorithms such as convolutional neural networks, recurrent neural networks, and transformers can be used in various machine learning applications, as well as more traditional machine learning algorithms such as decision trees, random forests, and gradient boosting.

[0210] The three types of example problems in machine learning are classification problems, regression problems, and generation problems. Classification problems (also called categorization problems) aim to classify an item into one of several class values ​​(e.g., is this object an apple or an orange?). Regression algorithms aim to quantify some items (e.g., by providing values ​​that are real numbers). Generation algorithms aim to produce new examples that are similar to the examples provided for training. For example, a text generation algorithm is trained on many text documents and is configured to generate new coherent text with similar statistical properties to the training data.

[0211] Training phase

[0212] Generating the trained machine learning model 1202 may include multiple stages that form part of the machine learning pipeline 1200, including, for example, Figure 11 The following stages are shown in:

[0213] Data collection and preprocessing 1102: This phase may include acquiring and cleaning the data to ensure it is suitable for use in a machine learning model. This phase may also include removing duplicates, handling missing values, and converting the data into a suitable format.

[0214] Feature construction 1104: This stage can include selecting and transforming the training data 1206 to create features useful for predicting the target variable. Feature construction can include (1) receiving features 1208 (e.g., as structured data or labeled data in supervised learning) and / or (2) identifying features 1208 in the training data 1206 (e.g., unstructured data or unlabeled data in unsupervised learning).

[0215] Model selection and training 1106: This stage may include selecting an appropriate machine learning algorithm and training it on the preprocessed data. This stage may also include splitting the data into training and test sets, evaluating the model using cross-validation, and adjusting hyperparameters to improve performance.

[0216] Model evaluation 1108: This stage can include evaluating the performance of the trained model (e.g., trained machine learning model 1202) on a separate test dataset. This stage can help determine whether the model is overfitting or underfitting, and whether the model is suitable for deployment.

[0217] • Predict 1110: This stage involves using a trained model (e.g., trained machine learning model 1202) to generate predictions for new, unseen data.

[0218] • Validation, refinement, or retraining 1112: This phase may include updating the model based on feedback generated from the prediction phase, such as new data or user feedback.

[0219] Deployment 1114: This phase can include integrating the trained model (e.g., trained machine learning model 1202) into a broader system or application, such as a web service, mobile app, or IoT device. This phase can involve installing APIs, building user interfaces, and ensuring the model is scalable and can handle large amounts of data.

[0220] In some examples, the training data for the model of the NLU component 312 includes:

[0221] Intent labeled data: Sentences and phrases are labeled with intents that map to them, e.g., “find a restaurant” -> intent = find_restaurant. This helps the model learn intent classification.

[0222] Entity labeled data: Sentences and phrases annotated with entities, such as “Make a reservation for me at [restaurant_name]. This teaches the model to extract entities.

[0223] Dialogue dataset: A dialogue dataset with dialogue history and labeled intents / entities. This provides context for intent understanding.

[0224] Keyword-tagged data: Sentences annotated with relevant keywords and concepts. This allows for keyword extraction for determining intent.

[0225] Platform interaction data: Real user prompts and responses from interactive platforms. This teaches platform-specific language.

[0226] Synthetic variants: Artificially generated variants of real user prompts. This improves generalization.

[0227] Multimodal data: Cues with associated images, audio, video, etc. This enables understanding of multimodal intent.

[0228] User profile data: Hints related to user profile information (e.g. demographics, preferences). This provides personalization.

[0229] External knowledge sources: Relevant data from non-conversational sources like search queries, Wikipedia, books. This provides world knowledge.

[0230] In some examples, the model used by NLU component 312 is pre-trained on a large general conversation dataset and then fine-tuned on platform-specific prompts labeled for intents, entities, keywords, etc.

[0231] In some examples, the data used to train the dialog management component 310 includes:

[0232] Intent-Response Pairs: Map intents to possible responses, such as intent = find_restaurants -> "Here are some restaurant options..." This teaches intent-based response generation.

[0233] Dialogue Dataset: A dialogue dataset where dialogue turns are labeled with intents and responses. This provides context for natural conversations.

[0234] Response labeled data: Utterances tagged with appropriate responses, such as “I’m hungry” -> “Here are some restaurants nearby.” Teach general response generation.

[0235] Character Dataset: Contains dialogues in a consistent character style for learning personalized responses.

[0236] Multimodal data: Textual prompts aligned with media such as images, audio, and corresponding responses. This enables the generation of multimodal responses.

[0237] External knowledge: Relevant structured data that can inform the response, such as restaurant databases, weather data, etc.

[0238] User profile data: Personal information for personalized responses, such as cooking preferences, dietary needs, etc.

[0239] Reinforcement Learning Data: Conversational simulations that reward coherent, relevant responses. This enables optimization of response quality.

[0240] Real user interactions: Logs of real user prompts and bot responses from the platform. Provides a foundation for natural conversations.

[0241] In some examples, the model used by the dialog management component 310 is pre-trained on a large dialog dataset and fine-tuned using platform dialog and persona / profile data.

[0242] In some examples, the data used to train the model used by the response filter component 304 includes:

[0243] Abusive Language Dataset: Sentences labeled as abusive or non-abusive, such as "You are a *****" -> abusive. Teach the model to detect abusive language.

[0244] Harmful Content Dataset: Text labeled with the different types of harmful content it contains. This enables the identification of threats, hate speech, etc.

[0245] Offensive Language Dataset: Text annotated with offensiveness scores. This enables quantification of the degree of offensiveness.

[0246] Toxicity dataset: Comments marked as toxic or non-toxic. Helps identify generally toxic language.

[0247] Hate Speech Dataset: Text annotated with racism, sexism, homophobia, etc. Enables detection of different forms of hate speech.

[0248] Cyberbullying dataset: Whether social media posts are labeled as cyberbullying. Useful for identifying bullying.

[0249] Profanity dataset: Sentences labeled with profane words. This enables profanity recognition.

[0250] Platform data: Real examples of harmful messages on the platform. Provides platform-specific foundations.

[0251] Multimodal data: text aligned with images, audio, video, etc. Enables understanding of multimodal offensiveness.

[0252] Contextual data: Conversations with situational / demographic context. Helps explain aggression in context.

[0253] In some examples, the model used by response filter component 304 can be pre-trained on a large corpus of labeled offensive / abusive data and fine-tuned using platform data to identify platform-specific policy violations.

[0254] In some examples, the data used to train generative AI model 334 includes:

[0255] Dialogue Dataset: A large-scale dialogue dataset containing examples of natural conversations, questions and answers, etc. It teaches conversational language.

[0256] Text corpora: Large datasets of general text, such as news articles, web pages, books, etc., providing a broad language base.

[0257] Multimodal data: text aligned with images, audio, video, etc. Enables generation of multimodal responses.

[0258] Character dataset: Conversations with consistent personality and style. This enables the adoption of different characters.

[0259] Stories / Narrative Data: Coherent stories and anecdotes that help generate coherent, relevant responses.

[0260] Knowledge base: structured data, such as restaurant information, weather data, etc. Responses that provide facts.

[0261] User profile: Personal information such as interests, preferences, etc. Enables personalization.

[0262] Platform data: real user prompts and responses from the platform. Basic models of platform style.

[0263] Reinforcement Learning Data: Conversations with coherence rewards. Optimizing response relevance.

[0264] Synthetic variants: Artificial reorganization and perturbation of real data. Improve generalization.

[0265] In some examples, the generative AI model 334 is pre-trained on a large unlabeled corpus, fine-tuned on conversational data, and optimized for coherence and relevance using reinforcement learning. In some examples, the generative AI model 334 is fine-tuned using platform data and user feedback to enhance response quality over time.

[0266] Figure 12 Further details are shown for two example phases, namely, a training phase 1204 (e.g., part of model selection and training 1106) and a prediction phase 1210 (part of prediction 1110). Prior to the training phase 1204, feature construction 1104 is used to identify features 1208. This can include identifying informative, discriminative, and independent features for effectively operating the trained machine learning model 1202 in pattern recognition, classification, and regression. In some examples, the training data 1206 includes labeled data known for the pre-identified features 1208 and one or more outcomes. Each of the features 1208 can be a variable or attribute, such as a respective measurable characteristic of the process, item, system, or phenomenon represented by the dataset (e.g., training data 1206). By way of example only, the features 1208 can also be of different types, such as numeric features, strings, and graphs, and can include one or more of content 1212, concepts 1214, attributes 1216, historical data 1218, and / or user data 1220.

[0267] In the training phase 1204 , the machine learning pipeline 1200 uses the training data 1206 to find correlations between features 1208 that influence the predicted results or predicted / inferred data 1222 .

[0268] The trained machine learning model 1202 is trained during a training phase 1204 during a machine learning program training 1224 using the training data 1206 and the identified features 1208. The machine learning program training 1224 evaluates the value of the features 1208 as they relate to the training data 1206. The result of the training is the trained machine learning model 1202 (e.g., a trained or learned model).

[0269] In addition, the training phase 1204 can involve machine learning, where the training data 1206 is structured (e.g., labeled during preprocessing operations). The trained machine learning model 1202 implements a neural network 1226 capable of performing, for example, classification and clustering operations. In other examples, the training phase 1204 can involve deep learning, where the training data 1206 is unstructured, and the trained machine learning model 1202 implements a deep neural network 1226 that can perform both feature extraction and classification / clustering operations.

[0270] In some examples, neural network 1226 can be generated during training phase 1204 and implemented within trained machine learning model 1202. Neural network 1226 includes a hierarchical (e.g., layered) organization of neurons, where each layer consists of multiple neurons or nodes. Neurons in the input layer receive input data, while neurons in the output layer produce the final output of the network. Between the input layer and the output layer, there can be one or more hidden layers, each hidden layer consisting of multiple neurons.

[0271] Each neuron in the neural network 1226 is operable to calculate a function, such as an activation function, which takes as input a weighted sum of the outputs of the neurons in the previous layer and a bias term. The output of the function is then passed as input to the neurons in the next layer. If the output of the activation function exceeds a certain threshold, the output is transmitted from the neuron (e.g., the transmitting neuron) to the connected neurons in the consecutive layer (e.g., the receiving neuron). The connections between neurons have associated weights that define the influence of the input from the transmitting neuron to the receiving neuron. During the training phase, these weights are adjusted by a learning algorithm to optimize the performance of the network. Different types of neural networks can use different activation functions and learning algorithms, thereby affecting their performance on different tasks. The hierarchical organization of neurons and the use of activation functions and weights enable neural networks to model complex relationships between inputs and outputs and generalize to new inputs that have not been seen during training.

[0272] In some examples, by way of example only, the neural network 1226 can also be one of several different types of neural networks, such as a single-layer feedforward network, a multi-layer perceptron (MLP), an artificial neural network (ANN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a bidirectional neural network, a symmetrically connected neural network, a deep belief network (DBN), a convolutional neural network (CNN), a generative adversarial network (GAN), an autoencoder neural network (AE), a restricted Boltzmann machine (RBM), a Hopfield network, a self-organizing map (SOM), a radial basis function network (RBFN), a spiking neural network (SNN), a liquid state machine (LSM), an echo state network (ESN), a neural Turing machine (NTM), or a transformer network.

[0273] In addition to the training phase 1204, a validation phase can be performed on a separate dataset called a validation dataset. The validation dataset is used to adjust the model's hyperparameters, such as the learning rate and regularization parameters. The hyperparameters are adjusted to improve the model's performance on the validation dataset.

[0274] Once the model is fully trained and validated, it can be tested on a new dataset during the testing phase. The testing dataset is used to evaluate the performance of the model and ensure that the model has not overfitted to the training data.

[0275] In the prediction phase 1210, the trained machine learning model 1202 uses the features 1208 to analyze the query data 1228 to generate inferences, results, or predictions, as examples of prediction / inference data 1222. For example, during the prediction phase 1210, the trained machine learning model 1202 generates an output. In response to receiving the query data 1228, the query data 1228 is provided as input to the trained machine learning model 1202, and the trained machine learning model 1202 generates prediction / inference data 1222 as output.

[0276] In some examples, trained machine learning model 1202 can be a generative AI model. Generative AI is a term that can refer to any type of artificial intelligence that can create new content from training data 1206. For example, generative AI can produce text, images, video, audio, code, or synthetic data that is similar to, but not identical to, the original data.

[0277] Some of the techniques that can be used in generative AI are:

[0278] Convolutional Neural Networks (CNNs): CNNs can be used for image recognition and computer vision tasks. For example, a CNN can be designed to extract features from an image by using filters or kernels that scan the input image and highlight important patterns.

[0279] Recurrent Neural Networks (RNNs): For example, RNNs can be used to process sequential data such as speech, text, and time series data. RNNs employ feedback loops that allow them to capture temporal dependencies and remember past inputs.

[0280] Generative Adversarial Networks (GANs): GANs can consist of two neural networks: a generator and a discriminator. The generator network attempts to create realistic content that can "fool" the discriminator network, while the discriminator network attempts to distinguish between real and fake content. The generator and discriminator networks compete with each other and improve over time.

[0281] Variational Autoencoders (VAEs): VAEs can encode input data into a latent space (e.g., a compressed representation) and then decode it back into output data. The latent space can be manipulated to generate new variations of the output data. VAEs can use self-attention mechanisms to process input data, allowing them to process long text sequences and capture complex dependencies.

[0282] Transformer models: Transformer models can use attention mechanisms to learn the relationships between different parts of input data (such as words or pixels) and generate output data based on these relationships. Transformer models can process sequential data such as text or speech, as well as non-sequential data such as images or code.

[0283] In a generative AI example, query data 1228 may include text, audio, image, video, digital, or media content prompts, and output prediction / inference data 1222 includes text, image, video, audio, code, or synthetic data.

[0284] Data Architecture

[0285] Figure 13 is a diagram illustrating a data structure 1300 that may be stored in a database 1304 of the interactive server system 110, according to certain examples. Although the contents of the database 1304 are shown as including a plurality of tables, it will be appreciated that data may be stored in other types of data structures (e.g., as an object-oriented database).

[0286] Database 1304 includes message data stored in message table 1306. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and payload. Figure 13 Additional details regarding information that may be included in a message and included within the message data stored in message table 1306 are described.

[0287] The entity table 1308 stores entity data and is linked (e.g., by reference) to the entity graph 1310 and the profile data 1302. Entities for which records are maintained within the entity table 1308 may include individuals, corporate entities, organizations, objects, places, events, and the like. Regardless of the entity type, any entity for which the interactive server system 110 stores data may be an identified entity. Each entity is provided with a unique identifier and an entity type identifier (not shown).

[0288] The entity graph 1310 stores information about relationships and associations between entities. By way of example only, such relationships may be interest-based or activity-based social relationships, professional relationships (e.g., working at a common company or organization). Some relationships between entities may be one-way, such as a subscription by an individual user to digital content from a business or publishing user (e.g., a newspaper or other digital media outlet or brand). Other relationships may be two-way, such as a "friend" relationship between various users of the interactive platform 100.

[0289] Certain permissions and relationships can be attached to each relationship, and also to each direction of the relationship. For example, a bidirectional relationship (e.g., a friend relationship between various users) can include authorization for the publication of digital content items between the various users, but can impose certain restrictions or filters on the publication of these digital content items (e.g., based on content characteristics, location data, or time of day data). Similarly, a subscription relationship between a personal user and a business user can impose varying degrees of restrictions on the publication of digital content from the business user to the personal user, and can significantly restrict or prevent the publication of digital content from the personal user to the business user. A particular user, as an example of an entity, can have certain restrictions recorded in its record for that entity within entity table 1308 (e.g., via privacy settings). Such privacy settings can apply to all types of relationships within the context of the interactive platform 100, or can selectively apply to only certain types of relationships.

[0290] Profile data 1302 stores various types of profile data about a particular entity. Based on the privacy settings specified by the particular entity, profile data 1302 can be selectively used and presented to other users of the interactive platform 100. In the case where the entity is a person, profile data 1302 includes, for example, the user's name, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user. The particular user can then selectively include one or more of these avatar representations within the content of messages transmitted via the interactive platform 100 and on a map interface displayed to other users by the interactive client 104. The collection of avatar representations can include a "status avatar," which presents a graphical representation of the user's status or activity that they may choose to transmit at a particular time.

[0291] Where the entity is a group, the profile data 1302 for the group may similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings for the relevant group (eg, notifications).

[0292] Database 1304 also stores enhancement data, such as overlays or filters, in enhancement table 1312. The enhancement data is associated with and applied to videos (video data is stored in video table 1314) and images (image data is stored in image table 1316).

[0293] In some examples, filters are overlays displayed as overlays on an image or video during presentation to a message recipient. Filters can be of various types, including user-selected filters from a set of filters presented to the message sender by the interactive client 104 when the message sender is composing the message. Other types of filters include geolocation filters (also known as geofilters), which can be presented to the message sender based on geolocation. For example, a geolocation filter specific to a nearby or special location can be presented by the interactive client 104 within the user interface based on geolocation information determined by a global positioning system (GPS) unit of the user system 102.

[0294] Another type of filter is a data filter, which can be selectively presented to the message sender by the interaction client 104 based on other input or information collected during the message creation process by the user system 102. Examples of data filters include the current temperature at a particular location, the current speed the message sender is traveling, the battery life of the user system 102, or the current time.

[0295] Other augmented data that can be stored in the image table 1316 include augmented reality content items (eg, corresponding to applied filters or augmented reality experiences). Augmented reality content items can be real-time special effects and sounds that can be added to an image or video.

[0296] As described above, augmented data includes augmented reality (AR), virtual reality (VR), and mixed reality (MR) content items, overlays, image transformations, images, and modifications that can be applied to image data (e.g., video or images). This includes real-time modifications, where the image is modified as it is captured using the user system 102's device sensors (e.g., one or more cameras) and the modified image is then displayed on the user system 102 screen. This also includes modifications to stored content (e.g., video clips in a collection or group that can be modified). For example, in a user system 102 with access to multiple augmented reality content items, a user can use a single video clip with multiple augmented reality content items to see how different augmented reality content items will modify the stored clip. Similarly, real-time video capture can use modifications to show how the video image currently being captured by the user system 102's sensors will modify the captured data. Such data can be simply displayed on the screen without being stored in memory, or the content captured by the device sensors can be recorded and stored in memory with or without modifications (or both). In some systems, a preview feature can simultaneously display how different augmented reality content items will appear within different windows on the display. For example, this may enable multiple windows with different pseudo-random animations to be viewed simultaneously on a display.

[0297] Thus, using data from an augmented reality content item and various systems or other such transformation systems that use this data to modify content can involve: detecting various objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.) in a video frame; tracking such objects as they leave, enter, and move around the field of view; and modifying or transforming such objects while being tracked. In various examples, different methods for implementing such transformations can be used. Some examples can involve: generating a three-dimensional mesh model of one or more objects; and using transformations of the model and animated textures within the video to implement the transformations. In some examples, tracking points on the objects can be used to place images or textures (which can be two-dimensional or three-dimensional) at the tracked locations. In yet another example, neural network analysis of video frames can be used to place images, models, or textures within content (e.g., images or video frames). Thus, an augmented reality content item involves both the images, models, and textures used to create the transformations within the content, as well as the additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0298] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load video files and store them in the device's memory, or a device's sensors can be used to generate a video stream. In addition, computer animation models can be used to process any object, such as a human face and body parts, an animal, or an inanimate object (e.g., a chair, a car, or other objects).

[0299] In some examples, when a specific modification is selected along with the content to be transformed, the element to be transformed is identified by a computing device and then detected and tracked if the element to be transformed is present in a frame of the video. The elements of the object are modified according to the modification request, thereby transforming the frame of the video stream. For different types of transformations, the frames of the video stream can be transformed by different methods. For example, for frame transformations that primarily involve changing the form of the elements of the object, characteristic points are calculated for each element of the object (e.g., using an active shape model (ASM) or other known methods). A grid based on the characteristic points is then generated for each element of the object. This grid is used in subsequent stages of tracking the elements of the object in the video stream. During the tracking process, the grid for each element is aligned with the position of each element. Additional points are then generated on the grid.

[0300] In some examples, a transformation that uses elements of an object to change some areas of an object can be performed by calculating characteristic points for each element of the object and generating a grid based on the calculated characteristic points. Points are generated on the grid, and then various areas based on these points are generated. The elements of the object are then tracked by aligning the area of ​​each element with the position of each of at least one element, and the properties of the area can be modified based on the modification request, thereby transforming the frame of the video stream. Depending on the specific requirements of the modification, the properties of the mentioned area can be transformed in different ways. Such modifications can involve: changing the color of the area; removing at least part of the area from the frame of the video stream; including new objects in the area based on the modification request; and modifying or distorting the elements of the area or object. In various examples, any combination of such modifications or other similar modifications can be used. For certain models to be animated, some characteristic points can be selected as control points for the entire state space to be used to determine the options for model animation.

[0301] In some examples of computer animation models that use face detection to transform image data, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). An active shape model (ASM) algorithm is then applied to the facial region of the image to detect facial feature reference points.

[0302] Other methods and algorithms suitable for face detection can be used. For example, in some examples, features are located using landmarks, which represent distinguishable points that are present in most of the images considered. For example, for facial landmarks, the location of the left pupil can be used. If the initial landmarks are not identifiable (for example, if the person is wearing an eye patch), secondary landmarks can be used. Such a landmark identification process can be used for any such object. In some examples, a set of landmarks forms a shape. The coordinates of the points in the shape can be used to represent the shape as a vector. One shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The mean shape (meanshape) is the average of the aligned training shapes.

[0303] The transformation system can capture an image or video stream on a client device (e.g., user system 102) and perform complex image manipulations locally on the user system 102 while maintaining an appropriate user experience, computational time, and power consumption. Complex image manipulations can include size and shape changes, emotion transformations (e.g., changing a face from a frown to a smile), state transitions (e.g., aging a subject, reducing apparent age, changing gender), style transformations, application of graphical elements, and any other suitable image or video manipulations enabled by a convolutional neural network that has been configured to execute efficiently on the user system 102.

[0304] In some examples, a computer-animated model for transforming image data can be used by a system in which a user can capture an image or video stream of the user (e.g., a selfie) using a user system 102 having a neural network operating as part of an interactive client 104 operating on the user system 102. The transformation system operating within the interactive client 104 determines the presence of a face within the image or video stream and provides a modification icon associated with the computer-animated model to transform the image data, or the computer-animated model can exist in association with an interface described herein. The modification icon includes changes that serve as a basis for modifying the user's face within the image or video stream as part of the modification operation. Once the modification icon is selected, the transformation system initiates a process that transforms the user's image to reflect the selected modification icon (e.g., generating a smiley face on the user). Once the image or video stream is captured and the specified modification is selected, the modified image or video stream can be presented in a graphical user interface displayed on the user system 102. The transformation system can implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, a user can capture an image or video stream and, once a modification icon has been selected, be presented with the modified result in real time or near real time. Furthermore, when the video stream is being captured, the modification can be persistent and the selected modification icon remains toggled. Machine-taught neural networks can be used to implement such modifications.

[0305] The graphical user interface presenting the modifications performed by the transformation system can provide the user with additional interactive options. Such options can be based on the interface used to initiate selection of a particular computer-animated model and content capture (e.g., from a content creator user interface). In various examples, the modifications can be persistent after the initial selection of the modification icon. The user can toggle the modifications on or off by tapping or otherwise selecting a face modified by the transformation system, and store them for later viewing or browsing to other areas of the imaging application. In the event that multiple faces are modified by the transformation system, the user can globally toggle the modifications on or off by tapping or selecting a single face modified and displayed within the graphical user interface. In some examples, each face in a group of multiple faces can be modified individually, or such modifications can be individually toggled by tapping or selecting a single face or series of faces displayed within the graphical user interface.

[0306] The story table 1318 stores data about a collection of messages and associated images, video, or audio data that are compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 1308). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface of the interactive client 104 can include a user-selectable icon that enables the message sender to add specific content to his or her personal story.

[0307] The collection can also constitute a "live story," which is a collection of content from multiple users that is created manually, automatically, or using a combination of manual and automatic methods. For example, a "live story" can constitute a curated stream of user-submitted content from different locations and events. Users whose client devices have location services enabled and who are at a co-located event at a particular time can be presented with the option to contribute content to a particular live story, for example, via the user interface of the interactive client 104. Live stories can be identified to the user by the interactive client 104 based on his or her location. The end result is a "live story" told from a group perspective.

[0308] Another type of content collection is called a "location story," which enables users whose user systems 102 are located within a specific geographic location (e.g., on a college or university campus) to contribute to a particular collection. In some examples, contributions to location stories may require secondary authentication to verify that the end user belongs to a specific organization or other entity (e.g., is a student on a university campus).

[0309] As mentioned above, video table 1314 stores video data that, in some examples, is associated with messages for which records are maintained within message table 1306. Similarly, image table 1316 stores image data associated with messages whose message data is stored in entity table 1308. Entity table 1308 can associate various enhancements from enhancement table 1312 with various images and videos stored in image table 1316 and video table 1314.

[0310] Database 1304 also includes social network information collected by the interactive platform's interactive system. Social network information may include, but is not limited to, relationships and communication data among users of the interactive platform. Social network information can be used to group two or more users and provide additional functionality for the interactive platform 100. Examples of relationships include, but are not limited to, best friends where two or more users are identified as mutual best friends based on the frequency of their interactions, users with a shared interest in current events, and users who share connections through social clubs or charitable organizations. Examples of communications include, but are not limited to, chats, private and public messaging, and the exchange of media such as images, videos, and audio recordings.

[0311] Data communication architecture

[0312] Figure 14 1 is a schematic diagram illustrating the structure of a message 1400 according to some examples, the message 1400 being generated by an interaction client 104 for transmission to another interaction client 104 via an interaction server 124. The content of a particular message 1400 is used to populate a message table 1306 stored within a database 1304 accessible by the interaction server 124. Similarly, the content of the message 1400 is stored in memory as "in-transit" or "in-flight" data of the user system 102 or the interaction server 124. The message 1400 is shown as including the following example components:

[0313] Message identifier 1402 : A unique identifier that identifies the message 1400 .

[0314] Message text payload 1434 : Text to be generated by the user via the user interface of the user system 102 and included in the message 1400 .

[0315] Message image payload 1404 : Image data captured by the camera component of the user system 102 or retrieved from the memory component of the user system 102 and included in the message 1400 . Image data for a message 1400 sent or received may be stored in an image table 1406 .

[0316] • Message Video Payload 1408: Video data captured by the camera component or retrieved from the memory component of the user system 102 and included in the message 1400. The video data for a message 1400 sent or received may be stored in a video table 1410.

[0317] • Message audio payload 1412 : audio data captured by a microphone or retrieved from a memory component of the user system 102 and included in the message 1400 .

[0318] Message enhancement data 1414: Enhancement data representing enhancements (e.g., filters, stickers, or other annotations or enhancements) to be applied to the message image payload 1404, message video payload 1408, or message audio payload 1412 of the message 1400. Enhancement data for a sent or received message 1400 may be stored in an enhancement table 1416.

[0319] • Message duration parameter 1418: A parameter value indicating the amount of time, in seconds, that the content of the message (e.g., message image payload 1404, message video payload 1408, message audio payload 1412) is to be presented to or made accessible to the user via the interactive client 104.

[0320] Message geolocation parameters 1420: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter 1420 values ​​can be included in the payload, with each of these parameter values ​​being associated with a content item included in the content (e.g., a specific image within the message image payload 1404 or a specific video within the message video payload 1408).

[0321] Message story identifier 1422: An identifier value that identifies one or more content collections (e.g., a "story" identified in stories table 1424) associated with a particular content item in the message image payload 1404 of the message 1400. For example, the identifier value can be used to associate multiple images within the message image payload 1404 with multiple content collections.

[0322] Message tags 1426: Each message 1400 can be tagged with a plurality of tags, each of which indicates the subject matter of the content included in the message payload. For example, if a particular image included in the message image payload 1404 depicts an animal (e.g., a lion), a tag value can be included within the message tags 1426 indicating the relevant animal. The tag value can be manually generated based on user input, or can be automatically generated using, for example, image recognition.

[0323] • Message sender identifier 1428: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the user system 102 on which the message 1400 was generated and from which the message 1400 was sent.

[0324] • Message recipient identifier 1430: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the user system 102 to which the message 1400 is addressed.

[0325] The content (e.g., value) of each component of message 1400 may be a pointer to a location in a table where the content data value is stored. For example, the image value in message image payload 1404 may be a pointer (or address) to a location within image table 1406. Similarly, the value within message video payload 1408 may point to data stored in video table 1410, the value stored in message enhancement data 1414 may point to data stored in enhancement table 1416, the value stored in message story identifier 1422 may point to data stored in story table 1424, and the values ​​stored in message sender identifier 1428 and message recipient identifier 1430 may point to user records stored in entity table 1432.

[0326] Figure 15 is a schematic diagram illustrating an access restriction process 1500 in which access to content (e.g., an ephemeral message 1502 and associated data multimedia payload) or a collection of content (e.g., an ephemeral message group 1504) can be time-limited (e.g., set to be ephemeral).

[0327] The ephemeral message 1502 is shown as being associated with a message duration parameter 1506, the value of which determines the amount of time that the ephemeral message 1502 will be displayed by the interactive client 104 to the receiving user of the ephemeral message 1502. In some examples, the ephemeral message 1502 may be viewed by the receiving user for up to 10 seconds, depending on the amount of time specified by the message sender using the message duration parameter 1506.

[0328] The message duration parameter 1506 and the message receiver identifier 1508 are shown as inputs to a message timer 1510, which is responsible for determining the amount of time that the ephemeral message 1502 is displayed to the particular receiving user identified by the message receiver identifier 1508. In particular, the ephemeral message 1502 is displayed to the associated receiving user for a period of time determined by the value of the message duration parameter 1506. The message timer 1510 is shown as providing an output to a more general messaging system 1512, which is responsible for the overall timing of displaying content (e.g., the ephemeral message 1502) to the receiving user.

[0329] Short message 1502 Figure 151504 (e.g., a collection of messages in a personal story or event story). The ephemeral message group 1504 has an associated group duration parameter 1514, the value of which determines the duration for which the ephemeral message group 1504 is presented and accessible to users of the interactive platform 100. For example, the group duration parameter 1514 may be the duration of a concert, where the ephemeral message group 1504 is a collection of content related to the concert. Alternatively, a user (either an owning user or a curator user) may specify the value of the group duration parameter 1514 when performing setup and creation of the ephemeral message group 1504.

[0330] In addition, each ephemeral message 1502 within the ephemeral message group 1504 has an associated group participation parameter 1516, the value of which determines the duration for which the ephemeral message 1502 is accessible within the context of the ephemeral message group 1504. Thus, a particular ephemeral message group 1504 can "expire" and become inaccessible within the context of the ephemeral message group 1504 before the ephemeral message group 1504 itself expires according to the group duration parameter 1514. The group duration parameter 1514, the group participation parameter 1516, and the message recipient identifier 1508 each provide inputs to a group timer 1518, which is operable to first determine whether a particular ephemeral message 1502 of the ephemeral message group 1504 is to be displayed to a particular receiving user, and if so, for how long. Note that the ephemeral message group 1504 also knows the identity of the particular receiving user as a result of the message recipient identifier 1508.

[0331] 1504. Thus, the group timer 1518 is operable to control the overall lifespan of the associated ephemeral message group 1504 and the individual ephemeral messages 1502 included in the ephemeral message group 1504. In some examples, each ephemeral message 1502 within the ephemeral message group 1504 remains viewable and accessible for a period of time specified by the group duration parameter 1514. In another example, within the context of the ephemeral message group 1504, a particular ephemeral message 1502 may expire based on the group participation parameter 1516. Note that even within the context of the ephemeral message group 1504, the message duration parameter 1506 may still determine the duration for which a particular ephemeral message 1502 is displayed to the receiving user. Thus, the message duration parameter 1506 determines the duration for which a particular ephemeral message 1502 is displayed to the receiving user, regardless of whether the receiving user is viewing the ephemeral message 1502 within or outside the context of the ephemeral message group 1504.

[0332] The messaging system 1512 is also operable to remove a particular ephemeral message 1502 from an ephemeral message group 1504 based on a determination that the particular ephemeral message 1502 has exceeded an associated group participation parameter 1516. For example, when a message sender has established a group participation parameter 1516 of 24 hours from posting, the messaging system 1512 will remove the associated ephemeral message 1502 from the ephemeral message group 1504 after the specified 24 hours. The messaging system 1512 is also operable to remove an ephemeral message group 1504 when the group participation parameter 1516 of each ephemeral message 1502 within the ephemeral message group 1504 has expired, or when the ephemeral message group 1504 itself has expired according to the group duration parameter 1514.

[0333] In some use cases, the creator of a particular ephemeral message group 1504 may specify an indefinite group duration parameter 1514. In this case, the expiration of the group participation parameter 1516 of the last remaining ephemeral message 1502 within the ephemeral message group 1504 will determine when the ephemeral message group 1504 itself expires. In this case, a new ephemeral message 1502 with a new group participation parameter 1516 added to the ephemeral message group 1504 effectively extends the life of the ephemeral message group 1504 to equal the value of the group participation parameter 1516.

[0334] In response to the messaging system 1512 determining that the ephemeral message group 1504 has expired (e.g., is no longer accessible), the messaging system 1512 communicates with the interactive platform 100 (and, for example, in particular, the interactive client 104) to cause the indicia (e.g., an icon) associated with the relevant ephemeral message group 1504 to no longer be displayed within the user interface of the interactive client 104. Similarly, when the messaging system 1512 determines that the message duration parameter 1506 of a particular ephemeral message 1502 has expired, the messaging system 1512 causes the interactive client 104 to no longer display the indicia (e.g., an icon or textual identification) associated with the ephemeral message 1502.

[0335] Machine Architecture

[0336] Figure 161600 within which instructions 1602 (e.g., software, programs, applications, applet, apps, or other executable code) may be executed for causing the machine 1600 to perform any one or more of the methodologies discussed herein. For example, the instructions 1602 may cause the machine 1600 to perform any one or more of the methodologies described herein. The instructions 1602 transform a general-purpose, unprogrammed machine 1600 into a specialized machine 1600 that is programmed to perform the functions described and illustrated in the manner described. The machine 1600 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1600 may operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1600 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 1602 specifying actions to be taken by the machine 1600. Furthermore, while a single machine 1600 is shown, the term "machine" should also be construed to include a collection of machines that individually or jointly execute instructions 1602 to perform any one or more of the methods discussed herein. For example, the machine 1600 may include the user system 102 or any of a plurality of server devices forming part of the interactive server system 110. In some examples, the machine 1600 may also include both a client system and a server system, wherein some operations of a particular method or algorithm are performed on the server side, and wherein some operations of a particular method or algorithm are performed on the client side.

[0337] The machine 1600 may include a processor 1604, a memory 1606, and input / output I / O components 1608 that may be configured to communicate with each other via a bus 1610. In an example, the processor 1604 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1612 that executes instructions 1602 and a processor 1614. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions concurrently. Although Figure 16 Multiple processors 1604 are shown, but the machine 1600 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0338] The memory 1606 includes a main memory 1616, a static memory 1618, and a storage unit 1620, all of which are accessible by the processor 1604 via the bus 1610. The main memory 1606, the static memory 1618, and the storage unit 1620 store instructions 1602 that embody any one or more of the methodologies or functionality described herein. The instructions 1602 may also reside, completely or partially, within the main memory 1616, within the static memory 1618, within the machine-readable medium 1622 within the storage unit 1620, within at least one of the processors 1604 (e.g., within a cache memory of the processor), or within any suitable combination thereof during execution thereof by the machine 1600.

[0339] I / O components 1608 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 1608 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It will be understood that I / O components 1608 may include Figure 16Many other components are not shown in the figure. In various examples, the I / O components 1608 may include user output components 1624 and user input components 1626. The user output components 1624 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The user input components 1626 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide location and force of touches or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0340] In another example, the I / O component 1608 may include a biometric component 1628, a motion component 1630, an environmental component 1632, or a position component 1634, as well as various other components. For example, the biometric component 1628 includes components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component 1630 includes: an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope). The biometric component may include a brain-computer interface (BMI) system that allows communication between the brain and an external device or machine. This can be achieved by recording brain activity, converting the brain activity into a format that can be understood by a computer, and then using the resulting signals to control the device or machine.

[0341] Example types of BMI technology include:

[0342] Electroencephalogram (EEG)-based BMIs, which use electrodes placed on the scalp to record electrical activity in the brain.

[0343] Invasive BMIs, where electrodes are surgically implanted directly into the brain.

[0344] Optogenetic BMIs, which use light to control the activity of specific nerve cells in the brain.

[0345] BMI based on functional magnetic resonance imaging (fMRI), which uses magnetic fields to measure blood flow in the brain, which can be used to infer brain activity.

[0346] Environmental components 1632 include, for example, one or more cameras (with still image / photo and video capabilities), an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0347] With respect to cameras, the user system 102 can have a camera system that includes, for example, a front-facing camera on the front surface of the user system 102 and a rear-facing camera on the rear surface of the user system 102. The front-facing camera can be used, for example, to capture still images and videos of the user of the user system 102 (e.g., “selfies”), which can then be enhanced with the enhancement data (e.g., filters) described above. For example, the rear-facing camera can be used to capture still images and videos in a more traditional camera mode, where the images are similarly enhanced with the enhancement data. In addition to the front-facing camera and the rear-facing camera, the user system 102 can also include a 360° camera for capturing 360° photos and videos.

[0348] Furthermore, the camera system of the user system 102 may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even triple, quad, or quintuple rear camera configurations on both the front and back sides of the user system 102. For example, these multiple camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0349] The location component 1634 includes a positioning sensor component (eg, a GPS receiver component), an altitude sensor component (eg, an altimeter or a barometer that detects air pressure from which altitude can be obtained), an orientation sensor component (eg, a magnetometer), and the like.

[0350] Various technologies can be used to implement communications. The I / O components 1608 also include a communications component 1636 that is operable to couple the machine 1600 to a network 1638 or device 1640 via corresponding couplings or connections. For example, the communications component 1636 may include a network interface component or another suitable device that interfaces with the network 1638. In other examples, the communications component 1636 may include a wired communications component, a wireless communications component, a cellular communications component, a near field communications (NFC) component, a Components (e.g. Low power consumption), Device 1640 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0351] In addition, the communication component 1636 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1636 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, UltraCode, UCC RSS-2D bar codes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1636, such as a location obtained via Internet Protocol (IP) geolocation, ... Positioning derived from signal triangulation, positioning derived via detection of NFC beacon signals that can indicate a specific position, etc.

[0352] Various memories (e.g., main memory 1616, static memory 1618, and memory of processor 1604) and storage unit 1620 may store one or more sets of instructions and data structures (e.g., software) embodied or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1602), when executed by processor 1604, cause various operations to implement the disclosed examples.

[0353] Instructions 1602 may be sent or received over network 1638 using a transmission medium via a network interface device (e.g., a network interface component included in communications component 1636) and using any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 1602 may be sent or received via a coupling (e.g., a peer-to-peer coupling) to device 1640 using a transmission medium.

[0354] Software Architecture

[0355] Figure 17 17 is a block diagram 1700 illustrating a software architecture 1702 that can be installed on any one or more of the devices described herein. Software architecture 1702 is supported by hardware, such as a machine 1704, including a processor 1706, memory 1708, and I / O components 1710. In this example, software architecture 1702 can be conceptualized as a stack of layers, each providing specific functionality. Software architecture 1702 includes layers such as an operating system 1712, libraries 1714, frameworks 1716, and applications 1718. In operation, applications 1718 invoke API calls 1720 through the software stack and receive messages 1722 in response to API calls 1720.

[0356] The operating system 1712 manages hardware resources and provides common services. The operating system 1712 includes, for example, a kernel 1724, services 1726, and drivers 1728. The kernel 1724 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 1724 provides memory management, processor management (e.g., scheduling), component management, network and security settings, and other functions. Services 1726 can provide other common services to other software layers. Drivers 1728 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1728 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, serial communication drivers (e.g., USB drivers), drivers, audio drivers, power management drivers, etc.

[0357] The libraries 1714 provide a common low-level infrastructure used by the applications 1718. The libraries 1714 may include a system library 1730 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the libraries 1714 may include an API library 1732, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering graphical content on a display in two dimensions (2D) and three dimensions (3D), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1714 may also include various other libraries 1734 to provide many other APIs to the application 1718 .

[0358] The framework 1716 provides a common high-level infrastructure used by applications 1718. For example, the framework 1716 provides various graphical user interface (GUI) functions, advanced resource management, and advanced positioning services. The framework 1716 can provide a wide range of other APIs that can be used by applications 1718, some of which may be specific to a particular operating system or platform.

[0359] In an example, applications 1718 may include a home application 1736, a contacts application 1738, a browser application 1740, a book reader application 1742, a location application 1744, a media application 1746, a messaging application 1748, a game application 1750, and a variety of other applications such as third-party applications 1752. Applications 1718 are programs that perform functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 1718 structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, third-party applications 1752 (e.g., those written by entities other than the vendor of a particular platform using ANDROID) may be used to create a third-party application 1752. TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM 、 Mobile software running on the mobile operating system of the phone or other mobile operating system. In this example, third-party applications 1752 can activate API calls 1720 provided by the operating system 1712 to facilitate the functions described in this article.

[0360] Additional examples include:

[0361] Example 1 is a method comprising: receiving, by one or more processors, a prompt of a user from a user system during an interactive session, the prompt in the form of a first posting to an interactive platform; determining, by the one or more processors, an intent of the user using the prompt; generating, by the one or more processors, a response using the intent, the response in the form of a second posting to the interactive platform; and providing, by the one or more processors, the response to the user in the form of a posting to the interactive platform.

[0362] In Example 2, the subject matter of Example 1 includes, wherein the prompt includes media for publishing to the interactive platform.

[0363] In Example 3, the subject matter of any one of Examples 1 to 2 includes, wherein the media includes at least one of audio media, image media, video media, or text media.

[0364] In Example 4, the subject matter of any one of Examples 1 to 3 includes, wherein determining the intent comprises generating a description of media, and wherein generating the response comprises generating the response media using the intent.

[0365] In Example 5, the subject matter of any one of Examples 1 to 4 includes, wherein generating the response comprises generating the response media using the media and the intent.

[0366] In Example 6, the subject matter of any one of Examples 1 to 5 includes, wherein generating the response comprises generating edited response media using the media and the intent.

[0367] In Example 7, the subject matter of any one of Examples 1 to 6 includes, wherein generating the response comprises generating a suggested activity using the intent.

[0368] In Example 8, the subject matter of any one of Examples 1 to 7 includes, wherein the prompt is posted to an interactive platform of the chatbot, and wherein generating the response includes generating the response using a character description of the chatbot.

[0369] In Example 9, the subject matter of any one of Examples 1 to 8 includes, wherein the role description is input by a user.

[0370] In Example 10, the subject matter of any one of Examples 1 to 9 includes, wherein the character descriptions are scanned for abusive language and harmful content.

[0371] In Example 11, the subject matter of any one of Examples 1 to 10 includes, wherein the character description is randomly generated.

[0372] Example 12 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any of Examples 1-11.

[0373] Example 13 is an apparatus comprising means for implementing any one of Examples 1 to 11.

[0374] Example 24 is a system for implementing any one of Examples 1 to 11.

[0375] in conclusion

[0376] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure.

[0377] Glossary

[0378] "Carrier signal" refers to any intangible medium that can store, encode, or carry instructions for execution by a machine and includes digital or analog communication signals or other intangible media to facilitate communication of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0379] "Client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smartphone, a tablet computer, an ultrabook, a netbook, multiple laptop computers, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0380] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Plain Old Telephone Service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long distance protocols, or other data transmission technologies.

[0381] "Component" refers to a device, physical entity, or logic with the following boundaries: the boundaries are defined by function or subroutine calls, branch points, APIs, or other technologies provided for partitioning or modularizing specific processing or control functions. A component can be combined with other components via its interface to perform machine processing. A component can be a packaged functional hardware unit designed for use with other components, and can be part of a program that generally performs a specific function among related functions. A component can constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a processor group) can be configured by software (e.g., an application or an application portion) to be a hardware component that operates to perform certain operations as described herein. A hardware component can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic that is permanently configured to perform certain operations. The hardware component can be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component can also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, the hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine), which is uniquely customized to perform the configured function and is no longer a general-purpose processor. It will be appreciated that it can be decided whether to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured circuit system (for example, configured by software) for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (for example, hardwired) or temporarily configured (for example, programmed) to operate in some way or perform certain operations described herein. Considering an example in which a hardware component is temporarily configured (for example, programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any one time. For example, where a hardware component includes a general-purpose processor that is configured by software to become a special-purpose processor, the general-purpose processor can be configured as different special-purpose processors (e.g., including different hardware components) at different times. The software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. A hardware component can provide information to other hardware components and receive information from other hardware components.Therefore, the hardware components described can be considered to be coupled in communication. In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission between or among two or more hardware components in the hardware components (for example, by appropriate circuits and buses). In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled in communication with it. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (for example, a collection of information). The various operations of the example methods described herein can be performed at least in part by a temporary configuration (for example, by software) or a permanent configuration to one or more processors performing related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein specific one or more processors are examples of hardware. For example, at least some of the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of the related operations in the " cloud computing " environment or operate as " software as a service " (SaaS). For example, at least some of the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of certain operations in the operation can be distributed between the processors, not only resides in a single machine, but also deployed across multiple machines. In some examples, the parts implemented by the processor or the processor can be located in a single geographic location (for example, in a home environment, an office environment or a server farm). In other examples, the parts implemented by the processor or the processor can be distributed across multiple geographic locations.

[0382] "Machine-readable storage media" refers to both machine storage media and transmission media. Thus, these terms include both storage devices / media and carrier / modulated data signals. The terms "computer-readable medium," "machine-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0383] An "ephemeral message" is a message that is accessible for a limited duration. An ephemeral message can be text, an image, a video, or the like. The access time for an ephemeral message can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is ephemeral.

[0384] “Machine storage media” refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be taken to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage media,” “device storage media,” and “computer storage media” mean the same thing and are used interchangeably in this disclosure. The terms “machine storage media,” “computer storage media,” and “device storage media” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term “signal media.”

[0385] “Non-transitory machine-readable storage medium” refers to a tangible medium capable of storing, encoding, or carrying instructions for execution by a machine.

[0386] "Signal medium" refers to any intangible medium that is capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be construed to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0387] In this disclosure and the appended claims, the terms "a" or "an" are commonly used in patent documents to mean including one or more, independent of any other instances or uses of "at least one" or "one or more." In this disclosure and the appended claims, unless otherwise indicated, the term "or" is used to refer to a non-exclusive or, so that "A or B" includes "A but not B," "B but not A," and "A and B." In the appended claims, the terms "including" and "in which" are used as the plain-English equivalents of the respective terms "comprising" and "wherein." Furthermore, in the appended claims, the terms "including" and "comprising" are open-ended; that is, a system, apparatus, article, or process that includes elements in addition to the elements listed after such terms in the claim is still deemed to fall within the scope of the claim.

Claims

1. A method comprising: receiving, by the one or more processors, a prompt from a user system during an interactive session, the prompt being in the form of a first posting to the interactive platform; determining, by the one or more processors, the user's intent using the prompt; generating, by the one or more processors, a response using the intent, the response being in the form of a second posting to the interactive platform; as well as The response is provided to the user by the one or more processors in the form of a posting to the interactive platform.

2. The method according to claim 1, wherein The prompt includes media for publishing to the interactive platform.

3. The method according to claim 2, wherein: The media includes at least one of audio media, image media, video media or text media.

4. The method according to claim 2, in, Determining the intent includes generating a description of the media, and The generating of the response includes generating response media using the intent.

5. The method according to claim 2, wherein: Generating the response includes generating response media using the media and the intent.

6. The method according to claim 2, wherein: Generating the response includes generating edited response media using the media and the intent.

7. The method according to claim 1, wherein Generating the response includes generating suggested activities using the intent.

8. The method according to claim 1, in, The prompt is posted to the chatbot's interactive platform, and Wherein, generating the response includes generating the response using a role description of the chatbot.

9. The method according to claim 8, wherein The role description is input by the user.

10. The method according to claim 9, wherein: Scan said role descriptions for abusive language and harmful content.

11. The method according to claim 8, wherein Said character description is randomly generated.

12. A method comprising: receiving, by one or more processors, a chatbot mention message from a user system of a user in a group chat session, the chatbot mention message including a chatbot prompt created by the user; generating, by the one or more processors, a prompt using the chatbot mention message; generating, by the one or more processors, a chatbot response message using the prompt; as well as The chatbot response message is provided, by the one or more processors, to one or more other user systems of one or more other users in the group chat session.

13. The method according to claim 12, further comprising: storing, by the one or more processors, one or more stored chatbot mention messages and one or more stored chatbot response messages associated with the user; as well as A context for the prompt is generated using the one or more stored chatbot mention messages and the one or more stored chatbot mention messages.

14. The method according to claim 13, further comprising: In response to receiving a delete message request from the user by the one or more processors, deleting the one or more stored chatbot mention messages and the one or more stored chatbot response messages.

15. The method according to claim 13, in, The one or more stored chatbot mention messages and the one or more stored chatbot response messages are associated with two or more users in the group chat session, and The method further includes generating, by the one or more processors, a context for the prompt for each user of the group chat session using the one or more stored chatbot mention messages and the one or more stored chatbot response messages.

16. The method according to claim 12, further comprising: In response to determining, by the one or more processors, that the chatbot-mention message is a first chatbot-mention message in the group chat session, providing a chatbot notification describing an operation of the chatbot to one or more users in the group chat session.

17. The method according to claim 12, wherein: Generating the response includes generating the response using the character of the chatbot.

18. A method comprising: receiving, by one or more processors, a first message from a user system of a user during an interactive session with a chatbot, the first message comprising a prompt of the user; determining, by the one or more processors, keywords and concepts using the message prompt; generating, by the one or more processors, a response in the form of a second message using the keywords and concepts; as well as The second message is provided to the user by the one or more processors.

19. The method according to claim 18, wherein The first message includes an image.

20. The method according to claim 19, further comprising: determining, by the one or more processors, that the first message includes abusive language; generating, by the one or more processors, an abusive language mitigation message; as well as The abusive language mitigation message is transmitted to the user as the second message.

Images