Facilitating interactions based on user access patterns

Abstract

In some examples, techniques according to the present disclosure include detecting a determinable interaction mode in which a first user interacts with a content item delivered by a second user on an interaction system via a user system. The interaction mode may be determined to include a plurality of access actions performed by the first user for the content item. In response to detecting that the interaction mode can be determined, an action graphical user interface element is automatically activated, the action graphical user interface element being selectable by the user to enable the first user to perform an action related to the content item.

Description

[0001] Priority Claim

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 365,187, filed on May 23, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to electronic content sharing. More specifically, the subject matter disclosed herein relates to systems and methods for facilitating user interaction based on a determinable interaction pattern with a content item. Background Art

[0004] The popularity of computer-implemented programs that allow users to access content online and interact with it and other users continues to grow. There are various computer-implemented applications that allow users to share content with other users through interactive applications such as messaging clients or content sharing clients. Some computer-implemented applications may be designed to run on mobile devices such as phones, tablets, or wearable devices, with back-end services provided on a server computer system for performing specific operations, for example, operations that may require more resources than can reasonably be performed on the user's device.

[0005] An interactive application, such as a messaging client executing on a user device, may enable a posting user to capture a photo or video and share the captured or viewed content to another device for viewing by a viewing user. The captured content may be enhanced with items such as filters, media overlays, augmented reality effects, links, etc. The viewing user may interact with the shared content or with the posting user via the interactive application in various ways, such as by sending a reply message to the posting user, or by capturing and sharing other photos or videos in response to content shared by the posting user. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In the accompanying drawings (which are not necessarily drawn to scale), the same reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, one or more of the highest digits in the reference numeral refers to the figure number in which the element is first introduced. Some non-limiting examples are shown in the accompanying drawings, in which:

[0007] Figure 1 is a diagrammatic representation of a networking environment in which the present disclosure may be deployed, according to some examples.

[0008] Figure 2 is a diagrammatic representation of a messaging system having both client-side and server-side functionality according to some examples.

[0009] Figure 3 is a flow chart illustrating a method for detecting a determinable interaction mode and activating an action graphical user interface element according to some examples.

[0010] Figure 4 is a flow chart illustrating a method for detecting a plurality of access actions with respect to content items forming part of a content collection and activating an action graphical user interface element, according to some examples.

[0011] Figure 5 is a user interface diagram illustrating a viewing user interface according to some examples.

[0012] Figure 6 is a user interface diagram illustrating a viewing user interface according to some examples.

[0013] Figure 7 is shown according to some examples Figure 6 A user interface diagram of a viewing user interface of the present invention, wherein an action graphical user interface element is presented in an adjusted display state.

[0014] Figure 8 is a diagrammatic representation of data structures as maintained in a database according to some examples.

[0015] Fig. 9 is a graphical representation of messages according to some examples.

[0016] Fig.10 is a diagrammatic representation of a system including a head wearable device according to some examples.

[0017] Fig.11 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein.

[0018] Fig.12 is a block diagram illustrating a software architecture in which examples may be implemented. DETAILED DESCRIPTION

[0019] In the context of interactive systems that allow users to share content, it is desirable to facilitate the ease with which messages and content items can be transmitted between users. There are many technical challenges with respect to increasing the ease and speed with which users can consume content and then interact with that content (e.g., by replying to the transmission or posting of an image or video posted by a user).

[0020] The viewing user often revisits content items, for example, the user may re-watch a video, or revisit (or review) a digital photo posted by another user of the interactive system (e.g., a friend of the viewing user). For example, where the content item is part of a navigable content collection that includes multiple other content items, the viewing user may navigate back to a specific content item to review the specific content item, or revisit the content collection on another occasion to review the specific content item. Such actions may indicate that the viewing user is interested in interacting with the specific content item or with the user who posted the content item, and therefore desires to facilitate such interaction.

[0021] Some examples described herein seek to provide various automations to remind or guide a viewing user of options for interacting with a content item (e.g., an image file, a video file, an audio file, or a combination thereof) that the viewing user has accessed (e.g., viewed or listened to) multiple times.

[0022] In some examples, technical barriers to improving the convenience or speed of communication between users of an interactive system are mitigated by providing a user interface that enables quick and easy replying to re-watched content items (e.g., re-watched items within a content collection). Various forms of action graphical user interface elements, including, for example, reply elements, may be provided within such a user interface. Examples of the present disclosure also include various user interface element transitions to address or mitigate the aforementioned technical barriers.

[0023] Action graphical user interface elements can be used to provide "reply prompts" to viewing users. For example, a quick reply tray can be automatically presented in response to detecting a re-watch. In some examples, in response to detecting a re-watch, the display state of a reply interface element (e.g., a reply button or a chat input field) can be automatically adjusted (e.g., to visually distinguish the reply interface element from other interface elements or its own default state). In examples of the present disclosure, activation of action graphical user interface elements and display state changes can be applied individually or in combination.

[0024] When considering the effects in the present disclosure, one or more methods described herein can avoid the need for certain work or resources that would otherwise be involved in executing an interactive application. For example, as a result of automatic interface transitions (e.g., by presenting automatic reminders or display state changes), or as a result of a reduction in the number of choices required to achieve an effect, the computing resources used by one or more machines, databases, or networks can be more efficiently utilized or even reduced. Examples of such computing resources may include processor cycles, network traffic, memory usage, graphics processing unit (GPU) resources, data storage capacity, power consumption, and cooling capacity.

[0025] Networked computing environment

[0026] Figure 1 1 is a block diagram illustrating an example interactive system 100 for facilitating interactions over a network (e.g., exchanging text messages, conducting text audio and video calls, or playing games). The interactive system 100 includes multiple user systems 102, each of which hosts multiple applications including an interactive client 104 (as an example of an interactive application) and other applications 106. Each interactive client 104 is communicatively coupled to other instances of the interactive client 104 (e.g., hosted on respective other user systems 102), an interactive server system 110, and a third-party server 112 via one or more communication networks including a network 108 (e.g., the Internet). The interactive client 104 can also communicate with the locally hosted application 106 using an application program interface (API).

[0027] Each user system 102 may include a plurality of user devices, such as a mobile device 114 , a head wearable device 116 , and a computer client device 118 , which are communicatively connected to exchange data and messages.

[0028] The interactive clients 104 interact with other interactive clients 104 and with the interactive server system 110 via the network 108. The data exchanged between the interactive clients 104 (e.g., interaction 120) and between the interactive clients 104 and the interactive server system 110 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video or other multimedia data).

[0029] The interactive server system 110 provides server-side functions to the interactive clients 104 via the network 108. Although certain functions of the interactive system 100 are described herein as being performed by the interactive clients 104 or by the interactive server system 110, whether certain functions are located within the interactive clients 104 or within the interactive server system 110 may be a design choice. For example, it may be technically preferred to initially deploy certain technologies and functions within the interactive server system 110, but to later migrate the technologies and functions to the interactive clients 104 where the user system 102 has sufficient processing power.

[0030] The interactive server system 110 supports various services and operations provided to the interactive clients 104. Such operations include sending data to the interactive clients 104, receiving data from the interactive clients 104, and processing data generated by the interactive clients 104. The data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the interactive system 100 is activated and controlled by functions available through the user interface (UI) of the interactive client 104.

[0031] Turning now specifically to the interaction server system 110, the API server 122 is coupled to and provides a programming interface to the interaction server 124, making the functionality of the interaction server 124 accessible to the interaction clients 104, other applications 106, and third-party servers 112. The interaction server 124 is communicatively coupled to a database server 126, thereby facilitating access to a database 128 that stores data associated with interactions processed by the interaction server 124. Similarly, a web server 130 is coupled to the interaction server 124 and provides a web-based interface to the interaction server 124. To this end, the web server 130 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0032] The API server 122 receives and sends interaction data (e.g., commands and message payloads) between the interaction server 124 and the user system 102 (and, for example, the interaction client 104 and other applications 106) and the third-party server 112. Specifically, the API server 122 provides a set of interfaces (e.g., routines and protocols) that the interaction client 104 and other applications 106 can call or query to activate the functions of the interaction server 124. The API server 122 exposes various functions supported by the interaction server 124, including account registration; login function; sending interaction data from a particular interaction client 104 to another interaction client 104 via the interaction server 124; transferring media files (e.g., images or videos) from the interaction client 104 to the interaction server 124; setting a media data collection (e.g., a story); retrieving a friend list of a user of the user system 102; retrieving information and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the interaction client 104).

[0033] The interactive server 124 hosts multiple systems and subsystems, as shown below. Figure 2 Give a description.

[0034] Linked Applications

[0035] Returning to the interactive client 104, the features and functions of the external resource (e.g., the linked application 106 or applet) are available to the user via the interface of the interactive client 104. In this context, "external" refers to the fact that the application 106 or applet is outside the interactive client 104. External resources are usually provided by a third party, but can also be provided by the creator or provider of the interactive client 104. The interactive client 104 receives a user selection of an option for launching or accessing the features of such an external resource. The external resource can be an application 106 (e.g., a "local app") installed on the user system 102, or a small-scale version of the application (e.g., a "applet") hosted on the user system 102 or located at the remote end of the user system 102 (e.g., on a third-party server 112). The small-scale version of the application includes a subset of the features and functions of the application (e.g., a full-scale local version of the application) and is implemented using a markup language document. In some examples, the small-scale version of the application (e.g., a "applet") is a web-based markup language version of the application and is embedded in the interactive client 104. In addition to using markup language documents (eg, .*ml files), applets may include scripting languages ​​(eg, .*js files or .json files) and style sheets (eg, .*ss files).

[0036] In response to receiving a user selection of an option for launching or accessing a feature of an external resource, the interactive client 104 determines whether the selected external resource is a web-based external resource or a locally installed application 106. In some cases, the application 106 installed locally on the user system 102 can be independent of the interactive client 104 and launched separately from the interactive client 104, such as by selecting an icon corresponding to the application 106 on a home screen of the user system 102. Small-scale versions of such applications can be launched or accessed via the interactive client 104, and in some examples, no portion of the small-scale application can be accessed outside of the interactive client 104 or only a limited portion of the small-scale application can be accessed outside of the interactive client 104. The small-scale application can be launched by the interactive client 104 receiving, for example, a markup language document associated with the small-scale application from the third-party server 112 and processing such a document.

[0037] In response to determining that the external resource is a locally installed application 106, the interactive client 104 instructs the user system 102 to launch the external resource by executing the locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interactive client 104 communicates with the third-party server 112 (for example) to obtain a markup language document corresponding to the selected external resource. The interactive client 104 then processes the obtained markup language document to present the web-based external resource within the user interface of the interactive client 104.

[0038] The interactive client 104 can notify the user of the user system 102 or other users (e.g., "friends") related to such users of activities occurring in one or more external resources. For example, the interactive client 104 can provide a notification related to the current or recent use of an external resource by one or more members of a group of users to a participant in a conversation (e.g., a chat session) in the interactive client 104. One or more users can be invited to join an active external resource or to start an external resource that was recently used but is currently inactive (in the group of friends). The external resource can provide the participants in the conversation who each use the corresponding interactive client 104 with the ability to share items, conditions, states, or locations in the external resource with one or more members of a group of users in a chat session. The shared item can be an interactive chat card, and the members of the chat can interact with the interactive chat card to, for example, start the corresponding external resource, view specific information in the external resource, or bring the members of the chat to a specific location or state in the external resource. Within a given external resource, a response message can be sent to the user on the interactive client 104. The external resource can selectively include different media items in the response based on the current context of the external resource.

[0039] The interactive client 104 may present a list of available external resources (e.g., applications 106 or applets) to the user to launch or access a given external resource. The list may be presented in the form of a context-sensitive menu. For example, icons representing different applications (or applets) of the application 106 (or applets) may vary based on how the user launches the menu (e.g., from a conversational interface or from a non-conversational interface).

[0040] System Architecture

[0041] Figure 21 is a block diagram showing additional details about the interactive system 100 according to some examples. Specifically, the interactive system 100 is shown as including an interactive client 104 and an interactive server 124. The interactive system 100 includes a plurality of subsystems that are supported on the client side by the interactive client 104 and on the server side by the interactive server 124. In some examples, these subsystems are implemented as microservices. A microservice subsystem (e.g., a microservice application) can have components that enable it to operate independently and communicate with other services. Example components of a microservice subsystem can include:

[0042] Functional logic: Functional logic implements the functionality of a microservice subsystem and represents the specific capabilities or functions provided by a microservice.

[0043] API interface: Microservices can communicate with each other through a well-defined API or interface using a lightweight protocol such as REST or messaging. The API interface defines the input and output of the microservice subsystem, as well as the way the microservice subsystem interacts with other microservice subsystems of the interactive system 100.

[0044] Data storage: The microservice subsystem may be responsible for its own data storage, which may be in the form of a database, cache, or other storage mechanism (e.g., utilizing database server 126 and database 128). This enables the microservice subsystem to operate independently of other microservices of interactive system 100.

[0045] Service discovery: A microservice subsystem can find and communicate with other microservice subsystems of the interactive system 100. The service discovery mechanism enables a microservice subsystem to locate and communicate with other microservice subsystems in a scalable and efficient manner.

[0046] Monitoring and logging: Microservice subsystems may need to be monitored and logged in order to ensure availability and performance. Monitoring and logging mechanisms enable tracking the health and performance of microservice subsystems.

[0047] The following discusses the example subsystem. First, and still with reference to Figure 2 , the image processing system 202 provides various functions that enable a user to capture and enhance (eg, annotate or otherwise modify or edit) media content associated with a message.

[0048] The camera system 204 includes control software (e.g., in a camera application) that interacts with and controls the hardware camera hardware of the user system 102 (e.g., directly or via operating system control) to modify and enhance real-time images captured and displayed via the interactive client 104.

[0049] The enhancement system 206 provides functionality related to the generation and publication of enhancements (e.g., media overlays) to images captured in real time by the camera of the user system 102 or retrieved from the memory of the user system 102. For example, the enhancement system 206 is operable to select, present, and display media overlays (e.g., image filters or image lenses) to the interactive client 104 for enhancing real-time images received via the camera system 204 or stored images retrieved from the memory of the user system 102. These enhancements are selected and presented to the user of the interactive client 104 by the enhancement system 206 based on some input and data, such as:

[0050] The geolocation of the user system 102; and

[0051] Social network information of users of the user system 102 .

[0052] Enhancement can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos or videos) at user systems 102 for transmission in messages, or applied to video content such as video content streams or feeds sent from interactive clients 104. In this way, image processing system 202 can interact with various subsystems of communication system 208 and support these subsystems, such as messaging system 210 and video communication system 212.

[0053] Media overlays may include text or image data that can be overlaid on a photo taken by the user system 102 or a video stream produced by the user system 102. In some examples, the media overlay may be a location overlay (e.g., Venice Beach), the name of a live event, or a business name overlay (e.g., Beach Cafe). In other examples, the image processing system 202 uses the geolocation of the user system 102 to identify a media overlay that includes the name of a business at the geolocation of the user system 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 128 and accessed by the database server 126.

[0054] The image processing system 202 provides a user-based publishing platform that enables a user to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify situations in which a particular media overlay should be provided to other users. The image processing system 202 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0055] The augmented creation system 214 supports the augmented reality developer platform and includes applications for content creators (e.g., artists and developers) to create and publish augmentations (e.g., augmented reality experiences) for the interactive clients 104. The augmented creation system 214 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking techniques, and templates.

[0056] In some examples, the enhancement creation system 214 provides a merchant-based publishing platform that enables merchants to select specific enhancements associated with a geographic location via a bidding process. For example, the enhancement creation system 214 associates the media overlay of the highest bidding merchant with the corresponding geographic location for a predefined amount of time.

[0057] The communication system 208 is responsible for enabling and processing various forms of communication and interaction within the interactive system 100, and includes a messaging system 210, an audio communication system 216, and a video communication system 212. The messaging system 210 is responsible for enforcing temporary or time-limited access to content by the interactive clients 104. The messaging system 210 includes multiple timers (e.g., in a transient timer system) that selectively enable access (e.g., for presentation and display) of messages and associated content via the interactive clients 104 based on duration and display parameters associated with a message or a collection of messages (e.g., a story). The audio communication system 216 enables and supports audio communication (e.g., real-time audio chat) between multiple interactive clients 104. Similarly, the video communication system 212 enables and supports video communication (e.g., real-time video chat) between multiple interactive clients 104.

[0058] The user management system 218 is operationally responsible for managing user data and profiles, and includes an entity relationship system 220 that maintains information about relationships between users of the interactive system 100 .

[0059] The collection management system 222 is operationally responsible for managing a collection or collection of media (e.g., a collection of text, image video, and audio data). A collection of content (e.g., messages, including images, videos, text, and audio) can be organized into an "event gallery" or "event story". Such a collection can be made available within a specified time period (e.g., the duration of the event to which the content relates). For example, content related to a concert can be available as a "story" for the duration of the concert. The collection management system 222 can also be responsible for publishing an icon that provides a notification of a specific collection to the user interface of the interactive client 104. The collection management system 222 includes a curation function that enables collection managers to manage and curate specific content collections. For example, a curation interface enables event organizers to curate a collection of content related to a specific event (e.g., deleting inappropriate content or redundant messages). In addition, the collection management system 222 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, compensation can be paid to users for including user-generated content in a collection. In such a case, the collection management system 222 operates to automatically pay such users to use their content.

[0060] The mapping system 224 provides various geolocation functions and supports the presentation of map-based media content and messages through the interactive client 104. For example, the mapping system 224 enables the display of a user icon or avatar (e.g., stored in the profile data 802) on a map to indicate the current or past locations of the user's "friends" within the context of the map, as well as media content (e.g., a collection of messages including photos and videos) generated by these friends. For example, a message posted by a user to the interactive system 100 from a particular geographic location can be displayed to the "friends" of a particular user on the map interface of the interactive client 104 within the context of that particular location on the map. The user can also share his or her location and status information with other users of the interactive system 100 via the interactive client 104 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to the selected user within the context of the map interface of the interactive client 104.

[0061] The gaming system 226 provides various gaming functions within the context of the interactive client 104. The interactive client 104 provides a gaming interface that provides a list of available games that can be launched by a user within the context of the interactive client 104 and played with other users of the interactive system 100. The interactive system 100 also enables a particular user to invite other users to participate in playing a particular game by sending an invitation to such other users from the interactive client 104. The interactive client 104 also supports voice, video, and text messaging (e.g., chat) within the context of game play, provides leaderboards for games, and also supports the provision of in-game rewards (e.g., game coins and items).

[0062] The external resource system 228 provides an interface for the interactive client 104 to communicate with a remote server (e.g., a third-party server 112) to launch or access an external resource (i.e., an application or applet). Each third-party server 112 hosts, for example, an application or a small-scale version of an application based on a markup language (e.g., HTML5) (e.g., a game application, a utility application, a payment application, or a ride-sharing application). The interactive client 104 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from a third-party server 112 associated with a web-based resource. The application hosted by the third-party server 112 is programmed in JavaScript using a software development kit (SDK) provided by the interactive server 124. The SDK includes an application program interface (API) with functions that can be called or activated by a web-based application. The interactive server 124 hosts a JavasScript library that provides access to a given external resource to specific user data of the interactive client 104. HTML5 is an example of a technology for programming games, but applications and resources programmed based on other technologies can be used.

[0063] In order to integrate the functionality of the SDK into the web-based resource, the SDK is downloaded from the interactive server 124 by the third-party server 112, or the SDK is received by the third-party server 112 in other ways. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functions of the SDK to integrate the features of the interactive client 104 into the web-based resource.

[0064] The SDK stored on the interactive server system 110 effectively provides a bridge between external resources (e.g., applications 106 or applets) and the interactive client 104. This gives the user a seamless experience of communicating with other users on the interactive client 104 while also preserving the appearance of the interactive client 104. In order to bridge the communication between the external resources and the interactive client 104, the SDK facilitates the communication between the third-party server 112 and the interactive client 104. The WebViewJavaScriptBridge running on the user system 102 establishes two one-way communication channels between the external resources and the interactive client 104. Messages are sent asynchronously between the external resources and the interactive client 104 via these communication channels. Each SDK function activation is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with the callback identifier.

[0065] By using the SDK, not all information from the interactive client 104 is shared with the third-party server 112. The SDK limits which information is shared based on the needs of the external resource. Each third-party server 112 provides an HTML5 file corresponding to a web-based external resource to the interactive server 124. The interactive server 124 can add a visual representation (e.g., box art or other graphics) of the web-based external resource in the interactive client 104. Once the user selects the visual representation or instructs the interactive client 104 to access the features of the web-based external resource through the GUI of the interactive client 104, the interactive client 104 obtains the HTML5 file and instantiates the resource for accessing the features of the web-based external resource.

[0066] The interactive client 104 presents a graphical user interface (e.g., a login page or title screen) for an external resource. During, before, or after presenting the login page or title screen, the interactive client 104 determines whether the launched external resource has been previously authorized to access the user data of the interactive client 104. In response to determining that the launched external resource has been previously authorized to access the user data of the interactive client 104, the interactive client 104 presents another graphical user interface of the external resource including the functions and features of the external resource. In response to determining that the launched external resource has not been previously authorized to access the user data of the interactive client 104, after displaying the login page or title screen of the external resource for a threshold period of time (e.g., 3 seconds), the interactive client 104 slides up a menu for authorizing the external resource to access the user data (e.g., animating the menu to emerge from the bottom of the screen to the middle or other part of the screen). The menu identifies the type of user data that the external resource will be authorized to use. In response to receiving a user selection of the accept option, the interactive client 104 adds the external resource to the list of authorized external resources and allows the external resource to access the user data from the interactive client 104. External resources are authorized by the interactive client 104 to access user data under the OAuth 2 framework.

[0067] The interaction client 104 controls the type of user data shared with the external resource based on the type of external resource that is authorized. For example, an external resource including a full-scale application (e.g., application 106) is provided with access to a first type of user data (e.g., a two-dimensional avatar of a user with or without different avatar characteristics). As another example, an external resource including a small-scale version of an application (e.g., a web-based version of the application) is provided with access to a second type of user data (e.g., payment information, a two-dimensional avatar of a user, a three-dimensional avatar of a user, and an avatar with various avatar characteristics). Avatar characteristics include different ways to customize the appearance of an avatar (e.g., different poses, facial features, clothing, etc.).

[0068] The advertising system 230 operatively enables third parties to purchase advertisements for presentation to end users via the interactive clients 104 and also handles the delivery and presentation of these advertisements.

[0069] The various functions and features of the interactive system 100 can be combined with or use artificial intelligence (AI) and machine learning (ML). The interactive system 100 can therefore include an AI / ML system 232. For example, the AI / ML system 232 operates with the image processing system 202 and the camera system 204 to analyze the image and extract information, such as objects, text, or faces. This information can then be used by the image processing system 202 to enhance, filter, or manipulate the image. The AI / ML system 232 can be used by the enhancement system 206 to generate enhanced content and augmented reality experiences, such as adding virtual objects or animations to real-world images. The communication system 208 and the messaging system 210 can use the AI / ML system 232 to analyze communication patterns and provide insights into how users interact with each other, as well as provide intelligent message classification and tagging, such as classifying messages based on emotions or topics. The AI / ML system 232 can also provide chatbot functionality to the message interactions 120 between the user systems 102 and between the user systems 102 and the interactive server system 110. The AI / ML system 232 may also cooperate with the audio communication system 216 to provide speech recognition and natural language processing functionality, thereby enabling a user to interact with the interactive system 100 using voice commands.

[0070] In some examples, the interaction mode management system 234 forms part of the interactive system 100. The interaction mode management system 234 is responsible for detecting determinable interaction modes of users of the interactive system 100, and initiating predefined responses based on the corresponding determinable interaction modes. For example, and as further described below, the interaction mode management system 234 can work with the messaging system 210 or the collection management system 222 to identify that a user has accessed a content item multiple times, and in response thereto, cause a specific action graphical user interface element to be activated, presented, and / or highlighted to enable the user to perform an action related to the content item (or facilitate the performance of such an action). In some examples, the interaction mode management system 234 can communicate with the AI / ML system 232 to apply machine learning techniques regarding detecting or responding to determinable interaction modes related to content items within the interactive system 100.

[0071] Figure 31 is a flow chart illustrating a method 300 according to some examples for enabling a user of the interactive system 100 to perform actions (e.g., reply to other users) related to content items (e.g., images or videos) published by other users via the interactive system 100. For ease of reference, a user who views a content item and is able to perform an action is referred to as a "viewing user," and a user who is responsible for delivering (e.g., sending, sharing, or publishing) a content item for viewing by a viewing user is referred to as a "publishing user." In some examples, the method 300 is performed by various subsystems of the interactive system 100, such as the messaging system 210, the collection management system 222, or the interaction mode management system 234.

[0072] Method 300 starts at open loop element 302 and then proceeds to operation 304, where the interactive system 100 enables a first user (e.g., a viewing user) to interact with a content item transmitted by a second user (e.g., a publishing user) via the interactive system 100. Both users use the interactive client 104 to participate in the interactive system 100. The content item may include different forms of user input content, such as images, videos, text or audio content, or a combination thereof. For example, the content item may include image data stored in the image table 804 or video data stored in the video table 806. In some examples, the user input content includes enhancements, such as media overlays or filters. For example, the publishing user may capture the content item using the camera function of the interactive client 104 (and apply enhancements if desired), or may load previously captured or stored content items from a storage location (e.g., a media library of the user system 102).

[0073] The content item may be part of a collection or collection of content items posted by a posting user within the interactive system 100, or posted to a feed (e.g., a "story") managed by the collection management system 222 or posted through messaging functionality provided by the messaging system 210. In some cases, a viewing user may navigate between items in a collection by performing predefined actions, such as tap gestures in a predefined navigation area of ​​a viewing user interface. In some examples, a collection may display each content item for a predefined period of time before automatically continuing to present the next content item in the collection (unless the user provides contrary user input during the predefined period of time, such as tapping to advance to the next content item or pausing presentation at a particular content item).

[0074] At operation 306 of the method 300, an interactive client 104 executing on a user device of a viewing user (e.g., on a suitable user system 102) detects a determinable interaction pattern of the viewing user interacting with a content item (or content items related thereto) published to the interactive client 104 of the viewing user via the interactive system 100. The interaction pattern management system 234 may be configured to examine the recorded interactions of the viewing user with each content item within the interactive system 100.

[0075] The interaction pattern may be determined to include a series of access actions, such as access actions performed on a content item posted by a posting user on the interactive system 100 within a determined time (e.g., 24 hours, 48 ​​hours, or a week). For example, in the case where the content item is an image or video, the interactive client 104 may detect multiple views (e.g., re-watches or re-views) of the relevant image or video by the viewing user using the interactive client 104. In the case where the content item is an audio file, the interactive client 104 may similarly detect that the viewing user listened to (e.g., re-listened to) the audio file using the interactive client 104. For example, when the viewing user accesses the content item, the interaction pattern management system 234 may perform a check in the database 128 to determine whether the viewing user has previously accessed the particular content item. If the interaction pattern management system 234 determines that the content item has indeed been accessed before, the interaction pattern management system 234 may trigger an automatic function of the interactive system 100 to facilitate user interaction, as further described with reference to operation 308.

[0076] At operation 308, in response to detecting the identifiable interaction mode, the interaction client 104 automatically activates an action graphical user interface element. The action graphical user interface element is selectable by the user to enable the viewing user to perform an action related to the content item. For example, the interaction mode management system 234 can detect a specific identifiable interaction mode that corresponds to an identifiable interaction mode with a corresponding function recorded within the interaction system 100. The action graphical user interface element can be a reply button that is presented and activated within the viewing user interface by the messaging system 210 in response to detecting the interaction mode. In some examples, multiple action graphical user interface elements or a group of interaction elements are activated or presented within the viewing user interface. Figure 5 , Figure 6 and Figure 7 Examples of such elements are shown in and described further below.

[0077] At operation 310, the interactive client 104 may detect a selection of an action graphical user interface element within the viewing user interface. In response, the interactive client 104 may automatically invoke a function of the interactive system 100 to enable the viewing user to perform one or more specific actions related to the content item (operation 312). For example, a reply function provided by the messaging system 210 may be activated by the interactive client 104 to enable the viewing user to reply to a request regarding the content item 504 ( Figure 5 ) reply to the posting user. The messaging functionality provided by the messaging system 210 may also (or alternatively) be activated by the interactive client 104 to enable the viewing user to generate and send a message about the content item 504 to the posting user. Other actions enabled by the invoked functionality may include, for example, sharing the content item with a third-party user or platform, reacting to the content item (e.g., electronically "liking" the content item), or posting a comment about the content item. The method 300 terminates at the closed loop element 314.

[0078] Figure 4 4 is a flow chart showing a method 400 according to some examples for detecting a plurality of access actions on content items forming part of a content collection and activating an action graphical user interface element. In some examples, the method 400 is performed by various subsystems of the interactive system 100, such as the messaging system 210, the collection management system 222, or the interactive mode management system 234.

[0079] Reference Figure 5 , Figure 6 and Figure 7 Method 400 is described using the example of a viewing user interface depicted in FIG. 4 , which may be presented to a viewing user via an interactive client 104 (as an example of an interactive application). A viewing user interface such as Figure 5 , Figure 6 and Figure 7 The viewing user interface depicted in FIG. 1 may be presented by an interactive client 104 executing on a device such as a mobile device 114 .

[0080] Note that although Figure 5 , Figure 6 and Figure 7 The interface in is described and shown as being presented on a touch screen, such as the screen of the mobile device 114, but the interface according to some examples may also be presented using other types of devices that can provide a suitable user interface or display, such as an optical display of a head wearable device 116, a desktop computer, or via a smart contact lens. Therefore, the examples of the present disclosure are not limited to user interfaces that require touch-based gestures.

[0081] The method 400 begins at an open-loop element 402 and then proceeds to operation 404 where the interactive system 100 enables a first user (eg, a viewing user) to interact with a content item transmitted via the interactive system 100 by a second user (eg, a publishing user).

[0082] At operation 406, the viewing user's first access to a content item, e.g., accessing a content collection and viewing a particular item therein for the first time, is detected by the interactive system 100 (e.g., the interactive mode management system 234 or the collection management system 222) and a record of the access to the content item is stored within the interactive system 100, e.g., in a database such as the database 128 or the database 808. For example, as described below with reference to Figure 8 As described, the interaction history may be stored as part of the profile data 802 .

[0083] Then, the same viewing user makes a choice to access the same content item again at operation 408. As mentioned above, in the case where the content item is part of a collection, the viewing user can navigate back to the content item, or the viewing user can re-access the content item in a different manner, such as reopening the interactive client 104, searching for the publishing user's profile, and accessing the profile to view the same content item again.

[0084] Figure 5 A viewing user interface 502 is shown according to some examples, within which an active content item 504 is displayed for viewing by a viewing user. Here, content item 504 is a video that forms part of a collection (e.g., a "story") of content items posted by a posting user ("Mary"), who is identified by user identification information 506 within viewing user interface 502. A set indicator 516 identifies content item 504 as part of a collection that is being automatically played (or, when re-watching occurs, replayed) in sequence, and the playback (or replay) position of content item 504 is indicated by an active content item identifier 518, which is continuously updated to serve as a progress bar.

[0085] In some examples, a collection (set) of content items is published by the interactive system 100 as a feed associated with a publishing user ("Mary"), and a viewing user is able to access the feed based on relationship information maintained by the interactive system 100 indicating a relationship between the first user and the second user. For example, the interactive system 100 may store data in the entity table 812 or entity graph 814 indicating an established relationship, such as a bidirectional ("friend") relationship or a unidirectional relationship in which the viewing user follows or subscribes to a posting made by the publishing user, but not vice versa. In some examples, the interactive mode management system 234 or the collection management system 222 only allows activation of an action graphical user interface element as described herein if a predefined type of relationship (e.g., a bidirectional relationship) exists between the two users.

[0086] The interaction client 104 can detect, for example, using the interaction mode management system 234, that the viewing user is re-watching a video within the viewing user interface 502 or has made a selection to re-watch a video. In response to detecting a second viewing of the content item 504, the interaction client 104 activates an action graphical user interface element (operation 410). As will be described below, in Figure 5 , Figure 6 and Figure 7 An example of an action graphical user interface element is shown.

[0087] In examples of the present disclosure, multiple action GUI elements or action GUI elements with multiple sub-elements may be presented within a viewing user interface. Figure 5 As shown in , the action graphical user interface elements may include a camera button 508 and a text input field 510, which are activated within the viewing user interface 502 to enable the viewing user to conveniently reply to the posting user or send a chat message related to the content item 504 to the posting user.

[0088] In some examples, a quick reply tray 514 including a camera button 508 , a text input field 510 , and a send to button 512 (as described below) may be automatically presented within viewing user interface 502 when a viewing user re-watches content item 504 .

[0089] In some examples, there can be a delay between the presentation of the revisited content item 504 and the presentation of the action graphical user interface element (e.g., reply tray 514). For example, the viewing user can begin re-watching the content item 504, and the interaction mode management system 234 detects the re-watching. If the re-watching continues for a predefined period of time, such as 1 or 2 seconds, the reply tray 514 is automatically activated and presented within the viewing user interface 502. In some examples, the action graphical user interface element can thus transition into the viewing user interface 502 or appear to be animated into the viewing user interface 502 while the viewing user is re-watching or re-viewing (or, in examples that include audio, re-listening to) the related content item.

[0090] In some examples, reply tray 514 may be manually closed by viewing a user performing a predefined closing action, such as a swiping motion relative to reply tray 514 .

[0091] As described below, the camera button 508 may be selected by the user to invoke a reply state facilitated by the reply camera interface. Figure 6 ) enables the viewing user to generate the text of the message and transition to the messaging state. The messaging state may cause a keyboard or other user input element to be presented to allow the viewing user to generate and send a reply message. The text added to the text input field 510 may therefore be transmitted along with the reply to the posting user. A send to button 512, also referred to as a share button, may be selected by the user to share the content item 504, for example, with a third-party user, as a direct message with a comment to the posting user, to the viewing user's own "story," or to a third-party messaging application.

[0092] Now refer to Figure 6 In the viewing user interface 602, in some examples, the camera button 508, the chat input field 604, and the send to button 512 are not part of the reply tray and can persist within the viewing user interface 602 during the re-viewing of the content item 504. Figure 6 The camera button 508 , chat input field 604 , and send to button 512 are presented superimposed on the content item 504 .

[0093] It will be apparent from the above description that in some examples, the action graphical user interface element does not appear in the viewing user interface when the viewing user first accesses or views the content item, and only appears when the viewing user accesses or views the content item again (or another predetermined time, depending on the settings of the interaction mode management system 234). Therefore, in some examples, reply elements such as camera button 508 or text input field 510 are only presented during re-watching or re-accessing of content item 504. As such, the viewing user interface 502 may not have reply tray 514 when the viewing user first accesses the content item. Therefore, when accessing the content item for the first time, an additional selection may be required to access reply tray 514. For example, the viewing user will need to perform a swipe gesture to open reply tray 514 during the first viewing.

[0094] It will be further apparent from the above description that in some examples, an action graphical user interface element may already be present when a viewing user first accesses or views a content item and persists on subsequent views, but its activation may cause certain changes.

[0095] Activating the action graphical user interface element when the viewing user accesses or views the content item again (or at another predetermined time) can cause a display state change within the action graphical user interface element or a portion thereof. For example, during re-watching of content item 504, the action graphical user interface element can be visually highlighted (e.g., by displaying the action graphical user interface element or a portion thereof in a particular color) or otherwise placed in a particular display state.

[0096] In some examples, the action graphical user interface element may be made to appear in the viewing user interface for the first time and be visually highlighted, or undergo a display state change, upon re-viewing to emphasize the usability of the action graphical user interface element upon re-viewing.

[0097] Go to again Figure 4 Method 400 and Figure 5 In the viewing user interface 502 of FIG. 4 , at operation 412 , camera button 508 (as an example of a reply element of an action graphical user interface element) is highlighted. For example, during re-viewing of content item 504, camera button 508 may be displayed in a different color, e.g., relative to the color of camera button 508 during initial viewing of content item 504 or relative to the colors of other elements in reply tray 514.

[0098] In some examples, there is a delay between the presentation of an action graphical user interface element (e.g., a reply element) and its display state change while content item 504 is being re-watched. For example, camera button 508 may initially be presented as a white camera image relative to a gray or black background, and in response to the viewing user continuing to re-watch content item 504 for a predefined period of time (e.g., 1 or 2 seconds), interactive client 104 may automatically adjust camera button 508 so that it undergoes a display state transition, e.g., the white image changes to yellow. As another example, and as described by displaying the camera button 508 relative to a gray or black background, the interactive client 104 may automatically adjust the camera button 508 so that it undergoes a display state transition, e.g., the white image changes to yellow. Figure 6 Highlight / Bold Figure 7 The boundaries and text of the chat input field 604 are conceptually shown, and Figure 6 The chat input field 604 may be displayed in a different color or otherwise highlighted or distinguished compared to the appearance of the chat output field 604 in FIG.

[0099] In some examples, the display state of an action graphical user interface element or its sub-elements or portions may vary depending on the number of visits (e.g., views or viewings) to a particular content item. Figure 5 , as an example, the interactive client 104 may adjust the display state of an action graphical user interface element based on the number of times the content item 504 has been accessed. In some examples, the camera button 508 or other reply element may be superimposed with a graphical numerical indication of the number of times the associated content item 504 has been re-viewed, or may have a border area that gradually becomes thicker or larger each time the content item 504 is re-viewed, or may have a color that changes based on the number of times the associated content item 502 has been re-viewed. The brightness, intensity, or other attributes of the color, border, or other visual element may be upgraded based on the number of re-views.

[0100] As mentioned, in some examples, the action graphical user interface element is a reply element that can be selected by a user to invoke a reply function of the interactive client 104. Therefore, the activation or presentation of such a reply element can be designed to provide a "reply prompt" to the viewing user, thereby facilitating interaction between users within the interactive system 100 (e.g., by requiring fewer manual selections to increase the speed or convenience of interaction), and encouraging users to communicate and share creative content (e.g., through a reply camera function). In some examples, multiple reply prompts can be presented simultaneously or at different times. For example, and with reference to Figure 7 , while the chat input field 604 may undergo a display state transition as described above to provide a reply prompt, the camera button 508 may also undergo a display state transition to provide an additional reply prompt within the same viewing user interface 602. These activations or transitions need not occur simultaneously.

[0101] Back to Figure 4 At operation 414, the viewing user selects the camera button 508, which causes the interactive client 104 to invoke a reply function (operation 416), such as a modular reply camera function. The reply camera function may cause a reply interface to be presented that allows the user to generate a user input content item by taking a picture (e.g., using a front-facing camera of the mobile device 114 hosting the interactive client 104) and sending the picture to a third-party user (e.g., a posting user).

[0102] In some examples, activation of an action GUI element for one content item may be linked to other content items in the same collection. Collection management system 222 may record activation of the action GUI element and apply correlation features between all such content items. This may be done, for example, as described below with reference to Figure 8 The description is recorded in the profile data 802 , or recorded in the entity table 812 .

[0103] Refer again Figure 5 Once the reply tray 514 is presented, activated, or opened for any of the plurality of content items included in a collection of content items (e.g., a "story"), the reply tray 514 may remain open and displayed for displaying any other content items of the associated collection. Similarly, if the reply element has undergone an automatic display state change, such as a color change, the change may persist throughout the content collection. Thus, at operation 418, once triggered, the interactive system 100 may maintain activation of the action graphical user interface element for all items in the content collection.

[0104] Note, for example, that in viewing user interface 502, setting indicator 516 identifies content item 504 as part of a collection being automatically played in sequence. Since activation applies only to a particular collection of content, reply tray 514 may be closed (e.g., no longer displayed within viewing user interface 502) as the viewing user accesses and views other different collections of content items, unless the first collection of content items is also being re-watched, or unless activation of a similar action graphical user interface element has been triggered for that other collection. Method 400 ends at closed loop element 420.

[0105] Data Architecture

[0106] Figure 8 8 is a schematic diagram illustrating a data structure 800 that may be stored in a database 808 of an interactive server system 110 according to some examples. Although the contents of the database 808 are illustrated as including a plurality of tables, it will be appreciated that data may be stored in other types of data structures (e.g., object-oriented databases).

[0107] Database 808 includes message data stored in message table 810. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and payload. Fig. 9 Additional details regarding information that may be included in a message and included within the message data stored in message table 810 are described.

[0108] The entity table 812 stores entity data and is linked (e.g., referenced) to the entity graph 814 and the profile data 802. The entity for which records are maintained within the entity table 812 may include a person, a corporate entity, an organization, an object, a place, an event, etc. Regardless of the entity type, any entity about which the interactive server system 110 stores data may be an identified entity. Each entity is provided with a unique identifier and an entity type identifier (not shown).

[0109] The entity graph 814 stores information about the relationships and associations between entities. As examples only, such relationships may be social relationships, professional relationships (e.g., working in a common company or organization) based on interests or activities.

[0110] The profile data 802 stores multiple types of profile data about a particular entity. Based on the privacy settings specified by the particular entity, the profile data 802 can be selectively used and presented to other users of the interactive system 100. In the case where the entity is a person, the profile data 802 includes, for example, the user name, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user. The particular user can then selectively include one or more of these avatar representations within the content of messages transmitted via the interactive system 100 and on a map interface displayed to other users by the interactive client 104. The collection of avatar representations can include a "status avatar" that presents a graphical representation of a status or activity that the user can choose to transmit at a particular time.

[0111] Where the entity is a group, profile data 802 for the group may similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings for the relevant group (eg, notifications).

[0112] As mentioned above, for a viewing user, profile data 802 may include interaction history information about previous interactions with content items. For example, interaction mode management system 234 may utilize profile data 802 to detect a viewing user's identifiable interaction pattern with a particular content item, so that a predefined function of interaction system 100 may be invoked in response thereto.

[0113] Database 808 also stores enhancement data, such as overlays or filters, in enhancement table 816. Enhancement data is associated with and applied to videos (data for videos is stored in video table 806) and images (data for images is stored in image table 804).

[0114] In some examples, filters are overlays that are displayed as superimposed on an image or video during presentation to a recipient user. Filters can be of various types, including filters selected by a user from a set of filters presented to a sending user by the interactive client 104 when the sending user is composing a message. Other types of filters include geolocation filters (also referred to as geofilters), which can be presented to the sending user based on geolocation. For example, a geolocation filter specific to a nearby or special location can be presented by the interactive client 104 within a user interface based on geolocation information determined by a global positioning system (GPS) unit of the user system 102.

[0115] Another type of filter is a data filter, which can be selectively presented to the sending user by the interaction client 104 based on other input or information collected during the message creation process by the user system 102. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, the battery life of the user system 102, or the current time.

[0116] Other augmented data that may be stored in the image table 804 include augmented reality content items (eg, corresponding to application "lenses" or augmented reality experiences). Augmented reality content items may be real-time special effects and / or sounds that may be added to an image or video.

[0117] As described above, augmented data includes augmented reality (AR), virtual reality (VR) and mixed reality (MR) content items, overlays, image transformations, images, and modifications that can be applied to image data (e.g., videos or images). This includes real-time modifications that modify images when they are captured using the device sensors (e.g., one or more cameras) of the user system 102 and then display the images with the modifications on the screen of the user system 102. This also includes modifications to stored content (e.g., video clips that can be modified in a collection or group). For example, in a user system 102 that can access multiple augmented reality content items, a user can use a single video clip with multiple augmented reality content items to see how different augmented reality content items will modify the stored clips. Similarly, real-time video capture can use modifications to show how the video image currently captured by the sensor of the user system 102 will modify the captured data. Such data can be displayed only on the screen without being stored in the memory, or the content captured by the device sensor can be recorded and stored in the memory with or without modifications (or in both cases). In some systems, the preview function can simultaneously display how different augmented reality content items will look in different windows of the display. For example, this can enable viewing multiple windows with different pseudo-random animations on the display at the same time.

[0118] Thus, data and various systems using augmented reality content items or other such transformation systems using the data to modify content may involve detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.) in video frames, tracking such objects as they leave, enter, and move around in the field of view, and modifying or transforming such objects as they are tracked. In various examples, different methods for implementing such transformations may be used. Some examples may involve: generating a three-dimensional mesh model of one or more objects; and using transformations of the models and animated textures within a video to implement the transformations. In some examples, tracking of points on an object may be used to place an image or texture (which may be two-dimensional or three-dimensional) at the tracked location. In yet another example, neural network analysis of a video frame may be used to place an image, model, or texture in content (e.g., an image or frame of a video). Thus, augmented reality content items refer to both images, models, and textures used to create transformations in content, as well as additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0119] Real-time video processing can be performed using any kind of video data (e.g., video streams, video files, etc.) stored in the memory of any kind of computerized system. For example, a user can load a video file and save it in the memory of the device, or the device's sensors can be used to generate a video stream. In addition, any object, such as human faces and body parts, animals, or inanimate objects (e.g., chairs, cars, or other objects) can be processed using computer animation models.

[0120] In some examples, when a specific modification is selected together with the content to be transformed, the elements to be transformed are identified by a computing device, and then if the elements to be transformed are present in a video frame, the transformed elements are detected and tracked. The elements of the object are modified according to the modification request, thereby transforming the frames of the video stream. For different types of transformations, the transformation of the frames of the video stream can be performed by different methods. For example, for frame transformations that mainly involve changing the form of the elements of the object, characteristic points for each element of the object are calculated (for example, using an active shape model (ASM) or other known methods). Then, a grid based on the characteristic points is generated for each element of the object. The grid is used for subsequent stages of tracking the elements of the object in the video stream. During the tracking process, the grid for each element is aligned with the position of each element. Then, additional points are generated on the grid.

[0121] In some examples, a transformation that changes some areas of an object using the elements of an object can be performed by calculating characteristic points of each element of the object and generating a grid based on the calculated characteristic points. Points are generated on the grid, and then various areas are generated based on these points. Then, the elements of the object are tracked by aligning the area of ​​each element with the position of each element of at least one element, and the properties of the area can be modified based on the modification request, thereby transforming the frame of the video stream. Depending on the specific modification requirements, the properties of the mentioned area can be transformed in different ways. Such modifications may involve: changing the color of the area; removing part of the area from the frame of the video stream; including new objects in the area based on the modification request; and modifying or distorting the elements of the area or object. In various examples, any combination of such modifications or other similar modifications may be used. For certain models to be animated, some characteristic points can be selected as control points of the entire state space to be used to determine options for model animation.

[0122] In some examples of computer animation models for transforming image data using face detection, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). An active shape model (ASM) algorithm is then applied to the face region of the image to detect facial feature reference points.

[0123] Other methods and algorithms suitable for face detection can be used. For example, in some examples, landmarks are used to locate features, and landmarks represent distinguishable points that exist in most of the images considered. For example, for facial landmarks, the location of the left pupil can be used. If the initial landmarks are not recognizable (for example, if the person is wearing an eye patch), secondary landmarks can be used. Such a landmark recognition process can be used for any such object. In some examples, a set of landmarks forms a shape. The shape can be represented as a vector using the coordinates of the points in the shape. One shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The mean shape (meanshape) is the average value of the aligned training shapes.

[0124] The transformation system can capture an image or video stream on a client device (e.g., user system 102) and perform complex image manipulations locally on the user system 102 while maintaining an appropriate user experience, computational time, and power consumption. Complex image manipulations can include size and shape changes, emotion transformations (e.g., changing a face from a frown to a smile), state transitions (e.g., aging a subject, reducing apparent age, changing gender), style transfer, application of graphical elements, and any other suitable image or video manipulations that have been configured to be efficiently executed on the user system 102.

[0125] In some examples, a computer animation model for transforming image data may be used by a system in which a user may capture an image or video stream of the user (e.g., a selfie) using a user system 102 having a neural network operating as part of an interactive client 104 operating on the user system 102. The transformation system operating within the interactive client 104 determines the presence of a face within the image or video stream and provides a modification icon associated with the computer animation model for transforming the image data, or the computer animation model may exist in association with an interface described herein. The modification icon includes as part of the operation a change that serves as a basis for modifying the user's face within the image or video stream. Once the modification icon is selected, the transformation system initiates a process of transforming the user's image to reflect the selected modification icon (e.g., generating a smiley face on the user). Once the image or video stream is captured and the specified modification is selected, the modified image or video stream may be presented in a graphical user interface displayed on the user system 102. The transformation system may implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, a user can capture an image or video stream, and once a modification icon is selected, the modified result can be presented in real time or near real time. In addition, the modification can be persistent while the video stream is being captured, and the selected modification icon remains toggled. Machine-taught neural networks can be used to implement such modifications.

[0126] The graphical user interface presenting the modifications performed by the transformation system can provide additional interactive options to the user. Such options can be based on the interface used to initiate content capture and selection of a particular computer animation model (e.g., initiated from a content creator user interface). In various examples, the modifications can be persistent after the modification icon is initially selected. The user can switch to turn the modification on or off by tapping or otherwise selecting the face being modified by the transformation system, and store it for later viewing or browsing other areas of the imaging application. In the case of modifying multiple faces by the transformation system, the user can globally switch to turn the modification on or off by tapping or selecting a single face modified and displayed in the graphical user interface. In some examples, each face in a group of multiple faces can be modified individually, or such modifications can be switched individually by tapping or selecting a single face or a series of single faces displayed in the graphical user interface.

[0127] The collection table 818 stores data about a collection of messages and associated images, video or audio data that are compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 812). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface of the interactive client 104 may include a user-selectable icon to enable the sending user to add specific content to his or her personal story.

[0128] The collection may also constitute a "live story", which is a collection of content from multiple users created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" may constitute a curated stream of user-submitted content from different locations and events. Users whose client devices have location services enabled and who are at a co-located event at a particular time may be presented with the option of contributing content to a particular live story, for example, via the user interface of the interactive client 104. Live stories may be identified to the user by the interactive client 104 based on his or her location. The end result is a "live story" told from a group perspective.

[0129] Another type of story is referred to as a "location story" that enables users whose user system 102 is located within a specific geographic location (e.g., on a college or university campus) to contribute to a specific collection. In some examples, contributions to location stories may require secondary authentication to verify that the end user belongs to a specific organization or other entity (e.g., is a student on a university campus).

[0130] As mentioned above, the video table 806 stores video data, which in some examples is associated with messages whose records are maintained within the message table 810. Similarly, the image table 804 stores image data associated with messages whose message data is stored in the entity table 812. The entity table 812 can associate various enhancements from the enhancement table 816 with the various images and videos stored in the image table 804 and the video table 806.

[0131] Data communication architecture

[0132] Fig. 9is a schematic diagram illustrating the structure of a message 900 according to some examples, the message 900 being generated by an interactive client 104 for transmission to another interactive client 104 via an interactive server 124. The content of a particular message 900 is used to populate a message table 810 stored in a database 808 accessible by the interactive server 124. Similarly, the content of the message 900 is stored in memory as "in-transit" or "in-flight" data of the user system 102 or the interactive server 124. The message 900 is shown to include the following example components:

[0133] Message identifier 902 : a unique identifier that identifies the message 900 .

[0134] Message text payload 904 : Text to be generated by the user via the user interface of the user system 102 and included in the message 900 .

[0135] Message image payload 906 : Image data captured by the camera component of the user system 102 or retrieved from the memory component of the user system 102 and included in the message 900 . The image data for a message 900 sent or received may be stored in the image table 804 .

[0136] Message video payload 908: Video data captured by the camera component or retrieved from the memory component of the user system 102 and included in the message 900. The video data for the message 900 sent or received may be stored in the video table 806.

[0137] Message audio payload 910 : audio data captured by a microphone or retrieved from a memory component of the user system 102 and included in the message 900 .

[0138] Message enhancement data 912: Enhancement data (e.g., filters, stickers, or other annotations or enhancements) representing enhancements to be applied to the message image payload 906, message video payload 908, or message audio payload 910 of the message 900. The enhancement data for a sent or received message 900 may be stored in the enhancement table 816.

[0139] Message duration parameter 914: A parameter value indicating the amount of time in seconds that the content of the message (eg, message image payload 906, message video payload 908, message audio payload 910) will be presented to or accessible to the user via the interactive client 104.

[0140] Message geolocation parameters 916: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter 916 values ​​may be included in the payload, with each of these parameter values ​​being associated with a content item included in the content (e.g., a specific image within a message image payload 906 or a specific video in a message video payload 908).

[0141] Message collection identifier 918: An identifier value that identifies one or more content collections (e.g., "Stories" identified in collection table 818) associated with a particular content item in message image payload 906 of message 900. For example, the identifier value may be used to associate multiple images within message image payload 906 with each of multiple content collections.

[0142] Message tags 920: Each message 900 may be tagged with a plurality of tags, each of which indicates the subject of the content included in the message payload. For example, where a particular image included in the message image payload 906 depicts an animal (e.g., a lion), a tag value may be included within the message tags 920 indicating the relevant animal. The tag value may be manually generated based on user input, or may be automatically generated using, for example, image recognition.

[0143] • Message sender identifier 922: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the user system 102 on which the message 900 was generated and from which the message 900 was sent.

[0144] • Message recipient identifier 924: Indicates an identifier of the user of user system 102 to which message 900 is addressed (eg, a messaging system identifier, an email address, or a device identifier).

[0145] The content (e.g., value) of each component of message 900 may be a pointer to a location in a table where the content data value is stored. For example, the image value in message image payload 906 may be a pointer (or address) to a location within image table 804. Similarly, the value within message video payload 908 may point to data stored within video table 806, the value stored within message enhancement data may point to data stored in enhancement table 816, the value stored within message set identifier 918 may point to data stored in set table 818, and the values ​​stored within message sender identifier 922 and message recipient identifier 924 may point to user records stored within entity table 812.

[0146] System with head wearable device

[0147] Fig.10A system 1000 is shown including a head wearable device 1018 with a selector input device according to some examples. Fig.10 is a high-level functional block diagram of an example head wearable device 1018 communicatively coupled to a mobile device 1014 and various server systems 1004 (e.g., interactive server system 110) via various networks 108.

[0148] The head wearable device 1018 includes one or more cameras, each of which may be, for example, a visible light camera 1006 , an infrared transmitter 1008 , and an infrared camera 1010 .

[0149] The mobile device 1014 is connected to the head wearable device 1018 using a low power wireless connection 1012 and a high speed wireless connection 1016. The mobile device 1014 is also connected to the server system 1004 and the network 1020.

[0150] The head wearable device 1018 also includes two image displays for display of images of the optical assembly 1022. The two image displays of the optical assembly 1022 include an image display associated with a left side of the head wearable device 1018 and an image display associated with a right side of the head wearable device 1018. The head wearable device 1018 also includes an image display driver 1024, an image processor 1026, a low power circuit system 1028, and a high speed circuit system 1030. The image display of the optical assembly 1022 is used to present images and videos to a user of the head wearable device 1018, including images that may include a graphical user interface.

[0151] The image display driver 1024 commands and controls the image display of the optical component 1022. The image display driver 1024 can deliver the image data directly to the image display of the optical component 1022 for presentation or can convert the image data into a signal or data format suitable for delivery to the image display device. For example, the image data can be video data formatted according to a compression format, such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc., and the static image data can be formatted according to a compression format (e.g., Portable Network Graphics Group (PNG), Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF) or Exchangeable Image File Format (EXIF), etc.).

[0152] The head wearable device 1018 includes a frame and a stem (or temple) extending from a lateral side of the frame. The head wearable device 1018 also includes a user input device 1032 (e.g., a touch sensor or a push button) including an input surface on the head wearable device 1018. The user input device 1032 (e.g., a touch sensor or a push button) is used to receive input selections from a user for manipulating a graphical user interface of a presented image.

[0153] Fig.10 The components for the head wearable device 1018 shown in are located on one or more circuit boards (e.g., PCBs or flexible PCBs) in the frame or temples. Alternatively or additionally, the depicted components may be located in chunks, frames, hinges, or nose bridges of the head wearable device 1018. The left visible light camera 1006 and the right visible light camera 1006 may include digital camera elements, such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, camera lenses, or any other corresponding visible light or light capturing elements that can be used to capture data (including images of scenes with unknown objects).

[0154] The head wearable device 1018 includes a memory 1002 that stores instructions for performing a subset or all of the functions described herein. The memory 1002 may also include a storage device.

[0155] like Fig.10As shown, the high-speed circuit system 1030 includes a high-speed processor 1034, a memory 1002, and a high-speed wireless circuit system 1036. In some examples, the image display driver 1024 is coupled to the high-speed circuit system 1030 and is operated by the high-speed processor 1034 to drive the left image display and the right image display of the image display of the optical component 1022. The high-speed processor 1034 can be any processor capable of managing high-speed communications and operations of any general computing system required by the head wearable device 1018. The high-speed processor 1034 includes the processing resources required to manage high-speed data transmission on the high-speed wireless connection 1016 to the wireless local area network (WLAN) using the high-speed wireless circuit system 1036. In some examples, the high-speed processor 1034 executes an operating system (e.g., a LINUX operating system) or other such operating system of the head wearable device 1018, and the operating system is stored in the memory 1002 for execution. In addition to any other duties, the high-speed processor 1034 that executes the software architecture of the head wearable device 1018 is used to manage data transmission with the high-speed wireless circuit system 1036. In some examples, high-speed wireless circuit system 1036 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as WiFi. In some examples, high-speed wireless circuit system 1036 can implement other high-speed communication standards.

[0156] The low-power wireless circuit system 1038 and the high-speed wireless circuit system 1036 of the head wearable device 1018 may include a short-range transceiver (Bluetooth TM ) and a wireless wide area network transceiver, a wireless local area network transceiver, or a wide area network transceiver (e.g., cellular or WiFi). Mobile device 1014, including a transceiver that communicates via low power wireless connection 1012 and a high speed wireless connection 1016, can be implemented using details of the architecture of head wearable device 1018, as can other elements of network 1020.

[0157] The memory 1002 includes any storage device capable of storing various data and applications, including camera data generated by the left visible light camera 1006 and the right visible light camera 1006, the infrared camera 1010, and the image processor 1026, and the generated images for display by the image display driver 1024 on the image display of the optical assembly 1022, etc. Although the memory 1002 is shown as being integrated with the high-speed circuit system 1030, in some examples, the memory 1002 can be a separate independent element of the head wearable device 1018. In some such examples, electrical wiring can provide a connection from the image processor 1026 or the low-power processor 1040 to the memory 1002 through a chip including the high-speed processor 1034. In some examples, the high-speed processor 1034 can manage the addressing of the memory 1002 so that the low-power processor 1040 will start the high-speed processor 1034 at any time when a read or write operation involving the memory 1002 is required.

[0158] like Fig.10 As shown, the low-power processor 1040 or the high-speed processor 1034 of the head wearable device 1018 can be coupled to a camera device (a visible light camera device 1006, an infrared emitter 1008, or an infrared camera device 1010), an image display driver 1024, a user input device 1032 (for example, a touch sensor or a push button), and a memory 1002.

[0159] The head wearable device 1018 is connected to a host computer. For example, the head wearable device 1018 is paired with the mobile device 1014 via a high-speed wireless connection 1016 or connected to the server system 1004 via a network 1020. The server system 1004 can be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the mobile device 1014 and the head wearable device 1018 through the network 1020.

[0160] The mobile device 1014 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication via a network 1020, a low-power wireless connection 1012, or a high-speed wireless connection 1016. The mobile device 1014 may also store at least a portion of the instructions for generating binaural audio content in a memory of the mobile device 1014 to implement the functions described herein.

[0161] The output components of the head wearable device 1018 include visual components, such as a display (e.g., a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide). The image display of the optical assembly is driven by the image display driver 1024. The output components of the head wearable device 1018 also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, etc. The input components of the head wearable device 1018, the mobile device 1014, and the server system 1004 (e.g., user input device 1032) may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide the location and force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0162] The head wearable device 1018 may also include additional peripheral elements. Such peripheral elements may include biometric sensors, additional sensors, or display elements integrated with the head wearable device 1018. For example, the peripheral elements may include any I / O components, including output components, motion components, position components, or any other such components described herein.

[0163] For example, the biometric component includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biological signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The position component includes a positioning sensor component for generating positioning coordinates (e.g., a global positioning system (GPS) receiver component), a Wi-Fi or Bluetooth receiver for generating positioning system coordinates, etc. TM transceiver, altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which altitude can be obtained), orientation sensor component (e.g., a magnetometer), etc. Such positioning system coordinates can also be received from the mobile device 1014 through the low-power wireless connection 1012 and the high-speed wireless connection 1016 via the low-power wireless circuit system 1038 or the high-speed wireless circuit system 1036.

[0164] Any biometrics collected through the biometric component are captured and stored only with the user's approval and are deleted upon user request. In addition, such biometric data may be used for very limited purposes, such as identification verification. In order to ensure limited and authorized use of biometric information and other personally identifiable information (PII), access to this data is limited to authorized personnel (if access to this data occurs). Any use of biometric data may be strictly limited to identification verification purposes, and the biometric data cannot be shared or sold to any third party without the user's explicit consent. In addition, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of this sensitive information.

[0165] Machine Architecture

[0166] Fig.111100 within which instructions 1102 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 1100 to perform any one or more of the methods discussed herein. For example, the instructions 1102 may cause the machine 1100 to perform any one or more of the methods described herein. The instructions 1102 convert a general purpose, unprogrammed machine 1100 into a specific machine 1100 that is programmed to perform the functions described and illustrated in the manner described. The machine 1100 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1100 may operate as a server machine or a client machine in a server-client network environment or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1100 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 1102 specifying actions to be taken by the machine 1100. In addition, although a single machine 1100 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 1102 to perform any one or more of the methods discussed herein. For example, the machine 1100 may include a user system 102 or any of a plurality of server devices that form part of an interactive server system 110. In some examples, the machine 1100 may also include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of the particular method or algorithm are performed on the client side.

[0167] The machine 1100 may include a processor 1104, a memory 1106, and an input / output (I / O) component 1108 that may be configured to communicate with each other via a bus 1110. In an example, the processor 1104 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1112 that executes instructions 1102 and a processor 1114. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.11 Multiple processors 1104 are shown, but the machine 1100 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0168] The memory 1106 includes a main memory 1116, a static memory 1118, and a storage unit 1120, which are all accessible by the processor 1104 via the bus 1110. The main memory 1106, the static memory 1118, and the storage unit 1120 store instructions 1102 that implement any one or more of the methods or functions described herein. The instructions 1102 may also reside, completely or partially, within the main memory 1116, within the static memory 1118, within the machine-readable medium 1122 within the storage unit 1120, within at least one of the processors 1104 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 1100.

[0169] The I / O components 1108 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 1108 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It should be appreciated that the I / O components 1108 may include Fig.111108. In various examples, the I / O components 1108 may include a user output component 1124 and a user input component 1126. The user output component 1124 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The user input component 1126 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides the location and force of a touch or touch gesture), an audio input component (e.g., a microphone), etc.

[0170] In other examples, the I / O component 1108 may include a biometric component 1128, a motion component 1130, an environmental component 1132, or a position component 1134, as well as a wide range of other components. For example, the biometric component 1128 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1130 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).

[0171] Environmental components 1132 include, for example, one or more cameras (with still image / photo and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors for detecting concentrations of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0172] With respect to cameras, user system 102 may have a camera system including, for example, a front-facing camera on a front surface of user system 102 and a rear-facing camera on a rear surface of user system 102. The front-facing camera may, for example, be used to capture still images and videos of a user of user system 102 (e.g., “selfies”), which may then be enhanced with the enhancement data (e.g., filters) described above. The rear-facing camera may, for example, be used to capture still images and videos in a more conventional camera mode, which images are similarly enhanced with the enhancement data. In addition to the front-facing camera and the rear-facing camera, user system 102 may also include a 360° camera for capturing 360° photos and videos.

[0173] Additionally, the camera system of the user system 102 may include dual rear cameras (e.g., a main camera and a depth sensing camera), or even triple, quad, or quintuple rear camera configurations on the front and back sides of the user system 102. For example, these multiple camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0174] The location component 1134 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.

[0175] Various technologies may be used to implement communications. The I / O components 1108 also include a communications component 1136 that is operable to couple the machine 1100 to a network 1138 or device 1140 via a corresponding coupling or connection. For example, the communications component 1136 may include a network interface component or another suitable device that interfaces with the network 1138. In other examples, the communications component 1136 may include a wired communications component, a wireless communications component, a cellular communications component, a near field communications (NFC) component, Components (e.g. Low energy consumption), Components and other communication components for providing communication via other forms. Device 1140 can be another machine or any peripheral device among various peripheral devices (eg, a peripheral device coupled via USB).

[0176] In addition, the communication component 1136 can detect the identifier, or include a component that is operable to detect the identifier. For example, the communication component 1136 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, data symbols (Dataglyph), MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying a tagged audio signal). In addition, various information can be obtained via the communication component 1136, such as positioning via Internet Protocol (IP) geographic location, obtaining location information via Positioning is obtained through signal triangulation, positioning is obtained via detection of NFC beacon signals that can indicate a specific position, etc.

[0177] Various memories (e.g., main memory 1116, static memory 1118, and memory of processor 1104) and storage unit 1120 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1102) when executed by processor 1104 cause various operations to implement the disclosed examples.

[0178] The instructions 1102 may be sent or received via a network interface device (e.g., a network interface component included in the communication component 1136) using a transmission medium and using any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over the network 1138. Similarly, the instructions 1102 may be sent or received via a coupling (e.g., a peer-to-peer coupling) to the device 1140 using a transmission medium.

[0179] Software Architecture

[0180] Fig.12 1200 is a block diagram illustrating a software architecture 1202 that may be installed on any one or more of the devices described herein. The software architecture 1202 is supported by hardware such as a machine 1204 including a processor 1206, a memory 1208, and an I / O component 1210. In this example, the software architecture 1202 may be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1202 includes layers such as an operating system 1212, a library 1214, a framework 1216, and an application 1218. In operation, the application 1218 invokes an API call 1220 through the software stack and receives a message 1222 in response to the API call 1220.

[0181] The operating system 1212 manages hardware resources and provides public services. The operating system 1212 includes, for example, a kernel 1224, services 1226, and drivers 1228. The kernel 1224 acts as an abstraction layer between the hardware layer and other software layers. For example, the kernel 1224 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1226 can provide other public services to other software layers. Drivers 1228 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1228 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communication drivers (e.g., USB drivers), Drivers, audio drivers, power management drivers, etc.

[0182] The libraries 1214 provide a common low-level infrastructure used by the applications 1218. The libraries 1214 may include a system library 1230 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the libraries 1214 may include an API library 1232, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering in two dimensions (2D) and three dimensions (3D) in graphics content on a display), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1214 may also include various other libraries 1234 to provide many other APIs to the application 1218 .

[0183] The framework 1216 provides a common high-level infrastructure used by the applications 1218. For example, the framework 1216 provides various graphical user interface (GUI) functions, advanced resource management, and advanced positioning services. The framework 1216 can provide a wide range of other APIs that can be used by the applications 1218, some of which can be specific to a particular operating system or platform.

[0184] In an example, applications 1218 may include a home application 1236, a contacts application 1238, a browser application 1240, a book reader application 1242, a positioning application 1244, a media application 1246, a messaging application 1248, a game application 1250, and a variety of other applications such as third-party applications 1252. Applications 1218 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 1218 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 1252 (e.g., those created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the mobile phone or another mobile operating system. In this example, the third-party application 1252 can activate the API call 1220 provided by the operating system 1212 to facilitate the functions described in this article.

[0185] Example

[0186] In view of the above-mentioned implementation methods of the subject matter, the present application discloses the following list of examples, wherein one feature of a separate example or more than one feature combination of an example and optionally combined with one or more features of one or more other examples are also other examples falling within the disclosure of the present application.

[0187] Example 1 is a method comprising: detecting a identifiable interaction pattern of a first user interacting via a user system with a content item transmitted by a second user on an interactive system, the identifiable interaction pattern comprising a plurality of access actions performed by the first user on the content item; and in response to detecting the identifiable interaction pattern, automatically activating an action graphical user interface element, which action graphical user interface element can be selected by a user to enable the first user to perform an action related to the content item.

[0188] In Example 2, the subject matter of Example 1 includes, wherein detecting the plurality of access actions comprises: detecting a plurality of views of the content item by the first user using an interactive client of the interactive system.

[0189] In Example 3, the subject matter of Example 2 includes, wherein the content item includes video content, and detecting multiple views of the content item includes detecting re-viewing of the video content by the first user using the interactive client.

[0190] In Example 4, the subject matter of Examples 2-3 includes, wherein the content item includes image content, and detecting the multiple views of the content item includes detecting a re-viewing of the image content by the first user using the interactive client.

[0191] In Example 5, the subject matter of Examples 1 to 4 includes, wherein the content item includes audio content, and detecting the plurality of access actions includes detecting re-listening of the audio content by the first user using an interactive client of the interactive system.

[0192] In Example 6, the subject matter of Examples 1 to 5 includes, wherein automatically activating the action graphical user interface element comprises visually highlighting the action graphical user interface element within a context of a viewing user interface of an interactive client of the interactive system.

[0193] In Example 7, the subject matter of Examples 1 to 6 includes, wherein automatically activating the action graphical user interface element comprises causing the action graphical user interface element to undergo a display state change within the context of a viewing user interface of an interactive client of the interactive system.

[0194] In Example 8, the subject matter of Example 7 includes, wherein the display state change comprises: adjusting a color of the action graphical user interface element.

[0195] In Example 9, the subject matter of Examples 1 to 8 includes, wherein automatically activating the action graphical user interface element comprises automatically adjusting a display state of the action graphical user interface element based on a number of times the first user accesses the content item.

[0196] In Example 10, the subject matter of Examples 1 to 9 includes, wherein automatically activating the action graphical user interface element includes: causing the action graphical user interface element to be presented in association with the content item within the context of a viewing user interface of an interactive client of the interactive system, wherein the action graphical user interface element is not presented in association with the content item within the context of the viewing user interface until a determinate interaction mode is detected.

[0197] In Example 11, the subject matter of Examples 1 to 10 includes detecting a selection of an action graphical user interface element; and in response to detecting the selection, automatically invoking a function of the interactive system to enable the first user to perform an action related to the content item.

[0198] In Example 12, the subject matter of Example 11 includes, wherein the functionality is a reply functionality for enabling the first user to reply to the second user in response to the second user transmitting the content item on the interactive system.

[0199] In Example 13, the subject matter of Examples 11-12 includes, wherein the functionality is a messaging functionality for enabling a first user to send a message to a second user in response to the second user transmitting a content item on the interactive system.

[0200] In Example 14, the subject matter of Examples 1 to 13 includes, wherein the content item is part of a collection of content items published by a second user on the interactive system, and wherein automatically activating the action graphical user interface element includes automatically activating the action graphical user interface element for each content item in the collection of content items when the first user accesses the content item on the interactive system.

[0201] In Example 15, the subject matter of Example 14 includes, wherein the interactive system publishes the collection of content items as a feed associated with the second user, the first user being able to access the feed based on relationship information maintained by the interactive system indicating a relationship between the first user and the second user.

[0202] In Example 16, the subject matter of Examples 1 to 15 includes, wherein the action graphical user interface element comprises at least one of a camera button, a text input field, or a share button.

[0203] In Example 17, the subject matter of Examples 1 to 16 includes, wherein the action graphical user interface element comprises a tray of user-selectable elements presented within the context of a viewing user interface of an interactive client of the interactive system.

[0204] In Example 18, the subject matter of Examples 1 to 17 includes, wherein detecting the multiple access actions includes detecting the multiple access actions within a determined time.

[0205] Example 19 is a computing device comprising: at least one processor; and at least one memory storing instructions, wherein the instructions, when executed by the at least one processor, configure the computing device to perform operations, the operations comprising: detecting a identifiable interaction pattern in which a first user interacts with a content item transmitted by a second user on an interactive system via a user system, the identifiable interaction pattern comprising multiple access actions performed by the first user on the content item; and in response to detecting the identifiable interaction pattern, automatically activating an action graphical user interface element, which action graphical user interface element can be selected by a user to enable the first user to perform an action related to the content item.

[0206] Example 20 is a non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising: detecting a identifiable interaction pattern in which a first user interacts with a content item transmitted by a second user on an interactive system via a user system, the identifiable interaction pattern comprising multiple access actions performed by the first user on the content item; and in response to detecting the identifiable interaction pattern, automatically activating an action graphical user interface element that is selectable by a user to enable the first user to perform an action related to the content item.

[0207] Example 21 is at least one machine-readable medium comprising instructions that, when executed by a processing circuit system, cause the processing circuit system to perform operations to implement any of Examples 1-20.

[0208] Example 22 is an apparatus comprising means for implementing any one of Examples 1 to 20.

[0209] Example 23 is a system for implementing any one of Examples 1 to 20.

[0210] Example 24 is a method for implementing any one of Examples 1 to 20.

[0211] As used in this disclosure, phrases of the form "at least one of A, B, or C," "at least one of A, B, or C," "at least one of A, B, and C," etc. should be interpreted as selecting at least one from the group consisting of "A, B, and C." Unless otherwise expressly stated in conjunction with a specific example in this disclosure, this wording does not mean "at least one of A, at least one of B, and at least one of C." As used in this disclosure, the example "at least one of A, B, or C" would encompass any of the following selections: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, and {A, B, C}.

[0212] Unless the context clearly requires otherwise, throughout the specification and claims, the words "include", "comprises", etc. should be interpreted as having an inclusive meaning, rather than an exclusive or exhaustive meaning, for example, in the sense of "including but not limited to". As used herein, the term "connection", "coupling" or any variation thereof means any connection or coupling between two or more elements, directly or indirectly; the coupling or connection between elements can be physical, logical, or a combination thereof. In addition, when used in this application, the words "herein", "above", "below" and words with similar meanings refer to this application as a whole rather than any particular part of this application. Where the context permits, words using the singular or plural may also include the plural or singular, respectively. With respect to a list of two or more items, the word "or" covers all the following interpretations of the word: any item in the list, all items in the list, and any combination of items in the list. Similarly, with respect to a list of two or more items, the term "and / or" covers all the following interpretations of the word: any item in the list, all items in the list, and any combination of items in the list.

[0213] The various features, steps or processes described herein can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. In addition, certain methods or processing blocks can be omitted in some implementations.

[0214] Although some examples, such as those depicted in the accompanying drawings, include a specific order of operations, the order may be changed without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different order that does not substantially affect the functions described in the examples. In other examples, different components of the example devices or systems that implement the example methods may perform functions substantially simultaneously or in a specific order.

[0215] Glossary

[0216] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communication signals or other intangible media that facilitates the transmission of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0217] "Client device" or "user equipment" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0218] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, other type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interconnection for microwave access (WiMAX), long term evolution (LTE) standard, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0219] "Component" refers to a logic or device, a physical entity with the following boundaries, which is defined by function or subroutine calls, branch points, APIs, or other technical definitions that provide partitioning or modularization for specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed to be used with other components and a part of a program that generally performs a specific function of related functions. Components can constitute software components (e.g., codes embodied on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform certain operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine), which is uniquely customized to perform the configured function and is no longer a general-purpose processor. It will be recognized that it can be determined for cost and time considerations whether to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system to implement the hardware component. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any one time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured at different times to be respectively different special purpose processors (e.g., including different hardware components). Software configures one or more specific processors accordingly, for example, to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission between or among two or more hardware components (e.g., by appropriate circuits and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., a collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether it is temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described in this article can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some operations of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in the "cloud computing" environment or as "software as a service" (SaaS) operation. For example, at least some operations in the operation can be performed by a computer group (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations can be distributed between processors, not only resident in a single machine, but deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0220] "Computer-readable storage media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0221] "Ephemeral messages" are messages that are accessible for a limited duration. Ephemeral messages can be text, images, videos, etc. The access time for ephemeral messages can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is ephemeral.

[0222] “Machine storage media” refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include non-volatile memory, including, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage media,” “device storage media,” and “computer storage media” mean the same thing and may be used interchangeably in this disclosure. The terms “machine storage media,” “computer storage media,” and “device storage media” explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term “signal media.”

[0223] “Non-transitory computer-readable storage medium” refers to tangible media capable of storing, encoding, or carrying instructions to be executed by a machine.

[0224] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine and includes digital or analog communication signals or other intangible media that facilitate the transmission of software or data. The term "signal medium" should be deemed to include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0225] "User input content" refers to any form of content that a user can publish, publish, share, send, receive or interact with, for example, using an online platform. The user input content may, for example, include digital content in the form of one or more images, photos, video clips, audio clips, text, or a combination thereof. In some examples, the user input content item may be a UGC (user generated content) item. The user input content may, for example, have an enhancement applied thereto (e.g., a lens, filter, media overlay, or other enhancement), and may therefore be an augmented reality content item. The user input content item may be generated by a user, for example, in the case where the content item is captured by a user using a digital camera of a user device, or, for example, in the case where the content item is created by a user by applying an enhancement to another content item. The user input content item may be user selected, for example, in the case where the content item is selected from a media library or file storage, or, for example, in the case where the content item originates from a third party or a third party device. The user input content may also include content that is partially or completely generated by a computer, for example, content created by generative artificial intelligence. The content item may, for example, be content data, such as a picture, video, or audio data (or a combination thereof).

Claims

1. A method comprising: detecting a determinable interaction pattern of a first user interacting via a user system with a content item delivered by a second user on an interactive system, the determinable interaction pattern comprising a plurality of access actions performed by the first user with respect to the content item; as well as In response to detecting the determinable interaction pattern, an action graphical user interface element is automatically activated, the action graphical user interface element being selectable by a user to enable the first user to perform an action related to the content item.

2. The method according to claim 1, wherein: Detecting the multiple access actions includes: detecting multiple views of the content item by the first user using an interactive client of the interactive system.

3. The method according to claim 2, wherein: The content item includes video content, and detecting multiple views of the content item includes detecting re-viewing of the video content by the first user using the interactive client.

4. The method according to claim 2, wherein: The content item includes image content, and detecting multiple views of the content item includes detecting a re-viewing of the image content by the first user using the interactive client.

5. The method according to claim 1, wherein: The content item includes audio content, and detecting the plurality of access actions includes detecting re-listening of the audio content by the first user using an interactive client of the interactive system.

6. The method according to claim 1, wherein: Automatically activating the action graphical user interface element includes visually highlighting the action graphical user interface element within the context of a viewing user interface of an interactive client of the interactive system.

7. The method according to claim 1, wherein: Automatically activating the action graphical user interface element includes causing the action graphical user interface element to undergo a display state change within the context of a viewing user interface of an interactive client of the interactive system.

8. The method according to claim 7, wherein: The display state change includes: adjusting the color of the action graphical user interface element.

9. The method according to claim 1, wherein: Automatically activating the action graphical user interface element includes automatically adjusting a display state of the action graphical user interface element based on a number of times the first user accesses the content item.

10. The method according to claim 1, wherein: Automatically activating the action graphical user interface element includes: causing the action graphical user interface element to be presented in association with the content item within the context of a viewing user interface of an interactive client of the interactive system, wherein the action graphical user interface element is not presented in association with the content item within the context of the viewing user interface before the determinate interaction mode is detected.

11. The method according to claim 1, comprising: detecting a selection of the action graphical user interface element; as well as In response to detecting the selection, a function of the interactive system is automatically invoked to enable the first user to perform an action related to the content item.

12. The method according to claim 11, wherein: The functionality is a reply functionality for enabling the first user to reply to the second user in response to the second user transmitting the content item on the interactive system.

13. The method according to claim 11, wherein: The functionality is a messaging functionality for enabling the first user to send a message to the second user in response to the second user transmitting the content item on the interactive system.

14. The method according to claim 1, wherein: The content item is part of a collection of content items published by the second user on the interactive system, and wherein automatically activating the action graphical user interface element comprises automatically activating the action graphical user interface element for each content item in the collection of content items when the first user accesses the content item on the interactive system.

15. The method according to claim 14, wherein: The interactive system publishes the collection of content items as a feed associated with the second user, the feed being accessible to the first user based on relationship information maintained by the interactive system indicating a relationship between the first user and the second user.

16. The method according to claim 1, wherein: The action graphical user interface element includes at least one of a camera button, a text input field, or a share button.

17. The method according to claim 1, wherein: The action graphical user interface element comprises a tray of user-selectable elements presented within the context of a viewing user interface of an interactive client of the interactive system.

18. The method according to claim 1, wherein: Detecting the plurality of access actions includes detecting the plurality of access actions within a determined time.

19. A computing device comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, configure the computing device to perform operations comprising: detecting a determinable interaction pattern of a first user interacting via a user system with a content item delivered by a second user on an interactive system, the determinable interaction pattern comprising a plurality of access actions performed by the first user with respect to the content item; as well as In response to detecting the determinable interaction pattern, an action graphical user interface element is automatically activated, the action graphical user interface element being selectable by a user to enable the first user to perform an action related to the content item.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising: detecting a determinable interaction pattern of a first user interacting via a user system with a content item delivered by a second user on an interactive system, the determinable interaction pattern comprising a plurality of access actions performed by the first user with respect to the content item; as well as In response to detecting the determinable interaction pattern, an action graphical user interface element is automatically activated, the action graphical user interface element being selectable by a user to enable the first user to perform an action related to the content item.

Images