User interface providing reply state transitions

Abstract

A user input content item is presented within a viewing interface of an interactive application executing on a user device. A user input content item is shared by a sending user with a viewing user via an interactive system. A push-hold operation viewing a presentation of a user about a user input content item is detected. In response to detecting the press hold operation, the interactive application automatically transitions to a reply state. In the reply state, a reply mechanism is activated to enable the viewing user to generate a reply message for the sending user.

Description

[0001] Priority claim

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 367,659, filed on July 5, 2022, the entire contents of which are incorporated herein by reference. Background Art

[0003] The popularity of computer-implemented programs that allow users to access and interact with content and other users online continues to grow. There are various computer-implemented applications that allow users to share content with other users through interactive applications, such as messaging clients. Some computer-implemented applications may be designed to run on mobile devices, such as phones, tablets, or wearable devices, while having back-end services located on a server computer system to perform certain operations, such as operations that may require more resources than are suitable for execution on the user's device.

[0004] An interactive application (e.g., a messaging client executing on a user device) can enable a sending user to capture a photo or video and share the captured or viewed content to another device for viewing by a viewing user. The viewing user can respond to the content using a reply user interface, for example, by sending a text reply to the sending user or responding to the content shared by the sending user by capturing and sharing other photos or videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the accompanying drawings (which are not necessarily drawn to scale), similar reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, the highest digit or digits in the reference numeral refer to the figure number in which the element is first introduced. Some non-limiting examples are shown in the figures of the accompanying drawings, in which:

[0006] Figure 1 is a diagrammatic representation of a networked environment in which the present disclosure may be deployed, according to some examples.

[0007] Figure 2 is a diagrammatic representation of an interactive system having both client-side and server-side functionality according to some examples.

[0008] Figure 3 is a diagrammatic representation of data structures maintained in a database according to some examples.

[0009] Figure 4 is a user interface diagram illustrating a viewing interface according to some examples.

[0010] Figure 5is a flow chart illustrating a method of facilitating communications between users of an interactive system, according to some examples.

[0011] Figure 6 is a user interface diagram illustrating a viewing interface with a tooltip superimposed on the viewing interface according to some examples.

[0012] Figure 7 is a user interface diagram showing a viewing interface in a viewing state according to some examples.

[0013] Figure 8 is a user interface diagram showing a viewing interface in a paused state according to some examples.

[0014] Fig. 9 is a user interface diagram showing a reply interface provided in a reply state of an interactive application according to some examples.

[0015] Fig.10 is a flow chart illustrating a method of facilitating communications between users of an interactive system, according to some examples.

[0016] Fig.11 is a graphical representation of messages according to some examples.

[0017] Fig.12 A system including a head mounted device according to some examples is shown.

[0018] Fig.13 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein.

[0019] Fig.14 is a block diagram illustrating a software architecture within which examples may be implemented. DETAILED DESCRIPTION

[0020] Examples of the present disclosure describe a user interface that provides an efficient and intuitive reply state transition, thereby facilitating communication between users of an interactive system. If a user is viewing a user input content item, such as viewing another user's "story," and wishes to reply, the user can perform a press-and-hold operation to initiate a reply state transition.

[0021] In some examples, the user input content item is presented in a viewing interface of an interactive application executed on a user device. The user input content item is shared by a sending user and a viewing user. A press-and-hold operation of the viewing user regarding the presentation of the user input content item is detected. In response to detecting the press-and-hold operation, the interactive application automatically switches to a reply state. In the reply state, a reply mechanism is activated to enable the viewing user to generate a reply message to the sending user.

[0022] The reply mechanism may include a reply interface that provides multiple interface components to facilitate the creation or transmission of a reply message. In some examples, the reply mechanism enables the viewing user to address the reply message to other users of the interactive system in addition to the sending user, for example, by presenting one or more user-added cards in the reply interface.

[0023] In some examples, a press-and-hold gesture allows the user to make the pause state active. In the pause state, a release gesture can complete the transition to the reply state, while an alternative gesture (e.g., a downward swipe gesture) can achieve a transition back to the viewing state, thereby canceling the reply.

[0024] The example technical problem of providing a user interface with intuitive reply state transitions can be solved by the example system described herein that automatically initiates such transitions based on a press-hold operation. In addition, in the case of presenting content in a time-limited manner, such as as part of a "story" or video, the user interface described herein can utilize a paused state to make it easier for a user to reply to a viewed content item.

[0025] Another example technical problem of incorporating a flexible and efficient reply mechanism into a reply state transition can be solved by the example system described herein, which enables a viewing user to generate other user input content items or add other users (in addition to the sending user) to a reply without having to make selections or input instructions outside of the reply mechanism. In addition, the example system described herein detects and responds to cancel actions such as swipe gestures, providing a solution to the example technical problem of enabling a user to cancel a reply state transition.

[0026] When considering the effects in the present disclosure as a whole, one or more methods described herein can eliminate the need for certain effects or resources other than those required to be involved in executing an interactive application. The computing resources used by one or more machines, databases, or networks can be more efficiently utilized or even reduced, for example, due to automatic interface conversion, or due to a reduced number of choices required to achieve the effect. Examples of such computing resources can include processor cycles, network traffic, memory usage, graphics processing unit (GPU) resources, data storage capacity, power consumption, and cooling capabilities.

[0027] Networked computing environment

[0028] Figure 11 is a block diagram illustrating an example interactive system 100 for facilitating interactions over a network (e.g., exchanging text messages, conducting text audio and video calls, or playing games). The interactive system 100 includes multiple user systems 102, each of which hosts multiple applications including an interactive client 104 and other applications 106. Each interactive client 104 is communicatively coupled to other instances of the interactive client 104 (e.g., hosted on respective other user systems 102), an interactive server system 110, and third-party servers 112 via one or more communication networks including a network 108 (e.g., the Internet). The interactive client 104 can also communicate with the locally hosted application 106 using an application program interface (API).

[0029] Each user system 102 may include multiple user devices, such as mobile devices 114, head mounted devices 116, and computer client devices 118, which are communicatively connected to exchange data and messages. The interactive clients 104 interact with other interactive clients 104 and with the interactive server system 110 via the network 108. The data exchanged between the interactive clients 104 (e.g., interactions 120) and between the interactive clients 104 and the interactive server system 110 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0030] The interactive server system 110 provides server-side functionality to the interactive clients 104 via the network 108. Although certain functions of the interactive system 100 are described herein as being performed by the interactive clients 104 or by the interactive server system 110, whether certain functions are located within the interactive clients 104 or within the interactive server system 110 may be a design choice. For example, it may be technically preferred to initially deploy certain technologies and functions within the interactive server system 110, but to migrate the technologies and functions to the interactive clients 104 where the user systems 102 have sufficient processing power at a later stage.

[0031] The interactive server system 110 supports various services and operations provided to the interactive clients 104. Such operations include sending data to the interactive clients 104, receiving data from the interactive clients 104, and processing data generated by the interactive clients 104. The data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, entity relationship information, and live event information. The data exchange within the interactive system 100 is activated and controlled by functions available through the user interface of the interactive client 104.

[0032] Turning now specifically to the interaction server system 110, the API server 122 is coupled to and provides a programming interface to the interaction server 124, making the functionality of the interaction server 124 accessible to the interaction clients 104, other applications 106, and third-party servers 112. The interaction server 124 is communicatively coupled to a database server 126, thereby facilitating access to a database 128 that stores data associated with interactions processed by the interaction server 124. Similarly, a web server 130 is coupled to the interaction server 124 and provides a web-based interface to the interaction server 124. To this end, the web server 130 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0033] The API server 122 receives and sends interaction data (e.g., commands and message payloads) between the interaction server 124 and the user system 102 (and, for example, the interaction client 104 and other applications 106) and the third-party server 112. Specifically, the API server 122 provides a set of interfaces (e.g., routines and protocols) that the interaction client 104 and other applications 106 can call or query to activate the functions of the interaction server 124. The API server 122 exposes various functions supported by the interaction server 124, including account registration; login functionality; sending interaction data from a particular interaction client 104 to another interaction client 104 via the interaction server 124; transferring media files (e.g., images or videos) from the interaction client 104 to the interaction server 124; setting a media data collection (e.g., a story); settings related to the state of the interaction client 104 (e.g., a viewing state, a pause state, or a reply state as described below); retrieving a friend list of a user of the user system 102; retrieving messages and content; retrieving entities to be added to a "conversation"; adding and removing entities (e.g., friends) from an entity graph (e.g., an entity graph); locating friends in an entity graph; and opening application events (e.g., related to the interaction client 104).

[0034] The interactive server 124 hosts multiple systems and subsystems, as shown below. Figure 2 Give a description.

[0035] System Architecture

[0036] Figure 2 1 is a block diagram showing additional details about an interactive system 100 according to some examples. Specifically, the interactive system 100 is shown to include an interactive client 104 and an interactive server 124. The interactive system 100 includes a plurality of subsystems that are supported on the client side by the interactive client 104 and on the server side by the interactive server 124. Example subsystems are discussed below.

[0037] The image processing system 202 provides various functions that enable a user to capture and enhance (e.g., annotate or otherwise modify or edit) media content associated with a message. The camera system 204 includes control software (e.g., in a camera application) that interacts with and controls the camera hardware of the user system 102 (e.g., directly or via operating system control) to modify and enhance real-time images captured and displayed via the interactive client 104.

[0038] The enhancement system 206 provides functionality related to the generation and publication of enhancements (e.g., media overlays) to images captured in real time by the camera of the user system 102 or retrieved from the memory of the user system 102. For example, the enhancement system 206 is operable to select, present, and display media overlays (e.g., image filters or image lenses) to the interactive client 104 for enhancing real-time images received via the camera system 204 or stored images retrieved from the memory 1202 of the user system 102. These enhancements are selected and presented to the user of the interactive client 104 by the enhancement system 206 based on some input and data, such as:

[0039] The geolocation of the user system 102; and

[0040] Entity relationship information of users of the user system 102 .

[0041] Enhancement can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos or videos) at user systems 102 for transmission in messages, or applied to video content such as video content streams or feeds sent from interactive clients 104. Therefore, image processing system 202 can interact with various subsystems of communication system 208, and support various subsystems of communication system 208, such as messaging system 210 and video communication system 212.

[0042] Media overlays may include text or image data that can be superimposed on a photo taken by the user system 102 or a video stream produced by the user system 102. In some examples, the media overlay may be a location overlay (e.g., Venice Beach), the name of a live event, or a business name overlay (e.g., Beach Cafe). In other examples, the image processing system 202 uses the geolocation of the user system 102 to identify a media overlay that includes the name of a business at the geolocation of the user system 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 128 and accessed by the database server 126.

[0043] The image processing system 202 provides a user-based publishing platform that enables a user to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify situations in which a particular media overlay should be provided to other users. The image processing system 202 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0044] The augmented creation system 214 supports the augmented reality developer platform and includes applications for content creators (e.g., artists and developers) to create and publish augmentations (e.g., augmented reality experiences) for the interactive clients 104. The augmented creation system 214 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking techniques, and templates.

[0045] In some examples, enhancement creation system 214 provides a merchant-based publishing platform that enables merchants to select specific enhancements associated with geographic locations via a bidding process. For example, enhancement creation system 214 associates the highest bidding merchant's media overlay with the corresponding geographic location for a predefined amount of time.

[0046] The communication system 208 is responsible for enabling and processing various forms of communication and interaction within the interactive system 100, and includes a messaging system 210, an audio communication system 216, and a video communication system 212. The messaging system 210 may be responsible for enforcing temporary or time-limited access to content by the interactive client 104. The messaging system 210 includes a plurality of timers (e.g., in a transient timer system 218) that selectively enable access (e.g., for presentation and display) of messages and associated content via the interactive client 104 based on durations and display parameters associated with a message or a collection of messages (e.g., a story). The messaging system 210 may also be responsible for providing a reply mechanism according to the examples described herein, facilitating replies to content items shared by users of the interactive system 100. In some examples, the messaging system 210 enables a user to initiate a reply state transition, e.g., where the user interface transitions from a viewing state to a paused state, and then to a reply state (or directly from the viewing state to the reply state, or from the paused state back to the viewing state).

[0047] The audio communication system 216 enables and supports audio communication (eg, real-time audio chat) between multiple interactive clients 104. Similarly, the video communication system 212 enables and supports video communication (eg, real-time video chat) between multiple interactive clients 104.

[0048] The user management system 220 is operationally responsible for the management of user data and profiles, and includes an entity relationship system 222 that maintains information about relationships between users of the interactive system 100 .

[0049] The collection management system 224 is operationally responsible for managing a collection or collection of media (e.g., a collection of text, image video, and audio data). A collection of content (e.g., messages, including images, videos, text, and audio) can be organized into an "event library" or "event story". Such a collection can be made available within a specified time period (e.g., the duration of the event to which the content relates). For example, content related to a concert can be available as a "story" for the duration of the concert. The collection management system 224 can also be responsible for publishing an icon that provides a notification of a specific collection to the user interface of the interactive client 104. The collection management system 224 includes a curation function that enables collection managers to manage and curate specific content collections. For example, a curation interface enables event organizers to curate a collection of content related to a specific event (e.g., deleting inappropriate content or redundant messages). In addition, the collection management system 224 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, compensation can be paid to users for including user-generated content in a collection. In such a case, the collection management system 224 operates to automatically pay such users to use their content. Collection management system 224 may communicate with messaging system 210 to enable users to reply to user-input content items that form part of a content collection, for example, for a user to reply to a particular item that forms part of another user's "story."

[0050] The mapping system 226 provides various geolocation functions and supports the presentation of map-based media content and messages by the interactive client 104. For example, the mapping system 226 enables the display of a user icon or avatar (e.g., stored in the profile data 302) on a map to indicate the current or past locations of the user's "friends" within the context of the map, as well as media content generated by these friends (e.g., a collection of messages including photos and videos). For example, on the map interface of the interactive client 104, a message posted by the user to the interactive system 100 from a particular geographic location can be displayed to the "friends" of a particular user within the context of that particular location on the map. The user can also share his or her location and status information with other users of the interactive system 100 via the interactive client 104 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to the selected user within the context of the map interface of the interactive client 104.

[0051] The gaming system 228 provides various gaming functions within the context of the interactive client 104. The interactive client 104 provides a gaming interface that provides a list of available games that can be launched by a user within the context of the interactive client 104 and played with other users of the interactive system 100. The interactive system 100 also enables a particular user to invite other users to participate in playing a particular game by sending an invitation to such other users from the interactive client 104. The interactive client 104 also supports voice, video, and text messaging (e.g., chat) within the context of game play, provides leaderboards for games, and also supports the provision of in-game rewards (e.g., game coins and items).

[0052] The external resource system 230 provides an interface for the interactive client 104 to communicate with a remote server (e.g., a third-party server 112) to launch or access an external resource (i.e., an application or applet). Each third-party server 112 hosts, for example, an application or a small-scale version of an application based on a markup language (e.g., HTML5) (e.g., a game application, a utility application, a payment application, or a ride-sharing application). The interactive client 104 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from a third-party server 112 associated with a web-based resource. The application hosted by the third-party server 112 is programmed in JavaScript using a software development kit (SDK) provided by the interactive server 124. The SDK includes an application program interface (API) with functions that can be called or activated by a web-based application. The interactive server 124 hosts a JavasScript library that provides a given external resource access to specific user data of the interactive client 104. HTML5 is an example of a technology for programming games, but applications and resources programmed based on other technologies can be used.

[0053] In order to integrate the functionality of the SDK into the web-based resource, the SDK is downloaded from the interactive server 124 by the third-party server 112, or the SDK is received by the third-party server 112 in other ways. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functions of the SDK to integrate the features of the interactive client 104 into the web-based resource.

[0054] The SDK stored on the interactive server system 110 effectively provides a bridge between external resources (e.g., applications 106 or applets) and the interactive client 104. This gives the user a seamless experience of communicating with other users on the interactive client 104 while also preserving the appearance of the interactive client 104. In order to bridge the communication between the external resources and the interactive client 104, the SDK facilitates the communication between the third-party server 112 and the interactive client 104. The WebViewJavaScriptBridge running on the user system 102 establishes two one-way communication channels between the external resources and the interactive client 104. Messages are sent asynchronously between the external resources and the interactive client 104 via these communication channels. Each SDK function activation is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with the callback identifier.

[0055] By using the SDK, not all information from the interactive client 104 is shared with the third-party server 112. The SDK defines which information is shared based on the needs of the external resource. Each third-party server 112 provides an HTML5 file corresponding to a web-based external resource to the interactive server 124. The interactive server 124 can add a visual representation (e.g., a box design or other graphics) of the web-based external resource in the interactive client 104. Once the user selects the visual representation or instructs the interactive client 104 to access the features of the web-based external resource through the graphical user interface of the interactive client 104, the interactive client 104 obtains the HTML5 file and instantiates the resource for accessing the features of the web-based external resource.

[0056] The interactive client 104 presents a graphical user interface (e.g., a login page or title screen) for an external resource. During, before, or after presenting the login page or title screen, the interactive client 104 determines whether the launched external resource has been previously authorized to access the user data of the interactive client 104. In response to determining that the launched external resource has been previously authorized to access the user data of the interactive client 104, the interactive client 104 presents another graphical user interface of the external resource including the functions and features of the external resource. In response to determining that the launched external resource has not been previously authorized to access the user data of the interactive client 104, after displaying the login page or title screen of the external resource for a threshold period of time (e.g., 3 seconds), the interactive client 104 slides up a menu (e.g., animating the menu to emerge from the bottom of the screen to the middle or other part of the screen) for authorizing the external resource to access the user data. The menu identifies the type of user data that the external resource will be authorized to use. In response to receiving a user selection of the accept option, the interactive client 104 adds the external resource to the list of authorized external resources and allows the external resource to access the user data from the interactive client 104. External resources are authorized by the interactive client 104 to access user data under the OAuth 2 framework.

[0057] The interaction client 104 controls the type of user data shared with the external resource based on the type of external resource that is authorized. For example, an external resource including a full-scale application (e.g., application 106) is provided with access to a first type of user data (e.g., a two-dimensional avatar of a user with or without different avatar characteristics). As another example, an external resource including a small-scale version of an application (e.g., a web-based version of the application) is provided with access to a second type of user data (e.g., payment information, a two-dimensional avatar of a user, a three-dimensional avatar of a user, and an avatar with various avatar characteristics). Avatar characteristics include different ways to customize the appearance of an avatar (e.g., different poses, facial features, clothing, etc.).

[0058] The advertising system 232 operatively enables third parties to purchase advertisements for presentation to end users via the interactive clients 104 and also handles the delivery and presentation of these advertisements.

[0059] Data Architecture

[0060] Figure 3 is a diagram illustrating a data structure 300 that may be stored in a database 304 of an interactive server system 110 according to some examples. Although the contents of the database 304 are illustrated as including a plurality of tables, it should be understood that data may be stored in other types of data structures (e.g., object-oriented databases).

[0061] Database 304 includes message data stored in message table 306. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and payload. Fig.11 Additional details regarding information that may be included in a message and included within the message data stored in message table 306 are described.

[0062] The entity table 308 stores entity data and is linked (e.g., referenced) to the entity graph 310 and the profile data 302. The entities for which records are maintained within the entity table 308 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the entity type, any entity about which the interactive server system 110 stores data may be an identified entity. Each entity is provided with a unique identifier and an entity type identifier (not shown).

[0063] The entity graph 310 stores information about relationships and associations between entities. Such relationships may be social, professional (e.g., working in a common company or organization), interest-based, or activity-based, by way of example only.

[0064] The profile data 302 stores multiple types of profile data about a particular entity. Based on the privacy settings specified by the particular entity, the profile data 302 can be selectively used and presented to other users of the interactive system 100. In the case where the entity is a person, the profile data 302 includes, for example, the user name, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user. The particular user can then selectively include one or more of these avatar representations within the content of messages transmitted via the interactive system 100 and on a map interface displayed to other users by the interactive client 104. The collection of avatar representations can include a "status avatar" that presents a graphical representation of a status or activity that the user can choose to transmit at a particular time.

[0065] Where the entity is a group, profile data 302 for the group may similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings for the relevant group (e.g., notifications). Database 304 also stores enhancement data, such as overlays or filters, in enhancement table 312. Enhancement data is associated with and applied to videos (data for videos is stored in video table 314) and images (data for images is stored in image table 316).

[0066] In some examples, the filter is an overlay displayed as an overlay on an image or video during presentation to the recipient user. The filter can be of various types, including a user-selected filter from a set of filters that the interactive client 104 presents to the sending user when the sending user is composing a message. Other types of filters include geolocation filters (also referred to as geofilters), which can be presented to the sending user based on geolocation. For example, a geolocation filter specific to a nearby or specific location can be presented by the interactive client 104 within the user interface based on geolocation information determined by a global positioning system (GPS) unit of the user system 102.

[0067] Another type of filter is a data filter, which can be selectively presented to the sending user by the interaction client 104 based on other input or information collected during the message creation process by the user system 102. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, the battery life of the user system 102, or the current time.

[0068] Other augmented data that may be stored in the image table 316 include augmented reality content items (eg, corresponding to application "lenses" or augmented reality experiences). Augmented reality content items may be real-time special effects and sounds that may be added to an image or video.

[0069] As described above, augmented data includes augmented reality (AR), virtual reality (VR) and mixed reality (MR) content items, overlays, image transformations, images, and modifications that can be applied to image data (e.g., videos or images). This includes real-time modifications that modify images when they are captured using the device sensors (e.g., one or more cameras) of the user system 102 and then display the modified images on the screen of the user system 102. This also includes modifications to stored content (such as video clips in a collection or group that can be modified). For example, in a user system 102 that can access multiple augmented reality content items, a user can use a single video clip with multiple augmented reality content items to see how different augmented reality content items will modify the stored clips. Similarly, real-time video capture can use modifications to show how the video image currently captured by the sensor of the user system 102 will modify the captured data. Such data can be displayed only on the screen without being stored in the memory, or the content captured by the device sensor can be recorded and stored in the memory with or without modification (or both). In some systems, the preview feature can simultaneously show how different augmented reality content items will look in different windows in the display. For example, this may enable multiple windows with different pseudo-random animations to be viewed simultaneously on the display.

[0070] Thus, data and various systems using augmented reality content items or other such transformation systems that use the data to modify content may involve detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.) in video frames; tracking of such objects as they leave, enter, and move around the field of view; and modification or transformation of such objects as they are tracked. In various examples, different methods for implementing such transformations may be used. Some examples may involve: generating a three-dimensional mesh model of one or more objects; and using transformations and animated textures of the models within the video to implement the transformations. In some examples, tracking of points on the objects may be used to place an image or texture (which may be two-dimensional or three-dimensional) at the tracked location. In other examples, an image, model, or texture may be placed in content (e.g., an image or video frame) using neural network analysis of the video frame. Thus, augmented reality content items involve both: images, models, and textures used to create transformations in the content; and additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0071] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load a video file and save it in the memory of the device, or a sensor of the device can be used to generate a video stream. Additionally, computer animation models can be used to process any object, such as a human face and parts of a human body, an animal, or a non-living thing such as a chair, a car, or other objects.

[0072] In some examples, when a specific modification is selected together with the content to be transformed, the element to be transformed is identified by a computing device, and then if the element to be transformed is present in a frame of the video, it is detected and tracked. The elements of the object are modified according to the modification request, thereby transforming the frame of the video stream. For different types of transformations, the transformation of the frame of the video stream can be performed by different methods. For example, for the transformation of the frame that mainly refers to changing the form of the elements of the object, the characteristic points of each element of the object are calculated (for example, using an active shape model (ASM) or other known methods). Then, a grid based on the characteristic points is generated for each element of the object. The grid is used in a subsequent stage in which the elements of the object in the video stream are tracked. During the tracking process, the grid for each element is aligned with the position of each element. Then, additional points are generated on the grid.

[0073] In some examples, the transformation of some areas of an object using the elements of the object can be performed by calculating the characteristic points of each element of the object and generating a grid based on the calculated characteristic points. Points are generated on the grid, and then various areas based on these points are generated. Then, the elements of the object are tracked by aligning the area of ​​each element with the position of each element of at least one element, and the properties of the area can be modified based on the request for modification, thereby transforming the frame of the video stream. Depending on the specific request for modification, the properties of the mentioned area can be transformed in different ways. Such modifications may involve: changing the color of the area; removing at least part of the area from the frame of the video stream; including new objects in the area based on the modification request; and modifying or distorting the elements of the area or object. In various examples, any combination of such modifications or other similar modifications may be used. For certain models to be animated, some characteristic points may be selected as control points to be used when determining the entire state space of options for model animation.

[0074] In some examples of computer animation models for transforming image data using face detection, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). An active shape model (ASM) algorithm is then applied to the face region of the image to detect facial feature reference points.

[0075] Other methods and algorithms suitable for face detection may be used. For example, in some examples, landmarks are used to locate features, and landmarks represent distinguishable points that are present in most of the images considered. For example, for facial landmarks, the location of the left pupil can be used. If the initial landmarks are not recognizable (for example, if the person is wearing an eye patch), secondary landmarks can be used. Such a landmark identification procedure can be used for any such object. In some examples, a set of landmarks forms a shape. The shape can be represented as a vector using the coordinates of the points in the shape. One shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The average shape is the average of the aligned training shapes.

[0076] The transformation system can capture an image or video stream on a client device (e.g., user system 102) and perform complex image manipulations locally on the user system 102 while maintaining an appropriate user experience, computational time, and power consumption. Complex image manipulations can include size and shape changes, emotion transformations (e.g., changing a face from a frown to a smile), state transitions (e.g., aging a subject, reducing apparent age, changing gender), style transformations, application of graphical elements, and any other suitable image or video manipulations implemented by a convolutional neural network that has been configured to execute efficiently on the user system 102.

[0077] In some examples, a computer animation model for transforming image data may be used by a system in which a user may capture an image or video stream of the user (e.g., a selfie) using a user system 102 having a neural network operating as part of an interactive client 104 operating on the user system 102. The transformation system operating within the interactive client 104 determines the presence of a face within the image or video stream and provides a modification icon associated with the computer animation model to transform the image data, or the computer animation model may exist in association with an interface described herein. The modification icon includes changes as a basis for modifying the user's face within the image or video stream as part of the modification operation. Once the modification icon is selected, the transformation system initiates a process of transforming the user's image to reflect the selected modification icon (e.g., generating a smile on the user). Once the image or video stream is captured and the specified modification is selected, the modified image or video stream may be presented in a graphical user interface displayed on the user system 102. The transformation system may implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, a user can capture an image or video stream, and once a modification icon is selected, the modified result can be presented to the user in real time or near real time. In addition, the modification can be persistent while the video stream is being captured and the selected modification icon remains toggled. Machine-taught neural networks can be used to implement such modifications.

[0078] The graphical user interface presenting the modifications performed by the transformation system can supply additional interactive options to the user. Such options can be based on an interface for initiating content capture and selection of a particular computer animation model (e.g., initiated from a content creator user interface). In various examples, the modification can persist after the initial selection of the modification icon. The user can turn the modification on or off by tapping or otherwise selecting a face modified by the transformation system, and store it for later viewing or browsing of other areas of the imaging application. In the case where multiple faces are modified by the transformation system, the user can globally turn the modification on or off by tapping or selecting a single face modified and displayed within the graphical user interface. In some examples, a single face in a group of multiple faces can be individually modified by tapping or selecting a single face or a series of single faces displayed within the graphical user interface, or such modifications can be individually switched.

[0079] The collection table 318 stores data about a collection of messages and associated images, video or audio data that are compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 308). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface of the interactive client 104 may include a user-selectable icon to enable the sending user to add specific content to his or her personal story.

[0080] A collection may also constitute a "live story," which is a collection of content from multiple users created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" may constitute a curated stream of user-submitted content from different locations and events. Users whose client devices have location services enabled and who are at a co-located event at a particular time may be presented with the option to contribute content to a particular live story, for example, via the user interface of the interactive client 104. Live stories may be identified to the user by the interactive client 104 based on his or her location. The end result is a "live story" told from a group perspective.

[0081] Another type of content collection is called a "location story" that enables users whose user system 102 is located within a specific geographic location (e.g., on a college or university campus) to contribute to a specific collection. In some examples, contributions to location stories may require secondary authentication to verify that the end user belongs to a specific organization or other entity (e.g., is a student on a university campus).

[0082] As mentioned above, video table 314 stores video data that, in some examples, is associated with messages for which records are maintained within message table 306. Similarly, image table 316 stores image data associated with messages whose message data is stored in entity table 308. Entity table 308 may associate various enhancements from enhancement table 312 with various images and videos stored in image table 316 and video table 314.

[0083] Figure 4 4 is a user interface diagram depicting a viewing interface 402 according to some examples, which can be presented by an interaction client 104 of an interaction application executed on a mobile device 404 (as an example of a user system 102). Although the viewing interface 402 is described as being presented on a screen of the mobile device 404, the viewing interface 402 can also be presented using other types of devices that can provide a suitable user interface, such as an optical display of a head mounted device 116, a desktop computer, smart contact lenses, etc.

[0084] The viewing interface 402 displays user input content items 406, which in some examples may be part of a collection of third-party content items generated and published (or posted) by a third-party user (sending user) of the interactive system 100. Such a collection may be referred to as a "story". The viewing user may browse the collection, for example by performing an appropriate click gesture to move forward to a subsequent content item in the collection (see arrow 424) or return to a previous content item in the collection (see arrow 422). In some examples, the collection may display each content item for a predefined period of time before automatically continuing to present the next content item in the collection. The user input content items 406 may include image data, video data, audio data, or a combination of these data types, as well as enhancement or modification data (e.g., stickers, filters, or lenses).

[0085] The viewing interface 402 includes an interactive area 408 in its lower area, and the interactive area 408 includes a camera button 410, which can be selected by the user to invoke the reply camera function. The reply camera function enables the user to respond to the publication or sharing of the user input content item 406 by a third-party user. The response may include a user-generated or user-selected content item (e.g., a digital image captured using the camera of the mobile device 404 via the reply camera function). To this end, the user's selection of the camera button 410 may result in the display of the camera user interface. The camera user interface may provide a continuous camera feed from the camera of the mobile device 404 and a capture button to facilitate the capture of image or video content. In addition, the invoked reply camera function may enable the user to use enhancements to enhance the captured or selected content item.

[0086] The interactive area 408 also includes a text input field 412, into which the user can provide text to be used as or to accompany a reply transmission to the third-party user, and a share button 414, which the user can select to share the user-input content item 406, for example, as a direct message to the third-party user, to a user viewing his or her own "story", or to a third-party messaging application.

[0087] The camera button 410, the text input field 412, and the share button 414 may form part of a tray element displayed in the interactive area 408, e.g. Figure 7 In addition to the interactive area 408, the viewing interface 402 also includes an action button 416, which can be selected by the user to present a set of additional available actions to the viewing user. These actions may include, for example, marking the user input content item 406, muting the sending user, unfollowing the sending user, etc.

[0088] Sending user information 418 can be included in viewing interface 402, for example, presented as an overlay on user input content item 406. For example, information 418 can include a username and avatar of the sending user. Sending user information 418 can be user-selectable to cause display of a profile or content feed of the sending user.

[0089] The examples described herein enable a user to invoke a reply mechanism without utilizing the interaction area 408 or the action button 416. For example, as further described below, a viewing user may perform a gesture directly on the user input content item 406 in the viewing interface 402, such as performing a press-hold gesture in the gesture area 420, to initiate a reply state transition that causes the reply mechanism to activate.

[0090] Figure 5 is a flow chart illustrating a method 500 of facilitating communication between users of an interactive system (eg, interactive system 100) according to some examples. Figure 4 and Figures 6 to 9 The method 500 is described with reference to the user interface shown in FIG. Figure 1 Components of the interactive system 100 described, such as the messaging system 210 or the collection management system 224, are executed.

[0091] The method 500 starts at the start loop element 502 and proceeds to operation 504, where the sending user shares a user input content item, such as the user input content item 406, via the interactive system 100. Since the user input content item 406 has been shared by the sending user via the interactive system 100, the user input content item 406 is presented in the viewing interface 402 (operation 506). For example, the sending user can generate, curate, and publish multiple user input content items in the form of a collection or narrative (e.g., an electronic "story"), and then other users (e.g., including the viewing user) can access the multiple user input content items due to the relationship data and associated access permissions stored in the entity table 308 and the entity graph 310. In some examples, the user input content item 406 can be presented in the viewing interface 402 due to the viewing user's selection of the message sender identification 702 associated with the sending user in a previous interface of the interactive client 104.

[0092] Thus, a particular user input content item (e.g., user input content item 406) is presented in the context of a viewing interface 402 of an interactive client 104 executing on a user system 102 (e.g., a mobile device). The user input content item 406 may be a video clip, in which case the reason for presentation may include performing playback of the video clip within the viewing interface 402. The user input content item 406 may also be a static image, in which case the static image is displayed within the viewing interface. As described, the user input content item may be part of a collection, in which case the item may be displayed for a predetermined period of time before the interactive client 104 transitions to presentation of the next content item in the collection.

[0093] Figure 6 A viewing interface 402 according to some examples is shown. Figure 6 As shown in , in conjunction with the display of the user input content item 406 within the viewing interface 402, in some examples, a tooltip in the example form of a notification card 602 is displayed superimposed on the user input content item 406. The notification card 602 is displayed by the interactive client 104 in response to the user selection of the message sender identification 702, and contains text and images notifying the viewing user of a reply function or reply mechanism, which the viewing user can activate from the background of the viewing interface 402 by performing a press-and-hold operation on the user input content item 406 displayed within the viewing interface 402. The viewing user can select the OK button 604 (presenting the text: "Sounds good") to clear the notification card 602. Figure 6 The notification card 602 shown in FIG. 6 is merely an example, and other suitable tooltips, information pop-ups, etc. may be presented in other examples to provide a notification indicating that a particular operation or gesture has resulted in activation of a reply mechanism.

[0094] Figure 7 4 shows a viewing interface 402 after a notification card 602 is cleared by a viewing user according to some examples. A user input content item 406 posted by a sending user (e.g., user "Isha" as identified by message sender identification 702) is presented to the viewing user. Figure 7 In , user input content item 406 is part of a collection of content items created by the sending user.

[0095] exist Figure 7 , user input content item 406 is shown to include a still image captured by the sending user, with enhancement 706 applied to the still image. Enhancement 706 is a media overlay including a rectangular bar and the following message: "@hanvish miss you". Figure 7In the example of , the sending user (“Isha”) has therefore tagged another user of the interactive system 100 (user “Hanvish”) in the user input content item 406 by using the “@” symbol in the enhancement 706 .

[0096] The viewing user can respond to or share the user input content item 406 using the options in the interaction tray 704. Specifically, the viewing interface 402 includes a camera button 410 that is user-selectable to invoke the modular camera function of the interaction client 104 and enables the viewing user to reply to the sending user with other user input content items in the form of image data. The viewing interface 402 also includes a text input field 412 and a share button 414 as part of the interaction tray 704. And as described below, the viewing user can perform a predefined gesture to cause the interaction application to transition to a reply state, rather than using the interaction tray 704 to initiate a response.

[0097] Returning to method 500, at operation 508, the interactive client 104 detects a first operation of the viewing user of the user system 102 regarding the presentation of the user input content item 406, an example form of the first operation being a press-and-hold operation. The press-and-hold operation may include a press-and-hold gesture applied to the viewing interface 402 in which the user input content item 406 is presented. For example, a press-and-hold gesture, also known as a long press gesture, may be performed when the touch input remains pressed for a period of time longer than a normal click (typically about 500 milliseconds or another predetermined period of time).

[0098] In operation 510, in response to detecting a press-and-hold operation, the interactive client 104 changes from a presenting state or a viewing state (eg, Figure 7 In the case where the user input content item 406 is a video clip, in response to detecting that the interactive client 104 has switched to the reply state, the playback of the video clip in the viewing interface 402 is also paused. Similarly, in the case where the user input content item 406 is presented as part of a content collection (or any transient message format) in a time-based manner, such as a predefined duration, the presentation can be paused so that the user input content item 406 remains presented in the viewing interface 402. Figure 8 As shown in , in some examples, the interactive system 100 can cause the interactive tray 704 to disappear from the viewing interface 402 as part of pausing playback or presenting the viewing interface 402. Thus, for example, when a press-hold gesture is in progress, the paused state can be entered as an intermediate state between the viewing state and the reply state.

[0099] According to some examples, the reply status is as follows Fig. 9 As shown in Figure 5In operation 512, and in response to entering the reply state, the interactive client 104 activates a reply mechanism that enables the viewing user to reply to the sending user of the user input content item 406, for example, by commenting on the user input content item 406 or providing other user input content items in response to the sending user's user input content item 406.

[0100] The response mechanism includes Fig. 9 902 shown in the reply interface. The reply interface 902 includes multiple interface components, and multiple interface components are presented as being superimposed on the viewing interface 402 so that the user input content item 406 is still partially visible in the reply interface 902. These interface components conveniently enable the viewing user to write and formulate the content of the reply message. Provide a reply tray 904 including various options (e.g., camera button, text input field, emoticon or icon selector, or media library access button) for making a reply, and allow the user to type the touch keyboard 906 of the reply message. In some examples, the reply interface 902 is configured to generate or receive other user input content items (e.g., static images or video clips or text), for inclusion in the reply message and transmission to the sending user. For this reason, the reply interface provides and supports the camera function, so that the viewing user can capture image data as a part of other user input content items for transmission to the sending user.

[0101] The reply message generated from the reply interface 902 can be a one-to-one message explicitly addressed from the viewing user to the sending user, or can be a one-to-many transmission issued by the interactive system 100 to multiple users of the interactive system based on the relationships and associated permissions managed by the interactive system 100. To this end, and as Fig. 9 As shown in , when the interactive client 104 enters the reply state and activates the reply mechanism, the interactive client 104 can also present multiple user-added graphical elements in the form of user cards 908. In addition to the sending user, the user card 908 enables the viewing user to add other users of the interactive system 100 as recipients of the reply message.

[0102] By selecting one or more user cards 908, the viewing user can conveniently add these additional recipients from within the same reply interface. Fig. 9 , each user card 908 includes an identification 910 of the other user, in the exemplary form of a user name (e.g., user "Hanvish") and a profile image, and an add button 912. The add button 912 can be selected by the user to address the reply message to the other user ("Hanvish") of the interactive system 100. In operation 514 of the method 500, the interactive system 100 can therefore receive from the viewing user a selection of additional users to add to the reply transmission.

[0103] exist Fig. 9 , user cards 908 are shown in a stacked presentation. If the viewing user selects the top card from the stack of user cards 908 by selecting add button 912, the associated user ("Hanvish") is added to the reply message, the top card is removed, and the next card in the stack can appear in reply interface 902 for the viewing user to consider. Alternatively, the user can perform a swipe gesture (e.g., swipe left or right) with respect to the top card to remove the top card from reply interface 902 (and thus not select the user associated with the card), and also cause the next card to appear through interaction client 104.

[0104] The interaction client 104 may selectively present user cards 908 identifying other users of the interaction system 100 who also receive the user input content item 406 or have access to the user input content item 406 because they have subscribed to the publication of the user input content item 406 from the sending user. In some examples, if another user is specifically tagged (or mentioned, or otherwise electronically identified) in the user input content item 406 (e.g., identifying the user "Hanvish" via the enhancement 706), one of the user cards 908 may be automatically presented and populated with the details of that user, such as Fig. 9 As shown in . In this way, a group discussion about a particular user input content item 406 is facilitated.

[0105] In some examples, when the interactive client 104 is in the reply state (e.g., Fig. 9 When the viewing user exits the reply state (the state in the reply interface 902), the viewing user can conveniently exit the reply state and convert the interactive client 104 from the reply state back to the viewing state by performing a second operation, which is exemplified as a downward sliding operation. Alternatively, the viewing user can exit the reply state and return to the viewing state by selecting the cancel button 914 in the reply interface 902.

[0106] In some examples, and as described below, Fig.10 As described, the interactive client 104 can enter the pause state before entering the reply state, and can enable the user to make a choice of converting from the pause state to the reply state or converting from the pause state back to the viewing state. In such an example, a second operation such as a downward sliding operation can be available to the viewing user in the pause state rather than the reply state.

[0107] Return again Figure 5At operation 516, the interactive client 104 receives or generates additional content items (e.g., images or text) for inclusion in the reply message, and transmits the reply message to the sending user (and any other users selected from the user card 908) via the interactive system 100. The method 500 then terminates at the end loop element 518.

[0108] As described above, in some examples, the interactive client 104 may enter a pause state before entering a reply state, and may enable the user to make a choice of switching from the pause state to the reply state, or switching from the pause state back to the viewing state. To illustrate various aspects thereof, Fig.10 A flow chart of a method 1000 for facilitating communication between users of an interactive system 100 is shown according to some examples. Figure 4 and Figures 7 to 9 In some examples, method 1000 is described with reference to the user interface shown in FIG. Figure 1 The components of the interactive system 100 are described as performing.

[0109] Method 1000 begins at start loop element 1002 and proceeds to operation 1004, where the interactive client 104 detects a press-and-hold gesture by a viewing user of the interactive system 100. As described above, the press-and-hold gesture may be applied to a user input content item displayed within a viewing interface, e.g. Figure 7 A user input content item 406 is presented in the viewing interface 402 in FIG.

[0110] In response to detecting that a press-hold gesture is in progress, at operation 1006, the interactive client 104 activates the pause state. Figure 8 As described above, while in the paused state, the interactive client 104 can remove selectable interface elements, such as the interactive tray 704, from the viewing interface 402. In some examples, the interactive client 104 remains in the paused state (with presentation of the user input content item 406 paused or shown as "frozen") as long as the press-hold gesture remains in progress (e.g., as long as the user maintains the press-hold gesture), and the interactive client 104 continuously checks to see the end of the user's press-hold gesture or another gesture (operation 1008), as described below.

[0111] If the viewing user releases the press-and-hold gesture, such as by lifting the pressing finger from the display screen, the interactive system 100 detects the release gesture at operation 1010 and invokes a transition to the resume state (operation 1012). The resume state may be substantially as described above with reference to Fig. 9 The described state includes a reply interface and various reply components to facilitate communication between users of the interactive system 100.

[0112] However, the viewing user may wish to return to the viewing state rather than continuing to the reply state. For this reason, when the interactive client 104 is in the pause state (e.g., when the viewing user still presses and remains on the user input content item 406, and presents the Figure 8 ), the viewing user may perform a predefined swipe gesture, such as a downward swipe gesture. In other words, rather than ending the press-and-hold gesture by simply releasing the finger or other input element (which would result in the presentation of the reply interface), the viewing user may swipe downward while maintaining the original gesture to cancel the reply state transition. The interactive client 104 detects the downward swipe gesture at operation 1014, and in response to detecting the swipe gesture, automatically transitions the interactive client 104 from the paused state back to the viewing state (e.g., Figure 7 ). Method 1000 ends at end loop element 1018.

[0113] Data communication architecture

[0114] Fig.11 1 is a schematic diagram illustrating the structure of a message 1100 according to some examples, which message 1100 is generated by an interactive client 104 for transmission to another interactive client 104 via an interactive server 124. The content of a particular message 1100 is used to populate a message table 306 stored in a database 128 accessible by the interactive server 124. Similarly, the content of the message 1100 is stored in memory as "in-transit" or "in-flight" data of the user system 102 or the interactive server 124. The message 1100 is shown as including the following example components:

[0115] Message ID 1102 : a unique ID that identifies the message 1100 .

[0116] Message text payload 1104 : Text to be generated by the user via the user interface of the user system 102 and included in the message 1100 .

[0117] Message image payload 1106 : Image data captured by the camera component of the user system 102 or retrieved from the memory component of the user system 102 and included in the message 1100 . The image data for a message 1100 sent or received may be stored in the image table 316 .

[0118] Message video payload 1108: Video data captured by the camera component or retrieved from the memory component of the user system 102 and included in the message 1100. The video data for a message 1100 sent or received may be stored in the video table 314.

[0119] Message audio payload 1110 : audio data captured by a microphone or retrieved from a memory component of the user system 102 and included in the message 1100 .

[0120] Message enhancement data 1112: Enhancement data (e.g., filters, stickers, or other annotations or enhancements) representing enhancements to be applied to the message image payload 1106, message video payload 1108, or message audio payload 1110 of the message 1100. The enhancement data for a sent or received message 1100 may be stored in the enhancement table 312.

[0121] · Message duration parameter 1114: A parameter value indicating the amount of time in seconds that the content of the message (e.g., message image payload 1106, message video payload 1108, message audio payload 1110) is to be presented to the user via the interactive client 104 or made accessible to the user.

[0122] Message geolocation parameters 1116: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter 1116 values ​​may be included in the payload, with each of these parameter values ​​being associated with a content item included in the content (e.g., a specific image within a message image payload 1106 or a specific video in a message video payload 1108).

[0123] Message collection identification 1118: An identification value that identifies one or more content collections (e.g., "Stories" identified in the stories table 318) associated with a particular content item in the message image payload 1106 of the message 1100. For example, the identification value can be used to associate multiple images within the message image payload 1106 with multiple content collections, respectively.

[0124] Message tags 1120: Each message 1100 may be tagged with a plurality of tags, each of which indicates the subject of the content included in the message payload. For example, where a particular image included in the message image payload 1106 depicts an animal (e.g., a lion), a tag value may be included within the message tags 1120 indicating the relevant animal. The tag value may be manually generated based on user input, or may be automatically generated using, for example, image recognition.

[0125] Message sender identification 702: Indicates the identification of the user of the user system 102 on which the message 1100 was generated and from which the message 1100 was sent (eg, a messaging system identification, an email address, or a device identification).

[0126] Message recipient identification 1122: Indicates the identification of the user of the user system 102 to which the message 1100 is addressed (eg, a messaging system identification, an email address, or a device identification).

[0127] The content (e.g., value) of each component of the message 1100 may be a pointer to a location in a table where the content data value is stored. For example, the image value in the message image payload 1106 may be a pointer (or address) to a location in the image table 316. Similarly, the value in the message video payload 1108 may point to data stored in the video table 314, the value stored in the message enhancement may point to data stored in the enhancement table 312, the value stored in the message collection identifier 1118 may point to data stored in the collection table 318, and the values ​​stored in the message sender identifier 702 and the message recipient identifier 1122 may point to user records stored in the entity table 308.

[0128] System with head wearable device

[0129] Fig.12 System 1200 is shown, which includes a head wearable device 116 with a selector input device according to some examples. Fig.12 is a high-level functional block diagram of an exemplary head wearable device 116 communicatively coupled to a mobile device 114 and various server systems 1204 (e.g., interactive server system 110) via various networks 1216.

[0130] The head wearable device 116 includes one or more cameras, each of which may be, for example, at least one of a visible light camera 1206, an infrared emitter 1208, and an infrared camera 1210. The mobile device 114 is connected to the head wearable device 116 using a low power wireless connection 1212 and a high speed wireless connection 1214. The mobile device 114 is also connected to the server system 1204 and the network 1216.

[0131] The head wearable device 116 also includes two image displays in the optical assembly's image displays 1218. The two optical assembly's image displays 1218 include an image display associated with a left lateral side of the head wearable device 116 and an image display associated with a right lateral side of the head wearable device 116. The head wearable device 116 also includes an image display driver 1220, an image processor 1222, low power circuits 1224 for low power consumption, and high speed circuits 1226. The optical assembly's image displays 1218 are used to present images and videos to a user of the head wearable device 116, including images that may include a graphical user interface.

[0132] The image display driver 1220 commands and controls the image display 1218 of the optical assembly. The image display driver 1220 can deliver the image data directly to the image display 1218 of the optical assembly for presentation or can convert the image data into a signal or data format suitable for delivery to the image display device. For example, the image data can be video data formatted according to a compression format such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc., while the static image data can be formatted according to a compression format such as Portable Network Graphics Group (PNG), Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF) or Exchangeable Image File Format (EXIF), etc.

[0133] The head wearable device 116 includes a frame and stems (or temples) extending from lateral sides of the frame. The head wearable device 116 also includes a user input device 1228 (e.g., a touch sensor or a push button), including an input surface on the head wearable device 116. The user input device 1228 (e.g., a touch sensor or a push button) is used to receive input selections from a user for manipulating a graphical user interface of a presented image.

[0134] Fig.12 The components of the head wearable device 116 shown in are located on one or more circuit boards (e.g., PCBs or flexible PCBs) in the frame or temples. Alternatively or additionally, the depicted components may be located in chunks, frames, hinges, or nose bridges of the head wearable device 116. The left visible light camera and the right visible light camera 1206 may include digital camera elements, such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, camera lenses, or any other corresponding visible light or light capturing elements that can be used to capture data (including images of scenes with unknown objects).

[0135] The head wearable device 116 includes a memory 1202 that stores instructions for performing a subset or all of the functions described herein. The memory 1202 may also include a storage device.

[0136] like Fig.12As shown, the high-speed circuit 1226 includes a high-speed processor 1230, a memory 1202, and a high-speed wireless circuit 1232. In some examples, the image display driver 1220 is coupled to the high-speed circuit 1226 and is operated by the high-speed processor 1230 to drive the left image display and the right image display in the image display 1218 of the optical assembly. The high-speed processor 1230 can be any processor capable of managing high-speed communications and operations of any general computing system required by the head wearable device 116. The high-speed processor 1230 includes the processing resources required to manage high-speed data transmission to a wireless local area network (WLAN) on the high-speed wireless connection 1214 using the high-speed wireless circuit 1232. In some examples, the high-speed processor 1230 executes an operating system (e.g., a LINUX operating system) or other such operating system of the head wearable device 116, and the operating system is stored in the memory 1202 for execution. In addition to any other duties, the high-speed processor 1230 that executes the software architecture of the head wearable device 116 is used to manage data transmission with the high-speed wireless circuit 1232. In some examples, high-speed wireless circuit 1232 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as WiFi. In some examples, high-speed wireless circuit 1232 can implement other high-speed communication standards.

[0137] The low-power wireless circuitry 1234 and the high-speed wireless circuitry 1232 of the head wearable device 116 may include a short-range transceiver (Bluetooth) and a wireless wide area network transceiver, a wireless local area network transceiver, or a wide area network transceiver (e.g., cellular or WiFi). The mobile device 114, including a transceiver that communicates via the low-power wireless connection 1212 and the high-speed wireless connection 1214, may be implemented using details of the architecture of the head wearable device 116, as may other elements of the network 1216.

[0138] The memory 1202 includes any storage device capable of storing various data and applications, including camera data generated by the left and right visible light cameras 1206, the infrared camera 1210, and the image processor 1222, and the images generated to be displayed on the image display 1218 of the optical assembly through the image display driver 1220. Although the memory 1202 is shown as being integrated with the high-speed circuit 1226, in some examples, the memory 1202 can be a separate independent element of the head wearable device 116. In some such examples, electrical wiring can provide a connection from the image processor 1222 or the low-power processor 1236 to the memory 1202 through a chip including the high-speed processor 1230. In some examples, the high-speed processor 1230 can manage the addressing of the memory 1202 so that the low-power processor 1236 will start the high-speed processor 1230 at any time when a read or write operation involving the memory 1202 is required.

[0139] like Fig.12 As shown, the low-power processor 1236 or the high-speed processor 1230 of the head wearable device 116 can be coupled to a camera device (a visible light camera device 1206, an infrared emitter 1208, or an infrared camera device 1210), an image display driver 1220, a user input device 1228 (for example, a touch sensor or a push button), and a memory 1202.

[0140] The head wearable device 116 is connected to a host computer. For example, the head wearable device 116 is paired with the mobile device 114 via a high-speed wireless connection 1214 or connected to a server system 1204 via a network 1216. The server system 1204 can be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the mobile device 114 and the head wearable device 116 through the network 1216.

[0141] The mobile device 114 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication via a network 1216, a low-power wireless connection 1212, or a high-speed wireless connection 1214. The mobile device 114 may further store at least a portion of the instructions for generating binaural audio content in a memory of the mobile device 114 to implement the functions described herein.

[0142] The output components of the head wearable device 116 include visual components, such as a display (e.g., a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide). The image display of the optical component is driven by the image display driver 1220. The output components of the head wearable device 116 also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, etc. The input components of the head wearable device 116, the mobile device 114, and the server system 1204 (e.g., user input devices 1228) may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, tracking balls, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide the location and force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0143] The head wearable device 116 may also include additional peripheral elements. Such peripheral elements may include biometric sensors, additional sensors, or display elements integrated with the head wearable device 116. For example, the peripheral elements may include any I / O components, including output components, motion components, position components, or any other such components described herein.

[0144] For example, the biometric component includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measuring biological signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The position component includes a positioning sensor component for generating position coordinates (e.g., a global positioning system (GPS) receiver component), a Wi-Fi or Bluetooth receiver for generating positioning system coordinates, etc. TM transceiver, altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which altitude can be obtained), orientation sensor component (e.g., a magnetometer), etc. Such positioning system coordinates can also be received from the mobile device 114 through the low-power wireless connection 1212 and the high-speed wireless connection 1214 via the low-power wireless circuit 1234 or the high-speed wireless circuit 1232.

[0145] Any biometric data collected by the biometric component is captured and stored only with the user's consent and deleted upon user request. In addition, such biometric data may be used for very limited purposes, such as identity verification. To ensure limited and authorized use of biometric information and other personally identifiable information (PII), access to such data is limited to authorized personnel. Any use of biometric data may be strictly limited to identity verification purposes, and biometric data may not be shared with any third party or sold without the user's explicit consent. In addition, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of such sensitive information.

[0146] Machine Architecture

[0147] Fig.131300 in which instructions 1302 (e.g., software, programs, applications, applet, app, or other executable code) may be executed to cause the machine 1300 to perform any one or more of the methodologies discussed herein. For example, the instructions 1302 may cause the machine 1300 to perform any one or more of the methodologies described herein. The instructions 1302 transform a general purpose, unprogrammed machine 1300 into a specific machine 1300 that is programmed to perform the functions described and illustrated in the manner described. The machine 1300 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1300 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1300 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 1302 specifying actions to be taken by the machine 1300. In addition, although only a single machine 1300 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 1302 to perform any one or more of the methods discussed herein. For example, the machine 1300 may include any of the user system 102 or a plurality of server devices forming part of the interactive server system 110. In some examples, the machine 1300 may also include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed on the server side, and certain operations of the particular method or algorithm are performed on the client side.

[0148] The machine 1300 may include a processor 1304, a memory 1306, and an input / output I / O component 1308 that may be configured to communicate with each other via a bus 1310. In an example, the processor 1304 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1312 that executes instructions 1302 and a processor 1314. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.13 Multiple processors 1304 are shown, but the machine 1300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0149] The memory 1306 includes a main memory 1316, a static memory 1318, and a storage unit 1320, all of which are accessible by the processor 1304 via the bus 1310. The main memory 1306, the static memory 1318, and the storage unit 1320 store instructions 1302 that implement any one or more of the methods or functions described herein. The instructions 1302 may also reside, completely or partially, within the main memory 1316, within the static memory 1318, within the machine-readable medium 1322 within the storage unit 1320, within at least one of the processors 1304 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 1300.

[0150] I / O components 1308 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 1308 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will most likely not include such a touch input device. It should be appreciated that I / O components 1308 may include Fig.131308. In various examples, the I / O components 1308 may include a user output component 1324 and a user input component 1326. The user output component 1324 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The user input component 1326 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a pointing-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or another pointing instrument), a tactile input component (e.g., a physical button, a touch screen that provides the location and force of a touch or touch gesture, or other tactile input component), an audio input component (e.g., a microphone), etc.

[0151] In other examples, the I / O component 1308 may include a biometric component 1328, a motion component 1330, an environment component 1332, or a position component 1334, as well as various other components. For example, the biometric component 1328 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1330 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).

[0152] Environmental components 1322 include, for example, one or more cameras (with still image / photo and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors for detecting concentrations of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0153] With respect to cameras, user system 102 may have a camera system including, for example, a front-facing camera on a front surface of user system 102 and a rear-facing camera on a rear surface of user system 102. The front-facing camera may, for example, be used to capture still images and videos of a user of user system 102 (e.g., “selfies”), which may then be enhanced with the enhancement data (e.g., filters) described above. For example, the rear-facing camera may be used to capture still images and videos in a more conventional camera mode, which images are similarly enhanced with the enhancement data. In addition to the front-facing camera and the rear-facing camera, user system 102 may also include a 360° camera for capturing 360° photos and videos.

[0154] Additionally, the camera system of the user system 102 may include dual rear cameras (e.g., a main camera and a depth sensing camera), or even triple, quad, or quintuple rear camera configurations on the front and back sides of the user system 102. For example, these multi-camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0155] The location component 1334 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.

[0156] Various technologies may be used to achieve communication. I / O components 1308 also include communication components 1336 that are operable to couple machine 1300 to network 1338 or device 1340 via corresponding couplings or connections. For example, communication components 1336 may include a network interface component or another suitable device that interfaces with network 1338. In other examples, communication components 1336 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth components (e.g., Bluetooth low energy), Wi-Fi components, and other communication components for providing communication via other forms. Device 1340 may be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0157] In addition, the communication component 1336 can detect the identification, or include a component that is operable to detect the identification. For example, the communication component 1336 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting a one-dimensional barcode such as a universal product code (UPC) barcode, a multi-dimensional barcode such as a quick response (QR) code, an Aztec code, a data matrix, a data symbol (Dataglyph), a maximum code (MaxiCode), PDF417, UltraCode, UCC RSS-2D barcode, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1336, such as obtaining a location via Internet Protocol (IP) geolocation, obtaining a location via Wi-Fi signal triangulation, obtaining a location via detecting an NFC beacon signal that can indicate a specific location, and the like.

[0158] Various memories (e.g., main memory 1316, static memory 1318, and memory of processor 1304) and storage unit 1320 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1302) when executed by processor 1304 cause various operations to implement the disclosed examples.

[0159] Instructions 1302 may be sent or received via a network interface device (e.g., a network interface component included in communications component 1336) using a transmission medium and using any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over network 1338. Similarly, instructions 1302 may be sent or received via a coupling (e.g., a peer-to-peer coupling) with device 1340 using a transmission medium.

[0160] Software Architecture

[0161] Fig.14 1400 is a block diagram illustrating a software architecture 1402 that may be installed on any one or more of the devices described herein. The software architecture 1402 is supported by hardware such as a machine 1404, which includes a processor 1406, a memory 1408, and an I / O component 1410. In this example, the software architecture 1402 may be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1402 includes, for example, the following layers: an operating system 1412, a library 1414, a framework 1416, and an application 1418. In operation, the application 1418 invokes an API call 1420 through the software stack and receives a message 1422 in response to the API call 1420.

[0162] The operating system 1412 manages hardware resources and provides public services. The operating system 1412 includes, for example, a kernel 1424, services 1426, and drivers 1428. The kernel 1424 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 1424 provides functions such as memory management, processor management (e.g., scheduling), component management, networking, and security settings. Services 1426 can provide other public services to other software layers. Drivers 1428 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1428 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communication drivers (e.g., USB drivers), Drivers, audio drivers, power management drivers, etc.

[0163] The libraries 1414 provide a common low-level infrastructure used by the applications 1418. The libraries 1414 may include a system library 1430 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the libraries 1414 may include an API library 1432, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presentation in two dimensions (2D) and three dimensions (3D) in graphics content on a display), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1414 may also include various other libraries 1434 to provide many other APIs to the application 1418 .

[0164] The framework 1416 provides a common high-level infrastructure used by the applications 1418. For example, the framework 1416 provides various graphical user interface (GUI) functions, high-level resource management, and high-level positioning services. The framework 1416 can provide a wide range of other APIs that can be used by the applications 1418, some of which can be specific to a particular operating system or platform.

[0165] In an example, applications 1418 may include a home application 1436, a contacts application 1438, a browser application 1440, a book reader application 1442, a positioning application 1444, a media application 1446, a messaging application 1448, a game application 1450, and a variety of other applications such as third-party applications 1452. Applications 1418 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 1418 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 1452 (e.g., provided by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, third-party applications 1452 can activate API calls 1420 provided by the operating system 1412 to facilitate the functions described herein.

[0166] Example

[0167] In view of the implementation of the above-mentioned subject matter, the present application discloses the following list of examples, in which one feature of a separate example or more than one feature of an example is combined together, and optionally, combined with one or more features of one or more additional examples, is an additional example that also falls within the disclosure of the present application.

[0168] Example 1 is a system, comprising: at least one processor; and at least one memory component storing instructions, wherein the instructions, when executed by the at least one processor, configure the at least one processor to perform operations, the operations comprising: causing a user input content item to be presented within a viewing interface of an interactive application executed on a user device, the user input content item having been shared by a sending user via an interactive system; detecting a press-and-hold operation of a viewing user of the interactive system regarding the presentation of the user input content item; in response to detecting the press-and-hold operation, automatically converting the interactive application to a reply state; and in the reply state, activating a reply mechanism to enable the viewing user to generate a reply message to the sending user for the user input content item.

[0169] In Example 2, the subject matter of Example 1 includes, wherein the press-and-hold operation comprises a press-and-hold gesture applied to a viewing interface during which the user input content item is presented.

[0170] In Example 3, the subject matter of Example 2 includes, wherein converting the interactive application to the reply state includes: detecting that a press-hold gesture is in progress; and in response to detecting that the press-hold gesture is in progress, automatically converting the interactive application from the viewing state to the paused state, the interactive application remaining in the paused state until the press-hold gesture ends.

[0171] In Example 4, the subject matter of Example 3 includes that the operations further include: in the pause state, detecting a release gesture of the viewing user; and in response to detecting the release gesture, automatically transitioning the interactive application from the pause state to the resume state.

[0172] In Example 5, the subject matter of any one of Examples 3-4 includes that the operations further include: in the paused state, detecting a sliding gesture of the viewing user; and in response to detecting the sliding gesture, automatically transitioning the interactive application from the paused state back to the viewing state.

[0173] In Example 6, the subject matter of Example 5 includes, wherein the swipe gesture is a downward swipe gesture applied to a viewing interface within which the user input content item is presented.

[0174] In Example 7, the subject matter of any one of Examples 1-6 includes, wherein the user input content item is the first user input content item in a collection of user input content items, the collection being configured to present the user input content items one after another in a viewing interface, each user input content item being presented for a predetermined time period, and the collection being paused in response to detecting a press-and-hold operation such that the first user input content item remains presented in the viewing interface.

[0175] In Example 8, the subject matter of any one of Examples 1-7 includes, wherein the user input content item is a video, causing presentation of the user input content item includes executing playback of the video within a viewing interface, and pausing playback of the video within the viewing interface in response to detecting a press-and-hold operation.

[0176] In Example 9, the subject matter of any of Examples 1-8 includes, wherein the reply mechanism comprises a reply interface to receive further user input content items from the viewing user for transmission to the sending user.

[0177] In Example 10, the subject matter of Example 9 includes, wherein the reply interface includes a camera function to enable the viewing user to capture image data as part of the other user input content item for transmission to the sending user.

[0178] In Example 11, the subject matter of Example 10 includes, wherein the image data includes at least one of still image data or video image data.

[0179] In Example 12, the subject matter of any of Examples 9-11 includes, wherein the reply interface includes text input functionality to enable the viewing user to generate text data as part of the other user-input content item for transmission to the sending user.

[0180] In Example 13, the subject matter of any one of Examples 1-12 includes, wherein the reply mechanism comprises a reply interface comprising a plurality of interface components superimposed on the viewing interface.

[0181] In Example 14, the subject matter of any one of Examples 1-13 includes, wherein the reply mechanism enables the viewing user to address the reply message to other users of the interactive system in addition to the sending user of the user input content item.

[0182] In Example 15, the subject matter of Example 14 includes, wherein activating the reply mechanism includes causing presentation of a user added graphical element identifying other users of the interactive system and selectable by the user to address the reply message to the other users of the interactive system.

[0183] In Example 16, the subject matter of Example 15 includes, wherein the user-added graphical element includes an identification of the other user and an add button selectable by the user to address the reply message to the other user of the interactive system.

[0184] In Example 17, the subject matter of any one of Examples 15-16 includes, wherein presentation of the user-added graphic element comprises: detecting that other users of the interactive system are marked in a user input content item of the sending user; and in response to detecting that other users of the interactive system are marked in the user input content item of the sending user, causing the user-added graphic element to be presented within the reply interface.

[0185] In Example 18, the subject matter of any one of Examples 1-17 includes, wherein the reply mechanism enables the viewing user to generate other user input content items and include the other user input content items in the reply message, and the operation further includes: transmitting the reply message to the sending user via the interactive system.

[0186] Example 19 is a method, comprising: causing a user input content item to be presented within a viewing interface of an interactive application executed on a user device, the user input content item having been shared by a sending user via an interactive system; detecting a press-and-hold operation of a viewing user of the interactive system regarding the presentation of the user input content item; in response to detecting the press-and-hold operation, automatically converting the interactive application into a reply state; and in the reply state, activating a reply mechanism to enable the viewing user to generate a reply message to the sending user for the user input content item.

[0187] Example 20 is a non-transitory computer-readable storage medium, which includes instructions that, when executed by at least one computer, cause the at least one computer to perform operations, the operations including: causing a user input content item to be presented within a viewing interface of an interactive application executed on a user device, the user input content item having been shared by a sending user via an interactive system; detecting a press-and-hold operation of a viewing user of the interactive system regarding the presentation of the user input content item; in response to detecting the press-and-hold operation, automatically converting the interactive application to a reply state; and in the reply state, activating a reply mechanism to enable the viewing user to generate a reply message to the sending user for the user input content item.

[0188] Example 21 is at least one machine-readable medium comprising instructions that, when executed by a processing circuit, cause the processing circuit to perform operations to implement any of Examples 1-20.

[0189] Example 22 is an apparatus comprising an apparatus for implementing any one of Examples 1-20.

[0190] Example 23 is a system that implements any one of Examples 1-20.

[0191] Example 24 is a method that implements any one of Examples 1-20.

[0192] in conclusion

[0193] In the examples described in the present disclosure, gestures such as a press-hold gesture or a slide gesture are touch gestures for a (touch-type) screen of a user device. It should be understood that although the examples described herein illustrate gesture-based interface transitions with reference to touch gestures, other types of operations or gestures that do not involve directly touching a screen or other components or devices can be used to apply the techniques described herein. The example gestures can be performed using any suitable user input element (e.g., a finger, a stylus, or other input device).

[0194] As used in this disclosure, phrases of the form "at least one of one A, one B, or one C", "at least one of A, B, or C", "at least one of A, B, and C", etc. should be interpreted as selecting at least one from the group including "A, B, and C". In this disclosure, unless otherwise explicitly stated in conjunction with a specific example, this wording does not mean "at least one of A, at least one of B, and at least one of C". As used in this disclosure, the example "at least one of one A, one B, or one C" will cover any of the following selections: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, and {A, B, C}.

[0195] Unless the context clearly requires otherwise, throughout the specification and claims, the words "include", "comprises", etc. should be interpreted as inclusive rather than exclusive or exhaustive; that is, "including but not limited to". As used herein, the term "connect", "couple", or any variation thereof means any direct or indirect connection or connection between two or more elements; the connection or connection between elements can be physical, logical, or a combination thereof. In addition, when used in this application, the words "herein", "above", "below" and words of similar meaning refer to the application as a whole, rather than to any particular part of the application. Where the context permits, words using the singular or plural may also include the plural or singular, respectively. When referencing a list of two or more items, the word "or" covers all of the following interpretations of the word: any item in the list, all of the items in the list, and any combination of the items in the list. Similarly, when referencing a list of two or more items, the term "and / or" covers all of the following interpretations of the word: any item in the list, all of the items in the list, and any combination of the items in the list.

[0196] Although some examples (such as the examples depicted in the accompanying drawings) include a specific order of operations, the order may be changed without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different order that does not substantially affect the functions described in the examples. In other examples, different components of the example devices or systems that implement the example methods may perform functions substantially simultaneously or in a specific order.

[0197] Glossary

[0198] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media that facilitates communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0199] "Client device" or "user equipment" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0200] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, other type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0201] "Component" refers to a logic or device, a physical entity with the following boundaries, which is defined by a function or subroutine call, a branch point, an API, or other technical definitions that provide partitioning or modularization for a specific processing or control function. A component can be combined with other components via its interface to perform machine processing. A component can be a packaged functional hardware unit designed for use with other components, and a part of a program that generally performs a specific function of a related function. A component can constitute a software component (e.g., a code implemented on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., an independent computer system, a client computer system, or a server computer system) or one or more hardware components (e.g., a processor or a processor group) of a computer system can be configured by software (e.g., an application or an application part) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit that is temporarily configured to perform certain operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine), which is uniquely customized to perform the function of the configuration and is no longer a general-purpose processor. It will be recognized that it can be determined whether to mechanically implement the hardware component in a dedicated and permanently configured circuit or in a temporarily configured (e.g., configured by software) circuit for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any one time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. The software configures a specific one or more processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission between or among two or more hardware components (e.g., by appropriate circuits and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., a collection of information). The various operations of the example methods described in this article can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether it is temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described in this article can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some of the various operations of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or as a "software as a service" (SaaS) operation. For example, at least some operations in the operation can be performed by a computer group (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations can be distributed between processors, not only resident in a single machine, but deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographic location (for example, in a home environment, an office environment, or a server cluster). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographic locations.

[0202] "Computer-readable storage media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier waves / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0203] "Ephemeral messages" are messages that are accessible for a limited duration. Ephemeral messages can be text, images, videos, etc. The access time for ephemeral messages can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is ephemeral.

[0204] “Machine storage media” refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage media,” “device storage media,” and “computer storage media” mean the same thing and may be used interchangeably in this disclosure. The terms “machine storage media,” “computer storage media,” and “device storage media” explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term “signal media.”

[0205] “Non-transitory computer-readable storage medium” refers to tangible media that is capable of storing, encoding, or carrying instructions that are executed by a machine.

[0206] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0207] "User input content" refers to any form of content that a user can publish, post, share, send, receive or interact with, for example, using an online platform. User input content may include, for example, one or more images, photos, video clips, audio clips, text, or digital content in the form of a combination thereof. In some examples, a user input content item may be a UGC (user generated content) item. For example, enhancements (e.g., lenses, filters, media overlays, or other enhancements) may be applied to the user input content. A user input content item may be generated by a user, for example, where the content item is captured by a user using a digital camera of a user device, or, for example, where the content item is created by a user by applying an enhancement to another content item. A user input content item may be user selectable, for example, where the content item is selected from a media library or file storage, or, for example, from a third party or third party device. User input content may also partially or completely include computer-generated content, such as content created by generating artificial intelligence. A content item may, for example, be content data, such as a picture, video, or audio data (or a combination thereof).

Claims

1. A system comprising: at least one processor; as well as at least one memory component storing instructions that, when executed by the at least one processor, configure the at least one processor to perform operations comprising: causing a user input content item to be presented within a viewing interface of an interactive application executed on a user device, the user input content item having been shared by a sending user via an interactive system; detecting a press-and-hold operation of a viewing user of the interactive system with respect to the presentation of the user input content item; In response to detecting the press-and-hold operation, automatically converting the interactive application into a reply state; and In the reply state, a reply mechanism is activated to enable the viewing user to generate a reply message to the sending user for the user-input content item.

2. The system according to claim 1, wherein: The press-and-hold operation comprises a press-and-hold gesture applied to the viewing interface within which the user input content item is presented.

3. The system according to claim 2, wherein: Converting the interactive application into the reply state comprises: detecting that the press-and-hold gesture is in progress; and In response to detecting that the press-and-hold gesture is in progress, the interactive application is automatically transitioned from a viewing state to a paused state, and the interactive application remains in the paused state until the press-and-hold gesture ends.

4. The system of claim 3, wherein the operations further comprise: In the pause state, detecting a release gesture of the viewing user; as well as In response to detecting the release gesture, automatically transitioning the interactive application from the pause state to the resume state.

5. The system of claim 3, wherein the operations further comprise: In the pause state, detecting a sliding gesture of the viewing user; as well as In response to detecting the slide gesture, automatically transitioning the interactive application from the paused state back to the viewing state.

6. The system according to claim 5, wherein: The swipe gesture is a downward swipe gesture applied to the viewing interface within which the user input content item is presented.

7. The system according to claim 1, wherein: The user input content item is the first user input content item in a collection of user input content items, the collection being configured to present the user input content items one after another in the viewing interface, each user input content item being presented for a predetermined time period, and the collection being paused in response to detecting the press-and-hold operation so that the first user input content item remains presented in the viewing interface.

8. The system according to claim 1, wherein: The user input content item is a video, causing the user input content item to be presented includes executing playback of the video in the viewing interface, and pausing the playback of the video in the viewing interface in response to detecting the press-and-hold operation.

9. The system according to claim 1, wherein: The reply mechanism includes a reply interface for receiving other user input content items from the viewing user for transmission to the sending user.

10. The system according to claim 9, wherein: The reply interface includes a camera function to enable the viewing user to capture image data as part of the other user input content item for transmission to the sending user.

11. The system according to claim 10, wherein: The image data includes at least one of static image data or video image data.

12. The system according to claim 9, wherein: The reply interface includes text input functionality to enable the viewing user to generate text data as part of the other user input content item for transmission to the sending user.

13. The system of claim 1, wherein: The reply mechanism includes a reply interface, and the reply interface includes a plurality of interface components superimposed on the viewing interface.

14. The system of claim 1, wherein: The reply mechanism enables the viewing user to address the reply message to other users of the interactive system in addition to the sending user of the user input content item.

15. The system of claim 14, wherein: Activating the reply mechanism includes: A user added graphical element is caused to be presented, the user added graphical element identifying the other user of the interactive system and selectable by a user to address the reply message to the other user of the interactive system.

16. The system of claim 15, wherein: The user-added graphical element comprises an identification of the other user and an add button selectable by a user to address the reply message to the other user of the interactive system.

17. The system of claim 15, wherein: The presenting of the user added graphical element comprises: detecting that the other user of the interactive system is marked in the user input content item of the sending user; and In response to detecting that the other user of the interactive system is marked in the user input content item of the sending user, the user-added graphical element is caused to be presented in a reply interface.

18. The system of claim 1, wherein: The reply mechanism enables the viewing user to generate other user input content items and include the other user input content items in the reply message, and the operation further includes: The reply message is transmitted to the sending user via the interactive system.

19. A method comprising: causing a user input content item to be presented within a viewing interface of an interactive application executed on a user device, the user input content item having been shared by a sending user via an interactive system; detecting a press-and-hold operation of a viewing user of the interactive system with respect to the presentation of the user input content item; In response to detecting the press-and-hold operation, automatically converting the interactive application into a reply state; as well as In the reply state, a reply mechanism is activated to enable the viewing user to generate a reply message to the sending user for the user-input content item.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by at least one computer, cause the at least one computer to perform operations comprising: causing a user input content item to be presented within a viewing interface of an interactive application executed on a user device, the user input content item having been shared by a sending user via an interactive system; detecting a press-and-hold operation of a viewing user of the interactive system with respect to the presentation of the user input content item; In response to detecting the press-and-hold operation, automatically converting the interactive application into a reply state; as well as In the reply state, a reply mechanism is activated to enable the viewing user to generate a reply message to the sending user for the user-input content item.

Images