Combining the first user interface content into the second user interface

Abstract

Aspects of the present disclosure relate to a system comprising a computer-readable storage medium storing a program and method for displaying augmented reality content. The program and method provide for displaying a first user interface for viewing a media content item within a messaging application running on a device; receiving user input via the first user interface, the user input corresponding to a predefined touch gesture; and, in response to receiving the user input, switching from the first user interface to a second user interface within the messaging application, wherein a camera of the device is activated to capture an image in the second user interface, and displaying a resized version of the media content item along with the image via the second user interface.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to U.S. patent application serial number 17 / 197,832, filed on March 10, 2021, which claims the benefit of U.S. provisional patent application serial number 63 / 000,185, filed on March 26, 2020, entitled “COMBINING FIRST USER INTERFACE CONTENT INTO SECOND USER INTERFACE,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to messaging applications, including providing for displaying augmented reality content within messaging applications. Background Art

[0004] Messaging systems provide for the exchange of message content between users. For example, messaging systems allow a user to exchange message content (e.g., text, images) with one or more other users. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] To easily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0006] Figure 1 is a pictorial representation of a networked environment in which the present disclosure may be deployed, according to some example implementations.

[0007] Figure 2 is a graphical representation of a messaging client application according to some example embodiments.

[0008] Figure 3 is a graphical representation of a data structure as maintained in a database, according to some example embodiments.

[0009] Figure 4 is a graphical representation of a message according to some example embodiments.

[0010] Figure 5 is a flowchart of an access restriction process according to some example implementations.

[0011] Figure 6 is an interaction diagram illustrating a process for combining media content from a first user interface into a second user interface, according to some example implementations.

[0012] 7A to 7BA first user interface is shown with media content combined into a second user interface according to some example embodiments.

[0013] Figures 8A to 8B A first user interface is shown with media content combined into a second user interface according to some example embodiments.

[0014] Figure 9 is a flow diagram illustrating a process for combining media content from a first user interface into a second user interface, according to some example implementations.

[0015] Figure 10 is a block diagram illustrating a software architecture in which the present disclosure may be implemented, according to some example embodiments.

[0016] Figure 11 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed, causing the machine to perform any one or more of the methodologies discussed herein, according to some example embodiments. DETAILED DESCRIPTION

[0017] Messaging systems typically allow users to exchange content items (e.g., messages, images, and / or videos) with each other in a message thread. Messaging systems can be implemented or otherwise operate in conjunction with an augmented reality system to display augmented reality content in connection with messaging. For example, the augmented reality content is combined with image data captured by a device camera when creating the message content. However, users may desire convenient selection of augmented reality content in connection with messaging.

[0018] The disclosed embodiments provide a messaging application that detects a predefined gesture (e.g., a pinch gesture) within a first user interface for viewing a media content item. In response to detecting the user input, the messaging application switches to a second user interface in which a device camera is activated to capture an image. The second user interface displays the captured image along with a resized version (e.g., a smaller version, such as a sticker) of the media content item.

[0019] In addition, the second user interface includes a carousel interface having icons representing available augmented reality content items (e.g., corresponding to application lenses or augmented reality experiences) for display with the resized media content items and the captured image. The icons are displayed in a ranked order based on objects detected in the media content items and the captured image.

[0020] Figure 11 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network 106. Messaging system 100 includes instances of client devices 102, each instance hosting a plurality of applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of messaging client applications 104 and a messaging server system 110 via a network 106 (e.g., the Internet).

[0021] The messaging client application 104 is capable of communicating and exchanging data with another messaging client application 104 and a messaging server system 110 via the network 106. The data exchanged between messaging client applications 104 and between other messaging client applications 104 and messaging server systems 110 includes functions (e.g., commands to call functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0022] In some embodiments, the messaging client application 104 activates the camera of the client device 102 (e.g., when the messaging client application 104 is launched). The messaging client application 104 allows the user to request a scan of one or more items in the camera feed captured by the camera. For example, the messaging client application 104 may receive a user selection of a dedicated scan option (e.g., a button) presented with the camera feed. In an alternative embodiment, the messaging client application 104 may detect physical contact between a finger of the user's hand and an area of ​​the touch screen for a threshold period of time. For example, the messaging client application 104 determines that the user touches the screen and holds their finger on the screen for more than three seconds. In response, the messaging client application 104 captures the image being displayed on the screen and processes the image to identify one or more objects in the image. In some embodiments, the messaging client application 104 uses one or more trained classifiers and / or environmental factors to identify the objects in the image.

[0023] The messaging server system 110 provides server-side functionality to specific messaging client applications 104 via the network 106. Although certain functionality of the messaging system 100 is described herein as being performed by the messaging client applications 104 or by the messaging server system 110, it will be understood that the location of certain functionality within the messaging client applications 104 or the messaging server system 110 is a design choice. For example, it may be technically preferable to initially deploy certain technologies and functionality within the messaging server system 110, but later migrate the technologies and functionality to the messaging client applications 104 where the client devices 102 have sufficient processing power.

[0024] The messaging server system 110 supports various services and operations provided to the messaging client applications 104. Such operations include sending data to the messaging client applications 104, receiving data from the messaging client applications 104, and processing data generated by the messaging client applications 104. By way of example, the data may include message content, client device information, graphical elements, geographic location information, media annotations and overlays, virtual objects, message content persistence conditions, social network information, and live event information. The exchange of data within the messaging system 100 is invoked and controlled through functionality available through a user interface (UI) (e.g., a graphical user interface) of the messaging client applications 104.

[0025] Turning now specifically to messaging server system 110, API server 108 (application programming interface server) is coupled to application server 112 and provides a programming interface to application server 112. Application server 112 is communicatively coupled to database server 116, which facilitates access to database 122 in which data associated with messages processed by application server 112 is stored.

[0026] Specifically, the API server 108 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the API server 108 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client application 104 in order to invoke functionality of the application server 112. The API server 108 exposes various functions supported by the application server 112, including account registration; login functionality; sending messages from a particular messaging client application 104 to another messaging client application 104 via the application server 112; sending media files (e.g., graphical elements, images, or videos) from a messaging client application 104 to the messaging server application 114, and for possible access by another messaging client application 104; lists of graphical elements; setting of collections of media data (e.g., stories); retrieval of such collections; retrieval of a friend list of a user of a client device 102; maintenance of augmented reality content items; retrieval of messages and content; adding and removing friends from a social graph; the location of friends within a social graph; access to user conversation data; access to avatar information stored on the messaging server system 110; and opening application events (e.g., related to a messaging client application 104).

[0027] The application server 112 hosts a number of applications and subsystems, including a messaging server application 114, an image processing system 118, a social networking system 120, and an augmented reality system 124. The messaging server application 114 implements a number of message processing techniques and functions, particularly relating to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client application 104. As will be described in greater detail, text and media content from multiple sources may be aggregated into collections of content (e.g., referred to as stories or libraries). These collections are then made available to the messaging client applications 104 by the messaging server application 114. Given the hardware requirements for such processing, the messaging server application 114 may also perform other processor- and memory-intensive processing of the data on the server side.

[0028] Application server 112 also includes an image processing system 118 dedicated to performing various image processing operations, typically on images or videos received within the payload of messages at messaging server application 114. In one or more embodiments, a portion of image processing system 118 may also be implemented by augmented reality system 124.

[0029] The social networking system 120 supports various social networking functions and services and makes these functions and services available to the messaging server application 114. To this end, the social networking system 120 maintains and accesses an entity graph within a database 122. Examples of functions and services supported by the social networking system 120 include identification of other users of the messaging system 100 with whom a particular user has a relationship or "follows," as well as identification of other entities and particular users of interest. Such other users may be referred to as the user's friends. The social networking system 120 may access location information associated with each of the user's friends to determine where they live or are currently geographically located. The social networking system 120 may maintain a location profile for each of the user's friends, which indicates the geographic location in which the user's friend resides.

[0030] The messaging client application 104 includes a set of functions that allow the client device 102 to access the augmented reality system 124. The augmented reality system 124 generates and maintains a list of augmented reality content items. The augmented reality content items may correspond to augmented reality experiences (e.g., lenses) that supplement captured image data with augmented reality content.

[0031] In one or more embodiments, the augmented reality system 124 provides for determining (e.g., receiving) one or more attributes (e.g., a name) of an object. The augmented reality system 124 provides for searching for one or more augmented reality content items (e.g., virtual objects) associated with the one or more attributes of the object, and ranking the virtual objects (e.g., based on the association and weights assigned to each attribute). The augmented reality system 124 causes one or more virtual objects or graphical elements of the highest-ranked augmented reality content items to be presented on the captured image.

[0032] Application server 112 is communicatively coupled to database server 116, which facilitates access to a database 122 storing data associated with messages processed by messaging server application 114. Database 122 may be a third-party database. For example, application server 112 may be associated with a first entity, and database 122, or a portion thereof, may be associated with and hosted by a second, different entity. In some implementations, database 122 stores user data collected by the first entity regarding each of the various users of a service provided by the first entity. For example, user data includes usernames, phone numbers, passwords, addresses, friends, activity information, preferences, videos or content consumed by the user, and the like.

[0033] Figure 2 1 is a block diagram illustrating additional details regarding messaging system 100 according to an example embodiment. Specifically, messaging system 100 is shown as including messaging client application 104 and application server 112, which in turn contains a number of subsystems, namely, transient timer system 202, collection management system 204, and annotation system 206.

[0034] The transient timer system 202 is responsible for enforcing transient access to content permitted by the messaging client application 104 and the messaging server application 114. To this end, the transient timer system 202 incorporates a plurality of timers that selectively display messages and associated content and enable access to the messages and associated content via the messaging client application 104 based on durations and display parameters associated with the message or collection of messages (e.g., a story). Additional details regarding the operation of the transient timer system 202 are provided below.

[0035] The collection management system 204 is responsible for managing collections of media (e.g., collections of text, image video, and audio data). In some examples, collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event libraries" or "event stories." Such collections can be made available for a specified time period (e.g., the duration of the event to which the content relates). For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 204 can also be responsible for publishing icons that provide notification of the existence of a particular collection to the user interface of the messaging client application 104.

[0036] The collection management system 204 also includes a curation interface 208 that allows a collection manager to manage and curate specific collections of content. For example, the curation interface 208 enables an event organizer to curate a collection of content related to a specific event (e.g., to remove inappropriate content or redundant messages). In addition, the collection management system 204 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some embodiments, users may be paid compensation to include user-generated content in a collection. In such cases, the curation interface 208 operates to automatically pay such users for use of their content.

[0037] The annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content associated with a message. For example, the annotation system 206 provides functions related to generating and publishing media overlays for messages processed by the messaging system 100. The annotation system 206 is operable to supply media overlays or supplements (e.g., image filters) to the messaging client application 104 based on the geographic location of the client device 102. In another example, the annotation system 206 is operable to supply media overlays to the messaging client application 104 based on other information (e.g., social network information of the user of the client device 102). Media overlays can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to content items (e.g., photos) at the client device 102. For example, a media overlay can include text that can be overlaid on a photo taken by the client device 102. In another example, a media overlay includes an overlay of a location identifier (e.g., Venice Beach), the name of a live event, or a business name (e.g., Beach Cafe). In another example, the annotation system 206 uses the geographic location of the client device 102 to identify a media overlay that includes the name of a business at the geographic location of the client device 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 122 and accessed through the database server 116.

[0038] In one example embodiment, the annotation system 206 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. Users can also specify the context in which specific media overlays should be provided to other users. The annotation system 206 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0039] In another example embodiment, the annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media coverage associated with a geographic location through a bidding process. For example, the annotation system 206 associates the media coverage of the highest-bidding merchant with the corresponding geographic location within a predefined amount of time.

[0040] Figure 3 is a schematic diagram illustrating a data structure 300 that may be stored in database 122 of messaging server system 110 according to certain example embodiments. Although the contents of database 122 are shown as including a plurality of tables, it should be understood that data may be stored in other types of data structures (e.g., as an object-oriented database).

[0041] The database 122 includes message data stored in a message table 316. The entity table 302 stores entity data, including an entity graph 304. The entities whose records are maintained in the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the type, any entity for which the messaging server system 110 stores data may be an identified entity. Each entity is provided with a unique identifier and an entity type identifier (not shown).

[0042] The entity graph 304 stores information about relationships and associations between entities. By way of example only, such relationships may be interest-based or activity-based social relationships, professional relationships (e.g., working in a common company or organization).

[0043] The message table 316 can store a collection of conversations between a user and one or more friends or entities. The message table 316 can include various attributes for each conversation, such as a list of participants, the size of the conversation (e.g., the number of users and / or the number of messages), the chat color of the conversation, a unique identifier for the conversation, and any other conversation-related features.

[0044] The database 122 also stores annotation data in the form of examples of filters in the annotation table 314. The database 122 also stores the annotation content received in the annotation table 314. The filters whose data is stored in the annotation table 314 are associated with videos (whose data is stored in the video table 312) and / or images (whose data is stored in the image table 310) and are applied to the videos and / or images. In one example, the filter is displayed as an overlay on the image or video during presentation to the recipient user. Filters can be of various types, including user-selected filters from a library of filters presented to the sending user by the messaging client application 104 when the sending user is composing a message. Other types of filters include geolocation filters (also known as geofilters), which can be presented to the sending user based on geographic location. For example, based on geographic location information determined by the global positioning system (GPS) unit of the client device 102, the messaging client application 104 can present a geolocation filter specific to a neighborhood or a particular location within the UI. Another type of filter is a data filter, which can be selectively presented to the sending user by the messaging client application 104 based on other input or information collected by the client device 102 during the message creation process. Examples of data filters include the current temperature at a particular location, the current speed the sending user is traveling, the battery life of the client device 102, or the current time.

[0045] Other annotation data that can be stored in the image table 310 is augmented reality content items (eg, corresponding to augmented reality experiences or lenses). Augmented reality content items can be real-time special effects and sounds that can be added to an image or video.

[0046] As described above, augmented reality content items, overlays, image transformations, AR images, and similar terms refer to modifications that can be made to a video or image. This includes real-time modifications, which modify an image as it is captured using a device sensor and then display the modified image on the device's screen. This also includes modifications to stored content, such as video clips in a library that can be modified. For example, on a device with access to multiple augmented reality content items, a user can use a single video clip with multiple augmented reality content items to see how different augmented reality content items will modify the stored clip. For example, by selecting different augmented reality content items for the content, multiple augmented reality content items with different pseudo-random motion models can be applied to the same content. Similarly, real-time video capture can be used with displayed modifications to illustrate how the video image currently captured by the device's sensors will modify the captured data. Such data can be simply displayed on the screen and not stored in memory, or the content captured by the device's sensors can be recorded and stored in memory with or without modifications (or both). In some systems, a preview feature can show how different augmented reality content items will appear simultaneously within different windows on the display. This can, for example, enable viewing multiple windows with different pseudo-random animations on the display simultaneously.

[0047] Thus, data and various systems that use augmented reality content items or other such transformation systems to modify content using such data may involve: detecting objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.); tracking such objects as they leave, enter, and move around the field of view in a video frame; and modifying or transforming such objects while being tracked. In various embodiments, different methods for implementing such transformations may be used. For example, some embodiments may involve generating a three-dimensional mesh model of one or more objects and using transformations and animated textures of the models within a video to implement the transformations. In other embodiments, tracking of points on an object may be used to place an image or texture (which may be two-dimensional or three-dimensional) at the tracked location. In yet another embodiment, neural network analysis of a video frame may be used to place an image, model, or texture within content (e.g., an image or frame of a video). Thus, augmented reality content items involve both images, models, and textures used to create transformations within the content, as well as the additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0048] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load video files and store them in the device's memory, or a device's sensors can be used to generate a video stream. In addition, computer animation models can be used to process any object, such as human faces and body parts, animals, or inanimate objects (e.g., chairs, cars, or other objects).

[0049] In some embodiments, when a specific modification is selected along with the content to be transformed, the computing device identifies the elements to be transformed and then detects and tracks them if present in the video frame. Elements of the object are modified based on the modification request, thereby transforming the frame of the video stream. Transformations of frames of the video stream can be performed using different methods for different types of transformations. For example, for frame transformations primarily involving changing the form of an object's elements, feature points are calculated for each element of the object (e.g., using an active shape model (ASM) or other known methods). A mesh based on the feature points is then generated for each of at least one element of the object. This mesh is used in subsequent stages of tracking the elements of the object in the video stream. During the tracking process, the mesh for each element is aligned with the position of each element. Additional points are then generated on the mesh. A first set of first points is generated for each element based on the modification request, and a second set of points is generated for each element based on the set of first points and the modification request. The frame of the video stream can then be transformed by modifying the elements of the object based on the set of first and second points and the mesh. In such methods, the background of the modified object can also be altered or distorted by tracking and modifying the background.

[0050] In one or more embodiments, a transformation that uses elements of an object to alter regions of the object can be performed by calculating feature points for each element of the object and generating a grid based on the calculated feature points. Points are generated on the grid, and then various regions based on these points are generated. The elements of the object are then tracked by aligning the region for each element with the position of each of at least one element, and the characteristics of the region can be modified based on a request for modification, thereby transforming the frames of the video stream. Depending on the specific request for modification, the characteristics of the mentioned region can be transformed in different ways. Such modifications can involve: changing the color of the region; removing at least some portions of the region from the frame of the video stream; including one or more new objects in the region based on the request for modification; and modifying or distorting elements of the region or object. In various embodiments, any combination of such modifications or other similar modifications can be used. For certain models to be animated, some feature points can be selected as control points for the entire state space used to determine options for model animation.

[0051] In some embodiments of computer animation models that use face detection to transform image data, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). An active shape model (ASM) algorithm is then applied to the facial region of the image to detect facial feature reference points.

[0052] In other embodiments, other methods and algorithms suitable for face and / or object detection may be used. For example, in some embodiments, features are located using landmarks that represent distinguishable points present in most of the images considered. For example, for facial landmarks, the location of the left pupil can be used. In cases where the initial landmarks are not recognizable (for example, if the person has an eye patch), secondary landmarks can be used. Such a landmark recognition process can be used for any such object. In some embodiments, a collection of landmarks forms a shape. The shape can be represented as a vector using the coordinates of the points in the shape. One shape is aligned to another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The average shape is the average of the aligned training shapes.

[0053] In some implementations, a landmark search begins with an average shape aligned with the position and size of a face determined by a global face detector. This search then iterates until convergence by adjusting the positions of shape points by template matching the image texture around each point to propose tentative shapes, and then fitting the tentative shapes to a global shape model. In some systems, template matching alone is unreliable, and the shape model aggregates the results of weak template matchers to form a stronger overall classifier. This entire search is repeated at each level in the image pyramid, from coarse to fine resolution.

[0054] Embodiments of the transformation system can capture an image or video stream on a client device (e.g., client device 102) and perform complex image manipulations locally on the client device 102 while maintaining an appropriate user experience, computational time, and power consumption. Complex image manipulations can include size and shape changes, emotion transfer (e.g., changing a face from a frown to a smile), state transfer (e.g., aging an object, reducing its apparent age, changing its gender), style transfer, application of graphical elements, and any other suitable image or video manipulations enabled by a convolutional neural network that has been configured to execute efficiently on the client device 102.

[0055] In some example embodiments, a computer-animated model for transforming image data may be used by a system in which a user may capture an image or video stream of the user (e.g., a selfie) using a client device 102 having a neural network operating as part of a messaging client application 104 operating on the client device 102. The transformation system operating within the messaging client application 104 determines the presence of an object (e.g., a face) within the image or video stream and provides a modification icon associated with the computer-animated model for transforming the image data, or the computer-animated model may exist in association with an interface described herein. The modification icon includes changes that can be used to modify the user's face within the image or video stream as part of the modification operation. Upon selection of the modification icon, the transformation system initiates a process of transforming the user's image to reflect the selected modification icon (e.g., generating a smiling face on the user). In some embodiments, once the image or video stream is captured and the specified modification is selected, the modified image or video stream may be presented in a graphical user interface displayed on the mobile client device. The transformation system may implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, a user can capture an image or video stream, and once a modification icon is selected, the modified result can be presented in real time or near real time. Furthermore, the modification can be persistent while the video stream is captured and the selected modification icon remains toggled. Machine-taught neural networks can be used to implement such modifications.

[0056] In some embodiments, the graphical user interface presenting the modifications performed by the transformation system can provide the user with additional interactive options. Such options can be based on the interface used to initiate the selection of a particular computer-animated model and content capture (e.g., from a content creator user interface). In various embodiments, the modifications can be persistent after the initial selection of the modification icon. The user can toggle the modifications on or off by tapping or otherwise selecting the face being modified by the transformation system, and save it for later review or browsing to other areas of the imaging application. In the event that the transformation system modifies multiple faces, the user can globally toggle the modifications on or off by tapping or selecting a single face that is being modified and displayed within the graphical user interface. In some embodiments, each face in a group of multiple faces can be modified individually, or such modifications can be individually toggled by tapping or selecting each face or a series of faces displayed within the graphical user interface.

[0057] As mentioned above, video table 312 stores video data, which, in one embodiment, is associated with messages whose records are maintained within message table 316. Similarly, image table 310 stores image data associated with messages whose message data is stored in entity table 302. Entity table 302 may associate various annotations from annotation table 314 with the various images and videos stored in image table 310 and video table 312.

[0058] The augmented reality content item table 318 stores indications (e.g., a list) of augmented reality content items that are available for selection and activation by the messaging client application 104. In one or more embodiments, each augmented reality content item in the augmented reality content item table 318 is associated with one or more object attributes. Each augmented reality content item in the augmented reality content item table 318 may also be associated with one or more predefined terms (e.g., using metadata tags, names, etc.). In one or more embodiments, the messaging client application 104 searches the object attributes and / or predefined terms stored in the augmented reality content item table 318 to identify one or more augmented reality content items associated with the scanned object or the object identified in the captured image. Each augmented reality content item stored in the augmented reality content item table 318 includes one or more graphical elements or virtual objects that may or may not be animated. Each augmented reality content item also includes instructions on where to position the graphical elements or virtual objects relative to other objects depicted in the captured image.

[0059] The story table 306 stores data about a collection of messages and associated images, videos, or audio data that are compiled into a collection (e.g., a story or library). The creation of a particular collection can be initiated by a particular user (e.g., each user whose record is maintained in the entity table 302). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the UI of the messaging client application 104 can include an icon that can be selected by the user to enable the sending user to add specific content to his or her personal story.

[0060] A collection may also constitute a "Live Story," which is a collection of content from multiple users that is created manually, automatically, or using a combination of manual and automatic techniques. For example, a "Live Story" may constitute a curated stream of user-submitted content from different locations and events. Users whose client devices have location services enabled and who are at a common location event at a particular time may be presented with an option to contribute content to a particular Live Story, for example, via the UI of the messaging client application 104. Live Stories may be identified to users by the messaging client application 104 based on their location. The end result is a "Live Story" told from a community perspective.

[0061] Another type of content collection is called a "location story," which enables users whose client devices 102 are located in a particular geographic location (e.g., at a university or college campus) to contribute to a particular collection. In some embodiments, contributions to location stories may require a second degree of authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0062] The memory table 308 stores data regarding one or more user "memories," where each user memory corresponds to content (e.g., images, videos, and / or audio files) saved by a single user for subsequent access by that user. The creation and saving of a particular user memory can be initiated by a single user (e.g., each user whose record is maintained in the entity table 302). Furthermore, each user memory stored in the memory table 308 can be viewed, edited, and / or sent by a single user.

[0063] For example, if a user creates content (e.g., for exchange with others in messaging), the created content may, by default, typically be automatically deleted and removed from storage after a specified period of time (e.g., one hour, one minute, one second, etc.). However, if, before the specified period of time at which the content is automatically deleted, the user (e.g., via a "save" interface) chooses to permanently retain the content, the content may be added as user storage to the storage table 308. The saved content may be accessed by a single user for viewing, editing, and / or sharing with other users.

[0064] Figure 4 is a schematic diagram illustrating the structure of a message 400 generated by a messaging client application 104 for transmission to another messaging client application 104 or a messaging server application 114, according to some embodiments. The content of a particular message 400 is used to populate a message table 316 stored within a database 122 accessible to the messaging server application 114. Similarly, the content of the message 400 is stored in memory as "in-transit" or "in-flight" data for the client device 102 or application server 112. The message 400 is shown as including the following components:

[0065] Message identifier 402 : A unique identifier that identifies the message 400 .

[0066] Message text payload 404 : Text to be generated by the user via the user interface of the client device 102 and included in the message 400 .

[0067] • Message image payload 406 : Image data captured by the camera component of the client device 102 or retrieved from the memory component of the client device 102 and included in the message 400 .

[0068] • Message video payload 408 : Video data captured by the camera component or retrieved from the memory component of the client device 102 and included in the message 400 .

[0069] • Message audio payload 410 : audio data captured by a microphone or retrieved from a memory component of the client device 102 and included in the message 400 .

[0070] Message annotation 412: Annotation data representing an annotation (eg, a filter, sticker, or other enhancement) to be applied to the message image payload 406, the message video payload 408, or the message audio payload 410 of the message 400.

[0071] Message duration parameter 414: A parameter value indicating the amount of time, in seconds, that the content of the message (eg, message image payload 406, message video payload 408, message audio payload 410) will be presented or made accessible to the user via the messaging client application 104.

[0072] Message geolocation parameters 416: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter values ​​416 may be included in the payload, each of which is associated with a content item included in the content (e.g., a specific image within the message image payload 406 or a specific video within the message video payload 408).

[0073] Message story identifier 418: An identifier value that identifies one or more content collections (e.g., "story") associated with a particular content item in the message image payload 406 of the message 400. For example, multiple images within the message image payload 406 may each be associated with multiple content collections using an identifier value.

[0074] Message tags 420: Each message 400 can be tagged with a plurality of tags, each of which indicates the subject matter of the content included in the message payload. For example, if a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value can be included within the message tags 420 indicating the relevant animal. The tag value can be manually generated based on user input, or can be automatically generated using, for example, image recognition.

[0075] • Message sender identifier 422: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the client device 102 on which the message 400 was generated and from which the message 400 was sent.

[0076] • Message recipient identifier 424: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the client device 102 to which the message 400 is addressed.

[0077] The contents (e.g., values) of various components of message 400 may be pointers to locations in a table where content data values ​​are stored. For example, the image value in message image payload 406 may be a pointer to a location (or address) within image table 310. Similarly, the value within message video payload 408 may point to data stored in video table 312, the value stored within message annotation 412 may point to data stored in annotation table 314, the value stored within message story identifier 418 may point to data stored in story table 306, and the values ​​stored within message sender identifier 422 and message recipient identifier 424 may point to user records stored in entity table 302.

[0078] Figure 5 is a schematic diagram illustrating an access restriction process 500 according to which access to content (e.g., a multimedia payload of an ephemeral message 502 and associated data) or a collection of content (e.g., an ephemeral message group 504) can be time-limited (e.g., ephemeral).

[0079] Ephemeral message 502 is shown associated with a message duration parameter 506, the value of which determines the amount of time that messaging client application 104 will display ephemeral message 502 to a receiving user of ephemeral message 502. In one embodiment, the receiving user may view ephemeral message 502 for up to a maximum of 10 seconds, depending on the amount of time specified by the sending user using message duration parameter 506.

[0080] The message duration parameter 506 and the message recipient identifier 424 are shown as inputs to a message timer 512, which is responsible for determining the amount of time that the ephemeral message 502 is displayed to the particular receiving user identified by the message recipient identifier 424. In particular, the ephemeral message 502 will only be displayed to the associated receiving user for the period of time determined by the value of the message duration parameter 506. The message timer 512 is shown as providing an output to a more generalized ephemeral timer system 202, which is responsible for the overall timing of displaying content (e.g., the ephemeral message 502) to the receiving user.

[0081] Figure 5, is included within an ephemeral message group 504 (e.g., a collection of messages in a personal story or event story). The ephemeral message group 504 has an associated group duration parameter 508, the value of which determines the duration for which the ephemeral message group 504 is presented and accessible to users of the messaging system 100. For example, the group duration parameter 508 may be the duration of a concert, wherein the ephemeral message group 504 is a collection of content belonging to the concert. Alternatively, a user (either an owning user or a curator user) may specify a value for the group duration parameter 508 when performing setup and creation of the ephemeral message group 504.

[0082] In addition, each ephemeral message 502 within the ephemeral message group 504 has an associated group participation parameter 510, the value of which determines the duration for which the ephemeral message 502 is accessible within the context of the ephemeral message group 504. Thus, a particular ephemeral message group 504 may "expire" and become inaccessible within the context of the ephemeral message group 504 before the ephemeral message group 504 itself expires according to the group duration parameter 508. The group duration parameter 508, the group participation parameter 510, and the message recipient identifier 424 each provide inputs to a group timer 514, which is operable to first determine whether a particular ephemeral message 502 of the ephemeral message group 504 is to be displayed to a particular receiving user, and if so, for how long. Note that, as a result of the message recipient identifier 424, the identity of the particular receiving user is also known to the ephemeral message group 504.

[0083] 504 .

[0084] The transient timer system 202 is also operable to remove a particular transient message 502 from the transient message group 504 based on a determination that an associated group participation parameter 510 has been exceeded. For example, when a sending user has established a group participation parameter 510 of 24 hours from posting, the transient timer system 202 will remove the associated transient message 502 from the transient message group 504 after the specified 24 hours. The transient timer system 202 is also operable to remove a transient message group 504 when the group participation parameter 510 for each transient message 502 within the transient message group 504 has expired, or when the transient message group 504 itself has expired according to the group duration parameter 508.

[0085] In some use cases, the creator of a particular ephemeral message group 504 may specify an indefinite group duration parameter 508. In this case, the expiration of the group participation parameter 510 for the last remaining ephemeral message 502 within the ephemeral message group 504 will determine when the ephemeral message group 504 itself expires. In this case, a new ephemeral message 502 with a new group participation parameter 510 added to the ephemeral message group 504 effectively extends the life of the ephemeral message group 504 to the value of the group participation parameter 510.

[0086] In response to the ephemeral timer system 202 determining that the ephemeral message group 504 has expired (e.g., is no longer accessible), the ephemeral timer system 202 communicates with the messaging system 100 (and, for example, in particular, the messaging client application 104) to cause the indicia (e.g., an icon) associated with the relevant ephemeral message group 504 to no longer be displayed within the user interface of the messaging client application 104. Similarly, when the ephemeral timer system 202 determines that the message duration parameter 506 for a particular ephemeral message 502 has expired, the ephemeral timer system 202 causes the messaging client application 104 to no longer display the indicia (e.g., an icon or textual identification) associated with the ephemeral message 502.

[0087] Figure 6 is an interaction diagram illustrating a process 600 for combining media content from a first user interface into a second user interface according to some example embodiments. Figure 1Process 600 is described with reference to the messaging client application 104 of the client device 102 and the messaging server system 110. However, one or more blocks (or operations) of process 600 can be performed by one or more other components and / or by other suitable devices. Further for the purpose of illustration, the blocks of process 600 are described herein as occurring in series or linearly. However, multiple blocks of process 600 can occur in parallel. In addition, the blocks of process 600 do not need to be performed in the order shown, and / or one or more blocks of process 600 do not need to be performed and / or can be replaced by other operations.

[0088] As described herein, the messaging client application 104 is configured to allow a first user to perform a predefined gesture (e.g., a pinch gesture) on a first user interface for viewing a media content item. In response to the user input, the messaging client application 104 switches to a second user interface in which the device camera is activated to capture an image. The second user interface displays the captured image with a smaller version (e.g., a sticker) of the media content item. The messaging client application 104 also displays a carousel interface with icons corresponding to available augmented reality content items. The icons are displayed in a ranked order based on objects detected in the media content item and the captured image.

[0089] Thus, at block 602, the messaging client application 104 displays a first user interface for viewing a media content item (e.g., an image and / or video with any annotations). For example, the media content item corresponds to a media message received from a second user (e.g., where the first user and the second user are contacts). As described above, the messaging server system 110 can be implemented and / or operate in conjunction with the social networking system 120 to determine that the first user and the second user are contacts (e.g., friends) with respect to the messaging server system 110.

[0090] In another example, the media content item corresponds to a user memory (eg, associated with the memory table 308). The user memory can be accessed by the first user for viewing, editing, and / or sharing with other users (eg, friends).

[0091] In another example, the media content item corresponds to a story (e.g., associated with the story table 306), such as a personal story, a live story, and / or a location story. As described above, a story corresponds to a collection of messages and associated image, video, or audio data. In one or more embodiments, the stories can be accessed via a friend list interface having available stories provided by the first user's friends. Alternatively or additionally, the stories can be accessed via a subscription list interface displaying available stories provided by publishers (e.g., merchants, social media influencers, businesses, advertisers, etc.).

[0092] In another example, the media content item corresponds to a message preview. For example, the message preview corresponds to a preview of a message including media content (e.g., images and / or videos) captured by a device camera. In one or more embodiments, the message preview provides one or more user-selectable editing tools (e.g., for drawing, adding text, adding stickers, cropping, etc.) for modifying / annotating the media content.

[0093] As described above, the above media content items correspond to examples. Process 600 is not limited to these examples, and process 600 can be applied to other types of media content items.

[0094] At block 604, the messaging client application 104 receives user input corresponding to a predefined touch gesture. For example, the user input corresponds to a pinch gesture performed in association with the first user interface, and the messaging client application 104 is configured to detect such a gesture.

[0095] In response to detecting the predefined touch gesture, the messaging client application 104 switches from the first user interface to a second user interface in which the camera of the client device 102 is activated to capture an image (block 606).

[0096] In one or more embodiments, the second user interface corresponds to an interface for composing / generating media content (e.g., for inclusion in a message, story, etc.) In one or more embodiments, the second user interface includes a carousel interface having user-selectable icons, each of the user-selectable icons corresponding to a respective augmented reality content item for display.

[0097] When showing the second user interface, messaging client application 104 displays the resized version of the media content item from the first user interface and the image (frame 608) captured by the device camera. For example, the resized version of the media content item corresponds to the smaller size version of the media content item. In one or more embodiments, when reducing the size of the media content item, messaging client application 104 provides the media content item to be depicted as contracting until reaching the smaller size animation. The smaller size can correspond to the predefined default size by messaging client application 104. Alternatively or additionally, the smaller size can be specified by the user (for example, wherein size can change as the user continues to pinch gestures and / or pull gestures).

[0098] In one or more implementations, the resized version of the media content item corresponds to a sticker superimposed on an image captured by the device camera. The messaging client application 104 can provide for the user to resize the sticker (e.g., in response to a pinch / pull input performed on the sticker). Alternatively or in addition, the messaging client application 104 can provide for the user to reposition the sticker (e.g., in response to a drag gesture performed on the sticker). In one or more embodiments, the messaging client application 104 provides for storing a resized version of the media content item (e.g., an image and / or video with any annotations), e.g., to facilitate such resizing and / or repositioning of the media content item.

[0099] In one or more implementations, the display of the media content item within the first user interface can be subject to one or more constraints.The messaging client application 104 provides for complying with the constraints with respect to displaying the resized version of the media content item.

[0100] For example, the media content item may be subject to a timer (e.g., specified by the second user) associated with the transient timer system 202. Thus, the messaging client application 104 provides compliance with time-based restrictions on displaying the resized version of the media content item in the second user interface.

[0101] In another example, because the media content item is sent from the second user to the first user, privacy can be maintained by not allowing the first user to share the contents of the second user interface (e.g., the captured image data combined with the resized media content item) with any user other than the second user. In yet another example, if the first user screen captures or otherwise saves the captured image data combined with the resized media content item, the messaging client application 104 can provide for notifying the second user (e.g., the sending user) of such screen capture / save operation.

[0102] As described above, the second user interface includes a carousel interface having icons. Each icon is selectable by the user and corresponds to a corresponding augmented reality content item (e.g., an augmented reality experience) for displaying the corresponding augmented reality content. The augmented reality content is displayed together with the resized media content item and the image captured by the device camera. In one or more embodiments, the icons are displayed in a ranked order.

[0103] To perform ranking, the messaging client application 104 detects one or more objects in each of the media content items and the captured image (block 610). As described above, the messaging client application 104 is configured to implement or otherwise access an object recognition algorithm (e.g., including a machine learning algorithm) that is configured to scan the image and detect / track the movement of objects within the image. By way of non-limiting example, detectable objects within the image include: human faces, human body parts, animals and their parts, landscapes, objects in nature, inanimate objects (e.g., chairs, books, cars, buildings, other structures), illustrations of objects (e.g., on posters and / or flyers), text-based objects, equation-based objects, and the like. In addition, the messaging client application 104 is configured to determine or otherwise access attributes of the object, such as name, type, style, color, size, shape, texture, geographic location, and / or other supplemental information (e.g., song title / artist for the media object).

[0104] Thus, in one or more embodiments, the messaging client application 104 scans each of the media content items and the captured image data to detect corresponding objects therein. For example, the scan is performed for a combined display of the resized media content item and the captured image data. Alternatively, a separate scan is performed for each of the media content item (e.g., not necessarily resized) and the captured image data to detect corresponding objects therein.

[0105] At operation 612, the messaging client application 104 sends a request indicating the detected object to the messaging server system 110 for ranking available augmented reality content items. In response to the request, the messaging server system 110 ranks the augmented reality content items based on the detected object (block 614).

[0106] For example, the available augmented reality content items may be ranked by comparing one or more attributes of the detected object (from the media content item and / or the captured image) with corresponding augmented reality content item attributes / terms (e.g., as stored in the augmented reality content item table 318). Furthermore, where multiple augmented reality content items are available for a detected object (e.g., the user's face), the augmented reality content items may be ranked based on one or more parameters, such as user preferences / history of selected augmented reality content items, popularity of augmented reality content items among system users, newly available augmented reality content items, etc.

[0107] In one or more embodiments, certain augmented reality content items are configured to create effects for two separate objects (e.g., a combination of faces and / or other objects). For example, a face swap effect provides for digitally swapping detected faces. Other effects provide for swapping entire bodies (e.g., human bodies) or parts thereof, swapping selected portions of other objects, segmenting out objects or parts thereof, and the like. The augmented reality content item table 318 can store properties about augmented reality content items that are configured to create such effects (e.g., a swap-type effect between two objects). In one or more embodiments, the messaging server system 110 is configured to provide a higher ranking for such augmented reality content items when an applicable object is detected in each of the media content items and the captured image data.

[0108] The messaging server system 110 sends an indication of the ranking to the messaging client application 104 (operation 616). For example, the ranking indicates the order in which the available augmented reality content items are to be displayed (e.g., within a carousel interface). Based on the received ranking, the messaging client application 104 displays a carousel interface with icons in the order of the ranking, each icon for selecting a corresponding augmented reality content item (block 618).

[0109] In one or more embodiments, the ranking further indicates the highest-ranked augmented reality content item. Thus, when combining the resized media content item and the captured image for display, the messaging client application 104 can additionally display the highest-ranked augmented reality content item (e.g., corresponding to the application's highest-ranked augmented reality experience). For example, based on user selection input (e.g., settings), the messaging client application 104 defaults to setting the highest-ranked augmented reality content item as the automatically activated augmented reality content item (e.g., in response to a pinch gesture to display the resized media content along with the captured image).

[0110] Thus, the messaging system 100 described herein provides for switching from a first user interface for viewing a media content item to a second user interface for viewing a captured image having a resized version of the media content item. Furthermore, the messaging system 100 provides for ranking augmented reality content items (e.g., augmented reality experiences) based on objects detected in both the media content item and the captured image. With these features, more convenient and / or more engaging messaging between users can be provided.

[0111] 7A to 7B A first user interface is shown with media content combined into a second user interface according to some example embodiments. Figure 7A User interface 702 and Figure 7BUser interface 704 corresponds to a different user interface of messaging client application 104 running on client device 102 of the first user.

[0112] exist Figure 7A In the example of FIG, user interface 702 is configured for a first user to view a media message received from a second user (e.g., sender 708), where the first user and the second user are friends / connections within messaging system 100. The media message includes captured image data 706 corresponding to an image provided / captured by the second user at their respective device.

[0113] Figure 7A The example of the user interface 702 is shown as corresponding to a received message. However, as mentioned above, the user interface 702 can also be applied to other types of media content items, such as but not limited to: user memories, stories and / or (for example, as described below with respect to Figures 8A to 8B A preview of the message in question.

[0114] In response to a predefined user input (e.g., a pinch gesture) detected at the device screen while displaying user interface 702, messaging client application 104 switches from display of user interface 702 to user interface 704 in which a camera of client device 102 is activated to capture an image (e.g., captured image data 710). In one or more implementations, user interface 704 corresponds to an interface for composing / generating media content (e.g., for inclusion in a message, story, etc.).

[0115] like Figure 7B As shown in the example of , user interface 704 also includes a resized image 712 corresponding to a resized version (e.g., a smaller version) of the media content item (e.g., captured image data 706) included in user interface 702. As described above, the first user can enlarge and / or reduce the resized image 712 (e.g., via corresponding pull / pinch gestures) and / or can reposition the resized image 712 within user interface 704 (e.g., via a drag gesture).

[0116] In one or more embodiments, the user interface 704 includes a carousel interface 714 that allows a user to cycle through and / or select augmented reality content items (e.g., augmented reality experiences) to apply to the captured image data 710. Each of the available augmented reality content items is represented by an icon that can be selected by the user to switch to the corresponding augmented reality content item. As described above, the icons of the carousel interface 714 can be presented in a ranked order (e.g., based on ranking data received from the messaging server system 110).

[0117] In one or more embodiments, the icon corresponding to the active augmented reality content item (e.g., the selected lens icon 716) is displayed differently relative to (e.g., larger than) the remaining icons. As described above, the active augmented reality content item can correspond to a default augmented reality content item selected by the messaging client application 104 based on augmented reality content item ranking data received from the messaging server system 110. Figure 7B In the example of , the selected lens icon 716 corresponds to an empty / blank icon that is not associated with displaying augmented reality content. For example, this can be based on the first user having opted out (e.g., based on user settings) to automatically activate the augmented reality experience.

[0118] In one or more embodiments, user selection of the selected lens icon 716 provides for generating a message including an image (e.g., in response to a press / tap gesture of the selected lens icon 716) and / or a video (e.g., in response to a press and hold gesture of the selected lens icon 716) of the screen content to, for example, send to a friend, include in a story, etc.

[0119] Figures 8A to 8B A first user interface is shown with media content combined into a second user interface according to some example embodiments. Figure 8A User interface 802 and Figure 8B The user interface 804 corresponds to a different user interface of the messaging client application 104 running on the user's client device 102.

[0120] exist Figure 8A In the example of , the user interface 802 corresponds to a message preview that includes media content (e.g., captured image data 806) captured by a camera of the client device 102. In one or more implementations, the user interface 802 includes editing tools 808 for modifying / annotating (e.g., drawing, adding text to, adding stickers to, cropping, etc.) the captured image data 806. In addition, the user interface 802 includes a save button 810 for saving the captured image data 806 with any annotations (e.g., as user storage, as discussed above). Furthermore, the user interface 802 includes a story button 812 for creating a story based on the captured image data 806 with annotations, and a send button 814 for sending a message including the captured image data 806 with annotations to a recipient (e.g., a contact / friend).

[0121] In response to user input (e.g., a pinch gesture) detected at the device screen while displaying user interface 702, messaging client application 104 switches from display of user interface 802 to user interface 804, in which the camera of client device 102 is activated to capture an image (e.g., captured image data 816). In one or more implementations, user interface 804 corresponds to an interface for composing / generating media content (e.g., for inclusion in a message, story, etc.).

[0122] like Figure 8B As shown in the example of , user interface 804 also includes a resized image 818 corresponding to a resized version (e.g., a smaller version) of the media content item (e.g., captured image data 706) included in user interface 702. As described above, the user can enlarge and / or reduce the resized image 818 (e.g., via corresponding pull / pinch gestures) and / or can reposition the resized image 818 within user interface 704 (e.g., via a drag gesture).

[0123] In one or more embodiments, the user interface 804 includes a carousel interface 820 that allows a user to cycle through and / or select augmented reality content items (e.g., augmented reality experiences) to apply to the captured image data 816. Each of the available augmented reality content items is represented by an icon that can be selected by the user to switch to the corresponding augmented reality content item. As described above, the icons of the carousel interface 820 can be presented in a ranked order (e.g., based on ranking data received from the messaging server system 110).

[0124] In one or more embodiments, the icon corresponding to the active augmented reality content item (e.g., the selected lens icon 822) is displayed differently relative to (e.g., larger than) the remaining icons. As described above, the active augmented reality content item can correspond to a default augmented reality content item selected by the messaging client application 104 based on augmented reality content item ranking data received from the messaging server system 110. Figure 8B In the example of , the selected lens icon 822 corresponds to an empty / blank icon that is not associated with displaying augmented reality content. For example, this can be based on the user having opted out (e.g., based on user settings) to automatically activate the augmented reality experience.

[0125] In one or more embodiments, user selection of the selected lens icon 822 provides for generating a message including an image (e.g., in response to a press / tap gesture of the selected lens icon 822) and / or a video (e.g., in response to a press and hold gesture of the selected lens icon 822) of the screen content to, for example, send to a friend, include in a story, etc.

[0126] Figure 9 is a flow chart illustrating a process 900 for combining media content from a first user interface into a second user interface according to some example embodiments. Figure 1 The process 900 is described with reference to the messaging client application 104 and the messaging server system 110. However, one or more blocks (or operations) of the process 900 can be performed by one or more other components and / or by other suitable devices. Further for the purpose of illustration, the blocks of the process 900 are described herein as occurring in series or linearly. However, a plurality of blocks of the process 900 can occur in parallel. In addition, the blocks of the process 900 do not need to be performed in the order shown, and / or one or more blocks of the process 900 do not need to be performed and / or can be replaced by other operations.

[0127] The messaging client application 104 displays a first user interface for viewing a media content item (block 902). At block 904, the messaging client application 104 receives user input via the first user interface, the user input corresponding to a predefined touch gesture (eg, a pinch gesture).

[0128] In response to receiving the user input, the messaging client application 104 switches from the first user interface to the second user interface, in which the device's camera is activated to capture an image (block 906), and the resized version of the media content item is displayed along with the image via the second user interface (block 908). The messaging client application 104 may display an animation that resizes the media content item to the resized version of the media content item in association with the switch from the first user interface to the second user interface.

[0129] The messaging client application 104 may detect a first object displayed in the media content item, detect a second object displayed in the image, and determine a ranking of multiple augmented reality content items for display within the second user interface, the ranking being based on the first object and the second object.

[0130] Determining the ranking may include sending a request to the messaging server system 110 to rank the plurality of augmented reality content items, the request indicating the first object and the second object, and receiving an indication of the ranking of the plurality of augmented reality content items from the messaging server system 110 in response to sending the request.

[0131] The messaging client application 104 may display a carousel interface having icons ordered based on ranking, the carousel interface including a plurality of icons, each icon selectable by a user to display a corresponding one of a plurality of augmented reality content items together with the image.

[0132] The messaging client application 104 may receive second user input to create a second media content item including the media content and the image, and generate the second media content item in response to receiving the second user input.

[0133] The media content item may correspond to at least one of: a received message, stored media content previously generated by a user of the device via a messaging application (e.g., user memory), a collection of multimedia content (e.g., a story), or a preview of a second image previously captured by a camera (e.g., a message preview).

[0134] The display of the media content item within the first user interface may be subject to one or more constraints. Displaying the resized version of the media content item complies with the one or more constraints.

[0135] Figure 10 10 is a block diagram 1000 illustrating a software architecture 1004 that can be installed on any one or more of the devices described herein. The software architecture 1004 is supported by hardware, such as a machine 1002, including a processor 1020, a memory 1026, and I / O components 1038. In this example, the software architecture 1004 can be conceptualized as a stack of layers, each of which provides specific functionality. The software architecture 1004 includes layers such as an operating system 1012, a library 1010, a framework 1008, and an application 1006. In operation, the application 1006 invokes an API call 1050 through the software stack and receives a message 1052 in response to the API call 1050.

[0136] The operating system 1012 manages hardware resources and provides common services. The operating system 1012 includes, for example, a kernel 1014, services 1016, and drivers 1022. The kernel 1014 acts as an abstraction layer between the hardware layer and other software layers. For example, the kernel 1014 provides memory management, processor management (e.g., scheduling), component management, network and security settings, and other functions. Services 1016 can provide other common services to other software layers. Drivers 1022 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1022 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0137] The libraries 1010 provide a low-level common infrastructure used by the applications 1006. The libraries 1010 may include a system library 1018 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the libraries 1010 may include an API library 1024, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering graphical content on a display in two dimensions (2D) and three dimensions (3D), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1010 may also include various other libraries 1028 to provide many other APIs to the application 1006 .

[0138] The framework 1008 provides a high-level common infrastructure used by the applications 1006. For example, the framework 1008 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 1008 can provide a wide range of other APIs that can be used by the applications 1006, some of which may be specific to a particular operating system or platform.

[0139] In an example embodiment, the applications 1006 may include a home application 1036, a contacts application 1030, a browser application 1032, a book reader application 1034, a location application 1042, a media application 1044, a messaging application 1046 (e.g., corresponding to the messaging client application 104), a game application 1048, and a broad category of other applications such as third-party applications 1040. The applications 1006 are programs that perform the functions defined in the program. Various programming languages ​​can be used to create one or more of the applications 1006 structured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a specific example, the third-party applications 1040 (e.g., applications developed by an entity other than the vendor of a particular platform using an ANDROID™ or IOS™ software development kit (SDK)) can be mobile operating systems such as IOS™, ANDROID™, Mobile software running on a mobile phone or another mobile operating system. In this example, a third-party application 1040 can activate an API call 1050 provided by the operating system 1012 to facilitate the functions described herein.

[0140] Figure 111 is a diagrammatic representation of a machine 1100 within which instructions 1108 (e.g., software, programs, applications, applet, apps, or other executable code) may be executed for causing the machine 1100 to perform any one or more of the methodologies discussed herein. For example, the instructions 1108 may cause the machine 1100 to perform any one or more of the methodologies described herein. The instructions 1108 transform a general-purpose, unprogrammed machine 1100 into a specialized machine 1100 that is programmed to perform the functions described and illustrated in the manner described. The machine 1100 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1100 may operate as a server or a client machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1100 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, web appliances, a network router, a network switch, a network bridge, or any machine capable of executing, sequentially or otherwise, the instructions 1108 specifying actions to be taken by the machine 1100. Furthermore, while only a single machine 1100 is shown, the term "machine" shall also be construed to include any collection of machines that individually or jointly execute the instructions 1108 to perform any one or more of the methodologies discussed herein.

[0141] The machine 1100 may include a processor 1102, a memory 1104, and an I / O component 1144, which may be configured to communicate with each other via a bus 1142. In an example embodiment, the processor 1102 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1106 that executes instructions 1108 and a processor 1110. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 11 Multiple processors 1102 are shown, but the machine 1100 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0142] The memory 1104 includes a main memory 1112, a static memory 1114, and a storage unit 1116, which are accessible to the processor 1102 via a bus 1142. The main memory 1104, the static memory 1114, and the storage unit 1116 store instructions 1108 that implement any one or more of the methodologies or functions described herein. The instructions 1108 may also reside, completely or partially, within the main memory 1112, within the static memory 1114, within the machine-readable medium 1118 within the storage unit 1116, within at least one of the processors 1102 (e.g., within a cache memory of the processor), or any suitable combination thereof during execution thereof by the machine 1100.

[0143] The I / O components 1144 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 1144 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It should be understood that the I / O components 1144 may include Figure 11 Many other components are not shown in the drawings. In various example embodiments, the I / O components 1144 may include output components 1128 and input components 1130. The output components 1128 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, and the like. The input components 1130 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide location and / or force of touches or touch gestures), audio input components (e.g., microphones), optical sensor components (e.g., cameras), and the like.

[0144] In another example embodiment, the I / O component 1144 may include a biometric component 1132, a motion component 1134, an environmental component 1136, or a positioning component 1138, as well as various other components. For example, the biometric component 1132 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component 1134 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1136 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety reasons or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1138 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer for detecting air pressure that can provide altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0145] A variety of technologies can be used to implement communications. The I / O components 1144 also include a communications component 1140 that is operable to couple the machine 1100 to the network 1120 or the device 1122 via the coupler 1126 and the coupler 1124, respectively. For example, the communications component 1140 may include a network interface component or another suitable device to interface with the network 1120. In other examples, the communications component 1140 may include a wired communications component, a wireless communications component, a cellular communications component, a near field communications (NFC) component, a Components (e.g. Low power consumption), Device 1122 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0146] In addition, the communication component 1140 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1140 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as a universal product code (UPC) bar code; multi-dimensional bar codes such as a Quick Response (QR) code, an Aztec code, a Data Matrix, a Dataglyph, a MaxiCode, a PDF417, a Hypercode, a UCC RSS-2D bar code; and other optical codes) or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). In addition, various information can be derived via the communication component 1140, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... Location of signal triangulation, location of NFC beacon signals that can indicate a specific location via detection, etc.

[0147] Various memories (e.g., memory 1104, main memory 1112, static memory 1114, and / or memory of processor 1102) and / or storage unit 1116 may store one or more sets of instructions and data structures (e.g., software) that embody or are used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1108) when executed by processor 1102 cause various operations to implement the disclosed embodiments.

[0148] The instructions 1108 may be sent or received over the network 1120 using a transmission medium via a network interface device (e.g., a network interface component included in the communication component 1140) and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 1108 may be sent or received using a transmission medium via a coupling 1124 (e.g., a peer-to-peer coupling) to the device 1122.

[0149] "Client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smartphone, a tablet computer, an ultrabook, a netbook, multiple laptop computers, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0150] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Plain Old Telephone Service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long distance protocols, or other data transmission technologies.

[0151] "Component" refers to a device, physical entity, or logic with the following boundaries: the boundaries are defined by function or subroutine calls, branch points, APIs, or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. A component can be a packaged functional hardware unit designed for use with other components and can be part of a program that generally performs a specific function among related functions. A component can constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a processor group) can be configured as a hardware component for performing certain operations as described herein by software (e.g., an application or an application portion). Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component can include dedicated circuits or logic that are permanently configured to perform certain operations. The hardware component can be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component can also include a programmable logic or circuit that is temporarily configured to perform certain operations by software. For example, the hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit or in a temporarily configured (e.g., configured by software) circuit can be promoted for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering the embodiment in which the hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any one time. For example, in the case where a hardware component includes a general-purpose processor that is configured as a special-purpose processor by software, the general-purpose processor can be configured as different special-purpose processors (e.g., including different hardware components) at different times. The software accordingly configures a specific processor or multiple specific processors to, for example, constitute a specific hardware component at one time and constitute different hardware components at different times. A hardware component can provide information to other hardware components and receive information from other hardware components. Accordingly, the described hardware components can be considered to be communicatively coupled.When there are multiple hardware components at the same time, communication can be achieved by signal transmission (for example, by appropriate circuits and buses) between or among two or more hardware components. In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation and store the output of the operation in a memory device coupled on the communication. Then, other hardware components can access the memory device at a subsequent time to retrieve and process the stored output. The hardware component can also initiate communication with an input or output device and can operate on resources (for example, the collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (for example, by software) or permanently configured to perform related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein specific one or more processors are examples of hardware. For example, at least some of the operations of the method can be performed by one or more processors or the components implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some of the operations in the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of certain operations can be distributed between processors, not only residing in a single machine, but also deployed across multiple machines. In some example embodiments, a processor or the components implemented by the processor can be located in a single geographical location (for example, in a home environment, an office environment, or a server farm). In other example embodiments, a processor or the components implemented by the processor can be distributed across multiple geographical locations.

[0152] "Computer-readable media" refers to both machine storage media and transmission media. Thus, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0153] An "ephemeral message" is a message that is accessible for a limited duration. An ephemeral message can be text, an image, a video, or the like. The access time for an ephemeral message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is ephemeral.

[0154] “Machine storage medium” refers to a single or multiple storage devices and / or media (e.g., a centralized or distributed database, and / or associated caches and servers) that store executable instructions, routines, and / or data. Thus, the above terms should be taken to include, but are not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage medium,” “device storage medium,” and “computer storage medium” mean the same thing and are used interchangeably in this disclosure. The terms “machine storage medium,” “computer storage medium,” and “device storage medium” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term “signal media.”

[0155] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine code," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0156] "Signal medium" refers to any intangible medium that is capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be construed to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same medium and may be used interchangeably in this disclosure.

[0157] Changes and modifications may be made to the disclosed embodiments without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure and are expressed in the following claims.

Claims

1. A method for message transmission, comprising: displaying a first user interface for viewing a media content item within a messaging application running on a device; receiving a user input via the first user interface, the user input corresponding to a predefined touch gesture; as well as In response to receiving the user input, switching from the first user interface to a second user interface within the messaging application, in which the device's camera is activated to capture an image, and displaying, via the second user interface, a resized version of the media content item along with the image; detecting a first object displayed in the media content item; detecting a second object displayed in the image; as well as A ranking of a plurality of augmented reality content items for display within the second user interface is determined, the ranking being based on the first object and the second object.

2. The method according to claim 1, wherein Determining the ranking includes: sending a request to a server for ranking the plurality of augmented reality content items, the request indicating the first object and the second object; and An indication of the ranking of the plurality of augmented reality content items is received from the server and in response to sending the request.

3. The method according to claim 1, further comprising: A carousel interface having icons ordered based on the ranking is displayed, the carousel interface including a plurality of icons, each icon selectable by a user to display a corresponding one of the plurality of augmented reality content items together with the image.

4. The method according to claim 1, further comprising: receiving second user input to create a second media content item comprising the media content and the image; as well as The second media content item is generated in response to receiving the second user input.

5. The method according to claim 1, further comprising: An animation is displayed that resizes the media content item to a resized version of the media content item in association with switching from the first user interface to the second user interface.

6. The method according to claim 1, wherein The predefined touch gesture is a pinch gesture.

7. The method according to claim 1, wherein The media content item corresponds to at least one of a received message, stored media content previously generated by a user of the device via the messaging application, a collection of multimedia content, or a preview of a second image previously captured by the camera.

8. The method according to claim 1, wherein display of the media content item within the first user interface is subject to one or more constraints, and Wherein displaying the resized version of the media content item complies with the one or more constraints.

9. A device comprising: processor; and a memory storing instructions that, when executed by the processor, cause the processor to: displaying a first user interface for viewing a media content item within a messaging application running on a device; receiving a user input via the first user interface, the user input corresponding to a predefined touch gesture; as well as In response to receiving the user input, switching from the first user interface to a second user interface within the messaging application, in which the device's camera is activated to capture an image, and displaying, via the second user interface, a resized version of the media content item along with the image; detecting a first object displayed in the media content item; detecting a second object displayed in the image; as well as A ranking of a plurality of augmented reality content items for display within the second user interface is determined, the ranking being based on the first object and the second object.

10. The apparatus according to claim 9, wherein Determining the ranking includes: sending a request to a server for ranking the plurality of augmented reality content items, the request indicating the first object and the second object; and An indication of a ranking of the plurality of augmented reality content items is received from the server and in response to sending the request.

11. The apparatus according to claim 9, wherein The instructions further cause the processor to: A carousel interface having icons ordered based on the ranking is displayed, the carousel interface including a plurality of icons, each icon selectable by a user to display a corresponding one of the plurality of augmented reality content items together with the image.

12. The apparatus according to claim 9, wherein The instructions further cause the processor to: receiving second user input to create a second media content item comprising the media content and the image; and The second media content item is generated in response to receiving the second user input.

13. The apparatus according to claim 9, wherein The instructions further cause the processor to: An animation is displayed that resizes the media content item to a resized version of the media content item in association with switching from the first user interface to the second user interface.

14. The apparatus according to claim 9, wherein The predefined touch gesture is a pinch gesture.

15. The apparatus according to claim 9, wherein The media content item corresponds to at least one of a received message, stored media content previously generated by a user of the device via the messaging application, a collection of multimedia content, or a preview of a second image previously captured by the camera.

16. The apparatus according to claim 9, wherein display of the media content item within the first user interface is subject to one or more constraints, and Wherein displaying the resized version of the media content item complies with the one or more constraints.

17. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to: displaying a first user interface for viewing a media content item within a messaging application running on a device; receiving a user input via the first user interface, the user input corresponding to a predefined touch gesture; as well as In response to receiving the user input, switching from the first user interface to a second user interface within the messaging application, in which the device's camera is activated to capture an image, and displaying, via the second user interface, a resized version of the media content item along with the image; detecting a first object displayed in the media content item; detecting a second object displayed in the image; as well as A ranking of a plurality of augmented reality content items for display within the second user interface is determined, the ranking being based on the first object and the second object.

Images