Content suggestion system

Abstract

Embodiments of the present disclosure generally relate to a system for presenting content suggestions within a messaging interface. According to certain example embodiments, a content suggestion system performs operations including: receiving an input in a messaging interface including a text string; in response to receiving the input, parsing a keyword from the input text string; retrieving message content comprising at least a media item based on the keyword; and presenting a chat suggestion including the message content in the message transmission interface.

Description

[0001] This application is a divisional application of a patent application with an application date of August 14, 2019, application number 201980053073.8, and invention name “Content Recommendation System”.

[0002] priority

[0003] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 16 / 103,626, filed on August 14, 2018, the benefit of which is hereby claimed and is incorporated herein by reference in its entirety. Technical Field

[0004] Embodiments of the present disclosure relate generally to messaging services and, more particularly, to a system for presenting content suggestions within a messaging interface. Background Art

[0005] Users are increasingly relying on messaging applications to achieve the purpose of communicating with other users, rather than traditional phone calls. The advantage of messaging applications is that they provide an enhanced interface through which users can communicate using various forms of media content (e.g., pictures, videos, and audio files).

[0006] While these messaging applications are superior to traditional communication methods for many reasons, the added functionality may often result in difficulty identifying and selecting appropriate media content to share in a conversation. As a result, users may interrupt the organic and natural flow of conversation because of the significant amount of time required to identify relevant media content to share. Therefore, a system that provides messaging suggestions would be an improvement over existing systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the drawing number in which the element is first introduced.

[0008] Figure 1 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network, wherein the messaging system includes a content suggestion system, in accordance with some embodiments.

[0009] Figure 2 is a block diagram showing further details regarding a messaging system according to an example embodiment.

[0010] Figure 3 is a block diagram illustrating various modules of a content suggestion system according to certain example embodiments.

[0011] Figure 4is a flow chart illustrating a method for presenting content suggestions in a messaging interface, according to certain example embodiments.

[0012] Figure 5 is a flow chart illustrating a method for presenting content suggestions in a messaging interface, according to certain example embodiments.

[0013] Figure 6 is a flow chart illustrating a method for generating a message based on content suggestions, according to certain example embodiments.

[0014] Figure 7 is a flow chart illustrating a method for presenting content suggestions in a messaging interface, according to certain example embodiments.

[0015] Figure 8 is a flow chart illustrating a method for associating index terms with media items in a media library, according to certain example embodiments.

[0016] Fig. 9 is an interface diagram depicting a messaging interface according to certain example embodiments.

[0017] Fig.10 is an interface diagram depicting messages generated using a messaging interface, according to certain example embodiments.

[0018] Fig.11 is an interface diagram depicting a messaging interface according to certain example embodiments.

[0019] Fig.12 is an interface diagram depicting messages generated using a messaging interface, according to certain example embodiments.

[0020] Fig.13 is a block diagram illustrating a representative software architecture that may be used in conjunction with the various hardware architectures described herein and to implement the various embodiments.

[0021] Fig.14 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. DETAILED DESCRIPTION

[0022] Messaging applications provide enhanced interfaces to enable users to communicate with each other by sharing text and various forms of media content. Although this is useful, due to the increase in the amount of available media content, searching and identifying relevant media content is often difficult and time consuming. Therefore, the natural flow of conversation may be interrupted.

[0023] Embodiments of the present disclosure generally relate to a system for presenting content suggestions within a messaging interface. According to certain example embodiments, the content suggestion system performs operations including: receiving input of message content including a text string included in a messaging interface; in response to receiving the message content, identifying keywords based on the string of text in the message content; retrieving one or more media items based on the keywords; and presenting chat suggestions including at least a presentation of the media items within the messaging interface.

[0024] In some embodiments, the content suggestion system generates and displays a messaging interface to facilitate communication between users. The messaging interface includes a display of message content, and a display of a message input field in which a user can provide the message content as input to the messaging interface. For example, a user can provide the message content as input into the message input field, where the input includes a text string, one or more graphical icons (e.g., emoticons), or a combination of text and graphical icons.

[0025] In response to receiving input in the message input field, and in real time, the content suggestion system displays the input message content within the messaging interface and parses the message content to identify one or more keywords. Keywords can include words or phrases, as well as various index items including numbers and graphical icons and letters. For example, keywords can include an emoji smiley face, a heart icon, a number string (e.g., 143), and a text string (e.g., "I love you").

[0026] In some embodiments, keywords identified by the content suggestion system can be emphasized in real time within the interface. For example, in response to receiving an input in a message input field, the content suggestion system causes a presentation of a text string received via the input to be displayed within the messaging interface. The presentation of the text string can simply include displaying the text string using a standard color or font (which can be defined by a user profile or an administrator of the messaging application). In response to identifying a keyword based on the text string, in some embodiments, the content suggestion system emphasizes the keyword within the presentation of the text string by displaying the keyword in a color or font that is different from the standard color or font, or in some example embodiments by animating the keyword to indicate motion.

[0027] In response to identifying one or more keywords within the input message content, the content suggestion system accesses a media index within the media content library to identify and retrieve relevant media content. The content suggestion system may query the media index based on keywords of the message content to identify one or more media items.

[0028] In some example embodiments, the media content library includes a media index that includes a collection of media items in various forms, including, but not limited to, images, graphics interchange format images (GIFs), videos, augmented reality filters (e.g., facial filters, lenses, content filters), and audio data. Media items can be tagged with one or more index items in the media index so that references to the index items can identify the storage location of the media item category or the corresponding media item. In other embodiments, media items can be organized in a collection tagged with index items. For example, a collection of media items can be tagged with the index item "love".

[0029] The content suggestion system presents chat suggestions to the user within the messaging interface based on one or more media items identified based on the keyword. The chat suggestion can be presented as a graphical icon within the messaging interface near the message input field. For example, in response to identifying and obtaining a media item based on the keyword, the content suggestion system can generate a chat suggestion within the messaging interface and cause display of the chat suggestion.

[0030] In some embodiments, the content suggestion system can curate and present a collection of media items as chat suggestions. Each media item in the collection of media items can be identified from the media content library based on a keyword. For example, the collection of media items can be displayed to the user in a suggestion carousel within the messaging interface, where the user can scroll through the collection of media items.

[0031] Figure 1 1 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and related content) over a network. The messaging system 100 includes a plurality of client devices 102, each client device hosting a plurality of applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0032] Thus, each messaging client application 104 is able to communicate and exchange data with another messaging client application 104 and a messaging server system 108 via a network 106. The data exchanged between messaging client applications 104 and between messaging client applications 104 and messaging server systems 108 includes functionality (e.g., commands to call functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0033] The messaging server system 108 provides server-side functionality to certain messaging client applications 104 via the network 106. Although certain functions of the messaging system 100 are described herein as being performed by the messaging client application 104 or by the messaging server system 108, it should be understood that the location of certain functions in the messaging client application 104 or the messaging server system 108 is a design choice. For example, it may be technically preferable to first deploy certain technologies and functions within the messaging server system 108 and then migrate the technologies and functions to the messaging client application 104 where the client device 102 has sufficient processing power.

[0034] The messaging server system 108 supports various services and operations provided to the messaging client applications 104. Such operations include sending data to the messaging client applications 104, receiving data from the messaging client applications 104, and processing data generated by the messaging client applications 104. In some embodiments, the data includes, by way of example, message content, client device information, geographic location information, media annotations and overlays, message content persistence conditions, social network information, and live event information. In other embodiments, other data is used. The data exchange within the messaging system 100 is invoked and controlled by functions available via the GUI of the messaging client applications 104.

[0035] Turning now specifically to the messaging server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

[0036] Specifically, the application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the application program interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client application 104 in order to invoke the functions of the application server 112. The application program interface (API) server 110 exposes various functions supported by the application server 112, including: account registration; login functionality; sending messages from a particular messaging client application 104 to another messaging client application 104 via the application server 112; sending media files (e.g., images or videos) from the messaging client application 104 to the messaging server application 114 for possible access by another messaging client application 104; setting of collections of media data (e.g., stories); retrieval of a friend list of a user of the client device 102; retrieval of such collections; retrieval of messages and content; adding and removing friends from a social graph; the position of friends in a social graph; and open application events (e.g., related to a messaging client application 104).

[0037] The application server 112 hosts a number of applications and subsystems, including a messaging server application 114, an image processing system 116, a social networking system 122, and a content suggestion system 124. The messaging server application 114 implements a number of message processing techniques and functions, particularly relating to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client application 104. As will be described in further detail, text and media content from multiple sources may be aggregated into collections of content (e.g., referred to as stories, galleries, or collections). The messaging server application 114 then makes these collections available to the messaging client application 104. Other processor and memory intensive processing of the data may also be performed on the server side by the messaging server application 114, given the hardware requirements of such processing.

[0038] The application server 112 also includes an image processing system 116 that is dedicated to performing various image processing operations, typically with respect to images or videos received within the payload of messages at the messaging application server 114 .

[0039] The social networking system 122 supports various social networking functions and services and makes these functions and services available to the messaging server application 114. To this end, the social networking system 122 maintains and accesses an entity graph 304 within the database 120. Examples of functions and services supported by the social networking system 122 include identifying other users of the messaging system 100 with whom a particular user has a relationship or who the particular user is "following," as well as identifying other entities and interests of a particular user.

[0040] The application server 112 is communicatively coupled to a database server 118 , which facilitates access to a database 120 in which data associated with messages processed by the messaging server application 114 is stored.

[0041] Figure 2 1 is a block diagram showing further details about the messaging system 100 according to an example embodiment. Specifically, the messaging system 100 is shown to include a messaging client application 104 and an application server 112, which in turn embody a number of specific subsystems, namely, a transient timer system 202, a collection management system 204, and an annotation system 206.

[0042] The transient timer system 202 is responsible for enforcing temporary access to content permitted by the messaging client application 104 and the messaging server application 114. To this end, the transient timer system 202 incorporates a plurality of timers that selectively display and enable access to messages and associated content via the messaging client application 104 based on durations and display parameters associated with the message, message collection (e.g., a SNAPCHAT story), or graphical element. Further details regarding the operation of the transient timer system 202 are provided below.

[0043] The collection management system 204 is responsible for managing collections of media (e.g., text, image video, and audio data). In some examples, collections of content (e.g., messages including images, videos, text, and audio) may be organized into "event galleries" or "event stories." Such collections may be available for a specified period of time (e.g., the duration of an event associated with the content). For example, content related to a concert may be available as a "story" for the duration of the concert. The collection management system 204 may also be responsible for publishing an icon that provides a notification of the existence of a particular collection to the user interface of the messaging client application 104.

[0044] The collection management system 204 also includes a curation interface 208 that allows a collection manager to manage and curate a particular collection of content. For example, the curation interface 208 enables an event organizer to curate a collection of content related to a particular event (e.g., to remove inappropriate content or redundant messages). In addition, the collection management system 204 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some embodiments, compensation (e.g., currency, non-monetary credits or points associated with a messaging system or a third-party reward system, travel miles, access to artwork or special lenses, etc.) may be paid to a user for including user-generated content in a collection. In this case, the curation interface 208 operates to automatically pay fees to such users for using their content.

[0045] The annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content associated with a message. For example, the annotation system 206 provides functions related to generating and publishing media overlays for messages processed by the messaging system 100. The annotation system 206 operably provides media overlays (e.g., SNAPCHAT filters) to the messaging client application 104 based on the geographic location of the client device 102. In another example, the annotation system 206 operably provides media overlays to the messaging client application 104 based on other information (e.g., social network information of the user of the client device 102). The media overlay may include audio and visual content and visual effects. Examples of audio and visual content include: pictures, text, logos, animations and sound effects, and animated facial models, and image filters. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos) at the client device 110. For example, the media overlay includes text that can be superimposed on a photo taken by the client device 102. In another example, the media overlay includes an identification of a location overlay (e.g., Venice Beach), the name of a live event, or the name of a business overlay (e.g., Beach Cafe). In another example, the annotation system 206 uses the geographic location of the client device 102 to identify a media overlay that includes the name of a business at the geographic location of the client device 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 120 and may be accessed by the database server 118.

[0046] In one example embodiment, the annotation system 206 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. Users can also specify under what circumstances specific media overlays should be provided to other users. The annotation system 206 generates a media overlay including the uploaded content and associates the uploaded content with the selected geographic location.

[0047] In another example embodiment, the annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media overlays associated with geographic locations via a bidding process. For example, the annotation system 206 associates the media overlays of the merchant with the highest bid with the corresponding geographic location within a predetermined amount of time.

[0048] Figure 3 is a block diagram illustrating components of content suggestion system 124 that configure content suggestion system 124 to identify and present content suggestions based on input received via a messaging interface, according to some example embodiments. Content suggestion system 124 is illustrated as including input module 302, keyword module 304, content module 306, and presentation module 308, each of which is configured to communicate with one another (e.g., via a bus, shared memory, or switch). Any one or more of these modules may be implemented using one or more processors 310 (e.g., by configuring such one or more processors to perform the functions described for that module), and thus may include one or more processors 310.

[0049] Any one or more of the described modules may be implemented using separate hardware (e.g., one or more processors 310 of the machine) or a combination of hardware and software. For example, any module described in the content suggestion system 124 may physically include an arrangement of one or more processors 308 (e.g., a subset of one or more processors of the machine or a subset thereof) configured to perform the operations described herein for the module. As another example, any module of the content suggestion system 124 may include software, hardware, or both, which configures an arrangement of one or more processors 310 (e.g., one or more processors of the machine) to perform the operations described herein for the module. Therefore, different modules of the content suggestion system 124 may include and configure different arrangements of such processors 310 or a single arrangement of such processors 310 at different points in time. In addition, any two or more modules of the content suggestion system 124 may be combined into a single module, and the functions described herein for a single module may be subdivided between multiple modules. In addition, according to various example embodiments, the modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0050] Figure 44 is a flow chart illustrating a method 400 for presenting content suggestions in a messaging interface according to some example embodiments. The operations of the method 400 may be described above with reference to Figure 3 The module described is executed. Figure 4 As shown, method 400 includes one or more operations 402 , 404 , 406 , and 408 .

[0051] At operation 402, input module 302 receives input into a messaging interface presented at a client device, such as client device 102. The input may include, for example, message content including text strings and graphical icons, such as emoticons, numbers, and punctuation marks.

[0052] At operation 404, the keyword module 304 identifies keywords based on text strings of the message content. For example, the keyword module 304 can parse the message content to identify one or more keywords in response to receiving input in the messaging interface. Keywords can include text strings and other index items based on combinations of letters, numbers, emoticons, or punctuation marks.

[0053] At operation 406, the content module 306 identifies and retrieves media items based on the keywords. The media items may include, for example, audio data, video data, images, and interactive media content such as augmented reality lenses and filters. For example, in response to the keyword module 304 identifying one or more keywords within the message content, the content module 306 may access a media library (e.g., database 120). The media library may include a collection of media items tagged with an index term so that a reference to the index term may identify a storage location of the media item. The content module 306 accesses the media library and retrieves one or more media items based on the keywords.

[0054] At operation 408, the presentation module 308 generates and causes display of a chat suggestion within the messaging interface, wherein the chat suggestion includes a media item from one or more media items retrieved from the media library. In some embodiments, the chat suggestion may include presentation of a graphical icon, wherein the graphical icon is based on the media item. In other embodiments, the chat suggestion may be presented as a curated selection of a collection of media items, wherein each media item in the collection of media items is retrieved based on a keyword. The sort order of the collection may be based on a ranking of the media items, wherein the ranking may be based on an overall usage metric (e.g., popularity), or based on relevance to a keyword (e.g., based on index term similarity).

[0055] Figure 5 is a flow chart illustrating a method 500 for presenting content suggestions in a messaging interface according to some example embodiments. The operations of the method 500 may be as described above with reference to Figure 3The modules described are executed. Figure 5 As shown, method 500 includes one or more operations 502 and 504 .

[0056] At operation 502, in response to receiving input in the messaging interface as discussed in operation 402 of method 400, presentation module 308 causes presentation of a text string from the message content to be displayed at a location within the messaging interface. Presentation of the message content may, for example, include display of the text string in a predetermined font, color, size, or display format.

[0057] In operation 504, in response to keyword module 304 identifying a keyword based on the text string (as in operation 404 of method 400), presentation module 308 emphasizes the display of the keyword within the presentation of the text string. For example, presentation module 308 may display the keyword in an emphasized font, color, or display format within the presentation of the text string.

[0058] Figure 6 6 is a flowchart illustrating a method 600 for generating a message based on content suggestions according to some example embodiments. The operations of the method 600 may be described above with reference to Figure 3 The modules described are executed. Figure 6 As shown, method 600 includes one or more operations 602 and 604 .

[0059] At operation 602, the input module 302 receives input via a messaging interface, wherein the input includes a selection of a content suggestion (i.e., a media item) to be presented to the user. For example, as in operation 408 of method 400, the content suggestion may be presented to the user of the client device 102. While viewing the content suggestion, in order to select the content suggestion, the user may provide input via the messaging interface, the input including a selection of the media item represented by the content suggestion.

[0060] At operation 604, in response to receiving a selection of a content suggestion, presentation module 308 generates a message based at least on the message content received from the user of the client device in operation 402 of method 400 and the content suggestion selected by the user. Presentation module 308 generates and causes display of a presentation of the message within the messaging interface.

[0061] Figure 7 is a flow chart illustrating a method 700 for presenting content suggestions in a messaging interface according to some example embodiments. The operations of method 700 may be as described above with reference to Figure 3 The modules described are executed. Figure 7 As shown, method 700 includes one or more operations 702 and 704 .

[0062] At operation 702, in response to the keyword module 304 parsing the message content received from the user (such as operation 404 of method 400), the content module 306 accesses a media library (e.g., database 120), wherein the media library includes a collection of media items, each of which is indexed and marked with one or more index terms. For example, media can be organized into one or more collections of media items in the media library, each of which is itself marked with one or more index terms.

[0063] At operation 704, the content module 306 identifies media items to be presented to the user as message content. The content module 306 accesses a media library and identifies media content based on keywords (eg, index terms).

[0064] Figure 8 800 is a flowchart illustrating a method 800 for generating a media index according to some example embodiments. The operation of the method 800 may be described above with reference to Figure 3 The module described is executed. Figure 8 As shown, method 800 includes one or more operations 802 and 804 .

[0065] The media index includes associations between media items having various index terms containing keywords. The media index may be hosted in a media library such as database 120 and at a storage location of client device 102. Thus, a reference to an index term within the media index identifies a storage location of one or more media items, which may be organized into a collection. For example, an index term including the text string "LOVE" may reference a storage location including multiple media items that have been tagged with the index term.

[0066] At operation 802, the input module 302 receives a request to generate a message from the client device 102. In some embodiments, the request to generate a message may occur before the method 400. The request to generate a message may, for example, include message content including a text string or other index term, and a selection of a chat suggestion including a media item.

[0067] In operation 804, in response to a request to generate a message including a selection of message content and a chat suggestion, the keyword module 304 generates a media index including a connection of index terms from the message content with media items so that references to the index terms identify storage locations of the media items. In some embodiments, the media index may correspond to a user profile associated with the client device 102, so that the media index is "personalized" to the user of the client device 102. In other embodiments, the media index may be located within the database 120 and may be updated and maintained based on requests to generate messages from multiple devices including the client device 102.

[0068] Fig. 9 FIG. 9 is an interface diagram 900 depicting a messaging interface 905 according to certain example embodiments. Fig. 9 As shown, the messaging interface 905 includes presenting a set of messages received from one or more users, message content 910 in the messaging interface 905, the message content 910 including a text string, and chat suggestions 915 generated based on the message content 910. The messaging interface 905 can be presented at the client device 102.

[0069] As in Figure 4 As discussed in method 400 of , a user may provide message content 910 to a messaging interface 905. In response to receiving the message content 910, various modules of the content suggestion system 240 parse the message content 910 to identify one or more keywords and identify one or more media items to present to the user as chat suggestions 915 in the messaging interface 905.

[0070] like Fig. 9 As shown, in this example embodiment, chat suggestions 915 include display of a graphical icon, wherein a particular shape and / or appearance of the graphical icon is based on message content derived based on keywords parsed from message content 910 .

[0071] In some embodiments, the user can generate a message, such as message 920, based on the message content 910 and the selection of the chat suggestion 915. For example, the chat suggestion 915 can be presented within an icon that includes a presentation of a media item 925, wherein the media item 925 includes an augmented reality filter element corresponding to a keyword parsed from the message content 910.

[0072] In further embodiments, the content suggestion system 240 can generate and display a collection 930, wherein the collection 930 includes a presentation of a set of media items corresponding to keywords parsed from the message content 910. For example, as discussed in operation 408 of the method 400, the chat suggestion can include a collection of curated media items, such as the collection 930. The user can scroll through the collection 930 to identify and select one or more media items from the collection 930, such as the media item 925.

[0073] Fig.10 FIG. 1 is an interface diagram 1000 depicting a message 920 generated using a messaging interface 905, according to some example embodiments. Fig.10 As shown, message 920 includes message content 910 (i.e., a text string) and a presentation of a media item 925 from chat suggestion 915. As shown in interface diagram 1000, media item 925 may include an augmented reality filter element.

[0074] Fig.11 FIG. 1100 is an interface diagram depicting a messaging interface 1105 according to certain example embodiments. Fig.11 As shown, the messaging interface 1105 includes a presentation of message content 1110 and a chat suggestion 1120 generated based on the message content 1110. The messaging interface 1105 can be presented at the client device 102.

[0075] As in Figure 5 As discussed in method 500 of , keywords 1115 within input 1110 may be emphasized in response to content suggestion module 240 parsing message content 1110 to identify keywords 1115 .

[0076] Fig.12 1 is an interface diagram 1200 depicting a message 1205 generated using the messaging interface 1105, according to some example embodiments. Fig.12 As shown, message 1205 includes message content 1110 and a presentation of media item 1210 from chat suggestion 1120 .

[0077] Software Architecture

[0078] Fig.13 is a block diagram illustrating an example software architecture 1306 that may be used in conjunction with the various hardware architectures described herein. Fig.13 is a non-limiting example of a software architecture, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 1306 may be implemented in a system such as Fig.14 1400 is executed on the hardware of the machine 1400, which includes, among other things, a processor 1404, a memory 1414, and I / O components 1418. A representative hardware layer 1352 is shown and may represent, for example, Fig.13 The machine 1300 of the embodiment of the present invention is shown in Figure 13. The representative hardware layer 1352 includes a processing unit 1354 having associated executable instructions 1304. The executable instructions 1304 represent executable instructions of the software architecture 1306, including implementations of the methods, components, etc. described herein. The hardware layer 1352 also includes a memory and / or storage module memory / storage device 1356, which also has executable instructions 1304. The hardware layer 1352 may also include other hardware 1358.

[0079] exist Fig.13In the example architecture of , software architecture 1306 can be conceptualized as a stack of layers, each of which provides specific functionality. For example, software architecture 1306 may include layers such as operating system 1302, library 1320, application 1316, and presentation layer 1314. In operation, applications 1316 and / or other components within these layers can call application program interfaces (APIs) API calls 1308 through the software stack and receive responses in response to API calls 1308. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 1318, while other operating systems may provide such a layer. Other software architectures may include other or different layers.

[0080] The operating system 1302 can manage hardware resources and provide common services. The operating system 1302 can include, for example, a kernel 1322, services 1324, and drivers 1326. The kernel 1322 can serve as an abstraction layer between the hardware and other software layers. For example, the kernel 1322 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 1324 can provide other public services to other software layers. Drivers 1326 are responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers 1326 include display drivers, camera drivers, drives, flash drives, serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0081] The libraries 1320 provide a common infrastructure used by the applications 1316 and / or other components and / or layers. The libraries 1320 provide functionality that allows other software components to perform tasks in an easier manner than directly interfacing with the underlying operating system 1302 functions (e.g., kernel 1322, services 1324, and / or drivers 1326). The libraries 1320 may include system libraries 1344 (e.g., C standard libraries), which may provide functionality such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the libraries 1320 may include API libraries 1346, such as media libraries (e.g., libraries that support the rendering and manipulation of various media formats (e.g., MPREG4, H.264, MP3, AAC, AMR, JPG, PNG)), graphics libraries (e.g., OpenGL frameworks that may be used to render 2D and 3D graphics content on a display), database libraries (e.g., SQLite that may provide various relational database functions), network libraries (e.g., WebKit that may provide web browsing functions), and the like. The library 1320 may also include a variety of other libraries 1348 to provide many other APIs to the applications 1316 and other software components / modules.

[0082] Framework / middleware 1318 (also sometimes referred to as middleware) provides a higher level common infrastructure that can be used by applications 1316 and / or other software components / modules. For example, framework / middleware 1318 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework / middleware 1318 can provide a wide range of other APIs that can be used by applications 1316 and / or other software components / modules, some of which can be specific to a particular operating system 1302 or platform.

[0083] Applications 1316 include built-in applications 1338 and / or third-party applications 1340. Examples of representative built-in applications 1338 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 1340 may include applications that are created by entities other than the seller of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) for developing applications and can be used on platforms such as IOS TM ANDROID TM , Mobile software running on a mobile operating system such as a mobile phone or other mobile operating system. Third-party applications 1340 can call API calls 1308 provided by a mobile operating system (eg, operating system 1302) to facilitate the functionality described herein.

[0084] Applications 1316 may use built-in operating system functionality (e.g., kernel 1322, services 1324, and / or drivers 1326), libraries 1320, and framework / middleware 1318 to create a user interface to interact with a user of the system. Alternatively, or in addition, in some systems, interaction with the user may occur through a presentation layer (e.g., presentation layer 1314). In these systems, the application / component "logic" may be separated from the various aspects of the application / component that interact with the user.

[0085] Fig.14 1 is a block diagram illustrating components of a machine 1400 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. Specifically, Fig.14A diagrammatic representation of a machine 1400 in the example form of a computer system is shown, in which instructions 1410 (e.g., software, programs, applications, applet, application program, or other executable code) for causing the machine 1400 to perform any one or more of the methods discussed herein may be executed. In this way, the instructions 1410 may be used to implement the modules or components described herein. The instructions 1410 convert a general, unprogrammed machine 1400 into a specific machine 1400 that is programmed to perform the functions described and illustrated in the described manner. In alternative embodiments, the machine 1400 operates as a standalone device, or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1400 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1400 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network appliance, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1410 in sequence or otherwise, which specify actions to be taken by the machine 1400. In addition, while only a single machine 1400 is shown, the term "machine" should also be taken to include a collection of machines that individually or collectively execute instructions 1410 to perform any one or more of the methodologies discussed herein.

[0086] The machine 1400 may include a processor 1404, a memory / storage device 1406, and an I / O component 1418, which may be configured to communicate with each other, for example, via a bus 1402. The memory / storage device 1406 may include a memory 1414 (such as a main memory, or other memory), and a storage unit 1416, both of which may be accessible to the processor 1404, such as via the bus 1402. The storage unit 1416 and the memory 1414 store instructions 1410 that embody any one or more of the methods or functions described herein. The instructions 1410 may also reside, in whole or in part, within the memory 1414, within the storage unit 1416, within at least one of the processors 1404 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution by the machine 1400. Thus, the memory 1414, the storage unit 1416, and the memory of the processor 1404 are examples of machine-readable media.

[0087] I / O components 1418 may include a variety of components to receive input, provide output, generate output, send information, exchange information, collect measurements, etc. The specific I / O components 1418 included in a particular machine 1400 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that I / O components 1418 may be included in Fig.14 1420. The I / O components 1418 are grouped according to function only for the purpose of simplifying the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 1418 may include output components 1426 and input components 1428. The output components 1426 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1428 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide touch position and force or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0088] In further example embodiments, the I / O component 1418 may include a biometric component 1430, a motion component 1434, an environmental component 1436, or a position component 1438, among various other components. For example, the biometric component 1430 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring bio-signals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1434 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental component 1436 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety or measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Position component 1438 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects the altitude from which the air pressure can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.

[0089] A variety of technologies may be used to achieve communication. I / O components 1418 may include a communication component 1440 that is operable to couple machine 1400 to network 1432 or device 1420 via coupling 1422 and coupling 1424, respectively. For example, communication component 1440 may include a network interface component or other suitable device that interfaces with network 1432. In further examples, communication component 1440 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g. Low energy consumption), Device 1420 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via a universal serial bus (USB)).

[0090] In addition, the communication component 1440 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1440 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as universal product codes (UPC) barcodes, multi-dimensional barcodes (e.g., Quick Response (QR) codes, Aztec codes, data matrix, digital graphics, Max codes, PDF417, Super codes, UCC RSS-2D barcodes), and other optical codes), or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). In addition, various information can be obtained via the communication component 1440, such as obtaining a location via Internet Protocol (IP) geolocation, ... Signal triangulation to obtain location, obtaining location via detection of NFC beacon signals that can indicate a specific location, etc.

[0091] Glossary

[0092] In this context, "carrier signal" refers to any intangible medium capable of storing, encoding or carrying instructions for execution by a machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device and using any of a number of well-known transmission protocols.

[0093] In this context, a "client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a portable computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a portable computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0094] In this context, a "communication network" refers to one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of a network may include a wireless or cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling may implement any of a variety of types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other standards defined by various standards development organizations, other remote protocols, or other data transmission technologies.

[0095] In this context, an ephemeral message is a message that is accessible for a limited duration. An ephemeral message can be text, an image, a video, etc. The access time for an ephemeral message can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, a message is ephemeral.

[0096] In this context, "machine-readable medium" refers to a component, device, or other tangible medium capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions. The term "machine-readable medium" should also be understood to include any medium or combination of multiple media capable of storing instructions (e.g., code) executed by a machine, such that these instructions, when executed by one or more processors of the machine, cause the machine to perform any one or more of the methods described herein. Therefore, "machine-readable medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0097] In this context, "component" refers to a device, physical entity, or logic with boundaries defined by function or subroutine calls, branch points, application program interfaces (APIs), or other partitioning or modularization techniques for specific processing or control functions. Components can be combined through their interfaces with other components to perform machine processes. A component can be a packaged functional hardware unit designed to be used with other components, and a part of a program that generally performs a specific function of related functions. A component can constitute a software component (e.g., a code embodied on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or an application portion) to operate as a hardware component that performs certain operations described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. A hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A hardware component may also include a programmable logic or circuit that is temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is specially customized to perform the configured function, and is no longer a general purpose processor. It should be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit or in a temporarily configured circuit (e.g., configured by software) may be made for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein. Considering an embodiment in which a hardware component is temporarily configured (e.g., programmed), each hardware component does not need to be configured or instantiated at any instance of time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Thus, the software configures one or more specific processors accordingly, for example, to constitute a specific hardware component at one time instance and to constitute a different hardware component at a different time instance. Hardware components may provide information to other hardware components, or receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components existing at the same time, communication can be achieved by signal transmission between two or more hardware components (for example, on appropriate circuits and buses). In embodiments in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing and obtaining information in a memory structure accessible to multiple hardware components. For example, a hardware component can perform an operation and store the output of the operation in a memory device to which it is coupled in communication. Then, another hardware component can access the memory device at a later time to obtain and process the stored output. The hardware component can also initiate communication with an input or output device and can operate on a resource (for example, a collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily (for example, by software) configured or permanently configured to perform related operations. Whether temporarily or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some operations of the method may be performed by one or more processors or processor-implemented components. In addition, one or more processors may also be operable to support the performance of related operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some operations may be performed by a group of computers (as an example of a machine including a processor), which can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs)). The performance of certain operations may be distributed among processors, not only residing in a single computer, but also deployed on multiple computers. In some example embodiments, the processor or the processor-implemented component may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processor or the processor-implemented component may be distributed in multiple geographic locations.

[0098] In this context, a "processor" refers to any circuit or virtual circuit (a physical circuit emulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and generates corresponding output signals for operating a machine. The processor may be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. The processor may further be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0099] In this context, a "timestamp" refers to a sequence of characters or encoded information that identifies when a specific event occurred, such as giving a date and time, sometimes accurate to a fraction of a second.

[0100] In this context, "lift" is a measure of the performance of a target model in a prediction or classification situation as having an enhanced response (relative to the population) as measured relative to a randomly selected target model.

[0101] In this context of "phoneme alignment", a phoneme is a unit of speech that distinguishes one word from another. A phoneme might consist of a sequence of closure, plosive, and aspiration events; or, a ligature might transition from a back vowel to a front vowel. Therefore, a speech signal can be described not only by the phonemes it contains, but also by the positions of the phonemes. Phoneme alignment can therefore be described as the "time alignment" of the phonemes in the waveform in order to determine the proper order and position of each phoneme in the speech signal.

[0102] In this context, "audio-to-visual conversion" refers to the conversion of an audible speech signal into visible speech, where the visible speech may include mouth shapes representing the audible speech signal.

[0103] In this context, a “Time Delay Neural Network (TDNN)” is an artificial neural network architecture whose main purpose is to process sequential data. An example is converting continuous audio into a stream of categorical phoneme labels for speech recognition.

[0104] "Bidirectional Long Short-Term Memory (BLSTM)" in this context refers to a recurrent neural network (RNN) architecture that can memorize values ​​over arbitrary time intervals. The stored values ​​are not modified as learning proceeds. RNNs allow for both forward and backward connections between neurons. BLSTM is well suited for classification, processing, and forecasting of time series given time lags of unknown size and duration between events.

Claims

1. A method comprising: receiving input to a messaging interface presented at a client device, the input comprising a text string; organizing a collection of message content based on the text string, the one or more media items comprising graphical content; determining a ranked order of the one or more media items from the collection of message content based on the ranking of the media items, wherein the ranking is based on relevance to the text string; and Based on the sort order, a display of the one or more media items is presented within a carousel of suggestions displayed at the location within the messaging interface.

2. The method according to claim 1, wherein: The method further comprises: receiving a selection of a media item from the one or more media items presented within the media carousel; and In response to selecting the media item from the media carousel, a message is generated that includes the media item and the input.

3. The method according to claim 1, wherein: The graphical content of the one or more media items includes an augmented reality filter.

4. The method according to claim 1, wherein: Organizing the collection of the message content based on the text string includes: In response to parsing the keyword from the text string, accessing a media library including a media index; and The collection of message content is organized from the media library based on the keyword.

5. The method according to claim 1, wherein: The set of message content includes a first media item and a second media item, and wherein presenting the media carousel includes: A first graphical icon representing the first media item and a second graphical icon representing the second media item are caused to be displayed within the media carousel.

6. The method according to claim 1, wherein: The method further comprises: receiving a request to generate a message from the client device, the request to generate a message including a keyword and a media item from the media carousel; and In response to receiving the request to generate the message, the keyword is associated with the media item within a media index.

7. The method according to claim 1, wherein: Determining the ranked order of the one or more media items based on the ranking of the media items further comprises: determining a usage metric associated with each of the one or more media items; and The rank order is determined based on the relevance and the usage metric.

8. A system comprising: Memory; as well as at least one hardware processor coupled to the memory and comprising instructions for causing the system to perform operations comprising: receiving input to a messaging interface presented at a client device, the input comprising a text string; organizing a collection of message content based on the text string, the one or more media items comprising graphical content; determining a ranked order of the one or more media items from the collection of message content based on the ranking of the media items, wherein the ranking is based on relevance to the text string; and Based on the sort order, a display of the one or more media items is presented within a carousel of suggestions displayed at the location within the messaging interface.

9. The system according to claim 8, wherein: The operations also include: receiving a selection of a media item from the one or more media items presented within the media carousel; and In response to selecting the media item from the media carousel, a message is generated that includes the media item and the input.

10. The system according to claim 8, wherein: The graphical content of the one or more media items includes an augmented reality filter.

11. The system according to claim 8, wherein: Organizing the collection of the message content based on the text string includes: In response to parsing the keyword from the text string, accessing a media library including a media index; and The collection of message content is organized from the media library based on the keyword.

12. The system according to claim 8, wherein: The set of message content includes a first media item and a second media item, and wherein presenting the media carousel includes: A first graphical icon representing the first media item and a second graphical icon representing the second media item are caused to be displayed within the media carousel.

13. The system according to claim 8, wherein: The operations also include: receiving a request to generate a message from the client device, the request to generate a message including a keyword and a media item from the media carousel; and In response to receiving the request to generate the message, the keyword is associated with the media item within a media index.

14. The system according to claim 8, wherein: Determining the ranked order of the one or more media items based on the ranking of the media items further comprises: determining a usage metric associated with each of the one or more media items; and The rank order is determined based on the relevance and the usage metric.

15. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: receiving input to a messaging interface presented at a client device, the input comprising a text string; organizing a collection of message content based on the text string, the one or more media items comprising graphical content; determining a ranked order of the one or more media items from the collection of message content based on the ranking of the media items, wherein the ranking is based on relevance to the text string; and Based on the sort order, a display of the one or more media items is presented within a carousel of suggestions displayed at the location within the messaging interface.

16. The non-transitory machine-readable storage medium of claim 15, wherein: The operations also include: receiving a selection of a media item from the one or more media items presented within the media carousel; and In response to selecting the media item from the media carousel, a message is generated that includes the media item and the input.

17. The non-transitory machine-readable storage medium of claim 15, wherein: The graphical content of the one or more media items includes an augmented reality filter.

18. The non-transitory machine-readable storage medium of claim 15, wherein: Organizing the collection of the message content based on the text string includes: In response to parsing the keyword from the text string, accessing a media library including a media index; and The collection of message content is organized from the media library based on the keyword.

19. The non-transitory machine-readable storage medium of claim 15, wherein: The set of message content includes a first media item and a second media item, and wherein presenting the media carousel includes: A first graphical icon representing the first media item and a second graphical icon representing the second media item are caused to be displayed within the media carousel.

20. The non-transitory machine-readable storage medium of claim 15, wherein: The operations also include: receiving a request to generate a message from the client device, the request to generate a message including a keyword and a media item from the media carousel; and In response to receiving the request to generate the message, the keyword is associated with the media item within a media index.

Images