Content suggestion system

Abstract

A content suggestion system generates and causes display of a set of chat suggestions based on messages received at a client device. The content suggestion system is configured to display, at a client device, a message including message content, and identify content selected by a user of the client device to be included in a response to the message received at the client device. A content suggestion system tracks and stores the number of times in which a particular content pair continuously appears in a chat context, and calculates an order of the content in a set of available content. When a subsequent message is displayed at the client device that includes the content in the pair of content, the content suggestion system retrieves and presents the set of content as a suggestion based on the corresponding ranking.

Description

[0001] This application is a divisional application of a patent application with an application date of April 10, 2019, application number 201980025185.2, and invention name “Content Recommendation System”.

[0002] priority

[0003] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 15 / 952,859, filed on April 13, 2018, the benefit of which is hereby claimed and is incorporated herein by reference in its entirety. Technical Field

[0004] Embodiments of the present disclosure relate generally to mobile computing technology and, more particularly and without limitation, to a system for managing the content of a chat session between users. Background Art

[0005] Mobile devices (e.g., smartphones, personal digital assistants, tablets, smart watches, etc.) are ubiquitous. Mobile devices provide users with advanced computing capabilities and services, such as voice communications, text and other messaging communications, video and other multimedia communications, streaming services, etc. Typically, users send and receive data (e.g., messages), or otherwise communicate with each other via mobile devices.

[0006] Existing text-based communication applications (e.g., text messaging, instant messaging, chat, email, etc.) provide users with user-selectable options to supplement input text with graphical elements (e.g., emojis and other ideograms, pictographs, images, GIFs, animations, videos, and other multimedia content). Users can search for and select various elements to add to their messages to provide additional context to the text-based content. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the drawing number in which the element is first introduced.

[0008] Figure 1 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network, wherein the messaging system includes a content suggestion system, in accordance with some embodiments.

[0009] Figure 2 is a block diagram showing further details regarding a messaging system according to an example embodiment.

[0010] Figure 3 is a block diagram illustrating various modules of a content suggestion system according to certain example embodiments.

[0011] Figure 4 is a flow chart illustrating a method for sorting chat suggestions according to certain example embodiments.

[0012] Figure 5 is a flow chart illustrating a method for presenting chat suggestions according to certain example embodiments.

[0013] Figure 6 is a flow chart illustrating a method for presenting chat suggestions according to certain example embodiments.

[0014] Figure 7 is a diagram illustrating a method for sorting chat content to be presented as chat suggestions, according to certain example embodiments.

[0015] Figure 8 is an illustration of an interface for displaying chat suggestions, according to certain example embodiments.

[0016] Fig. 9 is an illustration of an interface for displaying chat suggestions, according to certain example embodiments.

[0017] Fig.10 is a block diagram illustrating a representative software architecture that may be used in conjunction with the various hardware architectures described herein and to implement the various embodiments.

[0018] Fig.11 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. DETAILED DESCRIPTION

[0019] Conventional chat systems typically provide certain "chat suggestions" based on explicit (or implicit) input received from a user, where the chat suggestions typically include subsequent text elements (e.g., words, letters, punctuation marks, etc.). These conventional chat systems are capable of providing suggestions based (at best) on input received at a client device (e.g., a user of the client device begins drafting a response), or based on an existing dictionary or lexicon of commonly used words or phrases. Recently, chat systems have been developed that enable users to communicate with each other through unconventional message content that does not conform to existing chat suggestion frameworks, in part due to the abstract nature of the content and the constantly changing contextual meaning of the content itself.

[0020] For example, some platforms enable individuals to communicate with each other by generating and sharing content such as pictographs and ideograms via a chat interface, where the meaning or context of the pictograph or ideogram is not immediately obvious. As a result, it is challenging to generate chat and messaging suggestions that include such content, and currently it may require a significant amount of front-end work to categorize and define the content in a database. While this may be an adequate solution in the short term, such databases require significant maintenance and administrator input in order to regularly add or update content.

[0021] Embodiments described herein relate to a content suggestion system that generates and causes display of a chat suggestion set based on a message received at a client device. The content suggestion system is configured to display a message including message content at the client device and identify content selected by a user of the client device to be included in a response to the message received at the client device. The content suggestion system tracks and stores the number of times a particular content pair appears consecutively in a chat context and calculates a ranking of the content in a set of available content. When a subsequent message including the content of the content pair is displayed at the client device, the content suggestion system retrieves and provides the content set as a suggestion based on the corresponding ranking.

[0022] As discussed herein, "message content" may refer to various types of media items that may be included within a message (e.g., a text message, an ephemeral message, an email, a private message, a direct message, a forum post, etc.) displayed at a client device. For example, message content may include text strings, pictographs or ideograms (e.g., emoticons), images, audio data, video data, and personalized avatars (e.g., Bitmojis). Content pairs may include any combination of the various types of message content described above.

[0023] The content suggestion system is configured to cause a message including message content (e.g., first content) to be displayed at a client device, and to receive a selection of second content from a selection of content options. In response to receiving the selection of the second chat content, the content suggestion system indexes and stores a content pair including the first content and the second content, and increments a usage counter corresponding to the content pair and indicating a number of times the content pair occurs in the context of the message (i.e., when the second content is used in a response to a message including the first content). The content suggestion system ranks the second content in a content collection based on a usage metric for the content pair.

[0024] In response to receiving a subsequent message including the first content at the client device, the content suggestion system retrieves the content from the content database based on the ranking corresponding to the content pair including the first content. For example, the content suggestion system may retrieve the second content from the database based on the usage metrics of the content pair including the first content and the second content. The content suggestion system generates a presentation of the retrieved content, wherein the ranking (i.e., display order) of the content is based on the corresponding usage metrics. The presentation of the suggested content may be displayed in a portion of the chat interface, for example, near a keyboard element. In some embodiments, the presentation of the suggested content is displayed in response to receiving a user input displaying the keyboard element.

[0025] Sometimes, the ranking of content can result in identical or nearly identical content being displayed in the most prominent position in the suggested content. In some embodiments, the content suggestion system identifies the highest ranked portion of content (e.g., the top 6) and randomizes the display order of the highest ranked portion of content in the presentation of content to provide additional variety and a more interesting set of suggestions to the user.

[0026] Figure 1 1 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and related content) over a network. The messaging system 100 includes a plurality of client devices 102, each client device hosting a plurality of applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0027] Thus, each messaging client application 104 is able to communicate and exchange data with another messaging client application 104 and a messaging server system 108 via a network 106. The data exchanged between messaging client applications 104 and between messaging client applications 104 and messaging server systems 108 includes functionality (e.g., commands to call functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0028] The messaging server system 108 provides server-side functionality to certain messaging client applications 104 via the network 106. Although certain functions of the messaging system 100 are described herein as being performed by the messaging client application 104 or by the messaging server system 108, it should be understood that the location of certain functions in the messaging client application 104 or the messaging server system 108 is a design choice. For example, it may be technically preferable to first deploy certain technologies and functions within the messaging server system 108 and then migrate the technologies and functions to the messaging client application 104 where the client device 102 has sufficient processing power.

[0029] The messaging server system 108 supports various services and operations provided to the messaging client applications 104. Such operations include sending data to the messaging client applications 104, receiving data from the messaging client applications 104, and processing data generated by the messaging client applications 104. In some embodiments, the data includes, by way of example, message content, client device information, geographic location information, media annotations and overlays, message content persistence conditions, social network information, and live event information. In other embodiments, other data is used. The data exchange within the messaging system 100 is invoked and controlled by functions available via the GUI of the messaging client applications 104.

[0030] Turning now specifically to the messaging server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

[0031] Specifically, the application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the application program interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client application 104 in order to invoke the functions of the application server 112. The application program interface (API) server 110 exposes various functions supported by the application server 112, including: account registration; login functionality; sending messages from a particular messaging client application 104 to another messaging client application 104 via the application server 112; sending media files (e.g., images or videos) from the messaging client application 104 to the messaging server application 114 for possible access by another messaging client application 104; setting of collections of media data (e.g., stories); retrieval of a friend list of a user of the client device 102; retrieval of such collections; retrieval of messages and content; adding and removing friends from a social graph; the position of friends in a social graph; and open application events (e.g., related to a messaging client application 104).

[0032] The application server 112 hosts a number of applications and subsystems, including a messaging server application 114, an image processing system 116, a social networking system 122, and a content suggestion system 124. The messaging server application 114 implements a number of message processing techniques and functions, particularly relating to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client application 104. As will be described in further detail, text and media content from multiple sources may be aggregated into collections of content (e.g., referred to as stories, or galleries). The messaging server application 114 then makes these collections available to the messaging client application 104. Other processor and memory intensive processing of the data may also be performed on the server side by the messaging server application 114, given the hardware requirements of such processing.

[0033] The application server 112 also includes an image processing system 116 that is dedicated to performing various image processing operations, typically with respect to images or videos received within the payload of messages at the messaging application server 114 .

[0034] The social networking system 122 supports various social networking functions and services and makes these functions and services available to the messaging server application 114. To this end, the social networking system 122 maintains and accesses an entity graph 304 within the database 120. Examples of functions and services supported by the social networking system 122 include identifying other users of the messaging system 100 with whom a particular user has a relationship or who the particular user is "following," as well as identifying other entities and interests of a particular user.

[0035] The application server 112 is communicatively coupled to a database server 118 , which facilitates access to a database 120 in which data associated with messages processed by the messaging server application 114 is stored.

[0036] Figure 2 1 is a block diagram showing further details about the messaging system 100 according to an example embodiment. Specifically, the messaging system 100 is shown to include a messaging client application 104 and an application server 112, which in turn embody a number of specific subsystems, namely, a transient timer system 202, a collection management system 204, and an annotation system 206.

[0037] The transient timer system 202 is responsible for implementing temporary access to content, such as personalized avatars (e.g., emoticons), permitted by the messaging client application 104 and the messaging server application 114. To this end, the transient timer system 202 incorporates a plurality of timers that selectively display and enable access to messages and associated content via the messaging client application 104 based on durations and display parameters associated with the messages, message collections, or graphical elements. Further details regarding the operation of the transient timer system 202 are provided below.

[0038] The collection management system 204 is responsible for managing collections of media (e.g., collections of text, image video and audio data, personalized avatars). In some examples, collections of content (e.g., messages including images, videos, text, and audio) may be organized into "event galleries" or "event stories." Such collections may be available for a specified period of time (e.g., the duration of an event related to the content). For example, content related to a concert may be available as a "story" for the duration of the concert. The collection management system 204 may also be responsible for publishing an icon that provides a notification of the existence of a particular collection to the user interface of the messaging client application 104.

[0039] The collection management system 204 also includes a curation interface 208 that allows a collection manager to manage and curate a particular collection of content. For example, the curation interface 208 enables an event organizer to curate a collection of content related to a particular event (e.g., to remove inappropriate content or redundant messages). In addition, the collection management system 204 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some embodiments, compensation (e.g., currency, non-monetary credits or points associated with a messaging system or a third-party reward system, travel miles, access to artwork or special lenses, etc.) may be paid to a user for including user-generated content in a collection. In this case, the curation interface 208 operates to automatically pay fees to such users for using their content.

[0040] The annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content associated with a message. For example, the annotation system 206 provides functions related to generating and publishing media overlays for messages processed by the messaging system 100. The annotation system 206 operably provides media overlays to the messaging client application 104 based on the geographic location of the client device 102. In another example, the annotation system 206 operably provides media overlays to the messaging client application 104 based on other information (e.g., social network information of the user of the client device 102). The media overlay may include audio and visual content and visual effects. Examples of audio and visual content include: pictures, text, logos, animations and sound effects, and animated facial models, such as those generated by the content suggestion system 124. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos) at the client device 110. For example, the media overlay includes text that can be superimposed on a photo taken by the client device 102. In another example, the media overlay includes an identification of a location overlay (e.g., Venice Beach), the name of a live event, or the name of a business overlay (e.g., Beach Cafe). In another example, the annotation system 206 uses the geographic location of the client device 102 to identify a media overlay that includes the name of a business at the geographic location of the client device 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 120 and may be accessed by the database server 118.

[0041] In one example embodiment, the annotation system 206 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. Users can also specify under what circumstances specific media overlays should be provided to other users. The annotation system 206 generates a media overlay including the uploaded content and associates the uploaded content with the selected geographic location.

[0042] In another example embodiment, the annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media overlays associated with geographic locations via a bidding process. For example, the annotation system 206 associates the media overlays of the merchant with the highest bid with the corresponding geographic location within a predetermined amount of time.

[0043] Figure 3 1 is a block diagram illustrating components of a content suggestion system 124 that configure the content suggestion system 124 to perform operations including: displaying a message within a chat interface at a client device (e.g., client device 102A), wherein the message includes first content, such as a particular personalized avatar (e.g., an emoticon); receiving a selection of second content from client device 102A, wherein the second content may include a different personalized avatar to be assigned to a response to the message; in response to receiving the selection of the second content from client device 102A, incrementing a usage metric corresponding to a content pair including the first content and the second content; and, according to some example embodiments, ranking the second content in a content collection based on the usage metric of the corresponding content pair. The content suggestion system 124 is shown to include a presentation module 302, a ranking module 304, a content selection module 306, and a communication module 308, each of which is configured to communicate with each other (e.g., via a bus, shared memory, or switch). Any one or more of these modules may be implemented using one or more processors 310 (e.g., by configuring such one or more processors to perform the functions described for the module), and thus may include one or more processors 310.

[0044] According to some example embodiments, presentation module 302 is configured to generate and cause display of presentations of notifications and messages including content at a client device (eg, client device 102A and client device 102B).

[0045] According to some example embodiments, the ranking module 304 is configured to calculate usage metrics for corresponding content pairs based on user activities including selection of content via a client device.

[0046] According to some example embodiments, the content selection module 306 is configured to select and curate (eg, via the presentation module 302 ) a set of content to be presented as chat suggestions based on the ranking performed by the ranking module 304 .

[0047] According to certain example embodiments, the communication module 308 is configured to facilitate message exchanges and communications between client devices and the network 106 .

[0048] Any one or more of the described modules may be implemented using separate hardware (e.g., one or more processors 310 of the machine) or a combination of hardware and software. For example, any module described in the content suggestion system 124 may physically include an arrangement of one or more processors 310 (e.g., a subset of one or more processors of the machine or a subset thereof), which is configured to perform the operations described herein for the module. As another example, any module of the content suggestion system 124 may include software, hardware, or both, which configures an arrangement of one or more processors 310 (e.g., one or more processors of the machine) to perform the operations described herein for the module. Therefore, different modules of the content suggestion system 124 may include and configure different arrangements of such processors 310 or a single arrangement of such processors 310 at different points in time. In addition, any two or more modules of the content suggestion system 124 may be combined into a single module, and the functions described herein for a single module may be subdivided between multiple modules. In addition, according to various example embodiments, the modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0049] Figure 4 4 is a flow chart illustrating a method 400 for sorting chat suggestions according to some example embodiments. The operation of the method 400 may be described above with reference to Figure 3 The module described is executed. Figure 4 As shown, method 400 includes one or more operations 402 , 404 , 406 , and 408 .

[0050] At operation 402, the presentation module 302 causes a message to be displayed in a chat interface at the client device 102A, wherein the message includes a first content. For example, the message may be generated by the client device 102B and sent to the client device 102A via the network 106. The first content of the message includes media content, such as a personalized avatar, such as an emoticon.

[0051] At operation 404, the communication module 308 receives a selection of second content at the client device 102A, wherein the second content is to be assigned to a response to the message displayed at the client device 102A. For example, the user of the client device 102A may draft a response to the message by selecting from a selection of personalized avatars or generating a personalized avatar.

[0052] At operation 406, the sorting module 304 increments a usage metric associated with a content pair including the first content and the second content in response to receiving a selection of the second content from the client device 102A. In some embodiments, the database 120 may include a collection of content pairs based on the selection of the content. The sorting module 304 may index the content pairs based on the user's selection of the content and store the content pairs within the database 120. For example, a user of the client device 102A may receive a message including the first content (e.g., "emoji A"), and when drafting a response, a user of the client device 102B selects the second content (e.g., "emoji B"). In response to the user selecting the second content, the sorting module 304 indicates and stores the content pair including the first content and the second content, and increments a usage metric that defines the number of times "emoji B" was sent in response to "emoji A". At operation 408, the sorting module 304 sorts the content pairs in the collection of content pairs based on the corresponding usage metrics.

[0053] Figure 5 is a flow chart illustrating a method 500 for presenting chat suggestions according to some example embodiments. The operation of the method 500 may be described above with reference to Figure 3 Method 500 may be performed as part of or after one or more operations of method 400. Figure 5 As shown, method 500 includes one or more operations 502 , 504 , and 506 .

[0054] In operation 502, the presentation module 302 generates and causes display of a second message (e.g., a message subsequent to the message displayed in operation 402 of method 400) within the chat interface of the client device 102A, wherein the second message includes content such as the first content described in method 400.

[0055] In response to displaying a presentation of the second message at the client device 102A, the content selection module 306 generates a presentation of a set of suggested content based on the content displayed within the second message. The presentation of the set of suggested content includes displaying the second content at a position in the set of suggested content, wherein the position of the second content is based on the ranking of the second content. At operation 506, the presentation module 302 causes the presentation of the set of suggested content to be displayed within a portion of the chat interface at the client device 102A.

[0056] Figure 6 6 is a flow chart illustrating a method 600 for presenting chat suggestions according to some example embodiments. The operation of the method 600 may be described above with reference to Figure 3 Method 600 may be performed as a part of method 400 (e.g., a precursor or subroutine of one of the operations). Figure 6 As shown, method 600 includes one or more operations 602 , 604 , 608 , and 610 .

[0057] At operation 602 , the presentation module 302 causes a message to be displayed at the client device 102A, wherein the message includes message content including a text string.

[0058] In operation 604, in response to displaying a message including a text string at the client device 102A, the content selection module 306 converts the text string into a tag. The tag can be based on the text string itself or based on a category. For example, the text string of the message can be displayed as "Hello". The content selection module 306 can identify tags corresponding to various greetings, wherein the tags are associated with one or more media content items in the database 120. In operation 606, the content selection module 306 (e.g., first media content) identifies media content based on the text string and the tags.

[0059] In response to identifying the first content based on presentation of the second message including the text string at the client device 102A, the content selection module 306 generates a presentation of a set of suggested content based on the first content displayed within the message at operation 608. The content selection module 306 retrieves content pairs associated with the first content and generates a presentation of a set of suggested content based on the content pairs, wherein a ranked order of the set of suggested content is based on a usage metric corresponding to each content pair.

[0060] At operation 610 , the presentation module 302 causes a presentation of the suggested set of content to be displayed within a portion of the chat interface of the client device 102A.

[0061] Figure 7 is a diagram 700 illustrating a method for ordering chat content to be presented as chat suggestions, in accordance with certain example embodiments.

[0062] As described above, with respect to method 400, ranking module 304 calculates usage metrics for content pairs and ranks content based on the usage metrics. According to some example embodiments, ranking module 304 may generate one or more matrices (e.g., matrices 702 and 704), wherein each row of the matrix (e.g., row 706 and row 708) includes usage metrics for content pairs (i.e., the number of times a particular content pair was used). In some embodiments, ranking module 304 may generate a personalized matrix (e.g., matrix 702) and a general matrix (e.g., matrix 704), wherein the personalized matrix corresponds to specific usage metrics for a particular user of client device 102A, and the general matrix corresponds to usage metrics for a group of users.

[0063] The ranking module 304 identifies the highest ranked content pairs based on the usage metrics and generates ranked index lists 710 and 712, wherein the ranked index lists include lists of content pairs based on content pair identifiers such as content pair identifier 714 (e.g., "2500"). In some embodiments, the first element in the list (i.e., the content pair identifier) ​​corresponds to the highest ranked pair based on the usage metrics.

[0064] The sorting module removes duplicates between the sorted index lists 710 and 712 and interleaves the sorted index lists to generate a final interleaved index list 716 that includes the suggested content set. The presentation module 302 generates a presentation of the suggested content set based on the final interleaved index list 716.

[0065] Figure 8 is an illustration of an interface 800 for displaying a chat suggestion collection 802, according to certain example embodiments. Figure 4 As described in method 400, the content viewed in interface 800 is displayed at client device 102A.

[0066] The presentation module 302 can display a message in the interface 800, wherein the message includes the first content 804. Figure 8 As shown, the first content 804 includes a personalized avatar, such as an emoticon.

[0067] In response to receiving the message including the first content 804 , the presentation module 302 generates and causes display of a suggested content set 802 , wherein the suggested content set 802 is based on the first content 804 .

[0068] Fig. 9 is an illustration of an interface 900 for displaying a collection of chat suggestions, according to certain example embodiments. Figure 6 As described in method 600 of , presentation module 302 may generate and cause display of a message (eg, message 904) including first content, which may include a text string.

[0069] In response to receiving the message 904 including the text string, the content selection module 306 converts the text string of the message 904 into a tag and identifies media content (eg, first media content) based on the text string and the tag.

[0070] As described in operation 608 of method 600 , in response to identifying the first content based on the tag from message 904 , content selection module 306 generates a presentation of a set of suggested content 902 based on the first content associated with the tag.

[0071] Software Architecture

[0072] Fig.10 is a block diagram illustrating an example software architecture 1006 that may be used in conjunction with the various hardware architectures described herein. Fig.10 is a non-limiting example of a software architecture, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 1006 may be implemented in a variety of environments such as Fig.11 1100 is executed on the hardware of the machine 1100, which includes, among other things, a processor 1104, a memory 1114, and I / O components 1118. A representative hardware layer 1052 is shown and may represent, for example, Fig.10 The machine 1000 of the embodiment of the present invention is shown in FIG. 1000. The representative hardware layer 1052 includes a processing unit 1054 having associated executable instructions 1004. The executable instructions 1004 represent executable instructions of the software architecture 1006, including implementations of the methods, components, etc. described herein. The hardware layer 1052 also includes a memory and / or storage module memory / storage device 1056, which also has executable instructions 1004. The hardware layer 1052 may also include other hardware 1058.

[0073] exist Fig.10 In the example architecture of , software architecture 1006 can be conceptualized as a stack of layers, each of which provides specific functionality. For example, software architecture 1006 may include layers such as operating system 1002, library 1020, application 1016, and presentation layer 1014. In operation, applications 1016 and / or other components within these layers can call application program interface (API) API call 1008 through the software stack and receive responses in response to API call 1008. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 1018, while other operating systems may provide such a layer. Other software architectures may include other or different layers.

[0074] The operating system 1002 can manage hardware resources and provide common services. The operating system 1002 may include, for example, a kernel 1022, services 1024, and drivers 1026. The kernel 1022 may serve as an abstraction layer between the hardware and other software layers. For example, the kernel 1022 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 1024 may provide other common services to other software layers. Drivers 1026 are responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers 1026 include display drivers, camera drivers, drives, flash drives, serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0075] The library 1020 provides a common infrastructure used by the application 1016 and / or other components and / or layers. The library 1020 provides functions that allow other software components to perform tasks in an easier manner than directly interfacing with the underlying operating system 1002 functions (e.g., kernel 1022, service 1024, and / or driver 1026). The library 1020 may include a system library 1044 (e.g., a C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 1020 may include an API library 1046, such as a media library (e.g., a library that supports the rendering and manipulation of various media formats (e.g., MPREG4, H.264, MP3, AAC, AMR, JPG, PNG)), a graphics library (e.g., an OpenGL framework that can be used to render 2D and 3D graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a network library (e.g., WebKit that can provide network browsing functions), etc. The library 1020 may also include a variety of other libraries 1048 to provide many other APIs to the applications 1016 and other software components / modules.

[0076] The framework / middleware 1018 (sometimes also referred to as middleware) provides a higher level common infrastructure that can be used by the applications 1016 and / or other software components / modules. For example, the framework / middleware 1018 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. The framework / middleware 1018 can provide a wide range of other APIs that can be used by the applications 1016 and / or other software components / modules, some of which can be specific to a particular operating system 1002 or platform.

[0077] Applications 1016 include built-in applications 1038 and / or third-party applications 1040. Examples of representative built-in applications 1038 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 1040 may include applications that are created by entities other than the seller of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) for developing applications and can be used on platforms such as IOS TM ANDROID TM , or other mobile operating systems. The third-party application 1040 can call API calls 1008 provided by the mobile operating system (eg, operating system 1002) to facilitate the functionality described herein.

[0078] Applications 1016 may use built-in operating system functionality (e.g., kernel 1022, services 1024, and / or drivers 1026), libraries 1020, and framework / middleware 1018 to create a user interface to interact with a user of the system. Alternatively, or in addition, in some systems, interaction with the user may occur through a presentation layer (e.g., presentation layer 1014). In these systems, the application / component "logic" may be separated from the various aspects of the application / component that interact with the user.

[0079] Fig.11 1 is a block diagram illustrating components of a machine 1100 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. Specifically, Fig.11 A diagrammatic representation of a machine 1100 in the example form of a computer system is shown, in which instructions 1110 (e.g., software, programs, applications, applet, application program, or other executable code) for causing the machine 1100 to perform any one or more of the methods discussed herein may be executed. In this way, the instructions 1110 may be used to implement the modules or components described herein. The instructions 1110 convert a general, unprogrammed machine 1100 into a specific machine 1100 that is programmed to perform the functions described and shown in the described manner. In alternative embodiments, the machine 1100 operates as a standalone device, or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1100 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1100 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network appliance, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1116 in sequence or otherwise, which specify actions to be taken by the machine 1100. In addition, while only a single machine 1100 is shown, the term "machine" should also be taken to include a collection of machines that individually or collectively execute instructions 1110 to perform any one or more of the methodologies discussed herein.

[0080] The machine 1100 may include a processor 1104, a memory / storage device 1106, and an I / O component 1118, which may be configured to communicate with each other, for example, via a bus 1102. The memory / storage device 1106 may include a memory 1114 (such as a main memory, or other memory), and a storage unit 1116, both of which may be accessible to the processor 1104, such as via the bus 1102. The storage unit 1116 and the memory 1114 store instructions 1110 that embody any one or more of the methods or functions described herein. The instructions 1110 may also reside, in whole or in part, within the memory 1114, within the storage unit 1116, within at least one of the processors 1104 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution by the machine 1100. Thus, the memory 1114, the storage unit 1116, and the memory of the processor 1104 are examples of machine-readable media.

[0081] I / O components 1118 may include a variety of components to receive input, provide output, generate output, send information, exchange information, collect measurements, etc. The specific I / O components 1118 included in a particular machine 1100 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that I / O components 1118 may be included in Fig.11 110 and 1111. The I / O components 1118 are grouped according to function only for the purpose of simplifying the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 1118 may include output components 1126 and input components 1128. The output components 1126 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1128 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide touch position and force or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0082] In further example embodiments, the I / O component 1118 may include a biometric component 1130, a motion component 1134, an environmental component 1136, or a position component 1138, among various other components. For example, the biometric component 1130 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring bio-signals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1134 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental component 1136 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety or measuring pollutants in the atmosphere), or other components that may provide indications, measurements, or signals corresponding to the surrounding physical environment. Position component 1138 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects the altitude from which the air pressure can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.

[0083] A variety of technologies may be used to achieve communication. I / O components 1118 may include a communication component 1140 that is operable to couple machine 1100 to network 1132 or device 1120 via coupling 1122 and coupling 1124, respectively. For example, communication component 1140 may include a network interface component or other suitable device that interfaces with network 1132. In further examples, communication component 1140 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g. Low energy consumption), Device 1120 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via a universal serial bus (USB)).

[0084] In addition, the communication component 1140 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1140 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as universal product codes (UPC) barcodes, multi-dimensional barcodes (e.g., Quick Response (QR) codes, Aztec codes, data matrix, digital graphics, Max codes, PDF417, Super codes, UCC RSS-2D barcodes), and other optical codes), or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). In addition, various information can be obtained via the communication component 1140, such as obtaining a location via Internet Protocol (IP) geolocation, ... Signal triangulation to obtain location, obtaining location via detection of NFC beacon signals that can indicate a specific location, etc.

[0085] Glossary

[0086] In this context, "carrier signal" refers to any intangible medium capable of storing, encoding or carrying instructions for execution by a machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device and using any of a number of well-known transmission protocols.

[0087] In this context, a "client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a portable computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a portable computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0088] In this context, a "communication network" refers to one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of a network may include a wireless or cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling may implement any of a variety of types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other standards defined by various standards development organizations, other remote protocols, or other data transmission technologies.

[0089] In this context, an ephemeral message is a message that is accessible for a limited duration. An ephemeral message can be text, an image, a video, etc. The access time for an ephemeral message can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, a message is ephemeral.

[0090] In this context, "machine-readable medium" refers to a component, device, or other tangible medium capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions. The term "machine-readable medium" should also be understood to include any medium or combination of multiple media capable of storing instructions (e.g., code) executed by a machine, such that these instructions, when executed by one or more processors of the machine, cause the machine to perform any one or more of the methods described herein. Therefore, "machine-readable medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0091] In this context, "component" refers to a device, physical entity, or logic with boundaries defined by function or subroutine calls, branch points, application program interfaces (APIs), or other partitioning or modularization techniques for specific processing or control functions. Components can be combined through their interfaces with other components to perform machine processes. A component can be a packaged functional hardware unit designed to be used with other components, and a part of a program that generally performs a specific function of related functions. A component can constitute a software component (e.g., a code embodied on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or an application portion) to operate as a hardware component that performs certain operations described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. A hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A hardware component may also include a programmable logic or circuit that is temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is specially customized to perform the configured function, and is no longer a general purpose processor. It should be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit or in a temporarily configured circuit (e.g., configured by software) may be made for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein. Considering an embodiment in which a hardware component is temporarily configured (e.g., programmed), each hardware component does not need to be configured or instantiated at any instance of time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Thus, the software configures one or more specific processors accordingly, for example, to constitute a specific hardware component at one time instance and to constitute a different hardware component at a different time instance. Hardware components may provide information to other hardware components, or receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components existing at the same time, communication can be achieved by signal transmission (e.g., on appropriate circuits and buses) between two or more hardware components. In embodiments in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing and retrieving information in a memory structure accessible to multiple hardware components. For example, a hardware component can perform an operation and store the output of the operation in a memory device to which it is coupled in communication. Then, another hardware component can access the memory device at a later time to retrieve and process the stored output. The hardware component can also initiate communication with an input or output device and can operate on a resource (e.g., a collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily (e.g., by software) configured or permanently configured to perform related operations. Whether temporarily or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some operations of the method may be performed by one or more processors or processor-implemented components. In addition, one or more processors may also be operable to support the performance of related operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some operations may be performed by a group of computers (as an example of a machine including a processor), which can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs)). The performance of certain operations may be distributed among processors, not only residing in a single computer, but also deployed on multiple computers. In some example embodiments, the processor or the processor-implemented component may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processor or the processor-implemented component may be distributed in multiple geographic locations.

[0092] In this context, a "processor" refers to any circuit or virtual circuit (a physical circuit emulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and generates corresponding output signals for operating a machine. The processor may be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. The processor may further be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0093] In this context, a "timestamp" refers to a sequence of characters or encoded information that identifies when a specific event occurred, such as giving a date and time, sometimes accurate to a fraction of a second.

[0094] In this context, "lift" is a measure of the performance of a target model in a prediction or classification situation because of having an enhanced response (relative to the population) as measured relative to a randomly selected target model.

[0095] In this context of "phoneme alignment", a phoneme is a unit of speech that distinguishes one word from another. A phoneme might consist of a sequence of closure, plosive, and aspiration events; or, a ligature might transition from a back vowel to a front vowel. Therefore, a speech signal can be described not only by the phonemes it contains, but also by the positions of the phonemes. Phoneme alignment can therefore be described as the "time alignment" of the phonemes in the waveform in order to determine the proper order and position of each phoneme in the speech signal.

[0096] In this context, "audio-to-visual conversion" refers to the conversion of an audible speech signal into visible speech, where the visible speech may include mouth shapes representing the audible speech signal.

[0097] A “Time Delay Neural Network (TDNN)” in this context is an artificial neural network architecture whose main purpose is to process sequential data. An example is converting continuous audio into a stream of categorical phoneme labels for speech recognition.

[0098] "Bidirectional Long Short-Term Memory (BLSTM)" in this context refers to a recurrent neural network (RNN) architecture that can memorize values ​​over arbitrary time intervals. The stored values ​​are not modified as learning proceeds. RNNs allow for both forward and backward connections between neurons. BLSTM is well suited for classification, processing, and forecasting of time series given time lags of unknown size and duration between events.

Claims

1. A method comprising: causing display of a message at a client device, the message comprising first content; identifying at least the second content from a group of content based on a content pair association between the first content and the second content; determining a ranking of the content pair associations based on the usage metric; as well as A presentation of the second content is caused to be displayed in a collection of content at the client device based on the ranking.

2. The method according to claim 1, wherein: The usage metric includes a number of times the second content is included in a response to the first content.

3. The method according to claim 1, wherein: The method further comprises: determining, based on the ranking, that the second content is in a highest-ranked portion of a plurality of media content associated with the set of content; and The presentation of the content collection is caused to be displayed within an interface element at the client device, the presentation of the content collection comprising display of a highest ranked portion of the media content collection that includes the second content.

4. The method according to claim 3, wherein: The ranking order of the highest ranked parts is randomized.

5. The method according to claim 1, wherein: Identifying the second content further includes: determining a tag based on the first content of the message, the tag corresponding to a category associated with the first content; Based on a content pair association between the first content and the second content, the second content is identified from a group of content associated with the tag.

6. The method according to claim 1, wherein: The first content of the first message includes a graphical avatar.

7. The method according to claim 1, wherein: The first content of the first message includes a text string.

8. A system comprising: Memory; as well as at least one hardware processor coupled to the memory and comprising instructions for causing the system to perform operations comprising: causing display of a message at a client device, the message comprising first content; identifying at least the second content from a group of content based on a content pair association between the first content and the second content; determining a ranking of the content pair associations based on the usage metrics; and A presentation of the second content is caused to be displayed in a collection of content at the client device based on the ranking.

9. The system according to claim 8, wherein: The usage metric includes a number of times the second content is included in a response to the first content.

10. The system according to claim 8, wherein: The operations also include: determining, based on the ranking, that the second content is in a highest-ranked portion of a plurality of media content associated with the set of content; and The presentation of the content collection is caused to be displayed within an interface element at the client device, the presentation of the content collection comprising display of a highest ranked portion of the media content collection that includes the second content.

11. The system according to claim 10, wherein: The ranking order of the highest ranked parts is randomized.

12. The system according to claim 8, wherein: Identifying the second content further includes: determining a tag based on the first content of the message, the tag corresponding to a category associated with the first content; Based on a content pair association between the first content and the second content, the second content is identified from a group of content associated with the tag.

13. The system according to claim 8, wherein: The first content of the first message includes a graphical avatar.

14. The system according to claim 8, wherein: The first content of the first message includes a text string.

15. A machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: causing display of a message at a client device, the message comprising first content; identifying at least the second content from a group of content based on a content pair association between the first content and the second content; determining a ranking of the content pair associations based on the usage metric; as well as A presentation of the second content is caused to be displayed in a collection of content at the client device based on the ranking.

16. The machine-readable storage medium of claim 15, wherein: The usage metric includes a number of times the second content is included in a response to the first content.

17. The machine-readable storage medium of claim 15, wherein: The operations also include: determining, based on the ranking, that the second content is in a highest-ranked portion of a plurality of media content associated with the set of content; and The presentation of the content collection is caused to be displayed within an interface element at the client device, the presentation of the content collection comprising display of a highest ranked portion of the media content collection that includes the second content.

18. The machine-readable storage medium of claim 17, wherein: The ranking order of the highest ranked parts is randomized.

19. The machine-readable storage medium of claim 15, wherein: Identifying the second content further includes: determining a tag based on the first content of the message, the tag corresponding to a category associated with the first content; Based on a content pair association between the first content and the second content, the second content is identified from a group of content associated with the tag.

20. The machine-readable storage medium of claim 15, wherein: The first content of the first message includes a graphical avatar.

Images