Automatic remembering content item configuration

Abstract

An automated memory system can automatically identify and present content items that match a user's natural language (NL) input. The automated memory system can compute a match score between the NL input and individual memory content items of a plurality of potential memory content items. The automated memory system can compute match scores using various algorithms and / or machine learning models, such as using an image / NL matching process to obtain a first match score, using a tag matching process to obtain a second match score, and / or obtaining a combined match score from the first match score and the second match score. The automated memory system can select one or more content items of the plurality of content items that have one or more highest match scores. The automated memory system can provide the selected content items, such as by providing the content items to the NL, automatically displaying or playing the content items, inserting the content items into a conversation thread, etc.

Description

Technical Field

[0001] This disclosure pertains to automatically selecting content items that match natural language input. Background Technology

[0002] With the proliferation of mobile devices and other recording devices that capture fleeting moments in people's lives, a vast amount of "memory content items," such as photos, audio, and video, have become available. Sharing these memory content items further increases their quantity, allowing people to access not only their own captured memories but also those of their friends, acquaintances, and publicly shared content. People might spend hours simply organizing or searching through their large collections of memory content items, which can be cumbersome, for example, when a particular item is needed at any given moment, such as when someone is talking and you want to display that item. Summary of the Invention

[0003] In one aspect of the invention, a method for automatically presenting memory content items is provided, the method comprising: acquiring natural language (NL) input; calculating a plurality of matching scores, wherein each matching score indicates the degree of matching between the NL input and one of the plurality of memory content items, and wherein each matching score is based on a comparison between the NL input and one of the plurality of memory content items, the comparison comprising: segmenting the NL input into a plurality of segments; generating a plurality of sub-matching scores, each sub-matching score being calculated by applying a machine learning model to a specific memory content item and one of the plurality of segments; combining the plurality of sub-matching scores into one of the plurality of matching scores; selecting one or more memory content items corresponding to one or more of a plurality of highest calculated matching scores; and providing the selected one or more memory content items.

[0004] NL input can be obtained from text dialogues or text commands entered by the user.

[0005] NL input can be obtained in response to one or more of the following triggers: recognition of a specific verbal command, pressing of a specific button, or activation of a specific user interface (UI) element.

[0006] NL input can be obtained from recordings of spoken conversations or spoken commands.

[0007] The method may also include transcribing recordings of spoken conversations or commands into text.

[0008] Segmenting NL input into multiple segments can include selecting multiple segments from the transcribed text.

[0009] The NL input may be obtained from a user, and the method may further include: obtaining multiple memory content items from one or more of the following: A) posts associated with the user's account published on a social media platform; B) a repository on the user's device; C) an account associated with the user on a cloud storage provider; or D) any combination of the above.

[0010] The method may further include: retrieving at least two memory content items from a plurality of memory content items from posts associated with an account on a social media platform; wherein A) the account may be associated with a user providing NL input, B) the account is associated with one or more other participants in the thread providing NL input, or C) the account is associated with a user providing NL input or with one or more other participants in the thread at at least a minimum specified connection level on the social media platform.

[0011] The method may further include: obtaining multiple content items by acquiring multiple content items and filtering the multiple content items into multiple memory content items by excluding one or more content items from the multiple content items, wherein the multiple content items do not have a specified association with the user providing the NL input on the social graph; the multiple content items do not have a specified geographic location; or the multiple content items are not associated with a specified social media, which shares activities from or to the user providing the NL input.

[0012] Segmenting NL input into multiple segments can be performed by recognizing sentences, noun phrases, and / or independent clauses in the NL input.

[0013] The machine learning model can be a trained neural network model that uses training terms to determine a matching score between a content item and an NL text fragment, the training terms including: A) a content item and an NL text fragment input, and B) an indicator of whether the content item and the NL text fragment match.

[0014] Combining multiple submatch scores can include either adding the multiple submatch scores together or calculating the average of the multiple submatch scores.

[0015] Providing the selected one or more memory content items may include one or more of the following: automatically adding the selected one or more content items as part of a conversation from which the NL input is obtained; or receiving authorization to display the selected one or more content items by: providing an instruction to the user providing the NL input to provide the selected one or more content items; receiving permission from the user providing the NL input; and in response to the permission, causing the selected one or more content items to be displayed to one or more users, wherein at least one of the one or more users is a user other than the user providing the NL input.

[0016] In one aspect of the invention, a computer-readable storage medium is provided storing a plurality of instructions, which, when executed by a computing system, cause the computing system to perform a process for automatically presenting memory content items. The process includes: acquiring natural language (NL) input; identifying one or more key elements of the NL input; calculating a plurality of matching scores, wherein each matching score indicates the degree of matching between the NL input and one of the plurality of memory content items, and wherein each matching score is based on a comparison between the NL input and one of the plurality of memory content items, the comparison including: acquiring one or more item tags for a specific memory content item; and calculating a matching score between the specific memory content item and the NL input by determining the degree of matching between the one or more key elements and the one or more item tags for the specific memory content item; selecting one or more memory content items corresponding to one or more of the plurality of highest calculated matching scores; and providing the selected one or more memory content items.

[0017] The NL input may be obtained from the user, and the process may further include: obtaining multiple memory content items from one or more of the following: A) posts associated with the user's account published on a social media platform; B) a repository on the user's device; C) an account associated with the user on a cloud storage provider; or D) any combination of the above.

[0018] The process may also include: retrieving at least two memory content items from a plurality of memory content items from posts associated with an account on a social media platform; wherein A) the account is associated with a user who provided NL input, B) the account is associated with one or more other participants in the thread that provided NL input, or C) the account is associated with a user who provided NL input or with one or more other participants in the thread at at least a minimum specified connection level on the social media platform.

[0019] Identifying one or more key features of an NL input may include: applying a machine learning key feature model to the NL input, wherein the machine learning key feature model is trained to identify key features using multiple training item pairs, each training item pair including: a pair of A) NL text and B) key features mapped to the NL text.

[0020] Obtaining one or more content item labels for a specific memory content item may include: applying a machine learning labeling model to the specific memory content item, wherein the machine learning labeling model is trained to determine content item labels using training items comprising multiple pairs, each pair comprising: a pair of A) content items and B) one or more labels mapped to the content item.

[0021] The degree of matching can be determined by: embedding one or more key elements into an n-dimensional space; embedding one or more content item tags into the n-dimensional space; and calculating one or more distances between the embedded one or more key elements and the embedded one or more content item tags in the n-dimensional space.

[0022] Providing the selected one or more memory content items may include: providing an indication of the selected one or more content items to a user who provides NL input; receiving permission from the user who provides NL input; and, in response to the permission, causing the selected one or more content items to be displayed to one or more users, wherein at least one of the one or more users is a user other than the user who provided NL input.

[0023] In one aspect of the invention, a computing system for automatically presenting memory content items is provided. The computing system includes: one or more processors; and one or more memories storing a plurality of instructions, which, when executed by the one or more processors, cause the computing system to perform a process including the following steps: acquiring natural language (NL) input; calculating a plurality of matching scores, wherein each matching score indicates the degree of matching between the NL input and one of the plurality of memory content items, and wherein each matching score is calculated by: applying an image / NL matching module, the image / NL matching module: acquiring one or more content item labels for a specific memory content item; and... A matching score is calculated between a specific memory content item and the NL input by determining the degree of matching between one or more key elements identified for the NL input and one or more content item labels for the specific memory content item; or a label matching module is applied that: segments the NL input into multiple segments; generates multiple sub-matching scores, each sub-matching score being calculated by applying a machine learning model to the specific memory content item and one of the multiple segments; combines the multiple sub-matching scores into one of the multiple matching scores; selects one or more memory content items corresponding to one or more of the multiple highest calculated matching scores; and provides the selected one or more memory content items.

[0024] Calculating multiple matching scores can be performed by applying the image / NL matching module.

[0025] Calculating multiple matching scores can be performed using the application tag matching module. Attached Figure Description

[0026] Figure 1 This is a block diagram illustrating an overview of a device on which some implementations can run.

[0027] Figure 2 This is a block diagram illustrating an overview of the environment in which some implementations can operate.

[0028] Figure 3 This is a block diagram showing multiple components that, in some embodiments, can be used in systems employing the disclosed technology.

[0029] Figure 4 The flowchart illustrates a process used in some implementations for automatically identifying memorized content items in response to user natural language input.

[0030] Figure 5The flowchart illustrates a process used in a first implementation to determine a matching score between a memory content item and the user's natural language input.

[0031] Figure 6 The flowchart illustrates a process used in a second implementation to determine a matching score between a memory content item and the user's natural language input.

[0032] Figure 7 This is a concept diagram illustrating an example of automatic memory of content items based on real-time dialogue.

[0033] Figure 8A and Figure 8B This is a concept diagram illustrating an example of automatic memory of content items presented based on text-based dialogue.

[0034] The technology described herein can be better understood by referring to the following specific embodiments in conjunction with the accompanying drawings, in which similar reference numerals refer to the same or similarly functional elements. Detailed Implementation

[0035] Users may find it difficult to locate specific photos, videos, or other memory items, even if they remember having them. This is particularly challenging when users want to present an item as part of an ongoing conversation but cannot locate it, or when they don't want to pause the conversation while searching for the item. For example, a user might be sitting with a friend and want to tell them about a moment from a recent trip captured in a photo, but the conversation is interrupted for five minutes while the user is browsing her photo album. Automatic memory systems can mitigate this by automatically identifying and presenting items that match the user's natural language (NL) input.

[0036] The automatic memory system can retrieve NL input from a user, such as from a portion of a thread of recorded audio from a text conversation or user speech, which can be converted into text. The automatic memory system can also retrieve one or more content items that can be matched with the NL input. In various implementations, content items can be retrieved from various sources, such as content items posted to social media by users including NL input providers, one or more other participants in the thread, and users on the social media source with a specific level of connection to the NL input provider or the one or more other participants in the thread. As a further example, these content items can be retrieved based on categories or keywords that match categories or keywords determined for the NL input. Furthermore, content items can come from user-selected or default sources, such as personal repositories (e.g., a user's local folder or album, a designated cloud storage provider or a region on that cloud storage provider, etc.) or public sources (e.g., public social media posts, news reports, cloud image or video services, streaming services, etc.).

[0037] The automatic memory system can calculate a match score between the NL input and each of the multiple content items. The automatic memory system can select one or more content items with one or more highest match scores, which can be limited by criteria such as a minimum threshold match score or the highest match score of the selected content items must be higher than the other match scores by a threshold amount. The automatic memory system can provide these content items, for example, by recommending them to the NL user, automatically displaying or playing them, inserting them into the conversation thread, etc.

[0038] Automatic memory systems can use various algorithms and / or machine learning models to calculate matching scores. In some implementations, the automatic memory system can use an image / NL matching process to obtain a first matching score. In other implementations, the automatic memory system can use a label matching process to obtain a second matching score. Any of these matching scores can be used alone, or in yet another implementation, the automatic memory system can calculate a combined matching score based on the first and second matching scores. As an example, the automatic memory system can use an image / NL matching process with a machine learning model trained to generate a first matching score when it receives both a fragment of NL and an image or other content item. In this example, the automatic memory system can also use a label matching process that determines the second matching score by comparing key elements of the NL input with labels for the content item (e.g., the object, person, event, or associated time). The automatic memory system can then combine the first and second matching scores, for example, by adding the first and second matching scores together or by calculating the average of the first and second matching scores. In some implementations, the first and second matching scores can be weighted, for example, based on a confidence value for each process, where the confidence value is determined based on historical accuracy for each process. The following is about... Figure 4 Additional details on selecting content items by calculating the matching score between the content item and the NL input are discussed.

[0039] An automatic memory system can perform the image / NL matching process as follows: First, the NL input paragraph is segmented into multiple fragments (e.g., sentences, independent clauses, etc.) using an existing Natural Language Processing (NLP) model. Then, the automatic memory system can apply a trained model (e.g., the existing ViLBERT model—see https: / / arxiv.org / abs / 1908.02265) that takes pairs of NL model fragments and content items as input and generates partial matching scores. In some implementations, in addition to using NL fragments, the model can also use image content, audio content, video content, or a combination thereof. Multiple partial matching scores corresponding to the same content item can be combined (e.g., by calculating their average) to obtain a first matching score between that content item and the NL input. The following is about... Figure 5 Additional details are discussed on how to compute the first matching score by applying the model to inputs that include NL fragments and content items.

[0040] An automatic memory system can perform the label matching process using three steps: first, identifying key elements in the NL input; second, acquiring labels for content items; and third, performing a match between the key elements and the labels of specific content items. In various cases, these processes can utilize machine learning models and / or heuristics.

[0041] In the first process, the automatic memory system can extract key elements from the NL input, such as people, places, objects, activities, times / dates, and keywords. Machine learning models can be trained to identify these key elements, for example, by applying existing part-of-speech taggers or NLP semantic models. Further input priors that can be applied to the model or alternative heuristics for identifying key elements include: obtaining relationships between the NL input provider and other entities on the social graph (e.g., who the user's friends are, what activities the user participated in, where the user has checked in, etc.), obtaining relationships between the NL input provider and other entities through geolocation matching (e.g., where the user has been), obtaining relationships between the NL input provider and other entities through shared activities (e.g., who the user has interacted with, what types of content the user is most likely to share), etc. These identifiers can be used to determine which people, places, things, dates / times, or behaviors described in the conversation are more strongly associated with the speaker's tendency to share certain content items.

[0042] In the second process, the automatic memory system can identify content item tags, for example, by applying existing object, location, or person recognition models (e.g., models trained using manually labeled items, social media tags on content items, messages or posts provided by the content item, location check-ins using photos, street mapping data, audio-to-text models, etc.). User-provided tags on content items (e.g., titles, tags when the content item was posted, comments from the content item's publisher, or comments from others on social media platforms, etc.) or metadata about the content item (e.g., collection date, size, location, etc.) can also or alternatively be used as content item tags.

[0043] In some implementations of the third process, the automatic memory system may use a model that embeds the language of key elements and the language of memory tags into a shared n-dimensional space. The automatic memory system can then, for each specific content item, determine a combination of distances (e.g., cosine distances) between the embedded key elements from the NL input and the embedded tags of that specific content item. Alternatively or additionally, the automatic memory system may determine, for each specific content item, the closest such distance, or a threshold number of such distance combinations (e.g., a combination of the three closest distances between each key element and each tag of that specific content item).

[0044] In some implementations of the third process, additionally or alternatively, the automatic memory system may use a model trained to acquire a set of key elements and a set of content item labels and generate a matching score. This model may be trained, for example, based on previous shared activity on a conversation or social media platform. For instance, the training items for the model may be pairs of the following: a portion of the text from the conversation; and shared content items near that point in the conversation (where the first and second processes described above can be used to acquire the key elements and content item labels of the NL input). The key elements from that portion of the conversation can be matched with the labels of the shared content items to obtain training data items.

[0045] In some implementations, either of the two versions of the third process described above can be used, and the output of this process can be a second matching score. In other implementations, both versions of the third process described above can be used, and a combination of their outputs (e.g., sum, average, weighted average based on historical accuracy) can be used to calculate the second matching score. The following is about... Figure 6 Additional details are discussed regarding the calculation of the second matching score by applying a label matching process.

[0046] Existing content item storage and organization systems allow users to search and filter content items, for example, by name, length, or even by category or other tags. Alternatively, existing systems allow users to manually browse their content items. However, these systems require users to have a starting point and sufficient information within searchable categories to identify the content item they are looking for. Furthermore, these systems are a hindrance when a conversation requires content items, as loading procedures and performing searches consume significant user time. The automatic memory system and process described herein overcomes these problems associated with traditional content item selection systems and is expected to provide users with a greater ability to communicate more effectively by automatically selecting content items that match the natural language provided by the user. Automatic memory systems can automatically select content items that match the natural language provided by the user, for example, from auditory or written conversations, or from voice or text commands, in a way that significantly improves usability, where selection is faster, does not require consulting multiple repositories, and communication is more efficient. By using sophisticated selection models, content item constraints, and contextual signals to automatically select content items that match natural language expressions, conversations (or other content item selection processes) employing automatic memory systems can be more informative, efficient, faster, and more engaging. The automatic memory system and process described in this paper are rooted in computerized machine learning systems, rather than simulating human activity or other selection techniques for browsing content items. For example, existing content selection systems require a great deal of knowledge to search effectively: where to look and what content the user is looking for. In contrast, this automatic memory system allows the user to simply speak or type and provides relevant content items.

[0047] Several implementation methods are described in more detail below with reference to the accompanying drawings. Figure 1 A block diagram illustrating an overview of devices on which some embodiments of the disclosed technology may operate is provided. These devices may include hardware components of device 100, which calculates matching scores between each content item and an NL input and provides the content item with the highest score. Device 100 may include one or more input devices 120 that provide input to one or more processors 110 (one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more holographic processing units (HPUs), etc.) to notify the processors of multiple actions. These actions may be transmitted by a hardware controller that interprets signals received from the input devices and transmits information to processor 110 using a communication protocol. Input devices 120 may include, for example, a mouse, keyboard, touchscreen, infrared sensor, touchpad, wearable input device, camera-based or image-based input device, microphone, or other user input device.

[0048] Processor 110 may be a single processing unit or multiple processing units located in one device or distributed across multiple devices. Processor 110 may be coupled to other hardware devices, for example, via a bus such as a Peripheral Component Interconnect Standard (PCI) bus or a Small Computer System Interface (SCSI) bus. Processor 110 may communicate with hardware controllers for various devices (e.g., for display 130). Display 130 may be used to display text and graphics. In some embodiments, display 130 provides visual feedback to a user, including graphics and text. In some embodiments, display 130 includes an input device, which, when the input device is a touchscreen or equipped with an eye-direction monitoring system, is part of the display. In some embodiments, the display is separate from the input device. Examples of display devices include liquid crystal display (LCD) screens, light-emitting diode (LED) screens, projection displays, holographic displays, or augmented reality displays (e.g., heads-up displays or head-mounted displays). Other input / output (I / O) devices 140 may also be coupled to the processor, such as network cards, video cards, sound cards, Universal Serial Bus (USB), FireWire or other external devices, cameras, printers, speakers, compact optical disc read-only memory (CD-ROM) drives, digital video disc (DVD) drives, disk drives, or Blu-ray devices.

[0049] In some embodiments, device 100 also includes a communication device capable of wireless or wired communication with network nodes. This communication device may, for example, use Transmission Control Protocol / Internet Protocol (TCP / IP) to communicate with another device or server over a network. Device 100 can utilize this communication device to distribute operations across multiple network devices.

[0050] Processor 110 can access memory 150, which may be located in a single device or distributed across multiple devices. This memory includes one or more hardware devices of various types for volatile or non-volatile storage, and may include both read-only memory (ROM) and writable memory. For example, memory may include random access memory (RAM), various cache memories, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard disk drives, floppy disks, optical discs (CDs), DVDs, magnetic storage devices, and magnetic tape drives. Memory is not a propagating signal detached from the underlying hardware; therefore, memory is non-transitory. Memory 150 may include program memory 160, which stores programs and software, such as operating system 162, automatic memory system 164, and other application programs 166. The memory 150 may also include a data memory 170, such as various machine learning models (e.g., these machine learning models match NL fragments with content items, generate labels for content items, identify key features from NL inputs, and determine match scores between key features and content item labels), training data for these models, NL inputs, content items, identified key features, content item labels, configuration data, settings, user options or preferences, etc., all of which may be provided to the program memory 160 or any element of the device 100.

[0051] Some implementations can operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations that may be used with this technology include, but are not limited to, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments that include any of the above systems or devices.

[0052] Figure 2 This is a block diagram illustrating an overview of an environment 200 in which some embodiments of the disclosed technology may operate. Environment 200 may include one or more client computing devices 205A to 205D, examples of which may include device 100. Client computing devices 205 may operate in a network environment using a logical connection via network 230 to one or more remote computers (e.g., server computing devices).

[0053] In some implementations, server 210 (also referred to as server computing device) 210 may be an edge server that receives multiple client requests and coordinates the fulfillment of these requests through other servers (e.g., servers (also referred to as server computing devices) 220A to 220C). Server computing devices 210 and 220 may include computing systems, such as device 100. Although each server computing device 210 and 220 is logically presented as a single server, multiple server computing devices can be a distributed computing environment comprising multiple computing devices located in the same physical location or in geographically different physical locations. In some implementations, each server 220 corresponds to a set of servers.

[0054] Client computing device 205 and server computing devices 210 and 220 can each act as a server or client for other server / client devices. Server 210 can connect to database 215. Servers 220A to 220C can each connect to their respective databases 225A to 225C. As discussed above, each server 220 can correspond to a group of servers, and the individual servers among these servers can share a database or have their own database. Databases 215 and 225 can store (e.g., assemble) information. Although databases 215 and 225 are logically presented as a single unit, both databases 215 and 225 can be distributed computing environments containing multiple computing devices, which can be located within their respective servers, or in the same physical location or in geographically different physical locations.

[0055] Network 230 can be a local area network (LAN) or a wide area network (WAN), but it can also be other wired or wireless networks. Network 230 can be the Internet or some other public or private network. Client computing devices 205 can connect to network 230 via a network interface, for example, via wired or wireless communication. Although the connection between server 210 and the multiple servers 220 is shown as a separate connection, these connections can be any type of LAN, WAN, wired network, or wireless network that includes network 230 or a separate public or private network.

[0056] In some implementations, servers 210 and 220 can function as part of a social network. The social network can maintain a social graph and perform various actions based on it. The social graph can include a set of nodes (representing social network system objects, also known as social objects) interconnected by multiple edges (representing interactions, activities, or relevances). Social network system objects can be social network system users, non-human entities, content items, groups, social network system pages, locations, applications, topics, concept representations, or other social network system objects (e.g., movies, bands, books, etc.). Content items can be any digital data, such as text, images, audio, video, links, web pages, details (e.g., markers provided by client devices such as mood indicators, status text snippets, location indicators, etc.), or other multimedia. In various implementations, content items can be social network items or parts of social network items, such as posts, hobbies, mentions, news items, events, shares, comments, messages, other notifications, etc. In the context of the social graph, topics and concepts include nodes representing any person, place, thing, or idea.

[0057] Social networking systems allow users to input and display information related to their interests, age / date of birth, location (e.g., longitude / latitude, country, region, city, etc.), education information, life stage, marital status, name, commonly used device model, languages ​​familiar to the user, occupation, contact information, or other demographic or biometric information from the user profile. In various implementations, any such information can be represented by nodes in the social graph or edges between nodes. Social networking systems allow users to upload or create images, videos, documents, songs, or other content items, and allow users to create and schedule events. In various implementations, content items can be represented by nodes in the social graph or edges between nodes.

[0058] Social networking systems enable users to upload or create content items, interact with content items or other users, express interests or opinions, or perform other actions. Social networking systems can provide various means of interacting with non-user objects within the system. In various implementations, actions can be represented by nodes or edges between nodes in the social graph. For example, a user can form or join a group, or become a fan of a page or entity within the social networking system. Furthermore, users can create, download, view, upload, link to, tag, edit, or play social networking system objects. Users can interact with social networking system objects outside the context of the social networking system. For example, an article on a news website might have a clickable "like" button. In each of these instances, the interaction between the user and the object can be represented by an edge in the social graph connecting the user's node to the object's node. As another example, a user can use location detection (e.g., a GPS receiver on a mobile device) to "check in" at a specific location, and in the social graph, an edge can connect the user's node to the node at that location.

[0059] Social networking systems can provide users with various communication channels. For example, they can enable users to send emails, instant messages (IM), or text / SMS messages to one or more other users; they can allow users to post messages to their own wall or profile, or to another user's wall or profile; they can enable users to post messages to groups or fan pages; and they can allow users to comment on images, wall posts, or other content items created or uploaded by themselves or other users. In some embodiments, users can post a status message to their profile indicating a current event, mental state, thought, feeling, action, or any other expression relevant to the current time. Social networking systems enable multiple users to communicate not only within the social networking system but also outside of it. For example, a first user can send a message to a second user within the social networking system, send an email to a second user through the social networking system, send an email originating from the social networking system to a second user outside the social networking system, send an instant message to a second user within the social networking system, or send an instant message originating from the social networking system to a second user outside the social networking system. In addition, the first user can comment on the second user's profile page, or on objects associated with the second user (e.g., content items uploaded by the second user).

[0060] Social networking systems enable users to associate themselves with other users within the system and establish connections with them. When two users (e.g., social graph nodes) explicitly establish a social connection within a social networking system, they become "friends" (or "connections") within the context of the system. For example, a friend request accepted by "Jane Smith" from "John Doe" to "Jane Smith" is a social connection. This social connection can be an edge in the social graph. Becoming friends, or being a friend on the social graph, within a certain threshold number of edges, allows users access to more information about each other than the information available to multiple unrelated users. For example, becoming friends allows users to view another user's profile, their friends, or their photos. Similarly, becoming friends within a social networking system allows users more opportunities to communicate with another user, such as via email (both inside and outside the social networking system), instant messaging, text messaging, telephone, or any other communication interface. Becoming friends allows users to view, comment on, download, endorse, or otherwise interact with content uploaded by another user. The establishment of connections, access to user information, communication, and interaction within the context of a social network system can be represented by edges between nodes representing users of two social network systems.

[0061] In addition to explicitly establishing connections within a social network system, multiple users with shared characteristics can be considered connected (e.g., soft connections or implicit connections) for the purpose of determining social context (used when identifying communication topics). In some embodiments, multiple users belonging to a public network are considered connected. For example, multiple users attending the same school, working for the same company, or belonging to the same social network group can be considered connected. In some embodiments, multiple users with shared biometrics are considered connected. For example, a user's birth or residence geographic region, age, gender, and marital status can be used to determine whether multiple users are connected. In some embodiments, multiple users with shared interests are considered connected. For example, a user's movie preferences, music preferences, political views, religious views, or any other interests can be used to determine whether multiple users are connected. In some embodiments, multiple users who have performed the same action within the social network system are considered connected. For example, multiple users who endorse or recommend the same object, comment on the same content item, or respond to the same event (RSVP) can be considered connected. Social networking systems can utilize social graphs to identify multiple users connected to or similar to a specific user, in order to determine or assess the social context among these users. This social context and shared attributes can then be leveraged to facilitate the predictive selection of content items for caching on caching devices (associated with specific social network accounts) by content distribution and caching systems.

[0062] Figure 3This is a block diagram illustrating multiple components 300 that, in some embodiments, can be used in systems employing the disclosed technology. These multiple components 300 include hardware 302, general-purpose software 320, and multiple dedicated components 340. As discussed above, systems implementing the disclosed technology can use various hardware, including multiple processing units 304 (e.g., multiple CPUs, multiple GPUs, multiple APUs, etc.), working memory 306, storage memory 308 (local memory or as an interface to remote storage, e.g., database 215 or 225), and I / O devices 310. In various embodiments, storage memory 308 can be one or more of the following: a local device, an interface to a remote storage device, or a combination thereof. For example, storage memory 308 can be a collection of one or more hard disk drives (e.g., a redundant array of independent disks, RAID) accessible via a system bus, or it can be a cloud storage provider or other network storage (e.g., network accessible storage (NAS) device, such as database 215 or storage provided via another server 220) accessible via one or more communication networks. Multiple components 300 may be implemented in a client computing device (e.g., client computing device 205) or on a server computing device (e.g., server computing device 210 or 220).

[0063] General-purpose software 320 may include various applications, including operating system 322, multiple native programs 324, and basic input / output system (BIOS) 326. Specialized components 340 may be sub-components of general-purpose software 320 (e.g., native programs 324). Each specialized component 340 may include a content item retrieval and filtering module 344, an NL interface 346, a matching score module 348, a tag matching module 350, an image / NL matching module 352, and components (e.g., multiple interfaces 342) that can be used to provide a user interface (UI), transmit data, and control the specialized components. In some embodiments, the multiple components 300 may reside in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more of the specialized components 340. Although specialized components 340 are depicted as separate components, these specialized components may have logical functions or other non-physically distinct functions, and / or may be sub-modules or code blocks of one or more applications.

[0064] The content item retrieval and filtering module 344 can select one or more potential memory content items for the NL input. The content item retrieval and filtering module 344 can select potential memory content items from multiple sources, such as social media providers, local content item repositories, cloud storage providers, public sources of content items, etc. In some embodiments, the content item retrieval and filtering module 344 pre-filters the retrieved content items, for example, by selecting content items that have a specific relationship with the provider of the NL input (e.g., determined through social graph connections, geolocation connections, shared activities, etc.). The following is about... Figure 4 Box 404 describes the selection of potential content items in more detail.

[0065] The NL interface 346 can acquire natural language (NL) input. In various implementations, the NL interface 346 can achieve this by recording audio input via a microphone, or receiving text input via a keyboard or network connection. In some implementations, the NL interface 346 can convert audio NL input into text. The following is about... Figure 4 Box 402 describes the acquisition of NL input in more detail.

[0066] The matching score module 348 can generate a matching score between the NL input obtained by the NL interface 346 and the content items selected by the content item retrieval and filtering module 344. In various embodiments, the matching score module 348 can generate a matching score using the output from the tag matching module 350, the output from the image / NL matching module 352, or a combination of both (e.g., by calculating their average).

[0067] The tag matching module 350 can identify key elements from the NL input of the NL interface 346, such as people, objects, locations, dates or times, activities, etc., specified in the NL input. The following is about... Figure 6 Box 602 provides additional details for obtaining key elements of the NL input. The tag matching module 350 can also identify one or more tags for the content items selected by the content item retrieval and filtering module 344. The following is about... Figure 6 Box 606 provides additional details on retrieving content item tags. Finally, the tag matching module 350 can generate a matching score by comparing key elements with the tags of each content item. The following is about... Figure 6 Box 608 provides additional details on how to calculate a match score using content item tags and key elements.

[0068] The image / NL matching module 352 can segment the NL input from the NL interface 346 into multiple text fragments. The image / NL matching module 352 can then apply a model (which takes text fragments and content items as input) to each possible combination of: A) content items (from the content item retrieval and filtering module 344), and B) one text fragment from the multiple text fragments. Although this model is referred to herein as "image / NL" matching, in various implementations, in addition to using NL fragments, the model can also use image content, audio content, video content, or a combination thereof. The model can generate sub-matching scores, and the image / NL matching module 352 can combine (e.g., calculate their average) the sub-matching scores corresponding to each content item to obtain a matching score for that content item. The following is about... Figure 5 Additional details are provided on how to calculate the matching score by providing the model with content items and text fragments.

[0069] Those skilled in the art will understand that the above can be modified in various ways. Figures 1 to 4 The components shown are those shown in the flowcharts discussed below. For example, the order of the logic can be rearranged, multiple sub-steps can be executed in parallel, the logic shown can be omitted, and other logic can be included. In some embodiments, one or more of the components described above can perform one or more of the processes described below.

[0070] Figure 4 The diagram illustrates a flowchart of a process 400 used in several implementations for automatically recognizing memorized content items in response to user natural language input. In some implementations, process 400 may be executed automatically when a user provides NL input, for example, via instant messaging, video chat, email, text, or other digital dialogue systems. In some implementations, process 400 may be executed in response to a user command, such as when the user activates a control for a search item, speaks a search command expression, or selects a specific segment of the NL input. In some cases, process 400 may be executed on the user's local device (e.g., via a telephone or automated assistant device). In some cases, process 400 may receive audio or text of a conversation, or a command to search for a content item; such conversational NL input may originate from: face-to-face conversations, or conversations using local devices (e.g., instant messaging (IM), text messaging conversations, or video chat). In some implementations, process 400 may be executed by a server-side system facilitating the digital conversation, or by the server-side system after receiving such conversation or command data from a local system.

[0071] At box 402, process 400 may acquire user natural language (NL) input for selecting a memory content item. In some implementations, user-provided NL input may be acquired in response to a trigger, such as a user making a verbal command to input NL, a user pressing a button on the device, or a user activating a UI element. In various implementations, process 400 may acquire NL input from, for example, recordings of verbal conversations or commands, text conversations (e.g., emails, text messages, IMs, etc.), text commands (e.g., entering text in a search bar), or other natural language sources. In some cases where the NL input is in audio form, existing transcription processes can be used to convert the audio to text. In various implementations of these cases, process 400 may use only text, or it may use audio, or it may tag the text based on audio, using tone / mood indicators determined for different parts of the text.

[0072] At box 404, process 400 may calculate a match score between the NL input and each of one or more potential memory content items (i.e., content items that will be checked by process 400 to match the NL input from box 402). These potential memory content items can be retrieved from various sources, such as content items posted to social media by one or more of the following: A) the NL input provider, B) one or more other participants in the thread, or C) a user on the social media source with a specific level of connection to the NL input provider or the one or more other participants in the thread. As a further example, these content items can be retrieved based on one or more predefined categories or keywords for the content items, which match one or more categories or keywords determined for the NL input. Furthermore, content items can come from user-selected or default sources, such as personal repositories (e.g., a user's local folder or album, a specified cloud storage provider or a region on that cloud storage provider, etc.) or public sources (e.g., public social media posts, news reports, cloud image or video services, streaming services, etc.). In some implementations, process 400 may receive descriptive items of the content items (e.g., tags or other metadata determined for the content items) instead of retrieving the content items themselves. In some cases, process 400 may receive versions of the content items suitable for machine learning input, such as histograms.

[0073] For each content item, process 400 can calculate a matching score, which indicates how well the content item matches the expected NL input. Process 400 can use various algorithms and / or machine learning models to calculate the individual matching scores. In some implementations, process 400 can use an image / NL matching process to calculate the individual matching scores. This image / NL matching process may include: segmenting the NL input into multiple segments; and for each content item, for each text selection: applying a model trained to acquire NL text and content items and generate scores about their degree of matching. The image / NL matching process can then combine the segment matching scores corresponding to each content item. The following is about... Figure 5 Additional details are described regarding the use of an image / NL matching process to calculate matching scores. In some implementations, process 400 may use a label matching process to calculate individual matching scores. The label matching process may include: identifying key elements of the NL input (e.g., object, person, action, date, etc.); obtaining a label for each potential memory content item; and matching the labels with the key elements to obtain a matching score. The following is about... Figure 6 Additional details are described regarding the use of a label matching process to calculate the matching score.

[0074] While process 400 can use either the image / NL matching process or the tag matching process individually to calculate a matching score for a specific content item, in another embodiment, for each content item, process 400 can calculate a first matching score based on the image / NL matching process and a second matching score based on the tag matching process, and combine the first and second matching scores to form a final matching score for the content item. For a given image content item, process 400 can use the image / NL matching process by applying a machine learning model trained to generate multiple sub-scores of the first matching score when it receives the content item and individual segments from one or more NL segments. Process 400 can combine these sub-scores to form a first matching score for the content item. Process 400 can also use a tag matching process that determines a second matching score by comparing key features of the NL input with the tags of the content item. Process 400 can then combine the first and second matching scores, for example, by adding them together or calculating their average. In some implementations, the first and second matching scores can be weighted, for example, based on confidence values ​​for each process, determined based on historical accuracy for that process. For instance, whether a user selected each content item recommended by process 400 to share in a conversation can be identified as positive (selection occurred) or negative (no selection occurred) and can be used as further training data for various machine learning models, and / or compared to which matching process recommended the content item the user selected to calculate the accuracy score for each model. Accuracy scores can also be calculated based on alternative scores, such as, social media "likes" counts for the selected content item, manual user feedback on the accuracy of the recommended content item, or when the user chose to share a content item that was not among the recommended content items.

[0075] At box 406, process 400 can select one or more content items that meet the criteria and have the highest matching score. In some implementations, all analyzed content items are compliant with the criteria. In other implementations, a content item must have a matching score higher than a threshold (e.g., the threshold is in the range of 0 to 1, 0.5, 0.7, or 0.85) to be compliant with the criteria. In some cases, selected content items must have a matching score that is one threshold amount higher than the matching score of unselected content items to be compliant with the criteria. In some implementations, process 400 can select the maximum number of content items with the highest scores, such as the first, third, fifth, or ten highest-scoring content items.

[0076] At box 408, process 400 may provide one or more content items selected at box 406. In some embodiments, providing the selected content item may include automatically adding the selected content item as part of a dialog (from which the NL input is obtained) (e.g., as a new item in the dialog thread or at a specified content item location). In other embodiments, the selected content item may be automatically output, for example, by playing the selected video or audio file, or displaying the selected image. In yet another embodiment, the selected content item may be provided to the user who entered the NL input, allowing the user to make further selections or grant permissions for the selected content item before one or more content items are added to the dialog or otherwise displayed to other users.

[0077] As discussed above, these further choices made by the user can be used to identify additional training data that is used to update the training of the machine learning model. For example, if a user selects one of the recommended content items, that content item can be used as a further positive training term (pairs of the following: the selected content item and the NL input), while a user's selection of an alternative content item (which is not part of one or more of the recommended content items) can create one or both of positive training terms (pairs of the following: the selected content item and the NL input) and negative training terms (pairs of the following: one or more unselected content items and the NL input).

[0078] Figure 5 The diagram illustrates a process 500 used in a first embodiment to determine a matching score between a memory item and the user's natural language input. In some embodiments, process 500 may be executed as a subprocess of process 400, for example, starting from block 404.

[0079] At box 502, process 500 can segment the received NL input (e.g., from box 402) into multiple segments. In various implementations, process 500 can segment the NL input into multiple segments such as sentences, noun phrases, or independent clauses. Process 500 can achieve this using existing NL processing models (e.g., part-of-speech taggers and NL semantic models).

[0080] At box 504, process 500 can begin a loop between boxes 504 and 512, in which the operation is repeatedly performed on one selected text segment from the multiple text segments obtained by segmentation at box 502, thereby iterating through the entire set of text segments. At box 506, process 500 can begin a loop between boxes 506 and 510, in which the operation is repeatedly performed on the selected content item, thereby iterating through the entire set of content items (e.g., potential memory content items from box 404). Thus, these two loops operate on each selected text segment and each selected potential memory content item.

[0081] At box 508, process 500 can apply a model trained to determine a matching score between content items and NL text fragments. An example of such a model, as discussed below, could be a version of a neural network trained on pairs of: A) the content item and the NL text fragment input, and B) an indicator of whether they match. An example of such a model is the ViLBERT model (described in “ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks” by Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee, available at https: / / arxiv.org / abs / 1908.02265, which is incorporated herein by reference), but other models may also be used. In various implementations, in addition to using NL fragments, the model can also use image content, audio content, video content, or a combination thereof. Therefore, for the selected text fragment and the selected content item, box 508 can apply the model to generate a matching score.

[0082] At box 510, process 500 can select the next potential memory item from the set and return to box 506 to continue the inner loop. Once the inner loop between boxes 506 and 510 has operated on each of the multiple content items in the set, process 500 can proceed to box 512. At box 512, process 500 can select the next text fragment from the fragment set determined at box 502 and return to box 504 to continue the outer loop. Once the outer loop between boxes 504 and 512 has operated on each of the multiple text fragments in the set, process 500 can proceed to box 514.

[0083] At box 514, process 500 may return one or more highest matching scores. In some implementations, process 500 may return all matching scores, while in other implementations, it may only return the highest matching score of a threshold number. In some implementations, for a specific content item, the matching scores generated for each text fragment may be combined into a matching score between that content item and the entire NL input before being returned. For example, the average of these scores may be calculated.

[0084] Figure 6 The flowchart illustrates a process 600 used in a second embodiment to determine a matching score between a memory content item and the user's natural language input. In some embodiments, process 600 may be executed as a subprocess of process 400, for example, starting from block 404.

[0085] At box 602, process 600 can identify key elements of the NL input (e.g., the NL input acquired at box 402). Key elements can be portions of the NL input that are identified in relation to a specific category (e.g., person, place, object, event, or time / date). Process 600 can apply machine learning systems to identify these key elements, such as by applying existing part-of-speech taggers, natural language semantic models, etc. Additional or alternative heuristics can be applied to identify which words or expressions in the NL input are key expressions. For example, words or expressions can be advanced in a machine learning model to be more likely to be selected as key expressions when they correspond to other user data. For example, expressions in the NL input can be advanced when they correspond to: A) people, places, activities, etc., that the user is linked to on the social graph (e.g., who the user's friends are, what activities the user has participated in, where the user has checked in, etc.); B) people, places, activities, etc., that the user is linked to through identified geographic location similarities (e.g., locations the user is known to have visited); or C) people, places, activities, etc., that the user is linked to through shared activities (e.g., who the user has interacted with, what types of content the user is most likely to share), etc. This allows process 600 to identify people, places, things, or behaviors described in the NL input that have a special association with the source user as key elements or priors for machine learning models.

[0086] At box 604, process 600 may begin a loop between boxes 604 and 610, in which the operation is repeated on the selected content item each time, thereby iterating through the entire set of content items (e.g., potential memory content items from box 404). Thus, the loop operates on each selected potential memory content item.

[0087] At box 606, process 600 can obtain the tags of the currently selected content item from box 604. In some implementations, process 600 may use a machine learning system to identify tags for the content item, such as objects, people, places, actions, audio (which can be converted to text), dates / times, etc., described in the content item. For example, the following may be used to train the machine learning model: manually tagged content items, social media tags on content items, messages or posts provided in conjunction with content items, etc. Some of these tags may be determined using existing models (e.g., existing object, place, transcription, or person recognition models). Some tags may be based on the audio or video of the content item, such as by recognizing and tagging certain sounds (e.g., laughter, rain, waves, barking, etc.) or converting spoken language into text. Tags provided by the user on the content item (e.g., title, tags when the content item was published, comments from the content item publisher, or comments from others on the social media platform, etc.) or metadata about the content item (e.g., collection date, size, location, etc.) may also be used, or alternatively, as content item tags. In some implementations, these tags for content items can be predetermined using the machine learning model described above, user-provided tags, etc., and process 600 can retrieve these tags (in addition to or in place of content items).

[0088] At box 608, process 600 can determine a matching score between the key feature identified at box 602 and the label acquired at box 606. In some implementations, process 600 can achieve this using embedding matching, where process 600 samples a model trained to map key features and labels to the same n-dimensional semantic space. Process 600 can then compute a matching score based on the distance (e.g., cosine distance) between one or more key feature embedding points and multiple label embedding points. For example, process 600 can compute the average of the distances between the nearest or farthest embedding points (including only one or all) of a threshold number of key features and multiple labels. As another example, process 600 can take the distance between the center of multiple key feature embedding points and the center of multiple label embedding points.

[0089] In some implementations, additionally or alternatively, process 600 can determine a match score between key elements and tags by applying a model to key elements and tags that is trained to acquire a set of key elements and a set of content item tags and generate a match score. This model can be trained, for example, based on previous sharing activity on a conversation or social media platform. For example, the input to the positive training term could be a pair of A) key elements identified for a portion of the text in the conversation and B) tags for content items shared near that point in the conversation.

[0090] At box 610, process 600 can select the next potential memory item from the set and return to box 604 to continue the loop. Once the loop between boxes 604 and 610 has operated on each of the multiple items in the set, process 600 can proceed to box 612.

[0091] At box 612, procedure 600 may return one or more of the highest matching scores among a plurality of highest matching scores. In some implementations, procedure 600 may return all matching scores, while in other implementations, it may return only the highest matching score of a threshold number.

[0092] Figure 7 This is a conceptual diagram illustrating example 700 of an automatic memory content item presentation based on real-time dialogue. Example 700 includes two participants 702 and 704 engaging in an auditory dialogue. Device 708 is executing process 400, and the device has been configured to: receive NL input via a microphone, convert the NL input to text, and look up matching content items stored on the device, or matching content items posted to social media by one of multiple users. In example 700, participant 702 says the expression, “I had a great time at Niagara Falls with Uncle Jon last year.” Device 708 has taken this expression as NL input and has identified a key element (shown as 710; note that in some implementations, the key element is not shown). Device 708 calculates a first set of matching scores by applying a model to match the key element 710 with tags: tags identified for content items stored on device 708, and tags for content items posted to social media sites by participant 702. Device 708 has also segmented the NL input 706 into multiple fragments, compared these fragments with the same content item using the ViLBERT model, and generated a second set of matching scores for the content item. Finally, device 708 has combined the matching scores from the various sets corresponding to the same content item to obtain the final matching score for that content item. The resulting best match is image 712, depicting Niagara Falls, which is tagged with participant 702 and a user named Jon who is identified as participant 702's uncle in the social graph, and the image is tagged with a date taken 11 months ago. This content item is automatically displayed on device 708 for conversation participants 702 and 704 to view.

[0093] Figure 8A and Figure 8BThese are conceptual diagrams illustrating examples 800 and 850 of automatic memory content item presentation based on text dialogue. Example 800 includes device 808, which is executing an application that is having a text dialogue with device 858 of example 850. Device 808 is displaying a text input area 802, thread 804, and content item display area 806. In example 800, an intermediate device for the dialogue has matched the NL input “This morning my dog ​​went hiking with me” to a content item, which is provided by the intermediate device for display in area 806. The user is currently also entering another NL input, “Until we find a dense forest”. When this other NL input is sent, the intermediate device performs process 400 to select another content item, and areas 806 on device 808 and area 856 on device 858 are updated to display the selected content item with the best matching score for “Until we find a dense forest”.

[0094] As used herein, a "model" refers to a construct trained using training data to predict or provide probabilities for new data items, regardless of whether those new data items are included in the training data. For example, training data for supervised learning may include items with various parameters and a specified classification. New data items may have parameters that the model can use to assign a classification to them. As another example, a model may be a probability distribution derived from the analysis of training data, such as the probability of an n-gram appearing in a given language based on the analysis of a large corpus of that language. Examples of models include neural networks, support vector machines, decision trees, Parzen windows, Bayesian methods, clustering, reinforcement learning, probability distributions, decision trees, and decision tree forests. Models can be configured for various scenarios, data types, source and output formats.

[0095] In some implementations, the model described above may be a neural network with multiple input nodes that receive representations of, for example, content items, labels, key features, natural language, etc. These input nodes may correspond to functions that receive inputs and produce results. These results may be provided to one or more intermediate-level nodes, each producing further results based on a combination of results from lower-level nodes. Weighting factors may be applied to the output of each node before passing the results to the next layer. In the final layer (“output layer”), one or more nodes may generate values ​​that classify the inputs, for example, by providing matching scores between multiple inputs or providing new labels or key feature identifiers. In some implementations, neural networks referred to as deep neural networks may have multiple layers of intermediate nodes with different configurations; neural networks referred to as deep neural networks may be a combination of models that receive different parts of the input and / or models that receive inputs from other parts of the deep neural network, or neural networks referred to as deep neural networks may be convolutional (partially using the output from previous iterations to which the model is applied as further input to generate results for the current input, and / or operating on different parts of the model based on different parts of the input or at different resolutions).

[0096] Machine learning models can be trained using supervised learning, where training data includes pairs of sample inputs and desired outputs. For example, a representation of a content item and an NL input can be provided to the model in pairs, specifying whether the content item matches the NL input. The output from the model can be compared to the desired output of the training terms, and based on this comparison, the model can be modified, for example, by changing the weights between nodes in the neural network or the parameters of the function used at each node in the neural network (e.g., applying a loss function). After each of several training terms has been applied in this way and the model modified, it can be trained to evaluate new content item / NL input pairs to determine if they match. A similar process can be used, for example, to train additional models using: pairs of content items and labels, used to train the model to select labels for content items; pairs of key features and NL inputs, used to train the model to select key features for the NL input; or pairs of key feature / label inputs, and whether the key feature and label inputs match, used to train the model to calculate a match score between the key features of the NL input and the label of the content item.

[0097] Several embodiments of the disclosed technology have been described above with reference to the accompanying drawings. A computing device on which the described technology can be implemented may include one or more central processing units, memory, input devices (e.g., keyboards and pointing devices), output devices (e.g., display devices), storage devices (e.g., disk drives), and network devices (e.g., network interfaces). Memory and storage devices are computer-readable storage media that can store instructions for implementing at least a portion of the described technology. Furthermore, data structures and message structures may be stored or transmitted via data transmission media, such as signals on a communication link. Various communication links may be used, such as the Internet, local area networks, wide area networks, or point-to-point dial-up connections. Therefore, computer-readable media may include computer-readable storage media (e.g., "non-transitory" media) and computer-readable transmission media.

[0098] In this specification, references to "multiple embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that a particular feature, structure, or characteristic described with respect to that embodiment is included in at least one embodiment of this disclosure. The appearance of these expressions in different places in the specification does not necessarily refer to the same embodiment, nor is it necessarily a separate or alternative embodiment that is mutually exclusive with other embodiments. Furthermore, various features that may be presented by some embodiments but not by other embodiments are described. Similarly, various requirements that may be requirements of some embodiments but not other embodiments are described.

[0099] As used herein, above the threshold means that the value of a comparison item is higher than any other specified value, that the comparison item is among a specified number of items with a maximum value, or that the comparison item has a value within a specified highest percentage value. As used herein, below the threshold means that the value of a comparison item is lower than any other specified value, that the comparison item is among a specified number of items with a minimum value, or that the comparison item has a value within a specified lowest percentage value. As used herein, within the threshold means that the value of a comparison item is between two other specified values, that the comparison item is among a specified number of items in the middle, or that the comparison item has a value within a specified percentage range in the middle. When not otherwise defined, relative terms (e.g., high or unimportant) can be understood as assigning a value and determining how that value is compared to an established threshold. For example, the expression "select fast connections" can be understood as meaning: select connections with assigned values ​​corresponding to their connection speeds above the threshold.

[0100] As used herein, the word “or” refers to any possible permutation of a set of items. For example, expressing “A, B, or C” means at least one of the following: A, B, C, or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B, and C; or multiples of any items, such as A and A; B, B, and C; A, A, B, C, and C, etc.

[0101] Although the subject matter has been described in language specifically used for structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of these embodiments and implementations. The specific features and actions described above are disclosed as exemplary forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited beyond the scope of the appended claims.

[0102] All patents, patent applications, and other references mentioned above are incorporated herein by reference. Modifications may be made to various aspects to provide alternative implementations using the systems, functions, and concepts of the aforementioned references. In the event of any conflict between statements or subject matter in the documents incorporated by reference and statements or subject matter of this application, this application shall prevail.

Claims

1. A method for automatically presenting memorized content items, the method comprising: Obtain natural language (NL) input; Calculate multiple matching scores, where each matching score indicates the degree of match between the NL input and one of the multiple memory content items, and where each matching score is based on a comparison between the NL input and one of the multiple memory content items, the comparison including: The NL input is divided into multiple segments; Multiple sub-match scores are generated, each sub-match score being calculated by applying a machine learning model to a specific memory content item and one of the multiple segments; and The plurality of sub-matching scores are combined into one matching score from the plurality of matching scores; Select one or more memory content items corresponding to one or more of the highest computed matching scores from a plurality of highest computed matching scores; and Provide the selected one or more memory content items.

2. The method according to claim 1, wherein, The NL input is obtained from a text dialogue or text command entered by the user; or The NL input is acquired in response to one or more of the following triggers: recognition of a specific verbal command, pressing of a specific button, or activation of a specific user interface (UI) element. The NL input is obtained from a recording of a spoken conversation or spoken command; The method further includes: transcribing the recording of the spoken dialogue or spoken command into text; and The process of dividing the NL input into multiple segments includes selecting multiple segments from the transcribed text.

3. The method according to claim 1, further comprising: The multiple memory content items are obtained by acquiring multiple content items and filtering them into multiple memory content items by excluding one or more of the following content items: The one or more content items do not have a specified association with the user who provided the NL input on the social graph; One or more of the content items do not have a specified geographic location; or The one or more content items are not associated with a designated social media platform that shares activities from or to users who provided the NL input.

4. The method according to claim 1, wherein, The segmentation of the NL input into multiple segments is performed by identifying sentences, noun phrases, and / or independent clauses in the NL input.

5. The method according to claim 1, wherein, The machine learning model is a trained neural network model that uses training terms to determine a matching score between a content item and an NL text fragment. The training terms include the following pairs: A) the input content item and the NL text fragment, and B) an indicator of whether the content item and the NL text fragment match.

6. The method according to claim 1, wherein, Combining the multiple sub-match scores includes: adding the multiple sub-match scores together, or calculating the average of the multiple sub-match scores.

7. The method according to claim 1, wherein, The provision of the selected one or more memory content items includes one or more of the following: Automatically add the selected one or more content items as part of the conversation, where the NL input is obtained from the conversation; or Authorization to display the selected one or more content items can be received in the following ways: Provide the user who provided the NL input with an indication of the selected one or more content items; Receive permission from the user who provided the NL input; as well as In response to the permission, the selected one or more content items are displayed to one or more users, wherein at least one of the one or more users is a user other than the user who provided the NL input.

8. The method according to claim 1, wherein, The NL input is obtained from the user, and the method further includes: The plurality of memory content items may be obtained from one or more of the following: posts published on social media platforms that are associated with the user's account; repositories on the user's device; accounts associated with the user on cloud storage providers; or any combination thereof.

9. The method according to claim 1, wherein, The method further includes: Obtain at least two memory content items from posts associated with the account on social media platforms; The account is associated with the user who provided the NL input, the account is associated with one or more other participants in the thread that provided the NL input, or the account is associated with the user who provided the NL input or with one or more other participants in the thread at at least a minimum specified connection level on the social media platform.

10. A computer-readable storage medium storing a plurality of instructions, which, when executed by a computing system, cause the computing system to perform a process for automatically presenting a memory item, the process comprising: Obtain natural language (NL) input; Identify one or more key elements of the NL input; Calculate multiple matching scores, where each matching score indicates the degree of match between the NL input and one of the multiple memory content items, and where each matching score is based on a comparison between the NL input and one of the multiple memory content items, the comparison including: Retrieve one or more content item tags for a specific memory content item; and By determining the degree of matching between the one or more key elements and the one or more content item tags of the specific memory content item, a matching score between the specific memory content item and the NL input is calculated; one or more memory content items corresponding to one or more of the highest calculated matching scores are selected; and Provide the selected one or more memory content items.

11. The computer-readable storage medium according to claim 10, wherein, The NL input is obtained from the user, and the process further includes: The plurality of memory content items may be obtained from one or more of the following: posts published on social media platforms that are associated with the user's account; repositories on the user's device; accounts associated with the user on cloud storage providers; or any combination thereof.

12. The computer-readable storage medium according to claim 10, wherein, The process also includes: Obtain at least two memory content items from posts associated with the account on social media platforms; The account is associated with the user who provided the NL input, the account is associated with one or more other participants in the thread that provided the NL input, or the account is associated with the user who provided the NL input or with one or more other participants in the thread at at least a minimum specified connection level on the social media platform.

13. The computer-readable storage medium according to claim 10, wherein, The identification of the one or more key elements of the NL input includes: applying a machine learning key element model to the NL input, wherein the machine learning key element model is trained to identify key elements using multiple training item pairs, each training item pair including: a pair of NL text and a key element mapped to the NL text.

14. The computer-readable storage medium according to claim 10, wherein, The step of obtaining the one or more content item tags of the specific memory content item includes: applying a machine learning tag model to the specific memory content item, wherein the machine learning tag model is trained to determine content item tags using training items comprising multiple pairs, each pair comprising: a pair of A) content items and B) one or more tags mapped to the content item.

15. The computer-readable storage medium according to claim 10, wherein, The determination of the matching degree is performed in the following manner: Embed the one or more key elements into an n-dimensional space; Embed the one or more content item tags into the n-dimensional space; as well as Calculate one or more distances between the embedded one or more key elements and the embedded one or more content item tags in the n-dimensional space.

16. The computer-readable storage medium of claim 10, wherein, The provision of the selected one or more memory content items includes: Provide the user who provided the NL input with an indication of the selected one or more content items; Receive permission from the user who provided the NL input; and In response to the permission, the selected one or more content items are displayed to one or more users, wherein at least one of the one or more users is a user other than the user who provided the NL input.

17. A computing system for automatically presenting memorized content items, the computing system comprising: One or more processors; as well as One or more memories storing a plurality of instructions, which, when executed by the one or more processors, cause the computing system to perform a process including the following steps: Obtain natural language (NL) input; Calculate multiple matching scores, where each matching score indicates the degree of match between the NL input and one of the multiple memory content items, and where each matching score is calculated in the following manner: The image / NL matching module is applied, and the image / NL matching module: Obtain one or more item tags for a specific memory item; and A matching score between the specific memory content item and the NL input is calculated by determining the degree of matching between one or more key elements identified for the NL input and one or more content item tags of the specific memory content item; or The application tag matching module, wherein the tag matching module: The NL input is divided into multiple segments; Multiple sub-match scores are generated, each sub-match score being calculated by applying a machine learning model to the specific memory content item and one of the multiple segments; and The plurality of sub-matching scores are combined into one matching score from the plurality of matching scores; Select one or more memory content items corresponding to one or more of the highest computed matching scores from a plurality of highest computed matching scores; and Provide the selected one or more memory content items.

Images