Automatic message reply

Abstract

A method, computer storage medium and system for selecting an original message for an automatic reply result. An automatic message reply system can automatically select the message that a potential reply message is replying to. The automatic message reply system can obtain a message thread, a potential reply message and a context. The automatic message reply system can filter the message thread and generate a model input based on the following items: remaining messages, potential reply messages and context. The automatic message reply system can apply the model input to a machine learning model, which can generate a reply score for the remaining messages. After generating the reply score, the automatic message reply system can determine whether the remaining message with the highest reply score qualifies as the original message to be replied. The automatic message reply system can display the potential reply message as a reply to the determined original message.

Description

Technical Field

[0001] The present disclosure relates to a computerized messaging platform having automated reply functionality. Background Art

[0002] Messaging platforms and applications allow users to communicate with other users who are also online and logged into the same messaging platforms and applications. For example, social media platforms attract billions of active users per month around the world through various messaging functionalities. By facilitating the creation and sharing of information, social media platforms make the world more connected. Among the many forms of communication enabled through social media services, messaging applications are often the most commonly used means of communication between individuals. Users can send each other messages involving text, emoticons, links, images, videos, audio recordings, and various other forms of message content.

[0003] Many messaging platforms provide users with such communications with various features that enhance the communication experience. Some of the various messaging platforms include instant messaging, email, SMS (Short Message Service), MMS (Multimedia Messaging Service), group messaging, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 is a block diagram illustrating an overview of a device upon which some embodiments may operate.

[0005] Figure 2 is a block diagram illustrating an overview of an environment in which some embodiments may operate.

[0006] Figure 3 is a block diagram illustrating components that may, in some implementations, be used in a system employing the disclosed technology.

[0007] Figure 4 is a flow diagram illustrating a process for training a machine learning model to generate a response score in some implementations.

[0008] Figure 5 is a flow diagram illustrating a process for automatically selecting an originating message for a potential reply message in some implementations.

[0009] Figure 6 is a flow diagram illustrating a process used to filter message threads in some implementations.

[0010] Figure 7 is a conceptual diagram illustrating an example of a message thread with time stamps showing messages to be removed by filtering.

[0011] Figure 8 is a conceptual diagram illustrating an example of a user interface with a message shown as an originating message.

[0012] Fig. 9 is a conceptual diagram illustrating an example of a user interface with arrows showing messages as replies to other messages.

[0013] The technology described herein may be better understood by reference to the following detailed description in conjunction with the drawings, wherein like reference numbers indicate identical or functionally similar elements. DETAILED DESCRIPTION

[0014] Aspects of the present disclosure relate to an automatic message reply system. When a user is busy having a conversation on a messaging platform, multiple messages may be exchanged, so it may be difficult for the user to know which message is replying to which other message. A user may send a reply message in response to a previous message in a message thread. A "message thread" as used herein is a collection of messages exchanged between users on a messaging platform. A user may want the receiving user to know which previous message he / she has explicitly replied to. In some cases, a user may send a potential reply message and the receiving user may be confused as to whether the potential reply message refers to a previous message and / or which previous message the potential reply refers to. A "potential reply message" as used herein is a message that may or may not be a reply to any previous message in a message thread.

[0015] The automatic message reply system can automatically select the original message for the potential reply message. As used herein, "original message" is any message that the reply message is replying to. In other words, the automatic message reply system can automatically select which message in the message thread the potential reply message is replying to. The user receiving the message does not need to infer which sent message the received message is replying to. In addition, the user sending the message does not need to manually indicate which message they are selecting to reply. The automatic message reply system can identify the original message, and the original message is paired with the potential reply message so that the user can visualize the relationship between the messages. In some embodiments, the automatic message reply system can automatically select the original message group. As used herein, "original message group" is any message group that the reply message is replying to. This can explain when the potential reply message is a reply to more than one message.

[0016] In some embodiments, the automatic message reply system uses a machine learning model to automatically determine the originating message. To prepare data for training the machine learning model, the automatic message reply system can obtain a reply instance, which can be the following tuple: a reply message, a potential message to be replied, and a true label of True or False about whether the potential message to be replied is the originating message for the reply message (e.g., {reply message, potential message to be replied; True / False label}). In some embodiments, the reply instance can be a tuple generated based on all possible combinations of message pairs from the historical message thread of the user on the message platform. For example, a tuple with a True label can include the following instance: when the first user manually selects a message from the second user to reply on the message platform (e.g., {a reply message from the first user, a message from the second user manually selected by the first user to reply, True}), while a tuple with a False label includes all other instances where no manual selection has occurred. The automatic message reply system can also obtain a context for each reply instance, which can include various context data of the message in the reply instance. The context data can include, but is not limited to, the results of the topic analysis of the message, text features, analysis of the associated content items in the message, timing features, user message device context, or user attention. Based on the reply instances and their corresponding contexts, the automatic message reply system can generate model inputs for each reply instance and the corresponding context. The model input can be a text embedding of the reply message and the potential message to be replied to, concatenated with a feature vector of the context data. In some embodiments, the automatic message reply system can generate the model input by grouping reply instances that contain the same reply message and have potential message replies received by the user within a time window.

[0017] In order to train the machine learning model, the automatic message reply system can apply the model input to the machine learning model, and the machine learning model can update its model parameters to learn to generate a reply score. The reply score can indicate the degree of possibility that, for a reply instance, the potential reply message to be replied to is the original message for the reply message (for example, a higher score may mean a higher possibility, while a lower score may mean a lower possibility). Therefore, the machine learning model can learn to determine which messages are actual replies to which messages based on identifying patterns in the model input. In some embodiments, the reply score can indicate: for a reply instance, the degree of possibility that the message group (potential messages to be replied that the user receives within a time window of each other) is the original message group for the reply message. Once the machine learning model is trained to generate a reply score, the reply score can be provided for testing or deployment. The following is about Figure 4 Describes training machine learning models in more detail.

[0018] After the machine learning model is trained, the automatic message reply system can use the machine learning model to automatically select the originating message for the potential reply message. The automatic message reply system can obtain a message thread containing a message and a potential reply message and the context of the potential reply message. The message thread may contain candidates for the originating message. The automatic message reply system can determine which of the candidate messages are most likely to be the originating message. In some embodiments, the automatic message reply system can filter the messages of the message thread based on the context of the potential reply message. This can leave an unfiltered candidate message set. The automatic message reply system can first filter messages with retroactive restrictions. Retroactive restrictions can remove messages that are not on the screen of the replying user within a threshold time. By filtering based on retroactive restrictions, messages from the past that are unlikely to be replied can be removed. For example, assume that the automatic message reply system obtains the following message thread: User 1 sends the first message at 5:00 p.m. on Tuesday asking "How are you doing today?" and User 1 sends the second message at 9:00 a.m. on Wednesday asking "Do you want pizza?" If the lookback limit is 12 hours and the first message from user 1 was last seen by user 2 more than 12 hours ago, then in this example the first message from user 1 will be filtered out.

[0019] In addition to or as an alternative to filtering by backtracking limits, the automatic message reply system can filter messages with a cutoff limit. The cutoff limit can remove messages that are received after the replying user begins typing a potential reply. These messages can be removed because they appear after the user begins typing the potential reply message and are therefore unlikely to be messages that will be replied to (a user is unlikely to start replying to a message that doesn't exist yet). For example, assume that the automatic message reply system obtains the following message thread: User 1 sends a first message asking "How are you today?" at 5:00:00 PM on Tuesday, and User 1 sends a second message asking "Do you want pizza?" at 5:00:11 PM on Tuesday. If User 2 begins typing the potential reply message at 5:00:10 PM on the same day, then the second message from User 1 in this example will be filtered out because the second message was sent 1 second after User 1 began typing. The following text is combined with Figure 5 Box 508 and Figure 6 Describes filtering message threads in more detail.

[0020] After filtering the message thread based on the potential reply context, the automatic message reply system can generate a reply score for each remaining message in the remaining messages (unfiltered messages in the message thread or original messages (if no filtering is performed)) by: (1) generating a model input based on the potential reply messages, the remaining messages, and the context of the potential reply messages; and (2) applying the model input to a machine learning model trained to generate reply scores. Each reply score in the reply scores can indicate a degree of likelihood that the remaining message is an original message. In some embodiments, the automatic message reply system can group the remaining messages received by the user within a time window and generate a reply score for the message group. After generating the reply scores for the remaining messages, the automatic message reply system can identify the remaining messages with the highest reply scores. The remaining messages with the highest reply scores can be identified as original messages. In some embodiments, the automatic message reply system can identify the original message group with the highest reply score.

[0021] In some embodiments, before identifying the original message, the automatic message reply system can determine whether the remaining messages with the highest reply score qualify as the original message. The remaining messages with the highest reply score can qualify when the highest reply score is above a threshold confidence value and / or is a threshold amount above all other reply scores of the remaining messages. The threshold can ensure that there is sufficient confidence that the message is the original message.

[0022] In response to identifying the original message and / or determining that the remaining messages with the highest reply score are eligible, the automatic message reply system can cause the potential reply message to be displayed as a reply to the original message. For example, the potential reply message and the identified original message can be visually paired to show the reply relationship. This can allow the replying user to avoid manually indicating which message they are replying to, and the receiving user can visually understand which message the potential reply message is a reply to. In some embodiments, the automatic message reply system can display the potential reply message as a reply to the original message group with the highest reply score. Figure 5 Automatically selecting which message a potential reply message is a reply to is described in more detail.

[0023] As an example, assume that the automatic message reply system obtains a message thread containing the following: (user 1 and user 2 exchanged several messages), user 1 sent a message asking "how are you doing today" at 5:00:00 pm on Tuesday, and sent a message "do you want to eat pizza" at 5:00:10 pm on Tuesday. The automatic message reply system also obtains a potential reply message saying "yes, I do" and the context of the potential reply message from user 2 at 5:30:00 pm on Tuesday. Then, the automatic message reply system can obtain a machine learning model trained based on global user data and / or data from historical messages exchanged between user 1 and user 2. The automatic message reply system can filter the messages exchanged between user 1 and user 2 by a backtracking limit or a cutoff limit. After filtering, the automatic reply message can generate a model input for the remaining unfiltered messages, apply the model input to the machine learning model and generate the following reply scores: (all scores generated for the remaining unfiltered messages exchanged between user 1 and user 2 are lower than 40), the score for the message "how are you doing today" is 40, and the score for the message "do you want to eat pizza" is 85. Assume for this example that the threshold confidence is set to 70 and the threshold margin between reply scores is set to 20. The highest reply score is 85, and because the highest reply score is above the threshold confidence of 70 and is at least 20 higher than the reply scores of all other messages (85-20=65, and 65>40), the message "Do you want pizza?" will qualify as an originating message to be replied to. The automatic message reply system may display "Yes, I do" as a reply to the message "Do you want pizza?"

[0024] In some implementations, the automatic message reply system may suggest one or more "reply candidates" that the user may select as the actual original message she is replying to. For example, if the second-best candidate scores 70 (rather than 40 in the previous example), the automatic message reply system may display the best and second-best scoring messages for the user to choose from. Thus, instead of having a hard threshold on the margin, the automatic message reply system may have a "suggestion threshold" where when more than one message scores above this threshold, those messages are suggested to the user as suggested replies to the candidates.

[0025] Various messaging platforms allow users to exchange messages back and forth with other users. These platforms generally lack visual cues for users to distinguish which messages are replying to which other messages. Therefore, users may often be confused about which previous message in the thread is actually from another user's reply message. When users have different messaging habits and wording, this difficulty will escalate, requiring users to be more descriptive in their reply messages so that the recipient can understand which message they are responding to. Therefore, users usually have to exchange longer descriptive messages and larger amounts of messages in order to alleviate confusion, resulting in increased traffic on the messaging platform. Some messaging platforms allow users to manually select messages to reply. These platforms often require a lot of user interaction with the system, which can cause the message experience to be often intermittent, lack of fluency, and even annoying to use (e.g., users press many buttons to select many messages to reply, users constantly select messages to reply when talking back and forth quickly, and users scroll up and down in the conversation to find and select messages to reply). When users use their devices with one hand, this complexity becomes further exacerbated, resulting in incorrect manual selection of message reply functionality. Additionally, existing systems lack an option for a user to select multiple messages (message groups) to reply to all at once and display the messages (message groups) as to be replied to.

[0026] The automatic message reply system and process described herein are expected to overcome these problems associated with traditional message platforms, and are expected to provide users with an understanding of which messages are replies to which other messages in an automatic and robust manner. By selecting the originating message for the potential reply message, the automatic message reply system and process can eliminate the confusion about which messages in the thread will be replied. Due to less confusion in the communication, the receiving user no longer needs to spend as much time as possible to infer the message habits and wording of the replying user. Since the receiving user can automatically know what the message refers to, the replying user can spend less time to elaborate or be more descriptive in their reply messages. Therefore, shorter messages and less messages can be exchanged across message platforms, so that the required amount of computing resources is reduced. The message platform can then use the automatic reply system and process to provide communication with less network capacity and less waiting time. In addition, because the automatic reply system and process utilize reliable heuristics to filter messages and enrich context data to select the originating message, the burden of having to train a machine learning model (high computational cost and lack of data) can be reduced. Using the auto-reply message system and process, fewer training examples need to be provided because the model can filter unlikely candidates for the originating message and utilize various patterns with valuable contextual data. User-specific message data can also help reduce training time and save the model from having to be as general as possible for many different users. The auto-reply message system and process can therefore use fewer, lower-power, and lower-cost computing devices, as well as fewer, smaller-capacity, and lower-cost storage devices.

[0027] The automatic message reply system and process described herein provide many advantages over the user manual selection system. By automatically selecting the originating message for potential reply messages on behalf of the user, the automatic message reply system and process allow the user to focus only on thinking responses and typing reply messages. This can greatly improve the user experience during message delivery. Conversations can be better conducted because users can focus their attention only on thinking and replying, when users send messages and perform some other tasks at the same time (especially when messaging with one hand), users can better multitask, and when users want to respond to older messages, scroll fatigue can be eliminated because a choice has been made for the user. The ease of automatically selecting messages that can receive reply functionality can make message delivery smoother and easier, especially when multiple messages are exchanged in a short time and may be cumbersome for users who must manually select. Because the automatic message reply system and process can generate a reply score for a message, the possibility that a message is an originating message can be quantified, which is impossible to achieve with a manual selection system. This measurement is a valuable data point for understanding user reply behavior, which is lacking in existing systems and methods. In addition, because user correction feedback can be provided, the automatic reply message system and process can be personalized for the user and reduce errors in model selection, making it robust and reliable. In addition, when the user wants to respond to more than one message at a time, the automatic reply message system and process can select the originating message group and make the selection display, which is also lacking in existing messaging platforms.

[0028] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Figure 1 1 is a block diagram illustrating an overview of a device on which some embodiments of the disclosed technology can be run. The device may include hardware components of the device 100, which can automatically select messages to reply on the messaging platform. The device 100 may include one or more input devices 120, which provide input to (multiple) processors 110 (e.g., (multiple) CPUs, (multiple) GPUs, (multiple) HPUs, etc.), notifying them of actions. The actions can be mediated by a hardware controller that interprets signals received from the input devices and communicates the information to the processor 110 using a communication protocol. The input device 120 includes (for example) a mouse, keyboard, touch screen, infrared sensor, touchpad, wearable input device, camera or image-based input device, microphone, or other user input device.

[0029] The processor 110 can be a single processing unit or multiple processing units in a device, or distributed across multiple devices. The processor 110 can be coupled to other hardware devices, such as by using a bus, such as a PCI bus or a SCSI bus. The processor 110 can communicate with a hardware controller for the device, such as a display 130. The display 130 can be used to display text and graphics. In some embodiments, the display 130 provides graphical and textual visual feedback to the user. In some embodiments, the display 130 includes an input device as part of the display, such as when the input device is a touch screen or is equipped with an eye direction monitoring system. In some embodiments, the display is separated from the input device. Examples of display devices are: LCD display screens, LED display screens, projections, holographic or augmented reality displays (such as head-up display devices or head-mounted devices), etc. Other I / O devices 140 can also be coupled to the processor, such as a network card, a graphics card, a sound card, a USB, a FireWire or other external devices, a camera, a printer, a speaker, a CD-ROM drive, a DVD drive, a disk drive, or a Blu-ray device.

[0030] In some embodiments, the device 100 also includes a communication device capable of wireless or wired communication with a network node. The communication device can communicate with another device or server through a network using (for example) TCP / IP protocol. The device 100 can utilize the communication device to distribute operations across multiple network devices.

[0031] The processor 110 can access a memory 150 in a device or distributed across multiple devices. The memory includes one or more of various hardware devices for volatile and non-volatile storage, and can include both read-only memory and writable memory. For example, the memory can include random access memory (RAM), various cache memories, CPU registers, read-only memory (ROM), and writable non-volatile memory (e.g., flash memory, hard drive, floppy disk, CD, DVD, magnetic storage device, tape drive, etc.). The memory is not a propagation signal separated from the underlying hardware; therefore, the memory is non-transient. The memory 150 can include a program memory 160, which stores programs and software, such as an operating system 162, an automatic message reply system 164, and other applications 166. The memory 150 may also include a data storage 170, such as training data sets for reply instances, context data for reply instances, machine learning models to be trained or trained, machine learning models to be retrained or fine-tuned, message threads, potential reply messages, context data for potential reply messages, backtracking limits, cutoff limits, calculated reply scores, visual data for a user interface, configuration data, settings, user options or preferences, etc., which can be provided to the program storage 160 or any element of the device 100.

[0032] Some embodiments may operate with numerous other computing system environments or configurations. Examples of computing systems, environments, and / or configurations that may be suitable for use with the technology include, but are not limited to, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0033] Figure 2 2 is a block diagram illustrating an overview of an environment 200 in which some disclosed embodiments may operate. The environment 200 may include one or more client computing devices 205A-D, examples of which may include the device 100. The client computing device 205 may operate in a networked environment using logical connections to one or more remote computers (e.g., server computing devices) through a network 230.

[0034] In some embodiments, server 210 may be an edge server that receives client requests and coordinates the implementation of those requests by other servers (e.g., servers 220A-C). Server computing devices 210 and 220 may include computing systems, such as device 100. Although each server computing device 210 and 220 is logically shown as a single server, the server computing devices may each be a distributed computing environment, covering multiple computing devices located at the same or geographically different physical locations. In some embodiments, each server 220 corresponds to a server group.

[0035] The client computing device 205 and the server computing devices 210 and 220 can each act as a server or client of other server / client devices. The server 210 can be connected to a database 215. The servers 220A-C can each be connected to corresponding databases 225A-C. As described above, each server 220 can correspond to a server group, and each of these servers can share a database or can have its own database. The databases 215 and 225 can store (warehouse / store) information, such as training data sets for reply instances, context data for reply instances, machine learning models to be trained or trained, machine learning models to be retrained or fine-tuned, message threads, potential reply messages, context data for potential reply messages, backtracking limits, cutoff limits, calculated reply scores, and visualization data for user interfaces. Although the databases 215 and 225 are logically displayed as a single unit, the databases 215 and 225 can each be a distributed computing environment covering multiple computing devices, can be located in the corresponding servers of the database, or can be located at the same or geographically different physical locations.

[0036] The network 230 may be a local area network (LAN) or a wide area network (WAN), but may also be other wired or wireless networks. The network 230 may be the Internet or some other public or private network. The client computing device 205 may be connected to the network 230 via a network interface, such as via wired or wireless communication. Although the connections between the server 210 and the server 220 are shown as separate connections, these connections may be any kind of local, wide area, wired or wireless network, including the network 230 or a separate public or private network.

[0037] In some embodiments, servers 210 and 220 may be used as part of a social network. The automatic message reply system described herein may be used for any type of messaging system, not just those implemented by a social networking system. A social network may maintain a social graph and perform various actions based on the social graph. A social graph may contain a set of nodes (representing social networking system objects, also referred to as social objects) interconnected by edges (representing interactions, activities, or correlations). A social networking system object may be a social networking system user, a non-personal entity, a content item, a group, a social networking system page, a location, an application, a topic, a conceptual representation, or other social networking system object, such as a movie, a band, a book, etc. A content item may be any digital data, such as text, an image, audio, video, a link, a web page, a detail (e.g., a tag provided from a client device (e.g., an emotion indicator, a status text snippet, a location indicator, etc.)) or other multimedia. In various embodiments, a content item may be a social networking item or part of a social networking item, such as a post, a like, a mention, a news item, an event, a share, a comment, a message, other notifications, etc. Topics and concepts in the context of a social graph include nodes representing any person, place, thing, or idea.

[0038] The social networking system may enable a user to enter and display information related to the user's interests, age / date of birth, location (e.g., longitude / latitude, country, region, city, etc.), educational information, life stage, emotional state, name, commonly used device model, languages ​​identified as familiar to the user, occupation, contact information, or other demographic or biographical information in the user's profile. In various embodiments, any such information may be represented by nodes in the social graph or edges between nodes. The social networking system may enable a user to upload or create pictures, videos, documents, songs, or other content items, and may enable a user to create and schedule events. In various embodiments, content items may be represented by nodes in the social graph or edges between nodes.

[0039] The social networking system may enable users to upload or create content items, interact with content items or other users, express interests or opinions, or perform other actions. The social networking system may provide various means of interacting with non-user objects within the social networking system. In various embodiments, actions may be represented by nodes in a social graph or edges between nodes. For example, a user may form or join a group, or become a fan of a page or entity within the social networking system. In addition, a user may create, download, view, upload, link, tag, edit, or play a social networking system object. A user may interact with a social networking system object outside the context of the social networking system. For example, an article on a news website may have a "like" button that a user can click. In each of these instances, the interaction between a user and an object may be represented by an edge connecting the user's node to the object's node in the social graph. As another example, a user may use location detection functionality (e.g., a GPS receiver on a mobile device) to "check in" to a particular location, and an edge may connect the user's node to the node of the location in the social graph.

[0040] A social networking system may provide users with a variety of communication channels. For example, a social networking system may enable a user to send an email, instant message, or text / SMS message to one or more other users; may enable a user to post a message to a user's wall or profile or to another user's wall or profile; may enable a user to post a message to a group or fan page; may enable a user to comment on an image, wall post, or other content item created or uploaded by the user or another user, etc. In some embodiments, a user may post a status message to a user's profile to indicate a current event, mental state, thought, feeling, activity, or any other current time-related communication. A social networking system may enable users to communicate within and outside the social networking system. For example, a first user may send a second user a message within the social networking system, an email through the social networking system, an email outside the social networking system but originating from the social networking system, an instant message within the social networking system, or an instant message outside the social networking system but originating from the social networking system. In addition, a first user may comment on a second user's profile page, or may comment on an object associated with the second user (e.g., a content item uploaded by the second user).

[0041] A social networking system enables users to associate themselves and establish connections with other users of the social networking system. When two users (e.g., social graph nodes) explicitly establish a social connection in a social networking system, the two users become "friends" (or "connections") within the context of the social networking system. For example, a friend request from "John Doe" to "Jane Smith" accepted by "Jane Smith" is a social connection. A social connection can be an edge in a social graph. Becoming a friend or being within a threshold number of friend edges on a social graph can allow users to access more information about each other than other information available to unconnected users. For example, becoming a friend can allow a user to view another user's profile, view another user's friends, or view another user's pictures. Similarly, becoming a friend within a social networking system can allow a user to better access to communicate with another user, such as through email (internal and external to the social networking system), instant messaging, text messaging, phone, or any other communication interface. Becoming a friend can allow a user to access to view, comment on, download, approve, or otherwise interact with another user's uploaded content items. Establishing connections, accessing user information, communicating, and interacting within the context of a social networking system can be represented by an edge between nodes representing two social networking system users.

[0042] In addition to explicitly establishing a connection in a social networking system, in order to determine the purpose of the social context for determining a communication topic, users with common features can be considered to be connected (e.g., soft connections or implicit connections). In some embodiments, users belonging to a public network are considered to be connected. For example, users who go to a common school, work for a common company, or belong to a common social networking system group can be considered to be connected. In some embodiments, users with common biographical features are considered to be connected. For example, the geographic area where a user is born or lives, the age of the user, the gender of the user, and the emotional state of the user can be used to determine whether the user is connected. In some embodiments, users with common interests are considered to be connected. For example, the user's movie preferences, music preferences, political views, religious views, or any other interests can be used to determine whether the user is connected. In some embodiments, users who have taken common actions in a social networking system are considered to be connected. For example, users who recognize or recommend common objects, comment on common content items, or reply to common events can be considered to be connected. The social networking system can utilize a social graph to determine users who are connected or similar to a specific user in order to determine or evaluate the social context between users. The social networking system may utilize such social context and public attributes to facilitate content distribution systems and content caching systems that select content items in a predictable manner to be cached in cache devices associated with particular social networking accounts.

[0043] Figure 3 300 is a block diagram illustrating a component 300 that can be used in a system that employs the disclosed technology in some embodiments. Component 300 includes hardware 302, general software 320, and special components 340. As described above, a system that implements the disclosed technology can use various hardware, including a processing unit 304 (e.g., a CPU, a GPU, an APU, etc.), a working memory 306, a storage memory 308 (a local storage device or an interface to a remote storage device, such as a storage device 215 or 225), and an input and output device 310. In various embodiments, the storage memory 308 can be one or more of the following: a local device, an interface to a remote storage device, or a combination thereof. For example, the storage memory 308 can be a collection of one or more hard disk drives (e.g., a redundant array of independent disks (RAID)) accessible via a system bus, or can be a cloud storage device provider, or other network storage devices (e.g., a network accessible storage (NAS) device, such as a storage device 215 or a storage device provided by another server 220) that can be accessed via one or more communication networks. Component 300 may be implemented in a client computing device (eg, client computing device 205 ) or on a server computing device (eg, server computing devices 210 or 220 ).

[0044] The general software 320 may include various applications including an operating system 322, local programs 324, and a basic input-output system (BIOS) 326. The specialized components 340 may be subcomponents of the general software applications 320, such as local programs 324. The specialized components 340 may include, for example, a model trainer 344, a reply selector 346, a message filter 348, and components that may be used to provide a user interface, transfer data, and control the specialized components, such as an interface 342. In some implementations, the component 300 may be in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more specialized components 340. Although depicted as separate components, the specialized components 340 may be logical or other non-physical differences in functionality and / or may be submodules or code blocks of one or more applications.

[0045] The model trainer 344 can prepare training data for the machine learning model, train the machine learning model to generate a reply score for a message, and provide the trained machine learning model for testing or deployment. In some embodiments, the model trainer 344 can prepare training data based on a data set of reply instances, which can be obtained from the storage memory 308. The model trainer 344 can also determine the context for the reply instance, which contains various contextual data related to the message in the reply instance and can also be obtained from the storage memory 308. The model trainer 344 can generate model inputs based on the reply instance and its corresponding context, apply the model inputs to the machine learning model, and train the machine learning model. The machine learning model may include: neural networks, recurrent neural networks, convolutional neural networks, ensemble methods, cascade models, support vector machines, decision trees, random forests, logistic regression, linear regression, genetic algorithms, evolutionary algorithms, etc. After training the machine learning model, the model trainer 344 can provide the trained machine learning model to the reply selector 346 for testing or deployment. The following is about Figure 4 Boxes 402 through 412 provide additional details regarding training the machine learning model to generate response scores.

[0046] The reply selector 346 can automatically select which candidate message in the message thread is the originating message for the potential reply message. In order to make the selection, the reply selector 346 can obtain and use the machine learning model provided by the model trainer 344. In addition to obtaining the machine learning model, the reply selector 346 can obtain: potential reply messages, context data related to potential reply messages, and message threads with candidates for the originating messages for potential reply messages, all of which are from the storage memory 308 and / or the working memory 306. In some embodiments, the reply selector 346 can provide the message thread and context data to the message filter 348 for pre-processing the data via filtering. As described below, the message filter 348 can filter the messages in the message thread and provide the unfiltered messages back to the reply selector 346 for further processing. The reply selector 346 can generate a reply score for each remaining message in the remaining messages (unfiltered messages, or the originating messages in the message thread without filtering) by applying the obtained machine learning model to each remaining message in the remaining messages. If the message with the highest reply score qualifies as the message to be replied to, then the reply selector 346 can select the message with the highest reply score as the original message. An indication of this selection can be communicated to the computing device of the user who sent the original message via interface 342. In some embodiments, this selection can cause the potential reply message to be displayed as a reply to the selected original message via I / O 310 and / or via the computing device of the user who sent the original message. Figure 5 Boxes 502 through 514 provide additional details regarding the automatic selection of messages to be replied to.

[0047] The message filter 348 can filter messages in the message thread based on context data. In other words, the message filter 348 can pre-process the message thread before the message thread is further processed by the reply selector 346. The message filter 348 can first obtain the message thread and context data of the potential reply message provided by the reply selector 346. In order to filter the messages in the message thread, the message filter 348 can evaluate the context data to determine the backtracking limit and / or the cutoff limit. The backtracking limit can remove messages that are not on the replying user's screen within a threshold time. The cutoff limit can remove messages received after the replying user starts typing the potential reply message. After filtering, the message filter 348 can provide the remaining messages (unfiltered messages) back to the reply selector 346 for further processing. The following is about Figure 6 Blocks 602 through 608 provide additional details regarding filtering message threads.

[0048] Those skilled in the art will appreciate that the above-mentioned Figures 1 to 3 Components described in the flowcharts and in each of the flowcharts discussed below. For example, the order of the logic can be rearranged, sub-steps can be performed in parallel, the illustrated logic can be omitted, other logic can be included, etc. In some embodiments, one or more of the above components can perform one or more of the following processes.

[0049] Figure 4is a flowchart illustrating a process 400 used in some embodiments for training a machine learning model to generate a reply score. In some embodiments, process 400 can be executed in response to an administrator command to train a machine learning model on a specified data source. In other embodiments, process 400 can be executed or repeated in response to a time when another user-specific message data becomes available (e.g., a time when a user manually selects a message to reply to, a time when a user corrects a model selection for an originating message). In various embodiments, process 400 can be executed on a server device (e.g., a server of a messaging platform that sends messages between communicating users, processes message data between different users, and can calculate reply scores for messages) or a client device (e.g., a user device that sends messages to and receives messages from other user devices on a messaging platform, processes message data communicated with other user devices, and can calculate reply scores for messages). Process 400 can be executed on a client device to maintain the privacy of user-specific message data and accounts for instances when the client device is offline and cannot connect to the server. When completed, process 400 can provide a trained machine learning model, such as for Figure 5 In process 500 at block 502 .

[0050] At box 402, process 400 can obtain one or more reply instances. Reply instances can be data records, each of which is represented by a tuple data type of the following items: a reply message, a potential message to be replied to, and a true label of True or False as to whether the reply message is an actual reply to the potential message to be replied to (e.g., {reply message, potential message to be replied to; True / False label}). For example, a reply instance can be, {"I'm fine", "How are you today?", True}, while another reply instance can be {"I like playing basketball", "Want pizza for dinner", False}, etc. In some embodiments, process 400 can obtain reply instances from a historical message dataset (global message data) exchanged between all users or subsets of users on a messaging platform. The dataset can contain all (or alternatively, a subset of) possible combinations of labeled message pairs extracted from a user's historical message thread. For example, a tuple with a True label may include instances when a user manually selects a message from another user to reply to on a messaging platform (e.g., {reply message from first user, message from second user manually selected by first user to reply to, True}), while a tuple with a False label may cover all other instances where manual selection did not occur. In other embodiments, process 400 may obtain reply instances from a dataset of historical messages sent and received by only a single user on a messaging platform. The dataset may include all (or alternatively, a subset of) possible combinations of labeled message pairs extracted from a single user's historical message threads.

[0051] When the machine learning model is to be fine-tuned or retrained with user-specific message data, process 400 can also obtain reply examples from the data set of a single user. This can help the machine learning model better fit the message behavior and needs of a single user (because the model can see specific training examples of a single user's messages with other users), can better focus on learning a single user's reply behavior to other users, and can avoid having to rely on generalizing as much as possible from data from a variety of different users. In various embodiments, process 400 can obtain both global message data and user-specific message data for training machine learning models.

[0052] In some embodiments, additional user-specific training data may include instances when a user manually corrects a model selection. In other words, the acquired data record may include instances when a user manually corrects a model selection. Figure 5 The process 500 selects the instance of the originating message. User corrections can occur at Figure 5At box 514 of . For example, the user interface of the messaging platform may include an option for the user to indicate whether the selected originating message is incorrect (e.g., a "remove selection" user interface button, an "X" user interface element to be pressed). For example, assume that the message "Do you like pie" is selected as the originating message for the potential reply message "I like playing basketball". In this example, the user can press the "remove selection" button on their messaging platform to correct the model selection (basketball and pie are not related). User corrections can be applied to penalize the machine learning model trained to generate reply scores, because the user correction represents an instance when the trained machine learning model makes an incorrect selection of the originating message. The user correction can provide negative training examples for the model to be retrained and improved. The model can better understand what type of message is unlikely to be the originating message for certain reply messages. User corrections can also provide user feedback on how it performs to the machine learning model. Several user corrections can indicate that the model performs poorly, while some user corrections can indicate that the model performs well.

[0053] In some embodiments, process 400 can assign greater training weights to data records of user-specific training data to account for the training improvements provided by the user corrections to the machine learning model. Data records assigned with greater training weights can have a greater impact on the model than other data records during training.

[0054] At box 404, process 400 can obtain context for each reply instance in one or more reply instances. Process 400 can obtain context from a data set of various context data records collected for each reply instance in the reply instance. Process 400 can collect context data from the message of the reply instance (the reply message and / or the potential message to be replied), or from the reply message and / or the components of the receiving user's device, which includes but is not limited to: timer / clock, location / GPS service, Internet connection, camera, etc. Process 400 can then use all or a subset of the context data as features (context features) of the machine learning model (e.g., generating feature vectors based on the context data to input into the model). Context data may include but is not limited to: the results of the topic analysis of the message, text features, analysis of the associated content items in the message, timing features, user message device context, or user attention.

[0055] In some embodiments, the topic analysis of the message may include: determining what the topic discussed in the message is. For example, the analysis of the reply instance may determine that the message is talking about food, work, or any other topic. Process 400 may apply topic analysis to the message of the reply instance, and the topic analysis may be suitable for inclusion in the feature vector to be input to the machine learning model later. Using the topic analysis and other contextual features described herein, the model can learn patterns from, for example, the following items: the message is more likely to be the originating message for the reply message sharing the same discussion topic, or some users like to send reply messages with specific jargon / language / tone / phrase / grammar for specific topics. For example, a message discussing a food topic (e.g., "I like pasta") is more likely to be the originating message for a reply message that also discusses a food topic (e.g., "I prefer pizza"). On the other hand, a message discussing a food topic (e.g., "Do you like pizza?") is unlikely to be the originating message for a reply message that discusses a work topic in the opposite direction (e.g., "I had a hard day at the office"). As another example, a user may like to reply to an originating message containing a funny comment or joke as its topic with the message "LOL" representing "laughing out loud".

[0056] In some embodiments, the text feature may include various features related to the text content of the message. Examples of text features for messages may include, but are not limited to, the length of the message, the part-of-speech tag for the message, the word contained in the message, and / or the type analysis of the message. Type analysis may refer to determining whether the message contains questions, comments, etc. Process 400 can extract text features from the message of the reply instance, and the text features may be suitable for being included in the feature vector to be input to the machine learning model later. Using the text features and other context features described herein, the model can learn, for example, the following patterns: shorter messages tend to be replies to other shorter messages, longer messages tend to be replies to other longer messages, messages with verbs tend to be replies to messages asking for actions, messages with nouns tend to be replies to messages asking for objects or living things, messages with adjectives tend to be replies to messages asking for some descriptions, messages asking questions or making comments are more likely to be originating messages, messages with certain words tend to be replies to messages with specific words, depending on the user's jargon, language, tone, wording or syntax. As an example, a relatively short message saying "what's up" is more likely to be the originator of a relatively short reply message saying "nothing" than the originator of a relatively long reply message saying "it's been great, how about you?" As another example, a message asking "what did you do this week" is more likely to be the originator of a verb-focused reply message saying "I went fishing" than the originator of an adjective-focused message saying "the fish was delicious." As yet another example, a message saying "...?" is more likely to be the originator because it contains a question mark, or a message saying "I definitely prefer cats to dogs" is more likely to be the originator because it is a strong opinion that warrants a response. While in some embodiments, the system can explicitly make the above-mentioned types of inferences and the following types of inferences, in other embodiments, the inference between the message context (or a set of multiple message context features) and whether a message is a reply message can be encoded in the training of the machine learning model without explicitly determining the rationale for this relationship.

[0057] In some embodiments, the analysis of associated content items in a message may first include determining that the message contains content items. Content items may include, but are not limited to, links, pictures, audio clips, or videos. The analysis may also identify object identification tags, hash tags, topic identifiers, video length, sound length, who posted on social media, who liked social media posts, and / or any other metadata of the content items with respect to the content items. Process 400 may apply the analysis of associated content items to the message of the reply instance, and the analysis of associated content items may be suitable for inclusion in a feature vector to be input to a machine learning model later. Using analysis of associated content items and other contextual features described in this article, the model can learn, for example, the following patterns: a message with a picture / video link or attachment is more likely to be the originating message of a reply message that mentions something in a picture / video link, a message sharing a social media post or quoting a social media poster is more likely to be the originating message of a reply message that comments on a social media post or poster, a message sharing a long video / sound clip is more likely to be the originating message of a long reply message that comments on a video / sound clip, a message sharing a short video / sound clip is more likely to be the originating message of a short reply message that comments on a video / sound clip, a message with an object identification tag or hash tag is likely to be sharing information of the message it is replying to, and a message with a subject identified as the same or similar to another message is more likely to be a reply to that other message.

[0058] For example, a message that says "Watch this cool video: (insert link to video of someone doing stupid things)" might be the originator of a reply message that says "Wow, this guy is crazy." As another example, a message that says "Meghan is going to a game tomorrow" is more likely to be the originator of a reply message that says "#GoMeghan." As another example, a message with a funny video attachment is more likely to be the originator of a reply message that says "ROFL," which means "roll on the floor with laughter." As yet another example, a message containing a link to a movie is more likely to be the originator of a long analysis message of the movie that says "I really enjoyed the character development and the emotions captured. Certain moments gave me goosebumps. I think it might win a few awards."

[0059] In some embodiments, the timing feature may include various features, such as the time when the message is sent, the time when the message is read, and / or how long the message focus stays on the screen of the replying user. Process 400 can extract the timing feature from the timestamp of the record of the message of the reply instance, and the timing feature can be suitable for being included in the feature vector to be input to the machine learning model later. Using the timing features and other contextual features described herein, the model can learn, for example, the following patterns: the message sent earlier is more likely to be the originating message for the reply message that is also sent earlier, the message sent later is more likely to be the originating message for the reply message that is also sent later, the message that is read in seconds but answered for a long time is likely to be the originating message for the more complex reply message (the user considers the response for a period of time, because this is a complex answer), the message that is only read but not replied for a long time may be the originating message for the reply message sent when the replying user thinks more freely about the answer, and the message that focuses on the replying user's screen longer while the replying user is typing the reply message is more likely to be the originating message for the reply message that is replied. For example, suppose that user 1 sends a message collection to user 2, and user 2 replies to the message collection to user 1. Messages sent earlier in user 1's group are more likely to be the originating message for user 2's earlier sent messages in reply, while messages sent later in user 1's group are more likely to be the originating message for user 2's later sent messages in reply. This behavior of replying to messages in order may be common, although the reverse may also be true depending on the user's messaging behavior. As another example, suppose a message from user 1 remains focused on user 2's screen for a while, and user 2 begins typing a reply message. The message that remains focused is likely to be the originating message for the reply message.

[0060] In some embodiments, the device context may include information about where the replying user was when the message was sent and / or whether the message was sent after a period of Internet disconnection. Using the device context and other context features described herein, the model can capture scenarios such as: a message sent from a specific location of the replying user may be a reply to a message asking about something related to that specific location (e.g., weather, ongoing events, traffic, etc.), and a message sent after a period of Internet disconnection may be a reply to an older message. For example, suppose user 1 sends a message to user 2 saying "How is Brooklyn now?" If user 2 sends the message from Brooklyn, then the message is more likely to be a reply to user 1's message than a message not sent from Brooklyn.

[0061] In some embodiments, process 400 can determine user attention by first using a camera (e.g., a front-facing camera of a user device) to calculate where the replying user is looking. Process 400 can employ various computer vision techniques to track the eyes of the replying user and where they are looking on the screen. Next, process 400 can determine the message(s) that the replying user is viewing during a given time period and / or how long the replying user spent viewing the message(s) based on where the replying user is looking. Using user attention and other contextual features described herein, the model can capture scenarios such as the following: the user begins typing an answer, but will review the message they are replying to one or more times to ensure that they have answered the question correctly. For example, user 2 can review the message to be replied to from user 1, and then begin typing the reply message. While typing the reply message, user 2 can look back at the message from user 1 to ensure that they have understood the question correctly. In this scenario, the message from user 1 that user 2 is viewing may be the originating message for the reply message that user 2 is typing.

[0062] At block 406, process 400 can obtain a machine learning model to train on the reply instance and context. The machine learning model can include a neural network, a recurrent neural network (RNN), a long short-term memory network (LSTM), a convolutional neural network (CNN), an ensemble method, a cascade model, a support vector machine (SVM), a decision tree, a random forest, a logistic regression, a linear regression, a genetic algorithm, an evolutionary algorithm, or any other model or combination of models.

[0063] At box 408, process 400 can create a model input for each reply instance and the corresponding context pair in the reply instance. Process 400 can generate model input by representing the text of the reply message and / or the potential message to be replied as an embedding, and then connecting the text embedding in series with the feature vector of the context data. The form of the model input can depend on the architecture or model used. In some embodiments, process 400 can generate a model input for a recurrent neural network model, wherein the input cycle is a feature vector about text embedding and context data. For example, the input to the recurrent neural network can be a sequence of word embeddings of the reply message and / or the potential message to be replied, followed by a feature vector of the context data. In other embodiments, process 400 can generate a model input for an integrated model, wherein the text embedding can be used as an input to an integrated recurrent neural network and the feature vector of the context data can be used as an input to an integrated deep feedforward neural network. In various embodiments, process 400 can generate a model input for an architecture, which includes a convolutional neural network cascaded with a recurrent neural network. The convolutional neural network can take as input a feature vector of a video / photo of a link in a reply message and / or a potential message to be replied to, and output a vector representing the subject of the video / photo. For example, the convolutional neural network can recognize that the photo is a picture of a dog and output an embedding for "dog" or a more general embedding "pet". As another example, the convolutional neural network can recognize that the video is about a food review and output an embedding for the subject "food". Process 400 can then include the output subject as additional contextual data in its feature vector to be concatenated with the text embedding, and then use the subject for the recurrent neural network in the cascade. The above architectures and models are examples of possible configurations used, and are not the only configurations to which process 400 is limited.

[0064] In some embodiments, before applying the generated model input to the model, process 400 may group reply instances that contain the same reply message and have potential messages to be replied to by the user within a time window. In other words, a reply instance may be grouped if it shares the same reply message with another reply instance and has a potential message to be replied to within the time window of another reply instance. The time window may be a predefined threshold time value (e.g., in nanoseconds, milliseconds, seconds, minutes, etc.) that is fine-tuned to produce optimal message grouping. For example, assume that reply instance 1 has a reply message saying "Yes! I like this idea" and a potential message to be replied saying "Do you like our trip to Europe?" (received by the user at 09:00 am on December 12, 2020). Then assume that reply instance 2 has a reply message saying "Yes! I like this idea" and a potential message to be replied saying "Do you want to go there again next summer?" (received by the user at 09:01 am on December 12, 2020). In this example, if the time window is predefined to be greater than 1 minute, process 400 can group reply instance 1 and reply instance 2 together because the reply instance has the same reply message and their potential messages to be replied are less than 1 minute apart from each other. Grouping messages can take into account scenarios when a reply message potentially replies to more than one message (e.g., replies to a set of messages that are all received within a time window of each other). Providing training examples of grouped messages to a machine learning model can enable the machine learning model to learn these scenarios. For example, the reply message "Yes! I like this idea" can be a reply message to "Do you like our trip to Europe?" and "Do you want to go there again next summer?" both received within a narrow time window of each other. The phrase "Yes!" can refer to enjoying a trip to Europe, while "I like this idea" can refer to wanting to go again next summer. In order to group the reply instances, process 400 can combine feature vectors of text embedding and / or context data via concatenation, summation, averaging, weighted averaging, or any other aggregation method. The machine learning model can learn that these messages can be considered as a group during training.

[0065] At box 410, process 400 can apply the generated model input to the obtained machine learning model and update model parameters (e.g., weights, coefficients, hyperparameters, etc.) to train the model. Process 400 can train the machine learning model to output a reply score. The reply score output for the reply instance can be a numerical value indicating the degree of possibility that the potential message to be replied is the original message for the reply message. A higher value can indicate that the message is more likely to be the original message, while a lower value can indicate that the message is less likely to be the original message. By generating a reply score, the machine learning model can quantify the possibility that the message is the original message. During training, the machine learning model can learn to generate a high reply score for a reply instance with a True label, because the potential message to be replied is marked as the actual original message for the reply message. On the other hand, the machine learning model can learn to generate a low reply score for a reply instance with a False label, because the potential message to be replied is not marked as the actual original message for the reply message. The machine learning model can use various loss functions, including but not limited to mean absolute error loss, cross entropy loss, Huber loss, fair loss, asymmetric loss or mean square error loss to update its model parameters and implement training.

[0066] At block 412, process 400 may provide the trained machine learning model for testing, deployment, or another process. In some implementations, process 400 may provide the trained machine learning model to a Figure 5 Process 500 to generate a reply score for a potential reply message. Process 400 can provide a machine learning model when requested just in time or on a regular schedule (e.g., a predefined schedule for requesting a training model).

[0067] Figure 5 5 is a flowchart illustrating a process 500 used in some embodiments, which is used to automatically select an originating message for a potential reply message. In some embodiments, the process 500 can be performed in response to a replying user sending a potential reply message in an existing message thread. In various embodiments, the process 500 can be performed on a server device (e.g., a message sent between communicating users, a server of a message platform that processes message data between different users and can calculate a reply score for a message) or a client device (e.g., a user device that is used to send messages to other user devices and / or receive messages from other user devices on a message platform, processes message data communicated with other user devices, and can calculate a reply score for a message). The process 500 can be performed on a client device to maintain the privacy of user-specific message data and accounts for instances when the client device is offline and cannot be connected to the server.

[0068] At block 502, process 500 may obtain a computer trained to utilize at block 412 Figure 4 Process 400 generates a machine learning model for reply scores. In some embodiments, the machine learning model can be trained on a global dataset in process 400, and can also be retrained or fine-tuned with respect to user-specific message data of users who send and / or receive potential reply messages. The user-specific message data can include instances where a user manually selects a message to reply to or makes corrections to the model selection of the originating message at box 514. In some embodiments, the user-specific message data can also include selections made by process 500 that were not corrected by the potential reply messages sent / received by the user. Uncorrected model selections can represent instances when the machine learning model made a correct selection for the user and therefore serve as valuable data points. Correct selections can provide feedback to the model about instances when it performed well. In some embodiments, retraining or fine-tuning of the obtained machine learning model can occur when process 500 has been executed a predefined threshold number of times. This can be to ensure that process 500 has been executed a sufficient number of times to form a dataset of user-specific training examples that is large enough to be used to retrain or fine-tune the model. When the machine learning model is retrained, the model is repeated. Figure 4 of process 400.

[0069] At box 504, process 500 can obtain a message thread containing messages exchanged between a user who sent a potential reply message and a user who received the potential reply message. As an example, the message thread can be obtained from the following items: an instant messaging platform, a text message conversation, an email conversation, etc. The messages exchanged in the message thread (minus the potential reply message itself) can be candidates for the originating message selected for the potential reply message by process 500. In other words, the potential reply message can be a reply to any previous message in the message thread. The message thread can contain messages between users discussing one or more different topics (e.g., work, food, sports, emotional relationships, etc.). As "Example A" used in the further discussion of process 500 below, the message thread can contain the following: (a collection of messages exchanged between user 1 and user 2 before 5:00:00 p.m. on Tuesday), user 1 sends a message asking "How are you doing today" at 5:00:00 p.m. on Tuesday, and sends a message "Do you want to eat pizza" at 5:00:10 p.m. on Tuesday.

[0070] At block 506, process 500 may obtain the potential reply message itself and the context of the potential reply message from the replying user. The potential reply message may be the latest message from the replying user and may be a reply to any of the messages in the obtained message thread. The context of the potential reply message may include information about Figure 4The context data may be extracted from the potential reply message itself, extracted from the message in the message thread, obtained from the device of the user who sent and / or received the potential reply message, or any combination thereof. Returning to example A, assume that process 500 obtains a potential reply message saying "Yes, I do" from user 2 at 5:30:00 p.m. on Tuesday of the same day. The potential reply message may reply to any of the messages in the reply message thread, including "How are you doing today?" and "Do you want to eat pizza?" In this example, process 500 may also obtain the context of the potential reply message (e.g., topic analysis of the message "Do you want to eat pizza?" to determine that the message is about pizza, determining that the length of the potential reply message is 5 words, determining that the potential reply message was sent at 5:30:00 p.m. on Tuesday and the message "How are you doing today?" was sent at 5:00:00 p.m. on Tuesday, identifying that user 2 was looking at the message "Do you want to eat pizza?" when typing the potential reply message, etc.).

[0071] At block 508, process 500 may filter the messages of the obtained message thread based on the context of the potential reply message. Process 500 may filter the messages of the obtained message thread based on the context of the potential reply message in block 602 by providing the message thread and the potential reply context to the Figure 6 In some implementations, Figure 6 Process 600 may filter a message thread by removing messages that were not on the screen of the user sending the potential reply message within a threshold time. Returning to Example A, assume that User 2 receives a particular message from a set of messages exchanged between User 1 and User 2 before 5:00:00 PM on Tuesday at 9:00:00 AM. If the threshold time is predefined as 2 hours before the potential reply message was sent in Example A, then the particular message will be removed. In some other embodiments, process 600 may filter a message thread by removing messages received after the replying user begins typing the potential reply message. In Example A, assume that the particular message is received at 5:30:11 PM on Tuesday while User 2 is typing the potential reply message. Since the particular message was received 1 second after the replying user began typing, the particular message will be removed. After performing the filtering, process 500 may exit at box 608. Figure 6 The remaining messages (unfiltered messages) are obtained by process 600. Filtering can be performed as a data preprocessing step to remove messages that are unlikely to be the original message based on heuristics. This can reduce the burden on the machine learning model, which must also learn such heuristics on top of other patterns that are difficult to determine through heuristics alone. In some embodiments, process 500 can skip the filtering step, so that the remaining messages are just all the messages of the obtained message thread. Figure 6 Filtering message threads is described in more detail.

[0072] At block 510, process 500 may calculate a reply score for each of the remaining messages in the message thread by applying the obtained trained machine learning model to each of the remaining messages. Process 500 may calculate the reply score by first generating a model input based on the potential reply message, the remaining message, and the context of the potential reply message. The model input may be based on the previously generated model input. Figure 4 . Process 500 may generate a model input by representing the text of the potential reply message and / or the remaining message as an embedding, and then connecting the text embedding with the feature vector of the context data of the potential reply message. Returning to example A, assume that the messages "How are you today" and "Do you want to eat pizza" are both remaining messages. Then the text embedding may represent these messages and the potential reply message "Yes, I do". The form of the model input may depend on the architecture or model of the trained machine learning model. In some embodiments, the trained machine learning model is a recurrent neural network, wherein the input loop is on the text embedding and the feature vector of the context data. In other embodiments, the trained machine learning model is an integrated model, wherein the text embedding can be used as an input to the integrated recurrent neural network and the feature vector of the context data can be used as the input of the integrated deep feedforward neural network. In various embodiments, the trained machine learning model is a convolutional neural network cascaded with a recurrent neural network, wherein the text embedding and the feature vector of the context data can be used as input. The above architecture and model are examples of possible configurations that the trained machine learning model can be and process 500 is not limited to obtaining the architecture and model.

[0073] After generating the model input, process 500 can then calculate the reply score for each remaining message in the remaining messages by applying the model input to the obtained machine learning model trained to generate the reply score. Each reply score in the reply score represents the possibility that the remaining message is the original message for the potential reply message. The reply score output for the remaining message can be a numerical value representing the possibility that the remaining message is the original message for the reply message. A higher value can indicate that the remaining message is more likely to be the original message, while a lower value can indicate that the remaining message is less likely to be the original message. By generating a reply score, the trained machine learning model can quantify and predict the possibility that the remaining message is the original message for the potential reply message. After generating the reply score for each remaining message in the remaining messages, process 400 can identify the remaining messages with the highest reply score. Returning to example A, it is assumed that the filtered remaining messages have been applied to the machine learning model. The trained machine learning model can calculate the following reply scores: (all scores generated for the remaining unfiltered messages exchanged between user 1 and user 2 are below 40), the score for the message "How are you today?" is 40, and the score for the message "Do you want pizza?" is 85.

[0074] In some embodiments, before calculating the reply score, process 500 can group the remaining messages received by the replying user within each other's time window. In other words, if the remaining message is received by the replying user within the time window of another remaining message, then process 500 can group the remaining message and other messages also in the time window to form a message group. Returning to example A, it is assumed that two specific messages in the message set exchanged between user 1 and user 2 before 5:00:00 p.m. on Tuesday are received within 1 second of each other. If the time window is predefined as 2 seconds, then the two specific messages will be grouped together (1 second <2 seconds). The time window can be any predefined threshold time value (e.g., in nanoseconds, milliseconds, seconds, minutes, etc.) that is fine-tuned to produce the best message group. Grouping messages can explain the situation when a potential reply message potentially replies to more than one remaining message in the remaining message. In order to group the remaining messages, process 500 can combine the feature vectors of text embedding and / or context data via concatenation, summation, average, weighted average or any other aggregation method. After grouping the remaining messages, process 500 can calculate a reply score for the message group. A higher value can indicate that the message group is more likely to be the original message group, while a lower value can indicate that the message group is less likely to be the original message group. By generating a reply score, the trained machine learning model can quantify and predict the likelihood of the message group being the original message group for potential reply messages. In some embodiments, process 500 can identify the message group with the highest reply score (which is also higher than the reply scores of the remaining messages).

[0075] At box 512, process 500 can determine whether the remaining message or message group with the highest reply score is qualified as the originating message. In some embodiments, the remaining message or message group with the highest reply score can be qualified when the highest reply score is above a threshold confidence value. When high confidence is required when selecting the originating message, a higher threshold confidence value can be defined, while a lower threshold confidence value can be defined to make process 500 more flexible when selecting the originating message. In some embodiments, the remaining message or message group with the highest reply score is qualified when it is a threshold amount (margin) higher than all other reply scores of the remaining message or message group. When the highest reply score is required to be an appropriate score higher than other reply scores, a higher threshold margin can be defined, while when flexibility of the margin of error is required, a smaller threshold can be defined. The threshold can be a predetermined value that is fine-tuned to ensure that the remaining message or message group with the highest reply score is a predetermined value with sufficient confidence for the originating message of the potential reply message. Returning to example A, it is assumed that the threshold confidence is set to 70 and the threshold margin between reply scores is set to 20. The highest reply score is 85 and since this score is above the threshold confidence of 70 and is at least 20 higher than the reply scores of all other messages (85-20=65, and 65>40), the message "Do you want pizza?" would qualify as the originating message.

[0076] When the remaining message with the highest reply score is not qualified, process 500 does not select the originating message and process 500 can end. When the remaining message with the highest reply score is qualified, process 500 can select the remaining message as the originating message for the potential reply message, and process 500 can proceed to block 514.

[0077] At box 514, process 500 can make the potential reply message displayed as a reply to the remaining message or message group with the highest reply score. In other words, process 500 can make the potential reply message visually paired with the selected original message. By making the visual pairing of the message, the receiving user can visually understand which message the potential reply message is replying to. In some embodiments, process 500 can make the original message or message group directly displayed above the potential reply message. Returning to example A, process 500 can make the selected original message "Do you want to eat pizza?" be displayed directly above the potential reply message "Yes, I do." In other embodiments, process 500 can make it possible to display an arrow pointing to the potential reply message and the original message or message group. In example A, process 500 can make it possible to display an arrow pointing to the potential reply message "Yes, I do" and the selected original message "Do you want to eat pizza?". After making the reply displayed, if necessary, process 500 can make the display interface for the replying user to correct the model selection. In other words, process 500 can provide the replying user with an option to give feedback on how the selection of the machine learning model performed. When user corrections are made, process 500 can provide the user corrections to the replying user at block 402. Figure 4 The process 400 may be repeated as additional user-specific message data. The described visual elements for displaying potential reply messages as replies to the original message are not the only visual elements to which the process 500 is limited.

[0078] Figure 6 is a flow chart illustrating a process 600 for filtering message threads in some embodiments. In some embodiments, the process 600 may be responsive to Figure 5 Process 500 execution block 504 is performed to filter message threads based on the context of potential reply messages. In various embodiments, process 600 can be performed on a server device (e.g., a server of a messaging platform for sending messages between communicating users, processing message data between different users, and calculating reply scores for messages) or a client device (e.g., a user device for sending messages to other user devices and / or receiving messages from other user devices on a messaging platform, processing message data communicated with other user devices, and calculating reply scores for messages). Process 600 can be performed on a client device to maintain the privacy of user-specific message data and accounts for instances when the client device is offline and cannot connect to the server.

[0079] At block 602, process 600 may proceed from blocks 504 and 506. Figure 5The process 500 obtains a message thread and a context of potential reply messages (potential reply context). The message thread may contain messages to be filtered based on the context of the potential reply messages, such as timing characteristics, user device context, and / or user attention, such as about Figure 4 404 of FIG.

[0080] At box 604, process 600 can filter messages of message threads with backtracking restrictions. The backtracking restriction can remove messages that are not on the screen of the replying user within a threshold time when the replying user begins typing a potential reply message. In other words, process 600 can remove messages with timestamps that have not been seen on the screen of the replying user recently and may be too long away from when the replying user begins responding. The backtracking restriction can be a predefined amount of time that the replying user must have seen or "reviewed" the message on their screen recently. The time when the message of the message thread appears on the screen of the replying user can be determined according to the context based on context data, including but not limited to timing characteristics, user device context and / or user attention. In some embodiments, the time when the message appears on the screen of the replying user is the time when the user receives the message. In other embodiments, the time when the message appears on the screen of the replying user is the time when the replying user last opened the message and the message was presented to the user (e.g., when the user opens the message thread, the message is on the screen of the replying user, and the replying user scrolls up to the message). In various embodiments, the time when the message appears on the screen of the replying user is the time when the replying user last directly viewed the message based on the user attention context data (e.g., tracking where the replying user looks). By filtering based on a lookback limit, process 600 can remove older messages in the message thread history that are less likely to be the originating message because the replying user has not seen the message for a while. For example, assume process 600 obtains the following message thread: User 1 sent a first message at 5:00 PM on Tuesday asking "How are you today?" and User 1 sent a second message at 9:00 AM on Wednesday asking "Do you want pizza?" If the lookback limit is 12 hours and the first message from User 1 was last seen by User 2 more than 12 hours ago, then the first message from User 1 will be filtered out in this example.

[0081] At box 606, process 600 can filter messages of a message thread with a cutoff limit. The cutoff limit can remove messages received after the replying user begins typing a potential reply message. In other words, process 600 can "cut off" any messages with a timestamp received after the time the potential reply message began to be typed. By filtering based on the cutoff limit, process 600 can remove messages that are unlikely to be the originating message because the message appears after the user begins typing the potential reply message (a user is unlikely to begin replying to a message that does not yet exist). For example, assume that the automatic message reply system obtains the following message thread: User 1 sends a first message asking "How are you today?" at 5:00:00 PM on Tuesday, and User 1 sends a second message asking "Do you want pizza?" at 5:00:11 PM on Tuesday. If User 2 begins typing the potential reply message at 5:00:10 PM on the same day, then the second message from User 1 will be filtered out in this example because the second message was sent 1 second after User 1 began typing. In some embodiments, at Figure 5 Messages removed by the cutoff limit may still be included in the message group at block 510. In various implementations, the cutoff limit may allow a message to not be removed if the message is received within a threshold time limit (e.g., nanoseconds, milliseconds, seconds) after the user begins typing the potential reply message. This may account for a scenario where a replying user begins typing a potential reply message, sees a new message sent by another user after they begin typing, and then modifies their potential reply message as they type to answer the new message instead.

[0082] At block 608, process 600 may provide the remaining messages (unfiltered messages) of the filtered message thread for further processing. Process 600 may provide the remaining messages at block 508 to Figure 5 The process 500 is performed so that the process 500 can calculate the reply scores for the remaining messages.

[0083] Figure 7700 is a conceptual diagram illustrating an example of a message thread with a timestamp, which shows the message to be removed by filtering. Example 700 includes user A 702 and user B 704 exchanging messages on a messaging platform. User A has sent messages 706, 708, 712, 714, and 718 to user B, and user B has sent message 710 to user A and is currently entering a potential reply message 716. Example 700 also includes a backtracking limit 720 and a cutoff limit 722 to filter messages 706, 708, 710, 712, 714, and 718 based on the "seen" timestamps 724, 726, 728, 730, 732, and 734 of the message, respectively. The "seen" timestamp can be the time when user B received the message, the time when user B last saw the message on their screen (e.g., when user B opened example 700 and it appeared on the screen, user B scrolled up to the message), or the time when user B directly viewed the message. In example 700, the backtrack limit 720 is set to 24 hours from the current time of user B when he / she is currently typing potential reply message 716, so any messages with a "seen" timestamp earlier than 5:50:20 PM on Monday are removed (from 5:50:20 PM on Tuesday—24 hours when user B begins replying 736). The backtrack limit 720 may remove any messages above the dashed line 738, which includes removing message 706 (crossed out by the dashed line) because that message has a "seen" timestamp earlier than the backtrack limit 720 (3:00:00 PM on Monday is earlier than the backtrack limit of 5:50:20 PM on Monday).

[0084] In example 700, cutoff limit 722 is set to the time when user B starts replying 736, so any message from user A with a "seen" timestamp after cutoff limit 722 of 5:50:20 pm on Tuesday is removed. Cutoff limit 722 can remove any message below dashed line 740 from user A, which includes removing message 718 from user B (crossed out by dashed line) because the message has a "seen" timestamp later than cutoff limit 722 (5:50:25 pm on Tuesday is later than the cutoff limit of 5:50:20 pm on Tuesday). Message 718 is received by user B after user B starts typing potential reply message 716, so this is why the message can be removed by the cutoff limit. Messages 708, 710, 712, and 714 are the remaining messages that are not filtered by the cutoff limit and the backtracking limit.

[0085] Figure 8800 is a conceptual diagram illustrating an example of a user interface with a message shown as an originating message. Example 800 includes user A 802 and user B 804 exchanging messages on a messaging platform. User A has sent messages 806, 808, 810, and a potential reply message 818 to user B, while user B has sent potential reply messages 812, 816, and 824 to user A. Not shown in example 800 are the many messages (reserving space in this example) that are exchanged back and forth between user A and user B in 826 between the time message 806 and message 808 are sent / received by the users. Messages 806, 826, 808, and 810 may be candidates for originating messages for potential reply message 812; messages 806, 826, 808, 810, and 812 may be candidates for originating messages for potential reply message 816; messages 806, 826, 808, 810, 812, and 816 may be candidates for originating messages for potential reply message 818; and messages 806, 826, 808, 810, 812, 816, and 818 may be candidates for originating messages for potential reply message 824. Message 808 is selected as originating message 814 for potential reply message 812 and is displayed directly above potential reply message 812. Message 810 is selected as originating message 818 for potential reply message 816 and is displayed directly above potential reply message 816. Message 816 is selected as the originating message 820 for potential reply message 818 and is displayed directly above potential reply message 818. Message 806 is selected as the originating message 822 for potential reply message 824 and is displayed directly above potential reply message 824.

[0086] Fig. 9900 is a conceptual diagram illustrating an example of a user interface with arrows showing messages as replies to other messages. Example 900 includes user A 902 and user B 904 exchanging messages on a messaging platform. User A has sent messages 906, 908, 910 and potential reply message 916 to user B, and user B has sent potential reply messages 912 and 914 to user A. Messages 906, 908, and 910 may be candidates for originating messages for potential reply message 912; messages 906, 908, 910, and 912 may be candidates for originating messages for potential reply message 914; and messages 906, 908, 910, 912, and 914 may be candidates for originating messages for potential reply message 916. Messages 908 and 910 are selected as the originating message group for potential reply message 912, and an arrow 918 pointing between the messages may be displayed. No originating message for potential reply message 914 is selected. Message 914 is selected as the originating message for potential reply message 916, and an arrow 920 pointing between the messages may be displayed.

[0087] Several embodiments of the disclosed technology are described above with reference to the accompanying drawings. The computing device on which the described technology can be implemented may include one or more central processing units, memories, input devices (such as keyboards and pointing devices), output devices (such as display devices), storage devices (such as disk drives) and network devices (such as network interfaces). Memory and storage devices are computer-readable storage media that can store instructions for at least part of the implementation of the described technology. In addition, data structures and message structures can be stored or transmitted via data transmission media (such as signals on communication links). Various communication links can be used, such as the Internet, local area networks, wide area networks, or point-to-point dial-up connections. Therefore, computer-readable media can include computer-readable storage media (such as "non-transient" media) and computer-readable transmission media.

[0088] References in this specification to "embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "embodiment," etc.) mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present disclosure. These phrases appearing in different places in this specification do not necessarily all refer to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. In addition, various features are described that may be exhibited by some embodiments but not by other embodiments. Similarly, various requirements are described that may be requirements for some embodiments but not for other embodiments.

[0089] As used herein, above a threshold value means that the value for the item under comparison is higher than another specified value, and the item under comparison is among the items of a specific specified number with a maximum value or the item under comparison has a value within a specified maximum percentage value. As used herein, below a threshold value means that the value for the item under comparison is lower than another specified value, and the item under comparison is among the items of a specific specified number with a minimum value or the item under comparison has a value within a specified minimum percentage value. As used herein, within a threshold value means that the value of the item under comparison is between two other specified values, and the item under comparison is among the items of a middle specified number or the item under comparison has a value within a middle specified percentage range. When not otherwise defined, relative terms such as high or unimportant can be understood as assigning a value and determining how the value is compared with the established threshold value. For example, the phrase "selecting a fast connection" can be understood as meaning selecting a connection with a value higher than a threshold value assigned corresponding to its connection speed.

[0090] As used herein, the word "or" refers to any possible arrangement of a set of items. For example, the phrase "A, B or C" refers to at least one of A, B, C or any combination thereof, such as: A; B; C; A and B; A and C; B and C; any one of A, B and C; or any multiple of any one, such as A and A; B, B and C; A, A, B, C and C, etc.

[0091] Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the above-mentioned specific features or actions. For the purpose of illustration, specific embodiments and implementations have been described herein, but various modifications can be made without departing from the scope of the embodiments and implementations. The above-mentioned specific features and actions are disclosed as example forms for implementing the appended claims. Therefore, the embodiments and implementations are not limited by the appended claims.

[0092] Any patents, patent applications and other references mentioned above are incorporated herein by reference. If necessary, various aspects may be modified to adopt the systems, functions and concepts of the various references mentioned above to provide other embodiments. If the statements or themes in the documents incorporated by reference conflict with the statements or themes of the present application, the present application shall prevail.

Claims

1. A method for pairing messages sent in a message thread by selecting which originating messages in the message thread match reply messages in the message thread as automatic replies to a result, the method include: Get a message thread containing two or more messages; Obtaining a potential reply message and a context of the potential reply message; wherein the potential reply message is in the message thread and is a potential reply to one or more messages of two or more messages in the message thread; identifying one or more remaining messages by filtering out at least one of the two or more messages in the message thread based on the context of the potential reply message; A reply score for each of the one or more remaining messages and / or for a group of the one or more remaining messages is generated by: generating a model input based on: the potential reply message from the message thread, one or more of the remaining messages from the two or more messages in the message thread, and at least a portion of the context of the potential reply message; as well as applying the model input to a machine learning model trained to generate a response score; identifying a remaining message or group of remaining messages having a highest reply score; determining that the remaining message or remaining message group having the highest reply score qualifies as an original message to be replied to; as well as In response to determining that the originating message or group of originating messages having the highest reply score qualifies: pairing the potential reply message with the original message or message group having the highest reply score, such that in the message thread, the potential reply message is identified as a reply to the original message or message group having the highest reply score; and The potential reply message is caused to be displayed as a reply to the original message or group of original messages having the highest reply score.

2. The method of claim 1, wherein the filtering is performed by removing messages from the two or more messages that are not on the replying user's screen within a threshold time.

3. The method of claim 1, wherein the filtering is performed by removing messages of the two or more messages that are received after the replying user begins typing the potential reply. 4 . The method of claim 1 , wherein when the highest reply score is above a threshold confidence value, the remaining messages having the highest reply score qualify as the original message to be replied to.

5. The method of claim 1, wherein the remaining message with the highest reply score qualifies as the original message to be replied to when the highest reply score is a threshold amount higher than all other reply scores of the one or more remaining messages.

6. The method of claim 1, wherein the machine learning model is trained to generate a response score by: Get a collection of reply instances; Obtaining a corresponding context for each instance in the set of reply instances; Obtaining the machine learning model for training; generating a set of model inputs based on: the set of reply instances and the corresponding contexts for the set of reply instances; applying the set of model inputs and / or groups of the set of model inputs to the machine learning model and updating one or more model parameters; as well as The machine learning model trained to generate a response score is provided.

7. The method according to claim 1, wherein the context of the potential reply message includes one or more text features of the potential reply message and / or two or more messages; and The one or more text features are based on the following items: message length, included words, part-of-speech tags, type analysis, or any combination thereof.

8. The method according to claim 1, in, The context of the potential reply message comprises results of an analysis of associated content items of: the potential reply message and / or the two or more messages; The analysis of the associated content items is performed by: Determining that the potential reply message and / or two or more messages contain content items, wherein the content items include: links, pictures, videos, or any combination thereof; as well as The following items are identified that are related to the content item: an object identification tag, a hash tag, a subject identifier, a video length, a sound length, who posted on social media, who liked a social media post, or any combination thereof.

9. A method according to claim 1, wherein the context of the potential reply message includes one or more timing characteristics based on: the time when the potential reply message is sent, the time when one or more of the two or more messages are read, the length of time that one or more of the two or more messages remains focused on the replying user's screen, or any combination thereof.

10. A method according to claim 1, wherein the context of the potential reply message includes a device context based on the following items: where the replying user is when the potential reply message is sent and / or whether the potential reply message is sent after being disconnected from the network for a period of time.

11. The method of claim 1 , wherein the context of the potential reply message includes an indication of the replying user's attention as determined by: using a camera to calculate where the responding user is looking; and Based on where the replying user is looking: replying to one or more of the two or more messages that the user is viewing during a given time period, and / or How long it takes the replying user to view one or more of the two or more messages.

12. The method according to claim 1, wherein causing the potential reply message to be displayed as a reply to the remaining message or the remaining message group is performed by: causing the original message or original message group to be displayed directly above the potential reply message; or An arrow pointing between the potential reply message and the original message or group of original messages is displayed.

13. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for pairing messages sent in a message thread by selecting which originating messages in the message thread match reply messages in the message thread as automatic replies to a result, the process include: Get a message thread containing two or more messages; Obtaining a potential reply message and a context of the potential reply message; wherein the potential reply message is in the message thread and is a potential reply to one or more messages of two or more messages in the message thread; identifying one or more remaining messages by filtering out at least one of the two or more messages in the message thread based on the context of the potential reply message; A reply score for each of the one or more remaining messages and / or for a group of the one or more remaining messages is generated by: generating a model input based on: the potential reply message from the message thread, one or more of the remaining messages from the two or more messages in the message thread, and at least a portion of the context of the potential reply message; as well as applying the model input to a machine learning model trained to generate a response score; identifying a remaining message or group of remaining messages having a highest reply score; determining that the remaining message or remaining message group having the highest reply score qualifies as an original message to be replied to; as well as In response to determining that the originating message or group of originating messages having the highest reply score is eligible for: pairing the potential reply message with the original message or message group having the highest reply score, such that in the message thread, the potential reply message is identified as a reply to the original message or message group having the highest reply score; and The potential reply message is caused to be displayed as a reply to the original message or group of original messages.

14. The computer-readable storage medium of claim 13, wherein each of the groups of the one or more remaining messages comprises messages that have been received within a specified time window of each other.

15. The computer-readable storage medium according to claim 13, further comprising: include: receiving a user-specific message data set including one or more manual reply instances and / or one or more user corrections for modeling a selection of an original message to be replied to; The machine learning model is trained to generate a response score by: generating a set of model inputs based on: the one or more manual reply instances and / or the one or more user corrections for modeling a selection of an original message to be replied to; applying the set of model inputs to the machine learning model and updating one or more model parameters; as well as The machine learning model trained to generate a response score is provided.

16. The computer-readable storage medium of claim 13, wherein the filtering is performed by removing messages of the two or more messages that are not on the replying user's screen within a threshold time.

17. The computer-readable storage medium of claim 13, wherein the filtering is performed by removing messages of the two or more messages that were received after the responding user began typing the potential reply.

18. A computing system, include: one or more processors; as well as One or more memories for storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising: Get a message thread containing two or more messages; Obtaining a potential reply message and a context of the potential reply message; wherein the potential reply message is in the message thread and is a potential reply to one or more messages of two or more messages in the message thread; identifying one or more remaining messages by filtering out at least one of the two or more messages in the message thread based on the context of the potential reply message; A reply score for each of the one or more remaining messages and / or for a group of the one or more remaining messages is generated by: generating a model input based on: the potential reply message from the message thread, one or more of the remaining messages from the two or more messages in the message thread, and at least a portion of the context of the potential reply message; as well as applying the model input to a machine learning model trained to generate a response score; identifying the remaining message or message group having the highest reply score; as well as determining that the remaining message or remaining message group having the highest reply score qualifies as an original message to be replied to; as well as In response to determining that the originating message or group of originating messages having the highest reply score qualifies: pairing the potential reply message with the original message or message group having the highest reply score, such that in the message thread, the potential reply message is identified as a reply to the original message or message group having the highest reply score; and The potential reply message is caused to be displayed as a reply to the original message or group of original messages having the highest reply score.

19. The computing system of claim 18, wherein determining that the remaining message or group of remaining messages having the highest reply score is eligible as the original message to be replied to is also include: The original message having the highest reply score is determined to qualify as the message to be replied to by determining that the highest reply score is above a threshold confidence value.

Images