Communication Response Analyzer

Abstract

A method for classifying responses to communications is presented. One method includes: presenting, in a first user interface (UI), a message requesting a response from a sender to a recipient. The first UI includes a first classification, a second classification, and response options including an input field for entering a free-form text message. Another operation is to receive, as the recipient response, a selection of: (a) a first button or a second button, and (b) a free-form text message entered in the text input field. The method further includes: generating a characterized recipient response vector based on the free-form text message and the selection. A machine learning (ML) model computes a classification value for the recipient response from a set of possible classification values based on the characterized recipient response. The method also includes presenting, in a second UI for the sender, an indicator for the message and the computed classification value.

Description

Technical Field

[0001] The subject matter disclosed herein generally relates to methods, systems, and machine-readable storage media for using artificial intelligence to interpret people's responses. Background Art

[0002] In some systems for person-to-person electronic communication, distributed communication includes a request, and the system prompts the recipient to respond to the sender. Sometimes, the system provides the recipient with a limited set of options for the response (e.g., yes, no, I don't know).

[0003] In some cases, the sender sends a request to many users, and an online service analyzes and classifies the responses so that the sender can quickly evaluate the responses from the users, e.g., to select users for further interaction. If the online service can improve the classification of the responses, the overall system efficiency and operation are improved. The ability to quickly evaluate responses can greatly increase the productivity of the sender.

[0004] However, sometimes the system options provided to the recipient do not include the best response option (e.g., I'm not interested, but my friend Joe will be). Other times, the recipient may select a response option and later wish to select a different option, but there is no way to change the response. Additionally, some users may wish to expand their response options, but this is difficult when only predefined fixed options are available via the system.

[0005] In some cases, the recipient may be given the option to enter free text as a response, but this may complicate the system's ability to interpret the message for the sender, especially when the sender is only interested in categorizing a limited set of options (e.g., yes or no).

[0006] There is a need for a system that allows flexible options for users to respond to messages and that can also effectively categorize and analyze responses from users to allow the message sender to prioritize follow-up activities. Brief Description of the Drawings

[0007] The various drawings only illustrate exemplary embodiments of the present disclosure and should not be considered as limiting its scope.

[0008] Figure 1 is a user interface for responding to a request, where the recipient must categorize the response.

[0009] Figure 2 is a user interface for responding to a communication, where the recipient must classify the response and may optionally add additional free text.

[0010] Figure 3 is a general user interface according to some exemplary embodiments, wherein the recipient has an option to select from predefined answers or enter a custom response.

[0011] Figure 4 is a user interface for a sender according to some exemplary embodiments, wherein responses from the recipient are to be automatically categorized.

[0012] Figure 5 Illustrates the training and use of a machine learning program according to some exemplary embodiments.

[0013] Figure 6 Illustrates the process of obtaining training data for a receiving model according to some exemplary embodiments.

[0014] Figure 7 Illustrates using a receiving model to categorize recipient responses according to some exemplary embodiments.

[0015] Figure 8 is a block diagram of a networking system according to some exemplary embodiments, which illustrates an exemplary embodiment of an advanced client-server based network architecture.

[0016] Figure 9 is a flowchart of a method for classifying a recipient response as one of a finite set of possible response categories from a set of possible response categories according to some exemplary embodiments.

[0017] Figure 10 is a block diagram of an example of a machine on or by which one or more exemplary process embodiments described herein can be implemented or controlled. Detailed Description

[0018] Exemplary methods, systems, and computer programs are directed to interpreting a recipient's response to a communication in order to classify the response and present the response to the sender. The examples merely represent possible variations. Unless otherwise expressly stated, components and functions are optional and may be combined or subdivided, and the order of operations may be different or may be combined or subdivided. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the exemplary embodiments. However, it will be apparent to those skilled in the art that the subject matter may be practiced without these specific details.

[0019] A system enables a sender to interact with a number of recipients of a communication, and each communication includes a proposal for the recipient and a request for further interaction if the recipient is interested. The system gives each recipient the option to choose one from a number of options (e.g., select a button corresponding to the selected option), enter text for the response, or a combination thereof. The system analyzes the selections from the recipients and classifies them into one of a number of categories (e.g., interested or not interested). The system then presents the responses in a user interface and clearly marks the selected category so that the sender can quickly evaluate the intent of each recipient without having to read each response in the responses.

[0020] In one aspect, the sender is a recruiter who sends communications for a job opening to candidates, where the candidates are users or members of an online service. Each communication provides information about the job opening and requests a response from the recipient if interested in the presented job opening. Although embodiments are presented with reference to messages from a recruiter, the same system can be used for other types of communications that request a response, such as marketing communications, offers to buy or sell, offers to volunteer for an event, surveys, requests for donations, market research, etc.

[0021] Methods are presented for interpreting a recipient's response to a communication in order to classify the response. One method includes: presenting, in a first user interface (UI), a message from a sender to a recipient that requests a response from the recipient. The first UI includes response options for a first classification, a second classification, and a text input field for entering a free text message. Another operation is for receiving a recipient response that includes a selection of: (a) a first button or a second button, and (b) a free text message from the recipient entered in the text input field. The method further includes: generating a characterized recipient response vector based on the free text message (e.g., a vector for the entered text and button selection, if any). A machine learning (ML) model calculates a classification value for the received free text message based on the characterized recipient response vector according to a number of classification values. The method further includes presenting, in a second UI for the sender, an indicator for the message and the calculated classification value.

[0022] For the purposes of this specification, the phrases "online social network application", "online social network system", and "online social network service" may be referred to as "online social network" or simply "social network" and used interchangeably. It will also be noted that an online social network can be any type of online social network, such as, for example, a professional network, an interest-based network, or any online network system that allows users to join as registered members. For the purposes of this specification, registered members of an online social network may be referred to simply as members. Additionally, some online services provide services to their members (e.g., searching for jobs, searching for job candidates, job postings) rather than social networking, and the principles presented herein may also be applied to these online services.

[0023] Note that the embodiments are presented in the context of a recruitment scenario where a recruiter sends a message to a potential candidate. However, the same method can be used for any type of communication that requests a response, such as an invitation to connect to an online service, a sales offer, an invitation to participate in an event, an invitation to join a workgroup, a volunteer invitation, and so on. Accordingly, the illustrated embodiments should not be construed as exclusive or limiting, but rather as illustrative.

[0024] Figure 1 is a user interface 100 for responding to a request. In the illustrated example, a recruiter has sent a message 102 to a potential candidate, and the user interface 100 is presented to the potential candidate, also referred to herein as the recipient, with predefined options 104 - 106 for responding.

[0025] In some examples, the recipient is required to start by selecting one of the predefined options in buttons 104 - 106, which opens a dialog box (e.g., a text box with text sent when selected by the recipient). The recipient has the option to enter a text message in the input field 108. The recipient does not have to enter a text message, which is optional if the recipient wants to further qualify one of the predefined options.

[0026] In this example, there are three options: interested, possibly interested later, and not interested. In some exemplary cases, the response should be presented to the sender and classified as interested or not interested, which allows the sender to quickly sort through a large number of responses.

[0027] However, during the experiment, it has been found that receivers tend to provide an unusually high number of "interested" responses (also referred to herein as accepted), e.g., 90% of the responses are "interested". The reason may be that some receivers are not truly interested, or not very interested, but the receivers want to sound polite rather than give an obvious rejection. In this example, the selected classification is the classification reported to the sender, even if the receiver enters text in input field 108 that contradicts the button selection.

[0028] For the sender, these responses provide little value because the responses are not very discriminative. The sender has to read each response to determine whether the receiver is truly interested, or the receiver is just being polite or trying to convey some other information, such as "I'm not interested, but my friend Carla would be interested."

[0029] Figure 2 is a user interface 200 for making responses to communications, where the receiver has to classify the response. The user interface 200 is a communication from the sender to potential candidates for a job position. The framework 204 provides information about the company looking for candidates for a specific job. The message 202 is sent by the sender to the receiver.

[0030] In some examples, the receiver is given two or more options to make a response, and the receiver has to make an explicit choice from one of the options to respond because there is no availability to enter plain text as a response.

[0031] In the illustrated example in the user interface 200, two options are provided: Not Now and Yes. In other examples, additional options may be provided (e.g., may be interested in the future, interested in another job position from the same company).

[0032] It has been observed that some receivers may not respond because they are not satisfied with the options given, even though they may be somewhat interested in the job position or in jobs from the same company.

[0033] During the experiment, it was observed that the number of accepted responses from receivers was lower than Figure 1 the number of accepted responses in the user interface of. However, the quality of the accepted responses received was higher because the accepted responses signal true interest from the receiver. This allows the sender to follow up with further conversations with the receivers who accepted the request.

[0034] However, the problem with the user interface 200 is that the number of options is limited, and many recipients feel that the response they want to give is not represented in one of the options (e.g., I'm not interested but I know someone who is, I'm not interested in the data scientist job but I'm interested in the software developer job).

[0035] In Figure 2 the example of, the system does not intelligently categorize the response. If the recipient provides a categorization (e.g., button selection), then it is the value presented to the sender, even if the categorization may not match the recipient's intent based on the typed text message.

[0036] Figure 3 is for a general user interface 300 of a system for solving the problems explained above regarding Figure 1 and Figure 2 wherein the recipient has the option to: select from predefined answers; enter a custom response without selecting a predefined answer; or, according to some exemplary embodiments, enter both a predefined answer and a text custom response. If the recipient is interested in the presented offer, the message 302 invites the recipient to respond.

[0037] In some exemplary embodiments, the recipient has the option to select one of the predefined options 304, 306 or enter a free text response in the response text 308. That is, the recipient can select one of the buttons for the predefined options 304, 306, but the recipient is not forced to make a selection and can choose to only enter free text for the response.

[0038] By giving the recipient the flexibility to choose one of the options or enter text, the recipient can better choose how to respond to the invitation. This results in better communication from the recipient and increased user satisfaction.

[0039] However, it is important that the system can intelligently and correctly categorize incoming responses. The message sender prefers to quickly know whether the recipient is interested in the offer, but reading answers in text form takes more time than simply viewing an indicator summary (such as accepted or rejected). In some exemplary embodiments, the system analyzes the selected categorization (if any) and the free text response to categorize the response according to a predefined number of possibilities (e.g., accepted or not accepted, interested or not interested).

[0040] In some exemplary embodiments, the free text is classified from one of the options 304, 306 provided in the user interface. Note that the system provides the recipient with the option of entering the free text response without having to pre-select one of the predefined options 304, 306. This provides the recipient with maximum flexibility when entering the response.

[0041] In other exemplary embodiments, the response (selection of one of the predefined options or free text, or a combination thereof) is classified according to a predefined number of options, which may be the same as or different from the options provided in the button. For example, if three options are presented to the recipient, one of the options may be associated with acceptance and the other two options may be associated with rejection.

[0042] Sometimes, the recipient may select one of the predefined options, but the text may indicate a different intention (e.g., the recipient selects "interested", but the recipient is interested at some future time rather than now). Then, the system will select the classification that matches the recipient's intention without having to select the selected predefined option, i.e., the system will not use the classification indicated by the recipient by selecting one of the buttons, but will use the classification described in the text based on system analysis.

[0043] Other embodiments may utilize different layouts, different numbers of predefined answers, different predefined answers, etc. Therefore, the embodiments illustrated in Figure 3 should not be construed as exclusive or restrictive, but illustrative.

[0044] Figure 4 is a user interface 400 for a recruiter according to some exemplary embodiments, wherein the response from the recipient is classified according to the recipient's intention. On the left side of the user interface 400, the responses 402, 412 from the recipient are presented in a list.

[0045] In some exemplary embodiments, each recipient response is automatically classified by the system from one of a plurality of selections. In the example illustrated in Figure 4 the options are "Accept" and "Reject", but other embodiments may include additional categories. The accepted response 402 indicates that the recipient who has received a possible job offer has responded positively to the invitation and is interested in participating in the recruitment process. The rejected response 412 indicates that the recipient is not interested in the job described in the invitation.

[0046] When the sender selects one of the responses from the left - hand list, the content of the response is presented on the right - hand side of the user interface, which includes the original invitation 404, a message 406 indicating that the recipient has accepted the invitation (e.g., InMail), and an input field 408 for sending a new message to the user.

[0047] In some exemplary embodiments, a filter for the recipient's response is provided to the sender, such as a filter that presents only the accepted invitations. In this way, the sender does not have to waste time on recipients who reject the invitation. For example, the sender may have 100 open invitations and receive 30 acceptances and 70 rejections. The sender can quickly focus on the acceptances rather than wasting time on the rejections.

[0048] In some exemplary embodiments, the sender has two folders: one folder for acceptances and one folder for rejections. The sender can open the acceptance folder to quickly view the responses.

[0049] Presenting the recipient's intent regarding the invitation in such a clear manner allows the sender to save time by quickly focusing on high - quality recruits who are interested in continuing the hiring process.

[0050] Figure 5 Illustrated is the training and use of a machine - learning program according to some exemplary embodiments. In some exemplary embodiments, a machine - learning program (MLP) (also referred to as a machine - learning algorithm or tool) is used to perform operations associated with searching, such as job searching.

[0051] Machine learning is a field of study that gives computers the ability to learn without being explicitly programmed. Machine learning explores the study and construction of algorithms, also referred to as tools in this document, which can learn from existing data and make predictions about new data. Such machine - learning tools operate by building a model based on exemplary training data 512 in order to make data - driven predictions or decisions expressed as an output or evaluation 520. Although exemplary embodiments are presented with respect to some machine - learning tools, the principles presented herein can be applied to other machine - learning tools.

[0052] In some exemplary embodiments, different machine - learning tools can be used. For example, logistic regression (LR), naive Bayes, random forest (RF), neural network (NN), deep neural network (DNN), matrix factorization, and support vector machine (SVM) tools can be used to classify job postings or score them.

[0053] Two common problems in machine learning are classification problems and regression problems. Classification problems, also known as categorical problems, aim to classify items into one of several categorical values (e.g., is this object an apple or an orange?). Regression algorithms aim to quantify some item (e.g., by providing a value that is a real number). The machine learning algorithm utilizes training data 512 to find correlations between the identified features 502 that affect the outcome.

[0054] The machine learning algorithm utilizes features 502 to analyze the data to generate an assessment 520. Features 502 are individual measurable characteristics of the observed phenomena. The concept of a feature is related to the concept of an explanatory variable used in statistical techniques such as linear regression. Selecting informative, discriminative, and independent features is important for the effective operation of an MLP in pattern recognition, classification, and regression. Features can be of different types, such as numbers, strings, and graphics.

[0055] In one exemplary embodiment, features 502 can be of different types and can include one or more of the following: user features 504 (user features 504 can include information about the sender and / or recipient), job posting features 505; company features 506; message 102, message response 509, message text embedding 510, and other features 510 (user posts, online activities, followed companies, etc.).

[0056] User features 504 include user profile information, such as title, skills, experience, education, geography, the user's activities in the online service, etc. Job posting features 505 include information about the job posting, such as the company offering the job, the title of the job posting, the location of the job posting, the required skills, the description of the job, etc. Additionally, company features 506 include information about the company posting the job, such as the company's name, industry, revenue information, location, etc.

[0057] Message 102 includes information about the message sent from the sender to a potential candidate. Message response 509 includes information about the recipient's response to message 102, such as a selected predefined response, the time of the response, etc. Message text embedding 510 includes information about the text response entered by the recipient of the message. In some exemplary embodiments, the recipient's response should be embedded into a vector for processing by the ML algorithm.

[0058] The ML algorithm utilizes training data 512 to find correlations between the identified features 502 that affect the outcome or evaluation 520. In some exemplary embodiments, the training data 512 includes known data for one or more identified features 502 obtained from past activities of senders and receivers in the online system, such as responses to messages sent by the receiver and the classification (e.g., accepted or rejected) of the response by the sender or by the classification system.

[0059] In addition, the training data may include information about messages for which no response has been received from the receiver, such as an option classified as "possible".

[0060] Using the training data 512 and the identified features 502, the ML algorithm is trained at operation 514. The ML training evaluates the values of the features 502 as they relate to the training data 512. The result of the training is an accepted ML model 516.

[0061] When the accepted ML model 516 is used to perform an evaluation, new data 518 is provided as input to the accepted ML model 516, and the accepted ML model 516 generates an evaluation 520 as output. For example, the accepted ML model 516 can be used to classify a response that has been entered as text from the receiver as accepted or rejected. In some exemplary embodiments, additional classification categories may be included.

[0062] In some exemplary embodiments, the receiver of the message selects a predefined option (e.g., "Yes, interested" or "No, thank you") as well as a free text response. This response input is processed by the accepted ML model 516 to generate a classification of the response as one of a plurality of predefined values (e.g., "Yes, interested" or "No, thank you"). Thus, it is possible that as a result of the automated analysis of the free text response, the predefined option selected by the receiver may be overridden and changed to a different category. For example, the free response text indicates that the receiver is not really interested in the proposal, but the receiver may not want to sound negative by selecting the "No" option. However, the sender of the communication will find more value in the true classification compared to having to sort through "polite" affirmative button selections to discover that the receiver is not really interested.

[0063] In other cases, the recipient may not be interested but may choose "yes" because the recipient wants to provide a referral of a friend who may be interested. However, if the sender is not truly interested in the referral, such information may prove to be a waste of time. In some exemplary embodiments, the automatic categorization can include additional values that are not provided as explicit recipient options, such as "No, but I know someone else", "Yes, but not at this time", "No, but maybe later", etc. Thus, these categorizations will allow the sender to quickly traverse the responses without reading the response text.

[0064] Figure 6 Illustrated is a process for obtaining training data for an acceptance model according to some exemplary embodiments. Message 102 is sent to recipient 602. Each recipient 602 gives a binary option to accept 604 or reject 605 an invitation to apply for a job. In other exemplary embodiments, additional or different options may be provided to the recipient.

[0065] After the recipient has selected a response to message 102, the recipient 602 is given an input field 108 to enter additional text for the response. If the recipient enters text, the text is analyzed 606 by a natural language processor, which creates a characterized representation vector 608 of the text for use with the model, where the characterized response vector 608 is an expression of the text for features identified in the model, such as a vector created from the text and which can be used as training data or input into the acceptance model. In other exemplary embodiments, a matrix is used to create a characterized representation of the optional additional text 108 (e.g., representing each word in the text as a row of a matrix). In some exemplary embodiments, the characterized response vector 608 is a vector representing the text of the response, such as by assigning a vector to each word in the text and then combining the word vectors, such as by creating a matrix, adding vectors, concatenating vectors, etc. In some exemplary embodiments, the characterized response vector 608 is an embedding of the response text, where the embedding is a vector with a semantic representation of the text such that embedding vectors for text with similar meanings will be close to each other, while embedding vectors for text with different meanings will not be close to each other.

[0066] Natural language processing (NLP) is a subfield of linguistics, computer science, information engineering, and artificial intelligence that deals with the interaction between computers and human language, and in particular how to program computers to process and analyze large amounts of natural language data. Challenges in natural language processing often involve speech recognition, natural language understanding, and natural language generation.

[0067] Then, training data 610 is created using feature representation vectors 608 regarding message 102, recipient 602 (e.g., user profile data, recipient activities related to a job application in response to the message), the response to the message (accept 604 or reject 605), and text when available.

[0068] Since the recipient must choose to accept 604 or reject 605, all responses are classified because the recipient 602 has provided that classification. Additionally, optional additional text 108 provides information that matches the text response to the classification provided by recipient 602. This provides a clear one-to-one mapping between the options chosen by the recipient and the responses they type. This provides an initial set of training data 610 for the system. Over time, as the model is refined and as the system collects additional data (e.g., information from the recipient regarding the correct classification of the response), then this newer information will be used to improve the accuracy of training data 610.

[0069] In some exemplary embodiments, training data 610 includes the recipient classification for the response and the text input by the recipient providing the response, and the ML algorithm is a shallow neural network. In other exemplary embodiments, additional features may be included, such as those Figure 5 described.

[0070] In some exemplary embodiments, training data 610 can be refined by requesting confirmation from recipient 602 or the sender. For example, once the recipient inputs a text response, the recipient is prompted: "We believe you are interested in pursuing this opportunity, yes or no?" Then, the response can be used to improve the training data.

[0071] Similarly, when the sender is reviewing the response, information regarding the estimated classification can be used to prompt the sender, such as, "We think this candidate is interested in pursuing this opportunity, yes or no?" Then, the response can be used to improve the training data. For example, if the classification is incorrect, the training data is modified to reflect this incorrect classification.

[0072] At operation 514, training data 610 is used to create acceptance model 612, as described above with reference to Figure 5 In some exemplary embodiments, acceptance model 612 is a classifier to determine whether the response is an acceptance or a rejection. In other exemplary embodiments, acceptance model 612 provides a score or probability of acceptance (e.g., the probability that the response is an acceptance is 75% and the probability of rejection is 25%).

[0073] Figure 7Illustrated is the categorization of recipient responses using an acceptance model 612 according to some exemplary embodiments. In the illustrated example, the user interface described with reference to Figure 3 is presented to recipient 602 who receives message 102, i.e., the recipient can respond using response text 308, accept 304, reject 306, or a combination of response text 308 and one of accept 304 or reject 306.

[0074] When the recipient inputs response text, natural language processing 606 creates a feature representation vector 608 of the text based on the content of response text 308.

[0075] Information from one or more of message 102, recipient 602, response text 308, accept 304 or reject 306, and the feature representation vector 608 of the text is used as input data 702 for acceptance model 612.

[0076] Acceptance model 612 then provides an output indicating whether the response from a recipient 602 to a given message 102 is an acceptance 704 or a rejection 706. Then, categories are presented on the user interface, such as in the Figure 4 user interface 400 presented. Note that the output of acceptance model 612 is not necessarily equal to the recipient's selection of one of the predefined options, as acceptance model 612 can "re - categorize" the response based on response text 308.

[0077] In some exemplary embodiments, the response provided by recipient 602 can be used to enhance other functions in the online service besides sender communication. For example, the system measures the frequency with which the recipient responds to messages. If the recipient often responds with an acceptance, this is a signal that the recipient is interested in changing jobs. Then, this information is provided to a utility that ranks candidates, such that candidates more inclined to change jobs will see their ranking in the candidate list for jobs increase, based on their willingness to change jobs.

[0078] Furthermore, when the sender searches for candidates, the candidate search function utilizes the indication of the recipient's willingness to change jobs as one of the features for a machine learning model, where recipients willing to change jobs will see their scores increase.

[0079] In other exemplary embodiments, the online service analyzes the acceptances of a particular recipient to examine job opportunity parameters such as title, skills, industry, company, etc. The job search function (e.g., Jobs You Might Be Interested In (JYMBI)) can use this to find jobs for the recipient that match the identified desired job characteristics.

[0080] Figure 8FIG. is a block diagram of a networking system including a social network server 812 according to some exemplary embodiments, which illustrates an exemplary embodiment of an advanced client-server based network architecture 802. Referring to the online service rendering embodiments, and in some exemplary embodiments, the online service is a social network service.

[0081] The social network server 812 provides server-side functionality to one or more client devices 804 via a network 814 (e.g., the Internet or a wide area network (WAN)). Figure 8 Illustrated are, for example, a web browser 806, one or more client applications 808, and a social network client 810 executing on the client device 804. The social network server 812 is also communicatively coupled with one or more database servers 826 that provide access to one or more databases 816 - 224.

[0082] The social network server 812 includes modules such as a recruiter user interface (UI) 828, a message processor 830, and an acceptance model 612. The recruiter UI 828 provides a user interface for recruiters within the online service (e.g., Figure 4 user interface 400). The message processor 830 manages sending messages to candidates and their responses to the recruiter.

[0083] The client device 804 can include, but is not limited to: mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smart phones, tablet computers, netbooks, multi-processor systems, microprocessor-based or programmable consumer electronic systems, or any other communication device that a user can use to access the social network server 812. In some embodiments, the client device 804 can include a display module (not shown) to display information (e.g., in the form of a user interface).

[0084] In one embodiment, the social network server 812 is a network-based device that responds to initialization requests or search queries from the client device 804. One or more receivers 602 can be people, machines, or other means of interacting with the client device 804. In various embodiments, the receiver 602 interacts with the network architecture 802 via the client device 804 or other means.

[0085] The client device 804 may include one or more applications (also referred to as "apps"), such as but not limited to: a web browser 806, a social network client 810, and other client applications 808, such as a messaging application, an email application, a news application, etc. In some embodiments, if the social network client 810 is present in the client device 804, the social network client 810 is configured to locally provide a user interface for the application and communicate data and / or processing capabilities (e.g., access user profiles, authenticate recipient 602, identify or locate other connected recipients 602, etc.) that are not locally available as needed. Conversely, if the social network client 810 is not included in the client device 804, the client device 804 may use the web browser 806 to access the social network server 812.

[0086] In addition to the client device 804, the social network server 812 communicates with one or more database servers 826 and databases 816 - 224. In an exemplary embodiment, the social network server 812 is communicatively coupled to a member activity database 816, a social graph database 818, a member profile database 820, a job posting database 822, and a message database 824. The databases 816 - 224 may be implemented as one or more types of databases, including but not limited to: hierarchical databases, relational databases, object-oriented databases, one or more flat files, or combinations thereof.

[0087] The member profile database 820 stores user profile information about users who have registered on the social network server 812. With respect to the member profile database 820, a member can be an individual person or an organization, such as a company, enterprise, non-profit organization, educational institution, or other such organization.

[0088] In some exemplary embodiments, when the recipient 602 initially registers as a member of the social network service provided by the social network server 812, the recipient 602 is prompted to provide some personal information, such as name, age (e.g., date of birth), gender, interests, contact information, hometown, address, names of spouse and / or household users, educational background (e.g., school, major, enrollment and / or graduation dates, etc.), work experience (e.g., company worked for, length of employment for the corresponding job, job title), professional industry (also abbreviated as "industry" herein), skills, professional organizations, etc. This information is stored, for example, in the member profile database 820. Similarly, when a representative of an organization initially registers the organization with the social network service provided by the social network server 812, the representative may be prompted to provide specific information about the organization, such as the company industry.

[0089] When a member interacts with the social networking service provided by the social networking server 812, the social networking server 812 is configured to monitor these interactions. Examples of interactions include, but are not limited to: commenting on a post entered by another member, viewing a user profile, editing or viewing the member's own profile, sharing content outside of the social networking service (e.g., an article provided by an entity outside of the social networking server 812), updating a current status, posting content for other members to view and comment on, posting a work recommendation for a member, searching for job postings, and other such interactions. In one embodiment, records of these interactions are stored in the user activity database 816, which associates the interactions made by the member with his or her user profile stored in the user profile database 820.

[0090] The job posting database 822 includes job postings provided by companies. Each job posting includes information related to the job, such as any combination of employer, job title, job description, requirements for the job posting, salary and benefits, geographical location, one or more job skills required, the date the job posting was posted, relocation benefits, etc. In addition, the message database 824 stores messages and responses to the messages.

[0091] Although the (one or more) database servers 826 are illustrated as a single box, those of ordinary skill in the art will recognize that the (one or more) database servers 826 can include one or more such servers. Thus, and in one embodiment, the (one or more) database servers 826 implemented by the social networking service are also configured to communicate with the social networking server 812.

[0092] Figure 9 is a flowchart of a method 900 for classifying a recipient response into one of a finite set of possible response categories according to some exemplary embodiments. Although the various operations in this flowchart are presented and described in sequence, those of ordinary skill in the art will realize that some or all of the operations can be performed, combined, omitted, or executed in parallel in a different order.

[0093] Operation 902 is for causing a message from a sender to a recipient to be presented in a first user interface (UI). The message includes a request for a response from the recipient, and the first UI includes the following response options: a first button for selecting a first classification, a second button for selecting a second classification, and a text input field for entering free text.

[0094] From operation 902, method 900 flows to operation 904 for receiving a recipient response, the recipient response including a selection of: (a) a first button or a second button, and (b) a free text message entered in a text input field.

[0095] At operation 906, a characterized recipient response vector is generated based on the free text message and the selection.

[0096] From operation 906, the method flows to operation 908, where a machine learning (ML) model accepts a classification value of the recipient response calculated according to multiple classification values based on the characterized recipient response vector.

[0097] From operation 908, method 900 flows to operation 910 for presenting an indicator for the message and the calculated classification value in a second UI for the sender, where the presentation in the second UI is based on the calculated classification.

[0098] In one example, the accepting ML model is generated by an ML program based on training data, the training data including one or more of the following: (a) information about a recipient response to a request, the recipient response including a text response and a button selection, and (b) a classification value of the response to the request.

[0099] In one example, method 900 further includes generating training data based on a recipient response to a request, generating the training data including: resending in a third UI a message of the recipient response to the request; enabling the recipient to select one of a first classification or a second classification; and adding information about the message of the request response and the recipient's selection to the training data.

[0100] In one example, the accepting ML model is based on features of the following, including: a request to the recipient; a response to the request; a text embedding of the response to the request; and a classification value of the response to the request.

[0101] In one example, the features further include one or more of the following: user profile information, information associated with the request, and information about a company associated with the request.

[0102] In one example, generating the characterized recipient response vector includes: using a natural language processor to generate the characterized recipient response vector based on the semantic meaning of the free text message.

[0103] In one example, the first classification is accepting the request, and the second classification is rejecting the request.

[0104] In one example, the rendering in the second UI further includes: providing in the second UI a first folder for accepted requests and a second folder for rejected requests.

[0105] In one example, the message is a request for an employment opportunity from a recruiter.

[0106] In one example, the message is one of the following: an offer for a product for sale, an offer for a service for sale, or a survey.

[0107] Another general aspect is a system that includes a memory with instructions and one or more computer processors. The instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations including: causing a message from a sender to a receiver to be presented in a first user interface (UI), the message requesting a response from the receiver, the first UI including response options that include: a first button for selecting a first classification; a second button for selecting a second classification; and a text input field for entering a free-form text message; receiving a receiver response that includes a selection of: (a) the first button or the second button, and (b) the free-form text message entered in the text input field; generating a characterized receiver response vector based on the free-form text message and the selection; having a receiving machine learning (ML) model calculate a classification value of the receiver response based on the characterized receiver response vector according to multiple classification values; and presenting an indicator for the message and the calculated classification value in a second UI for the sender, wherein the presentation in the second UI is based on the calculated classification value.

[0108] In yet another general aspect, a machine-readable storage medium (e.g., a non-transitory storage medium) includes instructions that, when executed by a machine, cause the machine to perform operations including: causing a message from a sender to a receiver to be presented in a first user interface (UI), the message requesting a response from the receiver, the first UI including response options that include: a first button for selecting a first classification; a second button for selecting a second classification; and a text input field for entering a free-form text message; receiving a receiver response that includes a selection of: (a) the first button or the second button, and (b) the free-form text message entered in the text input field; generating a characterized receiver response vector based on the free-form text message and the selection; having a receiving machine learning (ML) model calculate a classification value of the receiver response based on the characterized receiver response vector according to multiple classification values; and presenting an indicator for the message and the calculated classification value in a second UI for the sender, wherein the presentation in the second UI is based on the calculated classification value.

[0109] Figure 10 FIG. is a block diagram illustrating an example of a machine 1000, on or by which one or more of the exemplary process embodiments described herein can be implemented or controlled. In alternative embodiments, machine 1000 can operate as a stand-alone device or can be connected (e.g., networked) to other machines. In a networked deployment, machine 1000 can operate in a server-client network environment as a server machine, a client machine, or both. In an example, machine 1000 can act as a peer machine in a peer-to-peer (or other distributed) network environment. Further, although only a single machine 1000 is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein, such as via cloud computing, software as a service (SaaS), or other computer cluster configurations.

[0110] As described herein, an example can include or be operated by a logic unit, a plurality of components or mechanisms. A circuit is a collection of circuits implemented in a tangible entity that includes hardware (e.g., simple circuits, gates, logic units). Circuit members can be flexible over time and underlying hardware variability. A circuit includes members that can perform specified operations individually or in combination when operating. In an example, the hardware of a circuit can be immutably designed to perform a particular operation (e.g., hardwired). In an example, the hardware of the circuit can include physically components that are variably connected (e.g., execution units, transistors, simple circuits), including a computer-readable medium that is physically modified (e.g., magnetically, electrically, by moveable placement of invariant mass particles) to encode instructions for a specific operation. When connecting the physical components, the underlying electrical properties of the hardware are changed (e.g., from an insulator to a conductor, or vice versa). The instructions enable the embedded hardware (e.g., execution unit or loading mechanism) to create members of the circuit in the hardware via the variable connections to perform portions of a specific operation when operating. Thus, when the device is operating, the computer-readable medium is communicatively coupled to other components of the circuit. In an example, any physical component can be used in more than one member of more than one circuit. For example, in operation, an execution unit can be used in a first circuit of a first circuit system at one point in time and reused by a second circuit in the first circuit system or by a third circuit in a second circuit system at a different time.

[0111] A machine (e.g., a computer system) 1000 can include a hardware processor 1002 (e.g., a central processing unit (CPU), a hardware processor core, or any combination thereof), a graphics processing unit (GPU) 1003, a main memory 1004, and a static memory 1006, some or all of which can communicate with each other via an interconnect (e.g., a bus) 1008. The machine 1000 can also include a display device 1010, an alphanumeric input device 1012 (e.g., a keyboard), and a user interface (UI) navigation device 1014 (e.g., a mouse). In an example, the display device 1010, the alphanumeric input device 1012, and the UI navigation device 1014 can be a touch screen display. The machine 1000 can additionally include a mass storage device (e.g., a drive unit) 1016, a signal generation device 1018 (e.g., a speaker), a network interface device 1020, and one or more sensors 1021, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. The machine 1000 can include an output controller 1028, such as a serial (e.g., universal serial bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC)) connection to communicate with or control one or more peripheral devices (e.g., a printer, a card reader).

[0112] The mass storage device 1016 can include a machine-readable medium 1022 on which is stored a set or sets of data structures or instructions 1024 (e.g., software) embodying any one or more of the techniques or functions described herein or used thereby. During execution of the instructions 1024 by the machine 1000, the instructions 1024 can also reside completely or at least partially within the main memory 1004, within the static memory 1006, within the hardware processor 1002, or within the GPU 1003. In an example, one or any combination of the hardware processor 1002, the GPU 1003, the main memory 1004, the static memory 1006, or the mass storage device 1016 can constitute a machine-readable medium.

[0113] Although the machine-readable medium 1022 is illustrated as a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store one or more instructions 1024.

[0114] The term "machine-readable medium" can include any medium that can store, encode, or carry instructions 1024 for execution by a machine 1000 and cause the machine 1000 to perform any one or more of the techniques of the present disclosure, or any medium that can store, encode, or carry a data structure used by or associated with such instructions 1024. Non-limiting examples of machine-readable media can include solid state memories and optical and magnetic media. In an example, a mass machine-readable medium includes machine-readable medium 1022, which includes a plurality of particles having invariant (e.g., stationary) mass. Thus, a mass machine-readable medium is not a transitory propagated signal. Specific examples of a mass machine-readable medium can include non-volatile memories such as semiconductor storage devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0115] Instructions 1024 can also be sent or received over a communication network 1026 via a network interface device 1020 using a transmission medium.

[0116] Throughout the specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed simultaneously, and the operations need not be performed in the order illustrated. Structures and functions presented as separate components in an exemplary configuration may be implemented as a combined structure or component. Similarly, structures and functions presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.

[0117] Embodiments illustrated herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and other embodiments may be derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Accordingly, the detailed description should not be construed as limiting, and the scope of the various embodiments is defined only by the appended claims and the full scope of equivalents given by such claims.

[0118] As used herein, the term "or" may be construed as inclusive or exclusive. Additionally, multiple instances may be provided for resources, operations, or structures that are described herein as a single instance. Further, the boundaries between various resources, operations, modules, engines, and data stores are somewhat arbitrary and a particular operation is illustrated in the context of a particular exemplary configuration. Other allocations of functionality are envisioned and may fall within the scope of various embodiments of the present disclosure. In general, structures and functionality that are presented as separate resources in an exemplary configuration may be implemented as a combined structure or resource. Similarly, structures and functionality that are presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within the scope of the embodiments of the present disclosure as represented by the appended claims. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A computer-implemented method for classifying responses to communications, comprising: causing a message from a sender to a recipient to be presented in a first user interface, the message requesting a response from the recipient, the first user interface including response options, the response options including: a first button for selecting a first classification; a second button for selecting a second classification; and a text input field for entering a free text message; receiving a recipient response including a selection of (a) the first button or the second button, and (b) the free text message entered in the text input field; generating a characterized recipient response vector based on the free text message and the selection; calculating a classification value of the recipient response by a receiving machine learning model based on the characterized recipient response vector according to a plurality of classification values, wherein the plurality of classification values indicate possible response categories of the recipient response, and the calculated classification value of the recipient response corresponds to one of the possible response categories; and presenting an indicator for the message and the calculated classification value in a second user interface for the sender, wherein the presentation in the second user interface is based on the calculated classification value.

2. The method according to claim 1, wherein, the receiving machine learning model is generated by a machine learning program based on training data, the training data including one or more of the following: (a) information about a recipient response to a request, the recipient response including a text response and a button selection, and (b) a classification value of the response to the request.

3. The method according to claim 2, further comprising: generating training data based on the recipient response to the request, the generating the training data including: presenting a message requesting a response from the recipient in a third user interface; enabling the recipient to select one of the first classification or the second classification; and adding information about the message requesting the response and the recipient's selection to the training data.

4. The method according to claim 2, wherein, the receiving machine learning model is based on the following features, the features including: the request to the recipient; the response to the request; a text embedding of the response to the request; and the classification value of the response to the request.

5. The method according to claim 4, wherein, the features further include one or more of the following: user profile information; information associated with the request; and information about the company associated with the request.

6. The method according to claim 1, wherein, generating the characterized recipient response vector includes: using a natural language processor to generate the characterized recipient response vector based on the semantic meaning of the free text message.

7. The method according to claim 1, wherein, the first classification is accepting the request, and the second classification is rejecting the request.

8. The method according to claim 7, wherein, The presentation in the second user interface further includes: Providing a first folder for accepted requests and a second folder for rejected requests in the second user interface.

9. The method according to claim 1, wherein, the message is a request for an employment opportunity from a recruiter.

10. The method according to claim 1, wherein, the message is one of the following: an offer for a product for sale, an offer for a service for sale, or a survey.

11. A system for classifying responses to communications, comprising: A memory including instructions; and One or more computer processors, wherein the instructions, when run by the one or more computer processors, cause the system to perform operations including the following: Causing a message from a sender to a recipient to be presented in a first user interface, the message requesting a response from the recipient, the first user interface including response options, the response options including: A first button for selecting a first classification; A second button for selecting a second classification; and A text input field for entering a free text message; Receiving a recipient response including a selection of (a) the first button or the second button and the free text message entered in the text input field; Generating a characterized recipient response vector based on the free text message and the selection; Calculating a classification value of the recipient response by an accepting machine learning model based on the characterized recipient response vector according to multiple classification values, wherein the multiple classification values indicate possible response categories of the recipient response, and the calculated classification value of the recipient response corresponds to one of the possible response categories; and Presenting an indicator for the message and the calculated classification value in a second user interface for the sender, wherein the presentation in the second user interface is based on the calculated classification value.

12. The system according to claim 11, wherein, the accepting machine learning model is generated by a machine learning program based on training data, the training data including one or more of the following: (a) information about a recipient response to a request, the recipient response including a text response and a button selection, and (b) a classification value of the response to the request.

13. The system according to claim 12, wherein, the instructions further cause the one or more computer processors to perform operations including the following: Generating training data based on the recipient response to the request, the generating of the training data including: Presenting a message requesting a response from the recipient in a third user interface; Enabling the recipient to select one of the first classification or the second classification; and Adding information about the message requesting the response and the recipient's selection to the training data.

14. The system according to claim 12, wherein, the accepting machine learning model is based on features including the following: The request to the recipient; The response to the request; A text embedding of the response to the request; and the classification value of the response to the request 15. The system according to claim 14, wherein, the feature further includes one or more of the following: user profile information; information associated with the request; and information about the company associated with the request.

16. The system according to claim 11, wherein, generating the characterized receiver response vector includes: using a natural language processor to generate the characterized receiver response vector based on the semantic meaning of the free text message.

17. The system according to claim 11, wherein, the first classification is to accept the request, and the second classification is to reject the request.

18. The system according to claim 11, wherein, the presentation in the second user interface further includes: providing a first folder for accepted requests and a second folder for rejected requests in the second user interface.

19. A non-transitory machine-readable storage medium including instructions that, when run by a machine, cause the machine to perform operations including the following: causing a message from a sender to a receiver to be presented in a first user interface, the message requesting a response from the receiver, the first user interface including response options, the response options including: a first button for selecting a first classification; a second button for selecting a second classification; and a text input field for entering a free text message; receiving a receiver response including a selection of (a) the first button or the second button, and (b) the free text message entered in the text input field; generating a characterized receiver response vector based on the free text message and the selection; calculating, by an acceptance machine learning model, a classification value of the receiver response based on the characterized receiver response vector according to a plurality of classification values, wherein the plurality of classification values indicate possible response categories of the receiver response, and the calculated classification value of the receiver response corresponds to one of the possible response categories; and presenting an indicator for the message and the calculated classification value in a second user interface for the sender, wherein the presentation in the second user interface is based on the calculated classification value.

20. The non-transitory machine-readable storage medium according to claim 19, wherein, the machine learning model is generated by a machine learning program based on training data including one or more of the following: (a) information about a receiver response to a request, the receiver response including a text response and a button selection, and (b) the classification value of the response to the request.

Images