Systems, methods, and computer-readable media for controlling a coalition of automated agents

Abstract

A method of controlling a federation of automated agents, comprising: receiving a ticket from a terminal, routing the ticket to some of a plurality of serving automated agents, measuring similarity of suggestions generated by some of the plurality of serving automated agents, the suggestions being suggestions for responding to the ticket, determining a subset of suggestions based on ratings of the some of the serving automated agents and the similarity of suggestions generated by the some of the serving automated agents, the ratings being based on previous scores indicating quality of previous subsets of suggestions, and providing the subset of suggestions to the terminal.

Description

Technical Field

[0001] Example embodiments generally relate to systems, methods, and / or computer-readable media for controlling a federation of automated agents.

[0002] Description of related fields

[0003] When a user wishes to get help for a technical problem, such as requesting technical support in installing and / or maintaining components of an infrastructure for a cellular phone, the user may wish to get help from a group of human experts. However, the user may request help at a time when a human expert is not immediately available.

[0004] Therefore, automated software agents (eg, automated robots (bots) and / or automated expert systems) have been developed to provide assistance when human experts are not available. Summary of the invention

[0005] According to some example embodiments, there is provided a method of controlling a federation of automated agents, the federation of automated agents comprising a plurality of serving automated agents, at least some of the plurality of serving automated agents comprising computer readable instructions that, when executed by a computer, cause the computer to generate suggestions for responding to tickets received from one or more terminals. The method comprises: receiving a ticket from a terminal, routing the ticket to some of the plurality of serving automated agents, measuring similarity of suggestions generated by some of the plurality of serving automated agents, the suggestions being suggestions for responding to the ticket, determining a subset of suggestions based on ratings of some of the serving automated agents and similarity of suggestions generated by some of the serving automated agents, the ratings being based on previous scores indicating quality of previous subsets of suggestions, and providing the subset of suggestions to the terminal.

[0006] The method may also include receiving a score from the terminal indicating the quality of the subset of suggestions, and updating a rating of some of the serving automated agents based on the score.

[0007] The ticket may include a plurality of fields. The method may also include measuring a complexity of the ticket based on at least one of the fields, and determining a number of serving automated agents based on the complexity.

[0008] Measuring the complexity of the document may include measuring the complexity of the document based on a number of words in a description field of the document.

[0009] The method may also include soliciting a score indicative of a quality of the subset of suggestions provided to the terminal.

[0010] The method may also include adding the ticket and the suggestion generated by at least one of the serving automated agents to the test set based on a similarity of the suggestions.

[0011] The method may also include removing at least one serving automation agent from the automation agent federation based on the rating of the at least one serving automation agent.

[0012] The method may also include generating, by some of the serving automated agents in the federation of automated agents, at least one suggestion for responding to the ticket.

[0013] Generating, by some of the agents in the federation of automated agents, at least one suggestion for responding to the ticket may include reviewing the description domain, comparing the description domain to a training set, and generating at least one suggestion based on the comparison of the description domain to the training set.

[0014] Determining a subset of suggestions based on ratings of some of the on-service automated agents and similarities of suggestions generated by some of the on-service automated agents may include merging suggestions from some of the on-service automated agents, reviewing ratings associated with some of the on-service automated agents, and adding suggestions to the subset of suggestions based on the review of the ratings.

[0015] The method may also include, in response to the score indicating the quality of the subset of suggestions being greater than a quality rating threshold, adding at least one of the ticket and the suggestion to the test set.

[0016] The method may also include, in response to the similarity level of the suggestion being greater than a similarity threshold, adding at least one of the ticket and the suggestion to the test set.

[0017] According to some example embodiments, a non-transitory computer-readable medium may include program instructions for causing an apparatus to perform any of the above-described methods.

[0018] According to some example embodiments, a method of adding a temporary automation agent to a federation of automation agents is provided, the automation agent federation comprising a plurality of serving automation agents, at least some of the plurality of serving automation agents comprising a list of computer-readable instructions, the computer-readable instructions, when executed by a computer, causing the computer to generate a recommendation for responding to a ticket received from one or more terminals. The method may include: receiving a ticket from a terminal, forwarding the ticket to a training manager, receiving a temporary recommendation generated by the temporary automation agent from the training manager, evaluating the temporary automation agent by comparing the temporary recommendation with recommendations generated by some of the serving automation agents in the automation agent federation, and, in response to the evaluation, adding the temporary automation agent to the plurality of serving automation agents.

[0019] According to some example embodiments, a non-transitory computer-readable medium includes program instructions for causing an apparatus to perform the above-described method.

[0020] According to some example embodiments, an apparatus for controlling an automated agent alliance may be provided, the automated agent alliance including a plurality of automated agents in service. The apparatus may include: at least one processor, and at least one memory including computer program code. The at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to receive a ticket from a terminal, route the ticket to some of the plurality of automated agents in service, measure similarity of suggestions generated by some of the plurality of automated agents in service, the suggestions being suggestions for responding to the ticket, determine a subset of suggestions based on ratings of some of the automated agents in service and similarity of suggestions generated by some of the automated agents in service, the ratings being based on previous scores indicating quality of previous subsets of suggestions, and provide the subset of suggestions to the terminal.

[0021] The at least one memory and the computer program may be configured to, with the at least one memory, cause the apparatus to receive from the terminal a score indicating a quality of the subset of suggestions, and to update a rating of some of the serving automated agents based on the score.

[0022] The at least one memory and the computer program may be configured to, with the at least one memory, cause the apparatus to solicit a score indicative of a quality of the subset of suggestions provided to the terminal.

[0023] The at least one memory and the computer program may be configured to, with the at least one memory, cause the apparatus to generate, by some of the serving automated agents in a federation of automated agents, at least one suggestion for responding to the ticket.

[0024] The at least one memory and the computer program may be configured to, with the at least one memory, cause the apparatus, in response to the score indicating the quality of the subset of suggestions being greater than a quality rating threshold, to add at least one of the ticket and the suggestion to the test set.

[0025] According to some example embodiments, there is provided an apparatus for controlling a federation of automated agents, the federation of automated agents comprising a plurality of automated agents in service, at least some of the plurality of automated agents in service comprising computer-readable instructions which, when executed by a computer, cause the computer to generate suggestions for responding to tickets received from one or more terminals, the apparatus comprising: means for receiving tickets from the terminals, means for routing the tickets to some of the plurality of automated agents in service, means for measuring similarities of suggestions generated by some of the plurality of automated agents in service, the suggestions being suggestions for responding to the tickets, means for determining a subset of suggestions based on ratings of some of the automated agents in service and similarities of suggestions generated by some of the automated agents in service, the ratings being based on previous scores indicating the quality of previous subsets of suggestions, and means for providing the subset of suggestions to the terminals.

[0026] According to some example embodiments, there is provided an apparatus for adding a temporary automation agent to an automation agent alliance, the automation agent alliance comprising a plurality of automation agents in service, at least some of the plurality of automation agents in service comprising a list of computer-readable instructions, the computer-readable instructions, when executed by a computer, causing the computer to generate a recommendation for responding to a ticket received from one or more terminals, the apparatus comprising: a component for receiving a ticket from a terminal, a component for forwarding the ticket to a training manager, a component for receiving a temporary recommendation generated by the temporary automation agent from the training manager, a component for evaluating the temporary automation agent by comparing the temporary recommendation with recommendations generated by some of the automation agents in service in the automation agent alliance, and a component for adding the temporary automation agent to the plurality of automation agents in service in response to the evaluation.

[0027] These and other aspects of the example embodiments will become apparent from the accompanying drawings and detailed description thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Some example embodiments may be more fully understood from the detailed description given herein below and the accompanying drawings, in which like elements are represented by like reference numerals, which are given by way of illustration and thus do not limit the present disclosure.

[0029] Figure 1 illustrates an environment in which one or more example embodiments may be implemented;

[0030] Figure 2 A method of controlling a federation of automated agents in service according to some example embodiments is shown;

[0031] Figure 3 shows an example of a user interface according to some example embodiments;

[0032] Figure 4 A method of routing tickets based on the complexity of the tickets according to some example embodiments is shown;

[0033] Figure 5 A method of generating a suggestion in response to a ticket according to some example embodiments is shown;

[0034] Figure 6 A method of generating a subset of suggestions for responding to a ticket by filtering a set of suggestions according to some example embodiments is shown;

[0035] Figure 7 A method of providing a subset of suggestions for responding to a ticket is shown according to some example embodiments;

[0036] Figure 8 A method of adjusting a rating of an automated agent in service is shown according to some example embodiments;

[0037] Fig. 9 A method of introducing a new software agent into a serving federation of automated agents is shown according to some example embodiments; and

[0038] Fig.10 An apparatus for performing methods associated with a serving automation agent federation is shown according to some example embodiments. DETAILED DESCRIPTION

[0039] Various example embodiments will now be described more fully with reference to the accompanying drawings, in which some example embodiments are shown.

[0040] Detailed illustrative embodiments are disclosed herein. However, for the purpose of describing the exemplary embodiments, the specific structural and functional details disclosed herein are only representative. However, the exemplary embodiments can be implemented in many alternative forms and should not be interpreted as being limited to the embodiments set forth herein.

[0041] Therefore, although the example embodiments are capable of various modifications and alternative forms, these embodiments are shown in the drawings by way of example and will be described in detail herein. However, it should be understood that the example embodiments are not intended to be limited to the specific forms disclosed. On the contrary, the example embodiments will cover all modifications, equivalents and alternatives that fall within the scope of the present disclosure. Throughout the description of the drawings, the same numbers refer to the same elements.

[0042] Figure 1 An environment is shown in which example embodiments may be implemented.

[0043] Reference Figure 1, environment 1 may include a software agent federation 100 and a device 1500 having a user interface 150. User interface 150 may be accessed by user 175.

[0044] The software agent federation 100 may include hardware and / or software running on a single computer, or alternatively, running on a collection of computers (not shown). The collection of computers may be distributed, such as distributed over a computer network.

[0045] The collection of computers may include a plurality of automated agents 101, 102, 103. At least some of the plurality of automated agents 101, 102, 103 may be implemented via computer-readable instructions that, when executed by a computer, cause the computer to provide recommendations for responding to a ticket. For example, at least some of the plurality of automated agents 101, 102, 103 may be implemented as software for executing a neural network and / or a machine learning algorithm and / or an expert system. At least some of the automated agents 101, 102, 103 may have corresponding ratings indicating the quality of the recommendations provided by the agent.

[0046] In the training phase, at least some of the plurality of automated agents 101, 102, 103 may be trained to generate responses to outstanding tickets. The training of the automated agents may be performed differently for different automated agents. For example, the training of the automated agent 101 may include training a neural network and / or a machine learning algorithm using a first training set, the first training set comprising a first example ticket and a corresponding first suggestion for responding to the first example ticket. The training of the automated agent 102 may include training a neural network and / or a machine learning algorithm using a second training set, the second training set comprising a second example ticket and a corresponding second suggestion for responding to the second example ticket.

[0047] The software agent alliance 100 may include a service manager 110. The service manager 110 may be implemented via computer-readable instructions that, when executed by a computer, cause the computer to perform one or more methods according to example embodiments. For example, the service manager 110 may receive a ticket 125 via a device 1500 displaying a user interface 150, determine the complexity of the ticket 125, and select a subset of the plurality of automated agents 101, 102, 103 for providing recommendations via the user interface 150. The service manager 110 may also request feedback from a user 175 via the user interface 150, including a rating indicating the quality of the recommendations.

[0048] The software agent alliance 100 may also include a training manager 1001. The training manager 1001 may manage a plurality of temporary agents (not shown) being trained. Fig. 9Training manager 1001 is discussed further.

[0049] The components of the software agent alliance 100 may communicate with each other. For example, each or at least some of the plurality of software agents 101, 102, 103 may communicate with the service manager 110. For example, the components of the software agent alliance 100 may communicate with each other through a network.

[0050] User interface 150 may be or include a graphical user interface (GUI), such as a graphical user interface presented on a website and / or within an app such as a mobile application. If user 175 wishes to receive advice on resolving a technical issue, user 175 may engage with user interface 150 to submit a ticket 125. Figure 3 The user interface 150 is described in more detail.

[0051] User 175 may be a user of the system and / or technology. As an example, user 175 may be a technician responsible for maintaining equipment associated with a mobile network. As an example, user 175 may be a user of software such as an operating system and / or word processing system and / or applications used in an office environment. As another example, user 175 may be a user of a computer game or a computer game system. As yet another example, user 175 may be a user of a system associated with healthcare delivery. The example embodiments are not limited to the above-described embodiments, and other example embodiments may be readily recognized by a person of ordinary skill in the art. The system may be operated based on suggestions provided by a software agent.

[0052] As an example, user 175 may request support for a new customer incident and request suggestions for responding to the new customer incident.

[0053] Figure 2 A method of controlling a federation of software agents according to some example embodiments is shown.

[0054] Reference Figure 2 In step 201, a user 175 may request support suggestions for resolving a technical problem by submitting a ticket 125 to the automation agent federation 100 via the device 1500. The user 175 may use the user interface 150 to submit the ticket 125 to the service manager 110 in the automation agent federation 100.

[0055] In step 202, the service manager 110 may determine the complexity of the document 125. For example, the service manager 110 may count the number of characters and / or the number of words within the description field 354 and may determine the complexity of the document based on the number of words.

[0056] Based on these fields, the automated agent federation 100 can determine whether the ticket 125 is simple or complex. Example embodiments are not limited to review of the description field 354 and may include review of other fields within the ticket 125.

[0057] In step 203, the service manager 110 may route the ticket 125 to a plurality of automated agents. The number of automated agents may be determined based on the complexity of the ticket 125. Figure 4 An example embodiment of a method for routing tickets 125 is described in more detail.

[0058] In step 204, the automated agent that receives the ticket 125 may generate one or more suggestions for responding to the ticket 125. Figure 5 Example methods for generating one or more recommendations are described in further detail.

[0059] In step 205, the service manager 110 may collate the suggestions from the automated agents, and may determine the amount of similarity by measuring the amount of overlap in the collated suggestions. For example, if the service manager 110 routes the ticket 125 to each of the automated agents 101, 102, 103, and each of the automated agents 101, 102, 103 provides the same suggestion for responding to the ticket 125, then the amount of overlap may be relatively high. However, if each of the automated agents 101, 102, 103 provides a different suggestion for responding to the ticket 125, then the amount of overlap may be relatively low.

[0060] In step 214, the service manager 110 may compare the amount of similarity to an upper similarity threshold. The upper similarity threshold may be determined, for example, by an operator (eg, a human or a network), and may be based on empirical data.

[0061] For example, the amount of overlap may be equal to the ratio between the number of identical suggestions generated by each of the in-service automation agents 101, 102, 103 and the total number of in-service automation agents 101, 102, 103 that generated suggestions. For example, if each of the in-service automation agents 101, 102, 103 generates the same suggestions for responding to the ticket 125, the overlap may be 100%. Alternatively, if each of the in-service automation agents 101, 102, 103 generates different suggestions for responding to the ticket 125, the overlap may be 0%.

[0062] For example, an upper similarity threshold may correspond to an 80% overlap.

[0063] If the overlap exceeds the upper similarity threshold, then in step 206, both the note 125 and the suggestion generated by the automated agent can be added to the test set. The test set can be used for training the temporary agent. Fig. 9 An example method for training a temporary agent is described in more detail.

[0064] Returning to step 214, if the overlap amount does not exceed the upper similarity threshold, then in step 215, the service manager 110 compares the overlap with the lower similarity threshold. The lower similarity threshold may be less than the upper similarity threshold. The lower similarity threshold may be defined by an operator (e.g., a person or a network) based on data.

[0065] For example, a lower similarity threshold may correspond to a 50% overlap in the same suggestions generated by agents within the automated agent federation 100 .

[0066] If the overlap is less than the lower similarity threshold, then the ticket 125 may be routed to a human expert for further review in step 216 .

[0067] Returning to step 215 , if the overlap is not less than the lower similarity threshold, the method proceeds to step 207 .

[0068] In step 207, the service manager 110 may select a subset of suggestions to present to the user 175 via the user interface 150. For example, the service manager 110 may select the subset of suggestions based on the ratings of the automated agents and / or the amount of overlap between the suggestions. Figure 6 An example method for determining a subset of suggestions is described in more detail.

[0069] In step 208, the service manager 110 may provide the subset of suggestions to the user 175 (eg, via the user interface 150). Figure 7 An example method for providing a subset of suggestions is described in more detail.

[0070] In step 209, the service manager 110 may solicit feedback from the user 175 through the user interface 150. For example, the service manager 110 may send a message to the user 175 through the user interface 150 and / or through some other communication method. The message may be or may include a request for feedback on the quality of the recommendations provided to the user 175 as described above with reference to step 208.

[0071] In step 210, the service manager 110 may receive feedback from the user 175 through the user interface 150. The feedback may indicate a level of satisfaction with each or some of the suggestions provided in step 208. The user 175 may indicate that he or she is very satisfied or extremely satisfied, somewhat satisfied, neutral, somewhat dissatisfied, or not satisfied at all with each or at least some of the suggestions provided in step 208.

[0072] In step 853, the service manager 110 can determine whether the response was helpful based on the feedback. For example, the user 175 can provide a numerical value indicating the degree of satisfaction with the advice provided in step 208. For example, the user 175 can indicate that he or she is satisfied at a certain level on a full-point scale. The numerical value can be compared to a feedback threshold. The feedback threshold can be defined by an operator (e.g., a person or a network) based on empirical data.

[0073] For example, the feedback threshold may correspond to a satisfaction rating of 8 out of 10 or higher.

[0074] If the value associated with the feedback exceeds the feedback threshold, then in step 854, the ticket 125 and the suggestions generated by the automated agent that were rated as helpful by the user 175 may be added to the test set.

[0075] In step 211, the service manager 110 may update the rating of the serving automated agent that provided the suggestion based on the feedback received from the user 175 in step 210. In addition, the membership of the serving automated agents in the automated agent alliance 100 may be adjusted based on the rating. For example, an underperforming automated agent may be removed from the automated agent alliance 100.

[0076] In addition, temporary automated agents that provide temporary suggestions may be included in the automated agent alliance 100. Fig. 9 An example method for coordinating an automated agent federation 100 is described.

[0077] Returning to step 853, if the value associated with the feedback is less than or equal to the feedback threshold, the process proceeds to step 211 and continues as described above.

[0078] Therefore, based on Figure 2By using the methods outlined in , more complex tickets can provide more helpful suggestions. In addition, by measuring the similarity level in the suggestions, new automated software agents can be added to the software agent federation without extensive training (e.g., training with human input). In addition, a decision can be made to exit software agents that provide outliers and / or inconsistent suggestions. This can help simplify the automated agent federation 100. In addition, by measuring the similarity level, a decision can be made to forward the ticket to request the help of a human expert, thereby improving the quality of the suggestions provided to the user.

[0079] Figure 3 Examples of user interfaces are shown in accordance with some example embodiments.

[0080] Reference Figure 3 , the user interface 150 can be presented as a graphical user interface (GUI). The GUI can include a plurality of fields that can be filled in by a user (e.g., user 175). For example, the user interface 150 can include a title field 351, a button field 352, a drop-down list field 353, and / or a description field 354.

[0081] The user 175 may identify a title for the ticket 125 to be submitted and enter it into the title field 351. In addition, the user 175 may check a box in the button field 352 and / or select an item from the drop-down list field 353. Further, the user 175 may provide a more detailed description of the problem in the description field 354. The description field 354 may allow the user to draft a free-form description of the problem for which he or she is requesting support. The description field 354 may include natural language.

[0082] Example embodiments are not limited in this regard. Figure 3 The design of the illustrated user interface 150 is not limited in this regard, and more or fewer fields may be presented to the user for request.

[0083] User interface 150 may be presented as a website on device 1500, and / or may be presented on a mobile app of a mobile device. Example embodiments are not limited to these examples.

[0084] Figure 4 A method of routing tickets to multiple automated agents according to some example embodiments is shown. The method may be performed by the service manager 110.

[0085] Reference Figure 4In step 405, the service manager 110 may determine whether the complexity of the ticket 125 is high, such as above a ticket complexity threshold. For example, if there are relatively many words in the description field 354, such as between 5 and 50 words, the service manager 110 may determine that the ticket 125 is complex. Alternatively, if there are relatively few words (e.g., less than 5 words) in the description field 354, the service manager 110 may determine that the ticket 125 is simple.

[0086] If the complexity of the ticket 125 exceeds the ticket complexity threshold, the service manager 110 may route (e.g., forward) the ticket 125 to a number of in-service automation agents in step 408. For example, if the complexity of the ticket 125 exceeds the ticket complexity threshold, the automation agent federation 100 may route the ticket 125 to three, four, or five in-service automation agents. For example, the service manager 110 may route the ticket to three, four, or five random in-service automation agents.

[0087] Returning to step 405, if the complexity of the ticket 125 does not exceed the ticket complexity threshold, the service manager 110 may route (e.g., forward) the ticket 125 to a smaller number of in-service automation agents in step 407. For example, if the complexity of the ticket 125 does not exceed the ticket complexity threshold, the service manager 110 may route the ticket to one or two in-service automation agents. For example, the service manager 110 may route the ticket to one or two random in-service automation agents.

[0088] Figure 5 A method of generating a suggestion for responding to a ticket is shown according to some example embodiments.

[0089] For example, we will discuss the Automation Agent 101 Figure 5 However, each automated agent may execute the method to generate one or more recommendations for responding to a ticket.

[0090] Reference Figure 5 At step 502 , the automation agent 101 may receive the ticket 125 .

[0091] In step 503, the automated agent 101 can review at least some of the fields included in the ticket 125. For example, the automated agent 101 can parse the words associated with the description field 354 to determine which problem is to be solved. For example, the automated agent 101 can parse the words associated with the description field 354 to determine keywords associated with the ticket 125.

[0092] In step 504, the automated agent 101 may compare the fields included in the ticket 125 with a training set used to train the automated agent 101. The training set may have been generated based on historical data and may be referenced later. Figure 6 The process described in more detail is further expanded upon.

[0093] Based on the comparison, the automated agent 101 can generate one or more suggestions for responding to the ticket 125. For example, if the automated agent 101 is implemented as a neural network trained to review free-form content describing the domain 354, the automated agent 101 can generate suggestions based on the characteristics of the trained neural network.

[0094] For example, the automated agent may generate suggestions based on a kth nearest neighbor algorithm and / or an algorithm that determines the joint complexity between the ticket 125 and the training set. The automated agent 101 may generate suggestions corresponding to suggestions having the closest neighbors and / or the highest joint complexity between the ticket 125 and the training set.

[0095] The suggestion may be a suggestion selected from a suggestion list. For example, the suggestion list may be previously generated. For example, the suggestion list may include a discussion in a reference manual. The suggestion list may be stored in a database and / or data structure.

[0096] In step 505 , the automation agent 101 may provide (eg, send) one or more suggestions to the service manager 110 .

[0097] Figure 6 A method of generating a subset of suggestions for responding to a ticket is shown according to some example embodiments.

[0098] Reference Figure 6 In step 606 , the service manager 110 may review a rating (eg, a quality rating) associated with the automated agent that generated the response.

[0099] For each automated agent, if the quality rating of the automated agent exceeds a quality rating threshold, the service manager 110 may add a suggestion to a subset of responses to be presented to the user through the user interface in step 607. The quality rating threshold may be defined by an operator (e.g., a human or a network) based on empirical data.

[0100] For example, based on user feedback, the quality rating threshold may have a value corresponding to an average satisfaction rating of 8 on a 10-point scale.

[0101] Figure 7 A method of providing a subset of suggestions in response to a ticket is shown according to some example embodiments.

[0102] The service manager 110 may prepare the recommendations to be presented to the user 175 via the user interface 150 .

[0103] Reference Figure 7 In step 702, the service manager 110 can format the suggestions for sending to the user 175 by arranging and ranking the suggestions based on the likelihood of being useful. The suggestions can be ranked based on the amount of overlap. For example, if the first suggestion has been generated by three of the five automated agents (e.g., automated agents 101, 102, 103), the first suggestion can be ranked first. If the second suggestion is generated by only two of the five automated agents (e.g., automated agents 101, 102), the second suggestion can be ranked second.

[0104] In step 703, service manager 110 may provide the subset of responses to user 175. The subset of responses may be provided through user interface 150, and / or through email, and / or through text messaging, and / or through some other method of communication with user 175. Example embodiments are not limited to these examples.

[0105] Figure 8 A method of adjusting a rating of an automated agent in service is shown according to some example embodiments.

[0106] At least some of the automated agents 101, 102, 103 may have corresponding ratings. In one example, the corresponding ratings may be corresponding quality ratings (e.g., numerical values ​​associated with the quality of the suggestions provided by the automated agents 101, 102, 103). The ratings may be adjusted based on feedback from the user 175.

[0107] For example, in step 902 , the service manager 110 may receive feedback from the user 175 .

[0108] In step 903, the service manager 110 may determine whether the user 175 is satisfied with the suggestion.

[0109] If user 175 indicates that he or she is dissatisfied with a suggestion, such as by giving the suggestion a low numerical value, then at step 904, the rating of the automated agent that provided the suggestion that user 175 is dissatisfied with may be lowered.

[0110] In step 906, the service manager 110 may compare the numerical value of the rating of each automated agent 101, 102, 103 to a delisting threshold. The delisting threshold may be defined by an operator (eg, a human or a network) based on empirical data.

[0111] For example, the delisting threshold may correspond to a rating threshold of 2 out of 10 on a 10-point scale.

[0112] In step 907, any automated agent among the automated agents 101, 102, 103 having a rating below the delisting threshold may be removed and / or exited from the automated agent federation 100. Thus, the automated agent federation 100 may be reduced and / or simplified.

[0113] Returning to step 903, if the user 175 is satisfied with the suggestion, then in step 905, the service manager 110 may increase the rating of the automated agent in service that provided the suggestion that satisfied the user 175. Further, as described with reference to step 206, the suggestion may be added to the test set.

[0114] Fig. 9 Methods of introducing new software agents into a software agent federation are shown according to some example embodiments.

[0115] The automated agent federation 100 may be comprised of automated agents in service. The automated agents in service may be trained in generating recommendations for responding to tickets, such as ticket 125. For example, the automated agents in service may be implemented as neural networks and / or machine learning algorithms and / or expert systems, and may be trained in generating recommendations based on a test set.

[0116] However, before a new automated agent is added to the established automated agent federation 100, the new automated agent may be considered a temporary agent.

[0117] Reference Fig. 9 There may be a training manager 1001 that manages a plurality of temporary agents (not shown) being trained.

[0118] Initially, in step 1050, a user 175 may submit a ticket 125 through the user interface 150. The automated agent federation 100 may receive the ticket.

[0119] In step 1051, the automated agent alliance 100 may generate one or more recommendations for responding to the ticket 125. The automated agent alliance 100 may generate one or more recommendations for responding to the ticket 125 based on the reference Figure 2-7 The outlined method generates one or more recommendations.

[0120] In step 1052, the service manager 110 may provide suggestions to the user 175 through the user interface 150. In addition, in step 1053, the automated agent federation 100 may forward the ticket 125 to the training manager 1001.

[0121] In step 1055, the training manager 1001 may provide the ticket 125 to the plurality of temporary automation agents. In step 1054, the training manager 1001 may forward a response from at least one temporary automation agent to the service manager 110 in the automation agent federation 100. The ticket 125 provided to the temporary agent may correspond to the ticket 125 marked as training data in step 206.

[0122] In step 1052, the service manager 110 may review and evaluate the responses generated by the temporary agent(s). For example, the service manager 110 may compare and determine the model precision and recall scores of the response(s) generated by the temporary agent(s).

[0123] For example, the service manager 110 may determine a precision score corresponding to the portion of suggestions provided by the transient agent(s) that are identical to suggestions generated by the plurality of servicing automation agents 101, 102, 103. The service manager 110 may determine a recall score corresponding to the portion of suggestions generated by the plurality of servicing automation agents 101, 102, 103 that are identical to suggestions generated by the transient agent(s).

[0124] The service manager 110 may determine a model precision and recall score that relates the precision score to the recall score to determine a measure of the quality of the recommendations from the temporary agent(s). The model precision and recall score may be twice the ratio of the product of the precision score and the recall score divided by the sum of the precision score and the recall score. Twice the ratio of the product of the precision score and the recall score divided by the sum of the precision score and the recall score may also be referred to as an F1 score. The model precision and recall score may be expressed as a ratio and / or a score and / or a percentage.

[0125] If the model precision and recall scores are greater than or equal to the entry threshold, then in step 1056 the temporary agent is accepted and deployed within the automated agent federation.

[0126] The entry threshold may be defined by an operator (eg, a human or a network) based on empirical data. For example, the entry threshold may correspond to an F1 score of 50%.

[0127] If the model precision and recall scores are less than the entry threshold, the ad hoc agent is not accepted and is not deployed within the automated agent federation.

[0128] Fig.10 Devices for implementing one or more service managers 110, one or more automation agents 101, 102, 103, one or more training managers 1001, etc. are shown according to some example embodiments.

[0129] As shown, device 1100 may include memory 1140 ; ​​a processor 1120 connected to memory 1140 ; ​​various interfaces 1160 connected to processor 1120 ; and one or more connections 1165 connected to the various interfaces 1160 .

[0130] The memory 1140 may be a computer-readable storage medium, which typically includes a random access memory (RAM), a read-only memory (ROM), and / or a permanent mass storage device, such as a disk drive. The memory 1140 also stores an operating system and any other routines / modules / applications for providing the functionality of the device 1100 to be executed by the processor 1120. These software components may also be loaded into the memory 1140 from a separate computer-readable storage medium using a drive mechanism (not shown). Such a separate computer-readable storage medium may include a disk, a tape, a DVD / CD-ROM drive, a memory card, or other similar computer-readable storage medium (not shown). In some example embodiments, the software components may be loaded into the memory 1140 via one of the various interfaces 1160, rather than via a computer-readable storage medium.

[0131] The processor 1120 may be configured to execute instructions of a computer program by performing arithmetic, logical, and input / output operations of the system. The instructions may be provided to the processor 1120 by the memory 1140 .

[0132] For example, the processor 1120 may be configured to perform the same Figure 2 and / or Figure 4-10 Any method described corresponds to the instruction.

[0133] The various interfaces 1160 may include components that interface the processor 1120 with other input / output components. As will be appreciated, the various interfaces 1160 and the programs stored in the memory 1140 to address the specific functions of the device 1100 will vary depending on the implementation of the device 1100.

[0134] The interface 1160 may also include one or more user input devices (eg, keyboard, keypad, mouse, touch screen, etc.) and user output devices (eg, display, speaker, touch screen, etc.).

[0135] One or more example embodiments provide mechanisms for determining when to direct (e.g., introduce) a newly trained automated software agent into a coalition of software agents. One or more example embodiments provide mechanisms for developing example test sets of tickets and corresponding recommendations. In addition, one or more example embodiments may provide mechanisms for determining when more than one software agent is involved in generating recommendations. In addition, it may be desirable for one or more example embodiments to provide mechanisms for determining when to deregister / exit an underperforming software agent from a coalition. Still further, one or more example embodiments provide mechanisms for determining when a ticket should be routed to an expert for review.

[0136] Although the terms first, second, etc. may be used to describe various elements herein, these elements should not be limited by these terms. These terms are only used to distinguish elements from each other. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of this disclosure. As used herein, the term "and / or" includes any and all combinations of one or more associated list items.

[0137] When an element is referred to as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intervening elements. Conversely, when an element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements. Other words used to describe the relationship between elements should be interpreted in a similar manner (e.g., "between" versus "directly between," "adjacent" versus "directly adjacent," etc.).

[0138] The terms used herein are only used to describe specific embodiments and are not intended to be restrictive. As used herein, the singular forms "one", "an" and "the" are intended to include plural forms as well, unless the context clearly indicates otherwise. It should also be understood that when the terms "include", "comprise", "contain" and / or "contain" are used herein, these terms specify the existence of stated features, integers, steps, operations, elements and / or components, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups.

[0139] It should also be noted that in some alternative implementations, functions / acts may not occur in the order shown in the figures. For example, two figures shown in succession may actually be executed substantially simultaneously, or may sometimes be executed in the reverse order, depending on the functions / acts involved.

[0140] Specific details are provided in the following description to provide a thorough understanding of the example embodiments. However, one of ordinary skill in the art will appreciate that the example embodiments may be practiced without these specific details. For example, the system may be shown in a block diagram so as not to obscure the example embodiments in unnecessary detail. In other cases, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the example embodiments.

[0141] As discussed herein, illustrative embodiments are described with reference to symbolic representations of actions and operations (e.g., in the form of flow charts, flowcharts, data flow diagrams, structure diagrams, block diagrams, etc.), which may be implemented as program modules or functional processes, including routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types, and may be implemented using existing hardware at, for example, existing network management devices, network management entities, clients, gateways, nodes, agents, controllers, computers, cloud-based servers, web servers, agents or proxy servers, application servers, load balancers or load balancing servers, device management servers, etc. As discussed later, such existing hardware may include one or more central processing units (CPUs), system-on-chip (SOC) devices, digital signal processors (DSPs), application specific integrated circuits, field programmable gate arrays (FPGAs), computers, etc.

[0142] Although a flowchart may describe operations as a sequential process, many operations may be performed in parallel, concurrently, or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operations are completed, but may also have other steps not included in the diagram. A process may correspond to a method, function, program, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0143] As disclosed herein, the term "storage medium", "computer-readable storage medium" or "non-transitory computer-readable storage medium" may refer to one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash memory devices and / or other tangible machine-readable media for storing information. The term "computer-readable medium" may include, but is not limited to, portable or fixed storage devices, optical storage devices, and various other media capable of storing, containing or carrying (multiple) instructions and / or data.

[0144] In addition, the example embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the necessary tasks may be stored in a machine or computer readable medium, such as a computer readable storage medium. When implemented in software, one or more processors will perform the necessary tasks.

[0145] A code segment may represent a program, function, subprogram, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0146] As used herein, the terms "include" and / or "contain" are defined as including (i.e., open language). As used herein, the term "coupled" is defined as connected, although not necessarily directly and not necessarily mechanically. Terms derived from the word "indicate" (e.g., "represent" and "indicate") are intended to cover all different technologies that can be used to communicate or reference the indicated object / information. Some, but not all, of the examples of technologies that can be used to communicate or reference the indicated object / information include the transmission of the indicated object / information, the transmission of an identifier of the indicated object / information, the transmission of information used to generate the indicated object / information, the transmission of certain components or parts of the indicated object / information, the transmission of certain derivatives of the indicated object / information, and the transmission of certain symbols representing the indicated object / information.

[0147] According to example embodiments, network management devices, network management entities, clients, gateways, nodes, proxy controllers, computers, cloud-based servers, web servers, application servers, agents or proxy servers, load balancers or load balancing servers, device management servers, etc. may be (or include) hardware, firmware, hardware that executes software, or any combination thereof. Such hardware may include one or more central processing units (CPUs), system-on-chip (SOC) devices, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) computers, etc., which are configured as special-purpose machines to perform the functions described herein and any other known functions of these elements. In at least some cases, CPUs, SOCs, DSPs, ASICs, and FPGAs may be generally referred to as processing circuits, processors, and / or microprocessors.

[0148] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments of the present invention. However, benefits, advantages, solutions to problems, and any (multiple) elements that may lead to or produce such benefits, advantages, or solutions, or make such benefits, advantages, or solutions more obvious, shall not be construed as key, required, or essential features or elements of any or all of the claims.

[0149] Reference is made in detail to embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. In this regard, example embodiments may have different forms and should not be construed as being limited to the description set forth herein. Therefore, example embodiments have been described above only by reference to the accompanying drawings to explain example embodiments. Aspects of various embodiments are specified in the claims.

Claims

1. A method of controlling an automated agent federation, the automated agent federation comprising a plurality of automated agents in service, at least some of the plurality of automated agents in service comprising computer readable instructions which, when executed by a computer, cause the computer to generate a proposal for responding to a ticket received from one or more terminals, the method comprising: Receive a ticket from a terminal; routing the ticket to some of the plurality of serving automation agents; measuring similarity of suggestions generated by the ones of the plurality of serving automated agents, the suggestions being suggestions for responding to the ticket; determining the subset of suggestions based on ratings of the some of the serving automated agents and the similarities of the suggestions generated by the some of the serving automated agents, the ratings being based on previous scores indicating quality of previous subsets of suggestions; as well as The subset of suggestions is provided to the terminal.

2. The method according to claim 1, further comprising: receiving from the terminal a score indicating a quality of the subset of suggestions; as well as The ratings of the serving ones of the automated agents are updated based on the scores.

3. The method according to any one of claims 1-2, wherein the ticket comprises a plurality of fields, and the method further comprises: measuring a complexity of the document based on at least one of the plurality of domains; as well as The number of serving automated agents is determined based on the complexity.

4. The method according to any one of claims 1-2, wherein said measuring the complexity of said document comprises: The complexity of the document is measured based on a number of words in a description field of the document.

5. The method according to any one of claims 1 to 2, further comprising: The score indicating the quality of the subset of suggestions provided to the terminal is solicited.

6. The method according to any one of claims 1 to 2, further comprising: The ticket and a suggestion generated by at least one of the plurality of serving automated agents are added to a test set based on the similarity of the suggestions.

7. The method according to any one of claims 1 to 2, further comprising: Based on the rating of at least one serving automation agent, the at least one serving automation agent is removed from the automation agent federation.

8. The method according to any one of claims 1 to 2, further comprising: At least one suggestion is generated by the serving ones of the automated agents in the federation of automated agents in response to the ticket.

9. The method of any one of claims 1-2, wherein generating, by the some agents in the automated agent federation, at least one suggestion for responding to the ticket comprises: Review description domain; Compare the description domain to the training set; as well as The at least one suggestion is generated based on the comparison of the description domain to the training set.

10. The method according to any one of claims 1-2, wherein determining a subset of suggestions based on ratings of the some serving automated agents and similarities of suggestions generated by the some serving automated agents comprises: merging said suggestions from said some of said serving automated agents; reviewing ratings associated with some of the automated agents in service; as well as The suggestion is added to the subset of suggestions based on the review of the rating.

11. The method according to any one of claims 1 to 2, further comprising: In response to the score indicating that the quality of the subset of suggestions is greater than a quality rating threshold, at least one of the ticket and the suggestions is added to a testing set.

12. The method according to any one of claims 1-2, further comprising: In response to the similarity level of the suggestions being greater than a similarity threshold, at least one of the ticket and the suggestions is added to a test set.

13. A non-transitory computer-readable medium comprising program instructions, wherein the program instructions are used to cause an apparatus to execute the method according to any one of claims 1 to 12.

14. A method of adding a temporary automated agent to an automated agent federation, the automated agent federation comprising a plurality of serving automated agents, at least some of the plurality of serving automated agents comprising a list of computer readable instructions that, when executed by a computer, cause the computer to generate a proposal for responding to a ticket received from one or more terminals, the method comprising: Receive a ticket from a terminal; forwarding the ticket to a training manager; receiving, from the training manager, interim recommendations generated by the interim automated agent; evaluating the temporary automated agent by comparing the temporary suggestion with suggestions generated by some serving automated agents in the federation of automated agents; as well as Responsive to the evaluating, the temporary automation agent is added to the plurality of serving automation agents.

15. A non-transitory computer readable medium comprising program instructions for causing an apparatus to execute the method according to claim 13.

16. A device for controlling an automated agent alliance, the automated agent alliance comprising a plurality of automated agents in service, the device comprising: at least one processor; as well as at least one memory including computer program code; The at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to receive a ticket from a terminal, routing the ticket to some of the plurality of serving automated agents, measuring similarity of suggestions generated by said some of said plurality of serving automated agents, said suggestions being suggestions for responding to said ticket, determining the subset of suggestions based on ratings of the some of the serving automated agents and the similarities of the suggestions generated by the some of the serving automated agents, the ratings being based on previous scores indicating the quality of previous subsets of suggestions, and The subset of suggestions is provided to the terminal.

17. The apparatus according to claim 16, wherein the at least one memory and the computer program are configured to, using the at least one memory, cause the apparatus to: receiving from the terminal a score indicating the quality of the subset of suggestions; and The ratings of the serving ones of the automated agents are updated based on the scores.

18. The apparatus according to any one of claims 16-17, wherein the at least one memory and the computer program are configured to, with the at least one processor, cause the apparatus to: A score is solicited indicating the quality of the subset of suggestions provided to the terminal.

19. The apparatus according to any one of claims 16-17, wherein the at least one memory and the computer program are configured to, using the at least one memory, cause the apparatus to: At least one suggestion is generated by the serving ones of the automated agents in the federation of automated agents in response to the ticket.

20. The apparatus according to any one of claims 16-17, wherein the at least one memory and the computer program are configured to, with the at least one processor, cause the apparatus to: In response to the score indicating that the quality of the subset of suggestions is greater than a quality rating threshold, at least one of the ticket and the suggestions is added to a testing set.

Images