Automatic chatbot language generation

By generating and verifying grammar-based pre-classified language expressions, the problem of insufficient training data of chat robots is solved, efficient multi-language chat robot development and quality verification are achieved, and the performance and deployment efficiency of chat robots are improved.

CN111914527BActive Publication Date: 2024-11-05SAP SE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201911050643.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-05-08
Filing Date
2019-10-31
Publication Date
2024-11-05
Estimated Expiration
2039-10-31

AI Technical Summary

Technical Problem

When developing chatbots, finding enough linguistic expressions for training and validation is a challenge, especially in multilingual environments, resulting in the ultimate quality being affected.

Method used

Generate a large number of pre-classified linguistic expressions through multiple stored linguistic expression generation templates based on grammar and trained and validated using selected generative linguistic expressions to indicate the benchmark value of the chatbot performance.

Benefits of technology

It realizes the rapid generation of large-scale language expressions, simplifies the development process of chat robots, supports multi-language environments, improves the quality and deployment efficiency of chat robots, and reduces development cycles and resource requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111914527B_ABST
    Figure CN111914527B_ABST
Patent Text Reader

Abstract

Language expressions for training a chatbot can be generated in an automated system via language expression templates associated with intents. The pre-classified language expressions can then be used for training and validation. Thus, chatbot development can be improved by making the chatbot more robust by having a large number of expressions used for development. In practice, the process can iterate modifications to the template until a suitable benchmark is met. The technology can be applied across human languages ​​to generate chatbots that can converse in any number of languages ​​and can be applied to a variety of fields.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This field generally relates to machine learning in the context of chatbot development. Background Art

[0002] Chatbots have become an increasingly common phenomenon in the online world. A typical chatbot combines artificial intelligence, natural language processing, and some way of interacting with users. Chatbots can be very effective because they tend to interact with human users in the same way that humans do (e.g., via voice, text, etc.). As a result, modern chatbots are very useful for providing information, solving problems, performing simple tasks, and more.

[0003] However, developing a chatbot can be quite challenging. In fact, a chatbot can be developed by feeding training data to an untrained chatbot model. However, finding good training data can be a barrier to chatbot development. Furthermore, chatbot technology is not limited to just one human language, such as English, so the barriers are even further multiplied when trying to support multiple languages. Summary of the invention

[0004] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] In one embodiment, a computer-implemented automatic language expression generation method includes: generating multiple generated language expressions for developing a chatbot based on multiple stored language expression generation templates that follow a grammar, wherein the generated language expressions have corresponding pre-classified intentions; training the chatbot using selected training language expressions from the generated language expressions; and verifying the trained chatbot using selected verification language expressions from the generated language expressions, wherein the verification generates a benchmark value indicating the performance of the chatbot.

[0006] In another embodiment, a computing system includes: one or more processors; a memory storing a plurality of stored language expression generation templates, wherein the templates are associated with corresponding intents; wherein the memory is configured to cause the one or more processors to perform operations, including generating a plurality of generated language expressions for developing a chatbot based on the plurality of stored language expression generation templates following a grammar, wherein the generated language expressions have corresponding pre-classified intents; and training the chatbot using selected training language expressions from the training language expressions and the corresponding pre-classified intents.

[0007] In another embodiment, one or more non-transitory computer-readable media include computer-executable instructions that, when executed, cause a computing system to perform a method comprising: generating a plurality of generated language expressions for developing a chatbot from a plurality of stored language expression generation templates, the language expression generation templates following a grammar supporting a plurality of optional alternative phrases and grouped by intent, wherein, according to the intent group of a given language expression generation template from which the generated language expression is generated, the generated language expression has a corresponding pre-classified intent; augmenting the generated language expressions via machine translation; filtering the generated language expressions based on edit distance; selecting a first subset of the generated language expressions for training; selecting a second subset of the generated language expressions for verification; training the chatbot using the generated language expressions selected for training; and verifying the trained chatbot using the generated language expressions selected for verification, wherein the verification generates one or more benchmark values ​​indicative of the performance of the chatbot.

[0008] The above and other objects, features and advantages will become more apparent from the following detailed description made with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram of an example system that implements automatic chatbot utterance generation.

[0010] Figure 2 is a flowchart of an example method for automatic chatbot utterance generation.

[0011] Figure 3 is a block diagram illustrating example language expression template grammars, example actual templates, and example language expressions generated therefrom.

[0012] Figure 4 is a block diagram illustrating numerous example language expressions generated from example language expression templates.

[0013] Figure 5 is a block diagram of an example system that employs a pre-classified utterance generator to generate pre-classified linguistic utterances from linguistic utterance generation templates associated with intents.

[0014] Figure 6 is a flow chart of an example method for generating pre-classified utterances from an utterance generation template associated with an intent.

[0015] Figure 7 is a block diagram of an example system for extending language expression.

[0016] Figure 8 is a flow chart of an example method of augmenting a language expression.

[0017] Fig. 9 is a block diagram of the example chatbot architecture.

[0018] Fig.10 is a block diagram of an example computing system in which the described embodiments may be implemented.

[0019] Fig.11 is a block diagram of an example cloud computing environment that can be used in conjunction with the techniques described herein. DETAILED DESCRIPTION

[0020] Example 1 - Overview

[0021] Conventional chatbot (or "bot" for short) training techniques cannot find a sufficient number of language utterances to use for training. For example, the number of actual conversations in chat or email may be limited, and it may be difficult to find from the countless possibilities.

[0022] The problem is further compounded in multilingual environments, such as for multinational companies seeking to support a large number of human languages ​​via chatbots.

[0023] Furthermore, due to the limited number of available expressions, developers may shorten or completely skip the chatbot validation. As a result, the final quality of the deployed chatbot may suffer as a result.

[0024] As described herein, automatic chatbot language expression generation can be used to generate a large number of pre-classified language expressions. For example, a rich language expression set with corresponding intent can be easily generated for a variety of chatbots in a variety of languages ​​described herein.

[0025] Since a large number of available expressions can be generated using the techniques described herein, validation can be easily performed.As described herein, any number of validation benchmarks can be performed.

[0026] If necessary, the development process can be iterated, with the iterations converging on an acceptable benchmark. Fine-tuning can be supported during the development cycle.

[0027] As described herein, other techniques may be used, such as expression augmentation, expression filtering, etc.

[0028] Thus, the described techniques provide considerable improvements over conventional chatbot development techniques.

[0029] Example 2 - Example system for implementing automatic chatbot language expression generation

[0030] Figure 11 is a block diagram of an example system 100 that implements automatic chatbot utterance generation. In this example, the system 100 may include a pre-classified utterance generator 120 that accepts an utterance generation template 110 and generates a pre-classified utterance 130. The utterance 130 may be pre-classified with an associated intent that is known for the corresponding utterance 130.

[0031] A subset of the language expressions is selected for training 135; another subset of the language expressions is selected for validation 137.

[0032] The expressions 135 used for training are input to the training engine 140 , and the training engine 140 outputs the trained chatbot 150 .

[0033] The verification engine 160 receives the trained chatbot 150 and the expression for verification 137, and outputs a verification result 180 (eg, a benchmark value as described herein).

[0034] As described herein, the system may be iteratively trained over time to converge to an acceptable baseline value.

[0035] In practice, the systems shown herein, such as system 100, may vary in complexity, with additional functionality, more complex components, etc. For example, additional functionality may be present within verification engine 160. Additional components may be included to implement security, redundancy, load balancing, reporting, etc.

[0036] The described computing systems can be networked via wired or wireless network connections, including the Internet. Alternatively, the systems can be connected via intranet connections (eg, in a corporate environment, a government environment, etc.).

[0037] System 100 and any other systems described herein can be implemented in conjunction with any hardware components described herein, such as the computing systems described below (e.g., processing units, memory, etc.). In any examples herein, templates, expressions, chatbots, verification results, etc. can be stored in one or more computer-readable storage media or computer-readable storage devices. The techniques described herein can be general techniques in operating systems or specific aspects of hardware, and can be applied in any of a wide variety of environments to utilize the described features.

[0038] Example 3 - Example method for implementing automatic chatbot language expression generation

[0039] Figure 2 is a flowchart of an example method 200 of automatic chatbot utterance generation, and may be, for example, Figure 1The automatic nature of method 200 allows for rapid generation of a large number of language expressions for use in developing a chatbot as described herein. In addition, the generation can be repeatedly used for various purposes, such as retraining a chatbot, training a chatbot in a different human language, etc.

[0040] In an example, at 240, based on a plurality of stored linguistic expression generation templates that follow a grammar, the method generates a plurality of generated linguistic expressions for use in developing a chatbot. The generated linguistic expressions may have corresponding pre-classified intents. For example, some linguistic expressions may be associated with a first intent, while other linguistic expressions may be associated with a second intent, and so on. The association between expressions and intents may take different forms in the templates and may take different forms in the expressions themselves, as described herein.

[0041] At 250, the method trains the chatbot using selected training linguistic expressions from the generated linguistic expressions. In practice, many generated linguistic expressions may be selected for training. The remaining (e.g., other) generated linguistic expressions may be used for verification. For example, many linguistic expressions may be randomly selected for training. In practice, the number of training expressions is typically lower than the number of verification expressions.

[0042] At 260, the method validates the trained chatbot using the selected validation utterance from among the generated utterances. The validation may generate a benchmark value (eg, a benchmark quantification) indicative of the performance of the chatbot.

[0043] Method 200 and any other methods described herein may be performed by computer executable instructions (e.g., causing a computing system to perform the method) stored in one or more computer readable media (e.g., storage or other tangible media) or stored in one or more computer readable storage devices. Such methods may be performed in software, firmware, hardware, or a combination thereof. Such methods may be performed, at least in part, by a computing system (e.g., one or more computing devices).

[0044] The illustrated actions may be described from alternative perspectives while still implementing the present technology.

[0045] Example 4 - Example Iteration

[0046] In any example herein, the training process can be iterated to improve the quality of the generated chatbot. For example, as the template is modified (e.g., attempted to be improved) and the benchmark converges to an acceptable value, training and validation can be repeated in multiple iterations.

[0047] Training and validation can be iterated (e.g., repeated) until an acceptable benchmark value is reached. This approach allows the templates to be modified until a suitable set of templates produces an acceptable chatbot.

[0048] Example 5 - Pre-classification example

[0049] In any example herein, the generated language expression can be pre-classified. The generated language expression is pre-classified because the corresponding intention of the expression is known. Such an intention can be associated with the language expression generation template that generates the expression.

[0050] Such an arrangement may be beneficial for training, since the corresponding intent is known and can be used as input to the training engine. Similarly, this arrangement is beneficial for verification, since the intent can be used as input to the verification engine to determine whether the chatbot has selected the correct intent.

[0051] In practice, the intent can be used when the chatbot is running to determine the task to be performed. If the trained chatbot can successfully identify the correct intent for a given language expression, it is considered to be processing the given language expression correctly.

[0052] Example 6 - Example Template

[0053] In any examples herein, a language expression generation template (or simply "template") can be used to generate language expressions for developing a chatbot. As described herein, such a template can be stored in one or more non-transitory computer-readable media and used as an input to an expression generator that outputs language expressions for use in chatbot training and / or validation techniques as described herein.

[0054] Figure 3 is a block diagram illustrating an example language expression template grammar 310, an example actual template 320, and example language expressions 330A-B generated therefrom.

[0055] In this example, the template grammar 310 supports multiple alternative phrases (e.g., multiple alternative phrases can be specified in the grammar, and the expression generator will select one of them). The example shown uses a vertical bar "|" as a delimiter between brackets, but other conventions can be used. In practice, a grammar is implemented as a grammar specification that can generate language expressions.

[0056] In practice, the generator can select from the alternative phrases in various ways. For example, the generator can use each alternative phrase (e.g., all possible combinations of expressions) to generate expressions. Other techniques can be random selection of expressions, weighted selection, etc. Example template 320 contains at least one instance of multiple alternative phrases. In practice, there can be any number of multiple alternative phrases, resulting in a surge in the number of expressions that can be generated therefrom. For example, two possibilities 330A and 330B are shown (e.g., "delete" and "remove"); however, in practice, more expressions can be generated due to the presence of a certain number of other multiple alternative phrases.

[0057] In this example, template grammar 310 supports optional phrases. Optional phrases specify terms that may (but need not) be included in the generated expression.

[0058] In practice, the generator may choose whether to include an optional phrase in a variety of ways. For example, the generator may generate an expression using an optional phrase, and may generate another expression without the optional phrase. Other techniques may be random selection of whether to include an expression, weighted inclusion, etc. Example template 320 includes an optional phrase. In practice, there may be any number of optional phrases, resulting in a further increase in the number of expressions that can be generated from potential templates. Multiple alternative phrases may also be included in the optional phrase mechanism, thereby generating optional multiple alternative phrases (e.g., no option needs to be included in the expression, or one of the options may be included in the template).

[0059] Figure 4 4 is a block diagram illustrating numerous example language expressions 420A-N generated from example language expression templates 410. In one example, a set of 20 templates are used to generate approximately 60,000 different expressions.

[0060] If necessary, the template text can be translated (e.g., by machine translation) into another human language to provide a template set in another language, or used as a starting point for a final template set in another language. Syntactic elements (e.g., delimiters, etc.) do not need to be translated and can be left unchanged by machine translation.

[0061] Example 7 - Additional syntax

[0062] The grammar (e.g., 310) may support regular expressions. Such regular expressions may be used to generate templates.

[0063] Example grammars may support optional elements, 0 or more iterations of a specified element (eg, a string), 1 or more iterations, from x to y iterations.

[0064] The grammar may allow metacharacters to be passed through that may be interpreted by downstream processing. Other grouping characters (e.g., "{" and "}") may be used to form blocks that are understood by other template rules as follows:

[0065] ({[Please] create}|{Add [new]})BUSINESS_OBJECT.

[0066] An example notation may include the following, but other arrangements are equally possible:

[0067] element

[0068] []: optional element

[0069] *: 0 or more iterations

[0070] +: 1 or more iterations

[0071] {x, y}: from x to y iterations

[0072] Example 8 - Additional syntax: dictionary

[0073] Dictionaries can also be supported, as follows:

[0074] dictionary

[0075] ATTRIBUTE_NAME: vendor, price, name

[0076] ATTRIBUTE_VALUE: Avantel, Green, Notebook

[0077] BUSINESS_OBJECT: products, sales orders

[0078] Example 9 - Additional template syntax

[0079] Additional syntax may be supported, as follows:

[0080] element

[0081] <>: any token (word)

[0082] []: optional element

[0083] *: 0 or more iterations

[0084] +: 1 or more iterations

[0085] {x, y}: from x to y iterations

[0086] ================

[0087] dictionary

[0088] ATTRIBUTE_NAME: vendor, price, name

[0089] ATTRIBUTE_VALUE: Avantel, Green, Notebook

[0090] BUSINESS_OBJECT: Product

[0091] ================

[0092] Core Entity

[0093] CURRENCY:$10,999 EUR

[0094] PERSON: John Smith,Mary Johnson

[0095] MEASURE: 1 mm, 5 inches

[0096] DATE: October 10, 2018

[0097] DURATION: 5 weeks

[0098] ================

[0099] Parts of Speech and Phrases

[0100] ADJECTIVE: small, green, old

[0101] NOUN: desk, computer

[0102] PRONOUN: it, he, she

[0103] NOUN_GROUP: Game of Thrones

[0104] Example 10 - Example Intent

[0105] In any of the examples herein, the system can support a wide variety of intents. Intents can vary based on the domain in which the chatbot operates and are not limited by the techniques described herein. For example, in the field of software development, intents can include "delete", "create", "update", "checkout", etc. Generated utterances have pre-classified intents, which can be obtained from a template (e.g., the template used to generate the utterance is associated with the intent).

[0106] From the perspective of the chatbot, the input language expression can be mapped to the intent. For example, "Submit a new leave request" can be mapped to "Create". "Show my leave request" can be mapped to "Read".

[0107] In practice, any number of other intents may be used in other fields, and they are not limited in number or subject matter.

[0108] In practice, providing sample language expressions for different intents is time-consuming because developers must generate many samples for training and even more samples for validation. If the validation is unsuccessful, the process must be performed again.

[0109] Example 11 - Example pre-classification language expression generation system

[0110] Figure 5 5 is a block diagram of an example system 500 that employs a pre-classified expression generator 520 to generate pre-classified language expressions 530 from language expression generation templates 517A-N associated with intents 515A-M. In practice, there may be more templates (e.g., per intent and overall) and more intents than those shown.

[0111] In this example, templates 517A-N can be associated with intents 515A-M in a variety of ways. For example, they can be simply grouped together in a single file or folder, in which case the location of the template refers to the schematic. Alternatively, the template can explicitly refer to the schematic (e.g., by name, intent identifier, etc.).

[0112] Pre-classified language expression generator 520 (e.g., Figure 1 The generator 120 of can generate the language expression 530 of pre-classification as described herein. Again, the expression 530 can be associated with the intent (e.g., the same intent of the group 515A-M) by position or by explicit indication.

[0113] Since the utterance is already associated with that intent when it is generated, the utterance is said to be "pre-classified." The associated intent indicates the intent to which the chatbot should be mapped.

[0114] Example 12 - Example pre-classification language expression generation method

[0115] Figure 6 is a flowchart of an example method for generating a pre-classified language expression from a language expression generation template associated with an intent, and can be, for example, Figure 5 The system 500 is executed.

[0116] At 640, a language expression is generated for the intent. The intent is known when the expression is generated.

[0117] At 650, a mapping of intents to which the generated language utterances should be mapped (e.g., by a correct chatbot) is represented. As described herein, such a mapping can be represented by a location or position (e.g., where the utterance appears), explicitly represented (e.g., by an intent name or intent identifier), etc. For example, the utterances can be stored as intent-expression pairs. In this case, the intents are stored as names or other identifiers, and the utterances are stored as text.

[0118] Example 13 - Language Expression Extension System Example

[0119] In any of the examples herein, the generated language expressions can be augmented to further increase the number of expressions that can be used for training and / or validation.

[0120] Figure 7 is a block diagram of an example system 700 for augmenting language expressions. In this example, a language expression 720 is extracted from a pre-classified language expression 710 and used as input to a machine translation 730 (e.g., to a different human language, such as from English to Italian), which outputs a (translated) language expression 750. The translated language expression 750 is then used as input to another machine translation 760 to translate the translated language expression 750 back to the original language (e.g., from Italian to English), which generates an output (re-translated) language expression 780, which serves as an additional language expression that can be added to the generated language expression 710.

[0121] The intent of the additional language expression 780 may be collected as (eg, equal to) the intent of the input expression 720 .

[0122] In practice, other intermediate translations may be included (eg, from English to Italian to Russian and back to English).

[0123] This technique can be applied to any number of generated expressions to generate additional expressions.

[0124] Example 14 - Example language expression expansion method

[0125] Figure 8 is a flow chart of an example method 800 of expanding a language expression and may be performed, for example, by a method such as Figure 7 The system 700 shown is implemented.

[0126] At 820 , the method translates the original (eg, generated) language expression from one human language to another human language via machine translation.

[0127] At 840, the translated expression is translated back into the original human language.

[0128] The output (re-translated expression) is added to the expression for training and / or validation at 860. The original expression is thus augmented, and such augmented expressions may be used for training or validation in addition to those expressions generated directly from the template.

[0129] As described above, one or more other translations may be made into one or more other intermediate human languages.

[0130] Example 15 - Example Language Selection

[0131] In any example herein, the expression selected for a certain stage of development can be variable. In one embodiment, a small amount (e.g., less than half, less than 25%, less than 10%, less than 5%, etc.) of available expressions are selected for use in the training set, and the remaining available expressions are used for verification. In another embodiment, the overlap between the training set and the verification set is allowed (e.g., a small amount of available expressions are selected for use in the training set, and all available expressions or filtered available expressions are used for verification). Based on the verification method and the developer's preference, any other arrangement is possible. If necessary, this selection can be configured by a user interface (e.g., one or more sliders).

[0132] Example 16 - Example language expression filtering

[0133] In any of the examples herein, it may be desirable to filter out some expressions. In some cases, such filtering can increase confidence in the chatbot being developed.

[0134] For example, a linguistic distance calculation may be performed on the available linguistic expressions. Certain expressions that are very close (eg, linguistically similar) to one or more other expressions may be removed.

[0135] Such filtering may be configurable to remove a configurable number (eg, an absolute number, a percentage, etc.) of expressions from the available expressions.

[0136] An example of such a linguistic difference calculation is the Levenshtein distance (e.g., edit distance), which is a string metric used to indicate the difference between two sequences. The distance can be specified in terms of the number of tokens (e.g., characters, words, etc.).

[0137] For example, a configuration user may specify that 20,000 very similar expressions should be removed from use during training and / or validation.

[0138] Example 17 - Example fine-tuning

[0139] In any of the examples herein, developers can fine-tune the development of a chatbot by specifying what percentage of generated utterances to use in a training set and the distance level (e.g., edit distance) of the utterances.

[0140] For example, if the distance is configured to be less than 3 tokens, then "please create an object" is considered the same as "please create objects", and only one of them will be used for training.

[0141] Example 18 - Example Benchmark

[0142] In any of the examples herein, various benchmarks can be used to measure the quality of the chatbot. Any one or more of the benchmarks can be measured during validation.

[0143] For example, the number of correct intents can be quantified as a percentage or other rating. Other benchmarks can be response time, the number of unclassified expressions (e.g., failures or crashes), etc.

[0144] In practice, as part of the validation process, one or more values ​​are generated and these values ​​can be compared to the benchmark values ​​to determine whether the chatbot is acceptable. As described herein, a chatbot that fails validation can be redeveloped by modifying the template.

[0145] Exemplary one or more benchmark values ​​that may be calculated during validation include accuracy, precision, recall, F1 score, or a combination thereof.

[0146] Accuracy can be a global rating of the performance of the chatbot. Accuracy is the proportion of successful classifications out of the total predictions made during the baseline period. While accuracy is a good performance metric for chatbots with balanced intent, it can be biased for unbalanced bots.

[0147] Precision can be a per-intent metric. For each intent, measure the proportion of correct predictions out of all the times the intent was stated during the baseline. Precision answers the question, "Out of all the times the bot predicted this intent, how many times was it correct?". Low precision usually indicates that the intent in question needs to be cleaned up, which means removing utterances that do not belong to that intent.

[0148] Recall can also be a per-intent metric. For each intent, it measures the proportion of correct predictions among all entries belonging to that intent. It answers the question: "Out of all the times my bot should have detected this intent, how many times did it detect it?" Low recall usually indicates that more training on the relevant intent is needed, for example, by enriching the training with more sentences.

[0149] The F1 score can be the harmonic mean of precision and recall. It can be a good indication of the performance of each intent and can be calculated as a range from 0 (bad performance) to 1 (good performance). The F1 scores for each intent can be averaged to create a global indication of the bot's performance.

[0150] Other measures of baseline value are possible.

[0151] Validation can also continue after using the expressions described in this article.

[0152] As described herein, benchmarks can be used to control when development iterations stop. For example, a development process (e.g., training and validation) can continue to iterate until a benchmark reaches a threshold level (e.g., a level indicating acceptable chatbot performance).

[0153] Example 19 - Example Chatbot

[0154] In any of the examples herein, the chatbot can be implemented via any number of chatbot architectures.

[0155] In practice, a chatbot can be constructed as a basic chatbot engine and an underlying internal representation of a knowledge base developed based on training data. The knowledge base is usually validated because it can be changed through additional training or retraining.

[0156] The chatbot may accept user input in the form of text or speech (e.g., converted to text), which is then recognized by the chatbot (e.g., finger gestures). The chatbot may extract parameters from the user input and then act on them.

[0157] For example, a user may say or type “Can you cancel auto-renewal?” and the chatbot, in response to recognizing the “cancel” intent, begins a cancellation workflow for the auto-renewal (e.g., asking for confirmation or other relevant information).

[0158] Fig. 9 9 is a block diagram of an example chatbot architecture that can be used with any of the examples herein. In this example, chatbot environment 900 includes a chatbot framework 950 , channels 960 , and external services 965 .

[0159] End users typically communicate with chatbots by connecting via a chatbot channel 960 (e.g., a messaging application such as Facebook Messenger, Slack Chat, a custom web chat, etc.; a voice-activated device or computing system, etc.). After the end user sends input 910 (e.g., by speaking, typing, etc.), it may be reformatted and redirected to the chatbot framework 950. A component 930 based on natural language processing can detect language, tokens, entities, and intents in the user input 920. Component 930 can use a trained machine learning model 955 based on a training data set provided by the chatbot developer. The results of natural language processing can be used in an action resolution process 940, which determines the next step of the process based on the dialog flow 957 defined by the chatbot developer at design time. The dialog flow may contain confirmation steps, slot filling for mandatory parameters, etc. Then, based on the results, a webhook call 960 is sent to an external API 965 (e.g., to complete the intended task).

[0160] The response 967 of the external API is adapted 970 to the specific robot channel and sent back 980 to the end user and delivered as a chatbot response 990.

[0161] In practice, any number of other variations are possible. For example, external services 965 can be integrated into chatbot framework 950.

[0162] Example 20 - Example Pre-classified Language Expression Generator

[0163] In any of the examples described herein, a pre-classified language utterance generator can be used to generate utterances for training and validating a chatbot.

[0164] In practice, the generator iterates over input templates. For each template, the generator reads the template and generates multiple output expressions based on the template grammar. The output expressions are then stored for later consideration during training or validation phases.

[0165] Example 21 - Example Training Engine

[0166] In any of the examples herein, the training engine can train the chatbot.

[0167] In practice, the training engine iterates over the input utterances. For each utterance, the training engine applies the utterance and the associated intent to the training technique that modifies the chatbot. Upon completion, the trained chatbot is output for validation. In practice, the validation can be performed using the internal representation of the trained chatbot (e.g., its knowledge base).

[0168] Example 22 - Example Validation Engine

[0169] In any of the examples herein, the validation engine can validate the trained chatbot or its internal representation.

[0170] In practice, the validation engine can iterate over input expressions. For each expression, the validation engine applies the expression to the trained chatbot and verifies that the chatbot selected the correct intent. There is a distinction between those instances where the chatbot selected the correct intent and those instances where the chatbot selected an incorrect intent (or did not select any intent at all). The benchmark can then be calculated as described herein based on the observed chatbot behavior.

[0171] Example 23 — Example Machine Translation

[0172] In any of the examples herein, machine translation can be achieved by creating an internal representation of an input language pathway, applying translation techniques to the internal representation, and outputting a language pathway in a different human language.

[0173] Example 24 - Example language expression generation template

[0174] The following provides a non-limiting example of a language expression generation template that can be used to generate language expressions for use in the technology described herein. The example is related to the "delete" intent, but any number of other intents can be supported.

[0175] Table 1 Example template

[0176]

[0177]

[0178] Example 25 - Example language expression

[0179] In any of the examples herein, a language expression (or simply "expression") may take the form of a text string that mimics what a user would or might type, enter, or say when interacting with a chatbot. In practice, a language expression takes the form of a sentence or sentence fragment (e.g., with a subject, a predicate; with a subject, a predicate, an object; a predicate, an object, etc.).

[0180] The following provides non-limiting examples of language expressions that can be used in the technology described herein. The examples are related to the "delete" intent (eg, as generated by the template of the above example), but any number of other language expressions can be supported.

[0181] Table 2 Example language expressions

[0182]

[0183]

[0184]

[0185] Example 26 - Example Implementation

[0186] In one embodiment, one or more non-transitory computer-readable media include computer-executable instructions that, when executed, cause a computing system to perform a method. Such a method may include the following:

[0187] Generating a plurality of generated utterances from a plurality of stored utterance generation templates for developing a chatbot, the utterance generation templates following a grammar supporting a plurality of optional alternative phrases and grouped by intent, wherein the generated utterances have corresponding pre-classified intents according to the intent group of a given utterance generation template for generating the generated utterances;

[0188] Augmenting generated language expressions via machine translation;

[0189] Filtering generated language expressions based on edit distance;

[0190] selecting a first subset of generated language utterances for training;

[0191] selecting a second subset of the generated language expressions for verification;

[0192] Training the chatbot using the generated language utterances selected for training; and

[0193] The trained chatbot is validated using the generated language utterances selected for validation, wherein the validation generates one or more benchmark values ​​indicative of the chatbot's performance.

[0194] Example 27 - Example Advantages

[0195] Many advantages can be achieved via the techniques described herein because they can quickly and easily generate a large number of expressions for chatbot development. For example, in any of the examples herein, the techniques can be used to develop chatbots in any number of human languages. The techniques described herein can greatly help deploy a large number of high-quality chatbots.

[0196] Such techniques could significantly reduce the development cycles and resources required to develop chatbots, leading to more widespread use of useful, accurate chatbots in a variety of fields.

[0197] The challenge of finding good training material for certain languages ​​in certain domains can be enormous. Therefore, this technique allows the development of high-quality chatbots for languages ​​that have fewer speakers than other languages.

[0198] Deploying chatbots to new use cases and calibrating current chatbots can be more easily achieved. Therefore, the effect of development advantages may be to increase the satisfaction of chatbot users and chatbot developers.

[0199] Example 28 - Example computing system

[0200] Fig.10 An example of a suitable computing system 1000 is depicted in which the described invention may be implemented. Because the invention may be implemented in a variety of computing systems, computing system 1000 is not intended to suggest any limitation as to the scope of use or functionality of the disclosure.

[0201] refer to Fig.10 , the computing system 1000 includes one or more processing units 1010, 1015 and memories 1020, 1025. Fig.10 , the basic configuration 1030 is included within the dashed line. Processing units 1010, 1015 execute computer executable instructions, such as to implement the features described in the examples herein. The processing unit may be a general-purpose central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. In a multi-processing system, multiple processing units execute computer executable instructions to increase processing power. For example, Fig.10 A central processing unit 1010 is shown, as well as a graphics processing unit or co-processing unit 1015. Tangible memory 1020, 1025 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both, which is accessed by the processing unit(s) 1010, 1015. The memory 1020, 1025 stores software 1080 implementing one or more innovations described herein in the form of computer-executable instructions suitable for execution by the processing unit(s) 1010, 1015.

[0202] The computing system 1000 may have additional functionality. For example, the computing system 1000 includes storage 1040, one or more input devices 1050, one or more output devices 1060, and one or more communication connections 1070, which include input devices, output devices, and communication connections for interacting with a user. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system 1000. Typically, operating system software (not shown) provides an operating environment for other software executed in the computing system 1000 and coordinates the activities of the components of the computing system 1000.

[0203] Tangible storage 1040 may be removable or non-removable and include disks, magnetic tapes or cartridges, CD-ROMs, DVDs, or any other medium that can be used to store information in a non-transitory manner and accessed within computing system 1000. Storage 1040 stores instructions for software 1080 that implement one or more innovations described herein.

[0204] The input device(s) 1050 may be an input device such as a keyboard, a mouse, a pen or a trackball, a voice input device, a scanning device, a touch device (e.g., a touchpad, a display, etc.), or another device that provides input to the computing system 1000. The output device(s) 1060 may be a display, a printer, a speaker, a CD burner, or another device that provides output from the computing system 1000.

[0205] Communication connection(s) 1070 enable communication with another computing entity via a communication medium. The communication medium conveys information, such as computer executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in a manner that encodes the information in the signal. By way of example and not limitation, the communication medium may use electrical, optical, RF or other carriers.

[0206] The present invention may be described in the context of computer executable instructions, such as computer executable instructions included in program modules, executed in a computing system on a target real or virtual processor (e.g., ultimately executed on one or more hardware processors). Typically, program modules or components include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules may be combined or split between program modules as desired. Computer executable instructions for program modules may be executed within a local or distributed computing system.

[0207] For presentation purposes, the detailed description uses terms such as "determine" and "use" to describe computer operations in a computing system. These terms are high-level descriptions of operations performed by a computer and should not be confused with operations performed by a human. The actual computer operations that correspond to these terms depend on the implementation.

[0208] Example 29 - Computer Readable Medium

[0209] Any computer-readable medium herein may be non-transitory (e.g., volatile memory such as DRAM or SRAM, non-volatile memory such as magnetic storage, optical storage, etc.) and / or tangible. Any storage action described herein may be implemented by storing in one or more computer-readable media (e.g., computer-readable storage media or other tangible media). Anything described as being stored (e.g., data created and used during implementation) may be stored in one or more computer-readable media (e.g., computer-readable storage media or other tangible media). Computer-readable media may be limited to implementations that do not include signals.

[0210] Any of the methods described herein may be implemented by computer-executable instructions in (e.g., stored in, encoded in, etc.) one or more computer-readable media (e.g., computer-readable storage media or other tangible media) or in one or more computer-readable storage devices (e.g., memory, magnetic storage, optical storage, etc.). Such instructions may cause a computing system to perform the method. The techniques described herein may be implemented in a variety of programming languages.

[0211] Example 30 - Example cloud computing environment

[0212] Fig.11 An example cloud computing environment 1100 is depicted in which the described techniques may be implemented, including, for example, Figure 1 The system 100 and other systems herein. The cloud computing environment 1100 includes a cloud computing service 1110. The cloud computing service 1110 may include various types of cloud computing resources, such as computer servers, data repositories, network resources, etc. The cloud computing service 1110 may be centralized (e.g., provided by a data center of an enterprise or organization) or distributed (e.g., provided by various computing resources located in different locations, such as different data centers, and / or located in different cities or countries).

[0213] Cloud computing services 1110 are utilized by various types of computing devices (e.g., client computing devices) such as computing devices 1120, 1122, and 1124. For example, computing devices (e.g., 1120, 1122, and 1124) may be computers (e.g., desktop computers or laptop computers), mobile devices (e.g., tablet computers or smartphones), or other types of computing devices. For example, computing devices (e.g., 1120, 1122, and 1124) may utilize cloud computing services 1110 to perform computing operations (e.g., data processing, data storage, etc.).

[0214] In practice, cloud-based, on-premises, or hybrid solutions can be supported.

[0215] Example 31 - Example Implementation

[0216] Although some disclosed method operations are described in a specific order for convenience, such a description includes rearrangement unless the specific language set forth herein requires a specific order. For example, operations described in sequence may be rearranged or performed simultaneously in some cases.

[0217] Example 32 - Example alternative implementation

[0218] Techniques from any example may be combined with techniques described in one or more of the other examples. Given that the principles of the disclosed technology can be applied to many possible embodiments, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be viewed as limiting the scope of the disclosed technology. On the contrary, the scope of the disclosed technology includes the scope and spirit of the appended claims.

Claims

1. A computer-implemented method for automatic language expression generation, include: generating a plurality of generated utterances for developing a chatbot based on a plurality of stored utterance generation templates that follow a grammar, wherein the grammar supports a plurality of alternative phrases, at least one of the stored utterance generation templates contains at least one instance of a plurality of alternative phrases, the plurality of alternative phrases having the same meaning, the plurality of stored utterance generation templates being associated with corresponding intents, and the generated utterances having corresponding pre-classified intents; training the chatbot using selected training utterances from among the generated utterances; and The trained chatbot is validated using a validation linguistic expression selected from the generated linguistic expressions, wherein the validation generates a benchmark value indicative of the performance of the chatbot.

2. The method according to claim 1, in: The grammar also supports optional phrases; and At least one of the stored utterance generation templates further comprises an optional phrase.

3. The method according to claim 1, further comprising: include: Prior to training, the generated language expressions are augmented.

4. The method according to claim 3, in, The expansion includes: One or more machine translations are performed on at least one of the generated language expressions.

5. The method according to claim 1, further comprising: include: Before training, the generated language expressions are filtered via language distance scores.

6. The method according to claim 1, in: The benchmark value includes one or more selected from the following group, the group consisting of: accuracy, precision, recall, and F1 score.

7. The method according to claim 1, further comprising: include: A subset of the generated language utterances is selected for training.

8. The method according to claim 1, in: The grammar supports regular expressions.

9. The method according to claim 1, in: The plurality of stored language expression generation templates are grouped by intent.

10. The method according to claim 1, further comprising: include: The training and the validation are repeated in multiple iterations as the utterance generation template is modified and the benchmark value converges to an acceptable value.

11. A computing system, include: one or more processors; a memory storing a plurality of stored language expression generation templates, wherein the plurality of stored language expression generation templates are associated with corresponding intents; The memory is configured to enable the one or more processors to perform the following operations, including: generating a plurality of generated utterances for developing a chatbot based on a plurality of stored utterance generation templates that follow a grammar, wherein the grammar supports a plurality of alternative phrases, at least one of the stored utterance generation templates contains at least one instance of a plurality of alternative phrases, the plurality of alternative phrases having the same meaning, and the generated utterances having corresponding pre-classified intents; and The chatbot is trained using training utterances selected from the generated utterances and the corresponding pre-classified intents.

12. The computing system according to claim 11, in, The operations also include: Authenticate the chatbot.

13. The computing system according to claim 11, in: The grammar also supports optional phrases.

14. The computing system according to claim 11, in, The operations also include: The generated linguistic expressions are filtered based on their edit distances.

15. The computing system of claim 14, in, The filtering includes: At least one linguistic utterance having a close edit distance to one or more other linguistic utterances is removed.

16. One or more non-transitory computer-readable media comprising computer-executable instructions that, when executed, cause a computing system to perform a method, include: Generating a plurality of generated utterances from a plurality of stored utterance generation templates for developing a chatbot, wherein the utterance generation templates follow a grammar supporting a plurality of optional alternative phrases and are grouped by intent, the plurality of alternative phrases having the same meaning, wherein the generated utterances have corresponding pre-classified intents according to the intent group of a given utterance generation template from which the generated utterances are generated; augmenting the generated language expression via machine translation; filtering the generated language expressions based on edit distance; selecting a first subset of the generated language utterances for training; selecting a second subset of the generated language expressions for verification; training the chatbot using the generated language utterances selected for training; and The trained chatbot is validated using the generated language utterance selected for validation, wherein the validation generates one or more benchmark values ​​indicative of the chatbot's performance.

17. One or more non-transitory computer-readable media comprising computer-executable instructions that, when executed, cause a computing system to perform the method of claim 1.