DYNAMIC GENERATION OF A CHAIN ​​OF FUNCTION MODULES OF A VIRTUAL ASSISTANT

Description

[0001] The present invention relates to the technical field of virtual assistants, and in particular to virtual voice assistants. It provides a method and a module for generating voice assistant functions.

[0002] It is known from the prior art that virtual assistants select a static command chain, specifically a so-called "voice chain," for a specific user to process requests. For example, if a user logs into a portal or their voice is recognized by the virtual assistant and assigned to a specific profile, the user's static command chain can be loaded. This means, in particular, that the specific selection of parameters, such as ASR ("automated speech recognition"), the NLU ("natural language understanding") type, and / or the speech synthesizer, depends solely on the user who made the request.Although virtual assistants offer users the possibility to define specific configurations for themselves, for example that a user wants to use different languages, these profiles are also statically assigned to the respective user, or rather they are statically bound to the user.

[0003] While a request is being processed, the previously loaded static command chain remains unchanged. While a user can explicitly switch from their first profile, which is assigned a first command chain, to a second profile, which is assigned a second command chain, these configurations also remain static. Thus, with currently available virtual assistants, it is possible for the user to switch to a different profile, but once this switch is complete, every subsequent request will be processed using the static command chain of the other profile.

[0004] However, such a static command chain has several disadvantages. It prevents the loading of specific ASR or NLU models or technical parameters that would otherwise be more suitable for a particular query. For example, a service for practicing vocabulary in multiple languages ​​should be able to understand any language the user wants to practice. However, the generated response to the user will still be displayed in the language corresponding to the initially loaded profile with its static command chain. Another example is the use of an implicit skill that allows the dictation of medical reports, which requires a specific vocabulary.

[0005] In this context, WO 2019 / 203795 A1 and WO 2018 / 067402 A1 also show virtual voice assistants that respond specifically to user requests.

[0006] The object of the invention is therefore to provide a chain of functional modules for a virtual assistant that can flexibly process various user requests.

[0007] This object is achieved by the features of the independent patent claims. The features of the various aspects of the invention or the various embodiments described below can be combined with one another, unless this is explicitly excluded or technically necessarily mutually exclusive. According to the invention, a method for dynamically generating a chain of functional modules of a virtual assistant is specified, wherein the virtual assistant is implemented as software on a computing unit, wherein a library of various functional modules for answering user requests is provided to the virtual assistant, wherein the method comprises the following steps: Transferring a user request to the virtual assistant; ∘ For example, the request can contain both English and German terms. Since the German language now has many anglicisms, this is quite common, for example: "Play favorite playlist on my iPhone." Analyzing the request using a selection algorithm to determine which function module from the library with the highest probability P Max is suitable for processing the user's request; ∘ In the above example, it is therefore necessary to decide whether a function module should be selected for the English or German language. However, it is also possible to initially select a function module for the intent "Play music on the device"; analyzing which function module is best suited for this request.For practical purposes, a user query received as an audio signal is first converted into text using common speech-to-text algorithms. Even to perform this conversion, it may be necessary to correctly select either the English or German function module. If the appropriate function module is selected for a language, elements of the audio signal can be compared with a library of stored audio elements in German and English. The speech function module with the most matches is selected as the PF,Max function module. If the query is already in text form at the beginning or after conversion, an analysis is carried out to determine which intent, domain, or entity the query has the most matches with – this is selected as the PF,Max.For this purpose, the so-called ground-truth dataset can be used, which maps the various functionalities of the virtual assistant into different categories. For example, "Play favorite playlist on my iPhone" could have more matches for the text elements "games," "playlist," and "iPhone" in a virtual assistant domain with the functionality "play music" than in a virtual assistant domain for operating household appliances. The selection algorithm selects the function module PF,Max with the highest probability from the library of various function modules; the selected function module PF,Max is loaded into the virtual assistant; ∘ If the library of functions is already implemented in the virtual assistant, the function module PF,Max does not need to be loaded separately; a single function call may suffice.The virtual assistant processes the user's request using the selected function module PF,Max. ∘ In this example, the German speech-to-text function module would first be selected and the text transcribed into German. In the next step, the most suitable function module could be selected again, and the favorite playlist could be played as the virtual assistant's "final" response.

[0008] The fact that the processing chain is no longer "tied" to the user, as is traditional, but to the request, offers many advantages. The virtual assistant is able to flexibly and dynamically select and apply the appropriate function modules based on the request. This enables dynamic adaptation to the request during runtime without the user having to manually change their profile. In particular, communication in multiple languages ​​is enabled. The virtual assistant's output voice can also modify the skill so that voice parameters such as mood, volume, and others are dynamically adapted to the situation. All of this enables a unique user experience.A further advantage of the method according to the invention is that even the first use of the virtual assistant by the user is significantly improved, as no complex setup scenario for creating a user profile, for example using a companion app, is necessary. Already with the first request, all of the virtual assistant's settings are determined by the request and not by the user. The use of different languages ​​​​thus flows seamlessly into one another as long as they are supported by the system. Technical skills will be able to provide special language models for ASR, STT and NLU and thus deliver better results and better understand the user's intention. The analysis steps, for example to determine the language of a request and further context, can be implemented using adapted machine learning models that are used to classify the request.This can be executed in parallel with standard processing. Parallel processing, in particular, enables good runtime performance.

[0009] Preferably, the selection algorithm additionally determines parameters, in particular, it selects those with the highest probability (P Para ,Max) of processing the user's request and passes these parameters (P Para ,Max) to the selected function module (PF,Max). When selecting the function module for transcribing, for example, parameters could be loaded that characterize the user's typical pronunciation characteristics. In this context, parameters for processing times (timeouts) can also be dynamically adapted to the respective function being performed. For example, a quiz skill can adjust the time available to the user to answer the question depending on the difficulty level. This impacts the entire speech processing chain (end of speech detection, beginning of speech detection, request timeouts, etc.), because the assistant must be placed in a "wait state."To achieve this, appropriate timeouts must be able to be adjusted at runtime of a request in order to do justice to the respective function.

[0010] This has the advantage that the processing of the request by the function module PF,Max is further improved.

[0011] According to one embodiment, for each processing step of the request, a function module PF,Max is again selected from the library with the highest probability P Max to process the request in this step, i.e. for the first step the function module P F1,Max with the probability P 1,Max , for the second step the function module P F2,Max with the probability P 2,Max and for the n-th step the function module P Fn,Max with the probability P n,Max .

[0012] This offers the advantage that no static, inflexible chain is loaded at the beginning, but rather that the most suitable functions P Fn, Max are dynamically and flexibly selected for each step at runtime and then used for processing. This makes it possible to process the query as effectively as possible.

[0013] According to the invention, for each processing step of the query, in addition to the function module PF,Max with the highest probability P Max , another function module PF,lower with a different probability P lower , in particular with the second-highest probability PF,2nd , is determined, in particular selected from the library. This procedure is preferably carried out for each processing step. In the example: "Play favorite playlist on my iPhone," the function module PF,lower for the English language could thus be selected in addition to the function module PF,Max for the German language.

[0014] This offers the advantage that another function module has been selected, thus being quickly available if it turns out that the choice of PF,Max as the "most suitable" function module for the processing step was incorrect. In some cases, due to ambiguities, it is not possible to immediately determine the most suitable function module with absolute certainty. The probabilities P n,max and P n,lower determined for each step n are stored in a memory unit in such a way that the algorithm can access the corresponding values ​​for evaluation at any time.

[0015] According to one embodiment, the virtual assistant evaluates the query in parallel using the function modules PF,Max and PF,lower and saves the resulting results. The results are marked accordingly so that they can be assigned to the function module with which they were generated.

[0016] This offers the advantage of implementing two dynamically generated processing chains in parallel. If it turns out at a certain step that processing the query with the function module PF,Max was the wrong choice, the query result generated by PF,lower can be accessed immediately without significant loss of time. For the user, this means a considerable

[0017] Improving runtime performance while improving correct intent detection

[0018] The virtual assistant preferably uses the result of the function module PF,lower instead of the result of the function module PF,Max for the further evaluation of the request if it turns out that the function module PF,Max was not the most suitable module for evaluating the request.

[0019] This offers the advantage of a correction option that allows the answer to be corrected once or several times before being output to the user.

[0020] According to the invention, the request is processed, in particular in parallel, with the function module PF,lower, with which another function module PF,lower is processed, if P lower exceeds a predetermined probability value. This predetermined probability value can be selected such that it is no more than 10%, particularly preferably no more than 25%, smaller than P Max . Or the predetermined probability value is selected such that it is at least 50%.

[0021] This offers the advantage that parallel processing is only performed when there is at least a certain probability that the other function module PF,lower will generate a relevant result for the query. This conserves the computing unit's resources when processing the query.

[0022] It is advisable to load only the function module P n,Max intended for this processing step in each processing step, or to load only the function modules P n,Max and P n,lower intended for this processing step in each processing step.

[0023] This offers the advantage of efficiently conserving resources by eliminating the need to reserve unnecessary memory space, as would be the case if the entire chain were loaded. Furthermore, it offers the advantage of reducing requests to external providers, thus reducing latency and costs.

[0024] When the selection algorithm selects the function module PF,Max, background information can be used in addition to the query content. For example, if a user is in a train station while using the virtual assistant, this can influence the selection of the most suitable domain.

[0025] This offers the advantage of providing the selection algorithm with additional information to help it select the most suitable function module PF,Max.

[0026] The selection algorithm's selection of the function module PF,Max can take into account not only the query content but also the user's command history. Since users typically repeat their queries, such as "What's the weather like in Darmstadt?", information from the command history can be very helpful in selecting the most suitable function module PF,Max.

[0027] This offers the advantage of providing the selection algorithm with additional information to help it select the most suitable function module PF,Max.

[0028] Advantageously, the chain of function modules predefined for the user is loaded to evaluate the query if the selection algorithm could not determine a function module PF,Max.

[0029] This offers the advantage that the virtual assistant essentially includes an emergency program that kicks in if the request would otherwise not be processed. This emergency program essentially corresponds to the functionality of common virtual assistants.

[0030] The virtual assistant can be a voice virtual assistant or a text-based virtual assistant.

[0031] This offers the advantage that the virtual voice assistant can be flexibly addressed via communication using a browser but also via a user device such as a smartphone, a smart speaker, a smartwatch, a tablet or similar devices. In particular, the request can be entered as speech via a microphone on the user device.

[0032] As an example, a virtual assistant is given, wherein the virtual assistant is configured to dynamically generate a chain of functional modules of the virtual assistant, wherein the virtual assistant is implemented on a computing unit of a server and / or a user terminal, wherein the virtual assistant receives a request from a user, wherein the virtual assistant has a selection algorithm configured to select the most suitable functional module PF,Max according to the method described above, wherein the virtual assistant has a processing module that processes the request using the functional module PF,Max and provides the result of the processing as a response, wherein the virtual assistant has means for outputting the response to the user. The means for outputting the response can in particular be a microphone and / or a screen of the user terminal.

[0033] The fact that the processing chain is no longer "tied" to the user, as is traditional, but to the request, offers numerous advantages for the virtual assistant. The virtual assistant is able to flexibly and dynamically select and apply the appropriate function modules based on the request. This enables dynamic adaptation to the request during runtime without the user having to manually change their profile. In particular, communication in multiple languages ​​is possible. The virtual assistant's output voice can also modify the skill so that voice parameters such as mood, volume, and others are dynamically adapted to the situation. All of this enables a unique user experience.

[0034] According to a third aspect of the invention, a user terminal is provided, wherein the virtual assistant according to the invention is implemented on the user terminal. The user terminal can be a smartphone, a tablet, a computer, a smartwatch, or a smart speaker.

[0035] Such a user terminal offers the advantage that the user can use the virtual assistant efficiently, as these devices are either able to carry out the method according to the invention themselves or to forward the request via a data channel to a server on which the virtual assistant, or the method, is implemented.

[0036] According to a fourth aspect of the invention, a computer program for dynamically generating a chain of functional modules of a virtual assistant is provided, wherein the computer program is configured to execute the steps of the above-described method according to the invention.

[0037] Further advantageous features of the present invention are defined in the patent claims.

[0038] In the following, preferred embodiments of the present invention are explained with reference to the accompanying figures: Fig. 1: shows a static command chain of a virtual assistant according to the known state of the art. Fig. 2: shows a concrete example of Fig. 1 . Fig. 3: shows a generic, flexible, dynamically generated command chain of a virtual assistant according to the invention. Fig. 4: shows a concrete example of Fig. 3 . Fig. 5: shows a computing unit on which the method according to the invention is implemented. Numerous features of the present invention are explained in detail below using preferred embodiments. The present disclosure is not limited to the specifically mentioned feature combinations. Rather, the features mentioned here can be combined in any desired way to form embodiments of the invention, unless expressly excluded below.

[0039] Fig. 1 shows how, according to the current state of the virtual assistant, a fixed configuration of a command chain 1 is loaded from function modules according to the recognized user. This command chain 1 is retained throughout the entire processing of the request, until the response to the request is issued to the user.

[0040] To change anything in Command Chain 1, the user or an administrator must manually change settings in the Voice Chain Configuration. For example, the user can select a different language via the app or enable a different skill for a specific purpose. However, once this change is made, it will be applied to every subsequent request until it is manually changed again.

[0041] A generic command chain 1 as in Fig. 1 can include the following steps: Step 5: Detection of the wake-up word and the request to the virtual assistant -> Step 7: Identify user, user device, provider -> Step 9: Load the language chain configuration -> Step 11: Execute the language chain configuration -> Schritt 13: Generate the response to the request -> Step 15: Output the response as a voice signal to the user.

[0042] Fig. 2 shows a concrete example of Fig. 1 and illustrates the static command chain 1'. At no point during the execution of the request can a change be made according to the context, the utterance, or an external parameter of the request. Thus, the same models, language settings, skills, and skill parameters are applied for every request the user utters. The example in Fig. 2 may include the following steps:

[0043] Step 17: "Hello Magenta, play music" as wake-up word and request -> Step 19: Provider "Telekom" -> Step 21: Load the language chain configuration -> Step 23: Transcribe German speech signal into text -> Step 25: Detect domain, intent, entity using German function modules -> Step 27: Execute the skill with the statically specified function modules -> Step 29: Generate an audio output based on the skill's response in German -> Step 31: Send the response -> Step 33: Output the response to the user.

[0044] Fig. 3 shows a generic, flexible, dynamically generated command chain 100 of a virtual assistant, in particular a virtual voice assistant, according to the invention.

[0045] At several steps of the command chain 100, in particular at each step of the command chain 100, the request and optionally additional available information can be analyzed to determine the appropriate functional module for this step and then select it for processing. The generic command chain 100 according to Fig. 3 may comprise the following steps:

[0046] Step 105: Detection of the wake-up word and the request -> Step 107: Analysis of the wake-up word -> Step 109: Generate the initial configuration of the virtual assistant based on the wake-up word -> Step 111: Start the transcription of the request or command -> Step 113: Flexible adjustment of the ASR module configuration -> Step 115: Loading of special models, modules and / or parameters, if necessary -> Step 117: Transcription of the request or command -> Step 119: Determination and setting of the NLU configuration (model selection, language selection), if necessary -> Step 121: Send text to the NLU model -> Step 123: Dynamic determination of the skill or function module for the intent of the current request -> Step 125: Send request to the selected skill function module,if necessary with additional parameters -> Step 127: Flexible determination of the TTS configuration according to the skill and the skill response -> Step 129: Generate audio signals for the output -> Step 131: Sends the response to the user device -> Step 133: Output of the virtual assistant's response to the user as a voice message.

[0047] The real-time language chain analyzes the input and derives requirements for each step of the command chain 100. Based on the requirements, the next step is configured accordingly, or even the current step is repeated for a better match.

[0048] In the example of Fig. 4 The user logs in with a French wake-up word in step 141. In step 143, the virtual assistant, or rather the selection algorithm, recognizes the French wake-up word. In step 145, the selection algorithm selects the corresponding function module and, specifically in the example of Fig. 4 loads the ASR configuration with the generic French model to transcribe the command of the request in step 147 following the wake-up word.

[0049] If the user decides to switch from French to German during communication with the virtual assistant and submits a corresponding request to the virtual assistant, this request is detected in step 149 and analyzed as a German command with German technical terms. For example, the user may request information on setting up an internet device. This request contains some technical terms related to a corresponding domain, so the selection algorithm loads the corresponding function module in step 151, which provides the functionality of the ASR module for German technical terms. The command chain 100' is therefore not static, but responds flexibly to the user's request. In step 153, this request is then transcribed into German. In step 155, the German dispatcher model for technical terms is loaded.

[0050] The information about the recognized domain and the model used is passed to the NLU model in step 157. In step 159, the correct dispatcher is selected as the appropriate functional module to optimize intent recognition. However, if no intent or skill is found (step 160), the request is repeated, and the command chain 100 returns to step 155. In this case, the standard dispatcher model can be used to locate the intent or skill during the next iteration. If a skill or intent has been successfully identified, the command chain 100 continues to step 161.

[0051] In step 161, the request and the determined parameters, possibly together with the payload, are passed to the corresponding skill. The skill, which is also implemented via a corresponding function module, processes the user's request. In particular, the skill also decides how to proceed to provide the best response for the user.

[0052] In step 163, the virtual assistant dynamically determines which output channel to select for the response. In particular, this does not have to be the same channel through which the request was submitted by the user.

[0053] As a first alternative, output via the screen can be selected in step 165. In this case, in step 167, the request is transmitted to a functional module of the user terminal, which implements a display of the response on the screen. In step 169, the response is presented to the user on the screen of the user terminal.

[0054] As a second alternative, you can specify that the output should be via a microphone.

[0055] In step 171, the algorithm determines that the query contains both German and English terms.

[0056] In step 173, the virtual assistant determines the best TTS model to use based on the given answer. Since the German sentence contains many English terms, the best model for this case is dynamically selected and the speech output is synthesized accordingly. This greatly reduces the need for special SSML tags to optimize pronunciation, which is handled by loading the best model. This also enables smaller and faster TTS models that can be optimized for specific domains.

[0057] In step 175, the audio signal for the response is generated.

[0058] Once the response has been provided, it can be routed to the correct output channel in step 177. This does not necessarily have to be the device that originated the request. Preferences and privacy settings are used to send the appropriate response to the user.

[0059] In step 179, the virtual assistant's response to the user is provided in the form of an audio signal via a loudspeaker of the user terminal.

[0060] The dynamically generated command chain 100' according to Fig. 4 This advantageously enables unprecedented flexibility in the virtual assistant's interaction with the user. User queries are answered as specifically as possible and with previously unattainable accuracy.

[0061] Fig. 5 shows a computing unit 201 on which the method according to the invention can run. This computing unit 201 can be implemented, for example, on a user terminal, such as a smartphone, a tablet, a smart speaker, and / or a computer, or even on a server of an internet service provider. A computer program implemented on the computing unit 201 executes the following method steps as an algorithm: Step 203: Transmitting a user's request to the virtual assistant; Step 205: Analyzing the request using a selection algorithm to determine which function module from the library with the highest probability P Max is suitable for processing the user's request; Step 207: Selecting the function module PF,Max with the highest probability from the library of different function modules by the selection algorithm; Step 209: Processing the user's request by the virtual assistant using the selected function module PF,Max .

Claims

1. A method for the dynamic generation of a chain of function modules of a virtual assistant, wherein the virtual assistant is implemented as software on a computing unit, wherein the virtual assistant is provided with a library of various function modules for answering requests from users, wherein the method comprises the following steps: • transmitting a request from a user to the virtual assistant; • analysing the request by means of a selection algorithm with regard to which function module from the library with the highest probability PMax is suitable for processing the user's request; • selection of the function module PF,Max with the highest probability from the library of different function modules by the selection algorithm; • processing of the user's request by the virtual assistant using the selected function module PF,Max, characterised in that, in addition to the function module PF,Max with the highest probability PMax, another function module PF, lower with another probability Plower is selected for each processing step of the request, wherein the request is also processed with the other function module PF,lower, if Plower exceeds a given probability value.

2. The method according to claim 1, wherein the selection algorithm also determines, in particular selects, parameters which, with the highest probability PPara,Max, are suitable for processing the user's request, and transfers these parameters PPara,Max to the selected function module PF,Max.

3. The method according to any one of the preceding claims, wherein, for each processing step of the enquiry, a function module PF,Max with the highest probability PMax is again selected from the library for processing the request in this step, i.e. for the first step the function module PF1,Max with the probability P1,Max, for the second step the function module PF2,Max with the probability P2,Max and for the n-th step the function module PFn,Max with the probability Pn,Max.

4. The method according to claim 1, wherein the virtual assistant evaluates the request in parallel with the function modules PF,Max and PF, lower and stores the results obtained in each case.

5. The method according to claim 4, wherein the virtual assistant takes the result of the function module PF,lower instead of the result of the function module PF,Max as a basis for the further evaluation of the request, if it emerges that the function module PF,Max was not the most suitable module for evaluating the enquiry.

6. The method according to any one of the preceding claims, wherein in each processing step only the function module Pn,Max provided for this processing step is loaded in each case or in each processing step only the function modules Pn,Max and Pn,lower provided for this processing step are loaded in each case.

7. The method according to any one of the preceding claims, wherein, in addition to the content of the request, background information is also used in the selection of the function module PF,Max by the selection algorithm.

8. The method according to any one of the preceding claims, wherein, in addition to the content of the request, the user's command history is also taken into account in the selection of the function module PF,Max by the selection algorithm.

9. The method according to any one of the preceding claims, wherein the chain of function modules predefined for the user is loaded for the evaluation of the request, if the selection algorithm could not ascertain a function module PF,Max.

10. The method according to any one of the preceding claims, wherein the virtual assistant is a virtual language assistant or a text-based virtual assistant.

11. A user terminal comprising a virtual assistant configured to perform the steps according to claim 1.

12. A computer program for the dynamic generation of a chain of function modules of a virtual assistant, wherein the computer program is configured to perform the steps of one of the methods 1 to 10.