METHOD AND DATA GENERATOR FOR GENERATING A BASIC DATA SET FOR A VIRTUAL ASSISTANT

Description

[0001] The invention relates to the technical field of virtual assistants to which users can address queries or with which users can communicate. In particular, the invention relates to the generation of a basic data set that forms the basis for the functionalities of these virtual assistants. Furthermore, the invention relates to the use of such a basic data set. This basic data set is also referred to as a ground-truth data set in the technical field of virtual assistants.

[0002] Virtual assistants are increasingly becoming part of users' daily lives and are therefore confronted with increasingly diverse requests. Users can communicate with the virtual assistants' various interfaces / touch points. Possible touch points include a smart speaker, a virtual assistant implemented on a smartphone, a support chat in a browser, and / or an automated telephone hotline. The assistants, and especially their specific speech recognition models (e.g., speech-to-text or natural language understanding), are constantly being developed further. For these developments, it is important to test changes, preferably using automated methods, to determine whether the improved virtual assistant functions better than the older virtual assistant and can successfully handle new user requests.

[0003] These tests are conveniently conducted based on an existing dataset of "utterances," also known as "utterances." Such utterances can be spoken language, i.e., audio signals, but also utterances in text form. This existing dataset is well-defined and consists of a set of utterances for which the virtual assistant's responses should be determined in advance, based on the "intent" and "entity" levels. The ground-truth dataset is, so to speak, a baseline of utterances that the system must ideally recognize 100% correctly. If new, specific models are to be integrated into the virtual assistant, these models are trained using an appropriately adapted ground-truth dataset. In this sense, a "new" ground-truth dataset forms the basis for new functions of the virtual assistant.

[0004] The term "entity" is familiar in relevant specialist circles. While the German term "entity" is occasionally used in this text, this does not imply any difference in meaning.

[0005] "Intents" are essentially the actions or tasks that a virtual assistant can perform for users. They are essentially categories into which, for example, messages or emails are categorized by an AI model. An intent should always reflect the user's goal or intention.

[0006] In addition to an intent, queries can also contain additional information that can be extracted as a so-called entity. In a query like "Book a flight to New York," New York could be extracted as the "Destination" entity, so that a virtual assistant already knows and can process this information.

[0007] In this sense, an intent and an entity form a kind of hierarchical structure on the request, in the sense that the intent is superior to the entity. In the above example, this would mean that if the virtual voice assistant simultaneously processes a user's request based solely on the entity, it would be unclear whether the user wants to book a flight to New York or be provided with a travel itinerary to New York.

[0008] When creating ground truth datasets, however, the problem arises that it is of fundamental importance to fill the ground truth dataset with only correct and accurate data and at the same time to ensure that this data is not duplicated or even contradicts itself.

[0009] Current solutions still rely entirely on appropriately trained employees, known as editors or annotators, to categorize and annotate user requests for the virtual assistant. Categorizing and annotating serves to correctly "classify" user requests, along with other important information, such as identifying the intent and entities within the request, within the ground truth dataset. The editors process and annotate the requests. If an utterance is included among these requests that is not yet present in the ground truth dataset, it is added to the ground truth dataset after consultation within the team. This consultation within the team is necessary to avoid errors and ensure correct classification.

[0010] Because "real" editors manually process user requests, the number of requests that can be processed scales with the size of these teams, or rather the number of teams. Typically, a team of 2-3 people can only process 3,000 requests per week, although currently, approximately 420,000 requests per week are submitted to Deutsche Telekom's "Hello Magenta" assistant. This means that a very large number of requests cannot be considered for processing. Another problem arises with regard to data protection, as the real editors can draw conclusions about users when processing queries.

[0011] Even "simple" automation offers no remedy in this regard. Such simple automation can be implemented by searching for keywords, assigning intents or entities to these keywords, and then adding them to the ground truth dataset. For example, if the keyword "next" is found within the query, this simple automation would result in the query being assigned both to an audio playback request "Play the following track next" and to the creation of a calendar entry, "Set the following appointment next." This would result in the ground truth database containing too many queries that are not precisely assigned and do not reflect the relevance of the original data.As a result, many incorrect statements are recognized as correct, which can subsequently lead to the problem of overfitting when creating the virtual assistant.

[0012] While it is important to maintain a sufficiently large ground-truth dataset for training the virtual assistant, an optimal compromise must be found between the size of the ground-truth dataset and the problem of overfitting discussed above. Ultimately, this means that not all possible variants of a query should be included in the ground-truth dataset; however, the relevant variants of a query should always be included preferentially in the ground-truth dataset.

[0013] DILEK HAKKANI-TUR ET AL: "An active approach to spoken language processing", ACM TRANSACTIONS ON SPEECH AND LANGUAGE PROCESSING (TSLP), ASSOCIATION FOR COMPUTING MACHINERY, INC, NEW YORK, USA, Vol. 3, No. 3, October 1, 2006 (2006-10-01), pages 1-31, XP058184508, ISSN: 1550-4875, DOI: 10.1145 / 11770551177056; shows an active approach in which the system for a virtual language assistant selects its own training data, evaluates itself, and retrains if necessary.

[0014] ARGAMON-ENGELSON S ET AL: "Committee-Based Sample Selection for Probabilistic Classifiers", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, June 1, 2011 (2011-06-01), XP080511688, DOI:10.1613 / JAIR.612; investigates methods for reducing annotation costs through example selection. In this approach, a learning program examines many unannotated examples during training and selects only those that are most informative for annotation at each stage. This avoids redundantly labeling examples that provide little new information.

[0015] It is therefore the object of the invention to provide techniques that make it possible to create a ground-truth dataset for training a virtual assistant effectively and with high quality.

[0016] This task is solved with the features of the independent claims.

[0017] The features of the various aspects of the invention or the various embodiments described below can be combined with one another unless this is explicitly excluded or technically mandatory.

[0018] According to the invention, a method for generating a ground-truth data set for a virtual assistant is specified, wherein the elements of the ground-truth data set include possible user requests to the virtual assistant, wherein the method can be executed at least partially automatically on a computer unit and comprises the following steps: Providing requests made by users to the computing unit, which are requests for the virtual assistant; o these requests are recorded from as many users as possible and, in particular, stored centrally in a database on a server; preferably, the server is provided by the company that offers the virtual assistant; analyzing the requests with regard to their suitability for inclusion in the ground truth data; whereby the analysis is carried out by at least two virtual editors, each of the virtual editors being trained based on the inputs of a single human editor and creating its own analysis result for each request; o a request is particularly suitable for inclusion in the ground truth data if it is "correct" data that is not duplicated or even contradictory;During the analysis, it can be determined, for example, whether the request has a specific intent and / or an entity that corresponds to a functionality to be provided by the virtual assistant. In particular, the analysis also includes the classification of which intent and / or entity is involved. The virtual editors are processes that run automatically on the computing unit and evaluate, or classify and annotate, the requests automatically like real editors. Implementing means for evaluating the analysis results on the computing unit, wherein the evaluation means calculate a value TS that represents a probability that the request is suitable for the ground truth data set, wherein the calculation is carried out based on the analysis results of the at least two virtual editors for the respective request;o For example, if a certain query has been unanimously assigned to a certain intent by all editors, it is obvious that this query will be added to the ground truth dataset. However, problems arise with this assessment, especially when there is not complete agreement between the editors. For this reason, it is useful to calculate a corresponding metric TS that allows an assessment of the quality of the query. Although the value TS represents a probability, this does not mean that if the value TS is, for example, 0.5, this corresponds to a 50% probability that the query can be included in the ground truth dataset. A higher numerical value TS merely indicates a higher probability that the query is suitable for the ground truth dataset. Comparing the calculated value TS with a predefined threshold TH;o The predefined threshold TH is chosen such that suitability for the ground-truth dataset is assumed if TH > TS. The predefined TH can be defined differently for intents / entities. Add the query to the ground-truth dataset if the calculated TS value is greater than the threshold TH. ;

[0019] This offers the advantage that the creation of the ground-truth dataset can no longer be scaled with the size of a team of real editors, but can be carried out automatically, while simultaneously ensuring a high quality of the dataset. This high quality is ensured by the fact that the process is modeled on the approach of real editors, and in particular teams of real editors. This also avoids the problem of overfitting, which would occur with simple classification algorithms. This offers the advantage that a significantly larger number of queries can be evaluated for their suitability for the ground-truth dataset. A key concept of the invention is that at least two virtual editors are used for this purpose.This also reflects the fact that even with real editors, a team of editors is used to maintain the quality of the dataset. It proves advantageous if the virtual editors have different characteristics, as is the case with real editors. If the virtual editors were based on exactly the same models, they would also make the same errors in categorization and annotation.

[0020] For practical purposes, the requests are taken from a history of real user requests stored in a database.

[0021] This offers the advantage that only those queries are used that are actually relevant and of interest to the user regarding the functionality of the virtual assistant.

[0022] In a preferred embodiment, means for filtering requests are implemented on the computing unit and / or a server, wherein the provided requests are a filtered subset of a total set of real user requests.

[0023] This offers the advantage that the process becomes significantly more efficient when queries that are clearly irrelevant to the ground-truth dataset are filtered out from the total set of real user queries. Such obviously irrelevant queries can be effectively filtered out using simple algorithms.

[0024] Preferably, the filtering means filter out requests to which the following tags, i.e., characteristics or features have been assigned: insufficient quality of the audio signal of the request, the length of the audio signal of the request is too long or too short, and / or the user has not provided a response.

[0025] This offers the advantage that only those audio signals from the requests that are highly likely to be usable for further processing are fed into the actual analysis and evaluation. If the quality of the audio signal of the request is insufficient, there is a risk that it will not be assigned to an intent at all or will be assigned to the wrong intent. If the length of the audio signal is too short, it can be assumed that it is not a request at all. If the request is too long, it can be assumed that the user accidentally activated the virtual voice assistant and the words in their auto-signal are not actually directed at the virtual voice assistant. These tags, i.e. the features, are preferably generated and assigned automatically by the virtual voice assistant as the request is entered.For example, the user can submit a request to the virtual voice assistant via a microphone on a device on which the virtual assistant is installed. If the auto-signal is less than 1 second long, the virtual assistant automatically generates the tag "Audio signal too short." The audio signal can then be stored in the central database along with this tag. Another option is for such a marked audio signal not to be forwarded to the central database at all, but to be discarded directly by the device. This offers the advantage of reducing data transfer over the network.

[0026] Furthermore, the filtering tools currently only consider requests marked with the "Success" tag to a lesser extent. While auto-signals marked as successful are valid requests to the virtual assistant, they are requests that are highly likely to already be present in the ground-truth dataset. However, audio signals marked as successful are particularly well-suited for training artificial intelligence as a virtual editor. Furthermore, "Not-Success" requests in particular are classified to analyze why the system failed to respond correctly—for example, because a function is not supported or a request is unsuitable for an existing intent.

[0027] According to a preferred embodiment, the requests are based on a text input or a voice input.

[0028] This offers the advantage that the virtual assistant can be used flexibly as a voice assistant or a text-based assistant. The text-based assistant can be provided to users in browsers, for example, and accessed via keyboard input, whereas the virtual voice assistant is preferably installed in smart speakers or user devices, allowing users to conveniently submit requests to the virtual assistant via audio.

[0029] Preferably, the threshold and / or a number of virtual editors are changed through appropriately configured APIs.

[0030] Changing the number of virtual editors offers the advantage that the system's performance, i.e. the speed at which user requests are processed, can be flexibly scaled. The more virtual editors are used, the more requests can be processed. The same applies to virtual editors: at least two virtual editors form a group, with each group processing identical requests. If more requests are to be processed at any given time, more groups of virtual editors must be formed. Increasing the number of editors within the groups offers the advantage of achieving more statistically significant results, because incorrectly marked requests by a single virtual editor no longer have such a significant impact. Ideally, a group should have at least five virtual editors.Changing the thresholds offers the advantage of flexibly varying the number of queries added to the ground truth dataset and the quality of these queries. Increasing the threshold results in fewer queries being added to the ground truth dataset, but these queries are of higher quality.

[0031] In a preferred embodiment, the queries are additionally analyzed by real editors with regard to their suitability for inclusion in the ground truth data. This offers the advantage of giving the automated process an additional quality boost, particularly when the virtual editors are confronted with new situations. The real editors are then grouped together with the virtual editors. For example, a combination of real editors and virtual editors can have the beneficial synergistic effect that the virtual editors compensate for careless errors made by humans, while the real editors can react better to new circumstances. A further advantage is that the queries processed by the real editors can be used to train the artificial intelligence of the virtual editors.In order for the real editors to be able to process the requests accordingly, they are provided with appropriate input and output devices (screen and keyboard) so that they can communicate with the computer unit via a corresponding data stream.

[0032] According to the invention, the virtual editors are implemented by artificial intelligence trained on the input of real editors. In particular, a single virtual editor is trained based on the input of a single real editor. In particular, machine learning methods such as deep learning and / or supervised training are used here. The virtual editor can also be trained on a selection of evaluated utterances for a specific domain. Thus, there could be a special virtual editor for a TV or just for the smart home domain. Other virtual editors are not as specialized and could annotate the other domains / categories. This results in the advantage that the virtual editors can be trained specifically for the respective application with specialized knowledge.

[0033] This offers the advantage of creating an efficient way to generate virtual editors that correspond as closely as possible to their human counterparts and are also sufficiently different.

[0034] The variability of these virtual editors can be further increased by training the artificial intelligence of the virtual editors with different models and / or different parameters. For statistical reasons, it is advantageous to provide virtual editors that differ sufficiently in their properties. This allows "mistakes" made by one virtual editor to be compensated for by another virtual editor. During automatic evaluation, separate confidences can in principle be created for the different machine learning models to form an overall confidence. If the overall confidence exceeds a certain threshold, the query is saved in the ground truth dataset, and the models are retrained using supervised training to further improve recognition.

[0035] The virtual editors can be updated regularly, especially when a new ground truth dataset has been generated.

[0036] This offers the advantage that the virtual editors can correctly analyze and evaluate requests with regard to new requests or new functionalities of the virtual assistant.

[0037] Preferably, the means for evaluating the analysis results include the calculation of the following metrics: Intent_Level_Score, o The Intent_Level_Score is used to assess how well the virtual and / or real editors agree with each other in assigning the request to a specific intent. The higher the agreement between the editors, the higher the corresponding score TSintent. If TS Intent is greater than a previously defined threshold TH Intent , it is assumed that the categorization of the request to a specific intent of the virtual assistant can be evaluated as successful with sufficient certainty, so that the request can be added to this intent in the ground truth dataset. Intent_Scoring_Metric, ∘ is an alternative metric to the Intent_Level_Score. Entity_Level_Score and / or ∘ The Entity_Level_Score is used to assess how well the virtual and / or real editors agree with each other in assigning the request to a specific entity.The higher the agreement between the editors, the higher the corresponding TS Entity score. If TS Entity is greater than a previously defined threshold TH Entity, it is assumed that the categorization of the request to a specific intent of the virtual assistant can be assessed as successful with sufficient certainty, so that the request to an entity can be added to the ground truth data set. ∘ Typically, a hierarchical approach is used here, so that the intent of a request is determined first, and then the entity within this intent. Thus, a rough categorization takes place first, namely the request is assigned to an intent. After the Entity_Level_Score has been determined, this classification may be further refined by adding the request to a specific entity within the intent of the ground truth data set.In this case, both scores must be higher than the previously defined thresholds: TS Entity > TH Entity AND TS Intent > TH Intent . ∘ In the event that it is difficult to assign an intent to the request in the first step, it may be advantageous to reverse the order of evaluation and first assign an entity to the request and then go back a step and attempt to categorize an intent for the request. Team / Annotator Confidence. ∘ This value indicates the degree to which the editors within a team "agree" on their assessments. A high "agreement" value among such a group of editors increases the probability that their categorization is correct.

[0038] In principle, the aforementioned metrics can be combined in any way. They advantageously provide a quantification of the quality of query categorization. This qualification is particularly useful if the method is intended to automatically decide whether a query should be added to the ground truth dataset or not. It is particularly advantageous if a specific threshold is set for each of the metrics used, whereby each of these thresholds must be exceeded for the query to be added to the ground truth dataset. This makes it possible to take into account that the metrics quantify the queries "differently." Preferably, an existing minimal set of ground truth already exists so that the metrics can be compared.This allows editors to be presented with not only new queries, but also existing ground-truth queries during the actual annotation process, to calculate a confidence vote for each editor. This confidence can be calculated for the entire team based on quality and quantity using Fleiss's cap or Krippendorf's alpha. This is referred to as "inter-annotator agreement."

[0039] Furthermore, a specific threshold can be defined for each domain or even each editor group. This offers the advantage of allowing annotations to be specified based on the importance or sensitivity of the domain. For example, a banking domain with a money transfer user case should have a very high threshold to ensure the intents and entities are correctly recognized. A music domain, for playing your favorite radio station, can have a low threshold.

[0040] According to one embodiment of the invention, the ground truth data set is extended by automatically adding variations of the queries contained in the ground truth data set based on dictionaries.

[0041] This offers the advantage of providing the most optimal compromise possible from the most comprehensive ground-truth dataset possible, while simultaneously not considering all variations of the queries, thus preventing so-called overfitting. Furthermore, the process can advantageously be automated by using appropriate databases. In addition to dictionaries, synonym lists, grammatically correct adjustments, filler words, and / or combinations can also be considered. Using words from dictionaries also prevents "too" personal data from being stored in the ground truth, such as "Turn off the light in Anna's room!"

[0042] This can be illustrated by an example: the utterance "What's the weather like in New York City?" already included in the ground truth can be modified as follows: Use of synonyms: "What's the weather like in the Big Apple?"; Dictionary adaptations: "What's the climate like in New York City?"; Use of filler words / different sentence structure: ∘ "What's the weather like in New York City, please?" ∘ "Tell me what the weather is like in New York City." Combining utterances: "What will the weather be like in the capital of the USA?"

[0043] According to a second aspect of the invention, a data set generator is provided, wherein the data set generator is configured to generate a ground truth data set as described above and comprises the following: a data input channel for receiving the provided queries; a computing unit with an algorithm implemented on the computing unit, wherein the algorithm is configured to execute the method described above; a data output channel for transmitting the queries to be added to the ground truth data set. Preferably, the queries are stored, for example, on the computing unit and / or a server in a storage module as a ground truth data set or added to an existing ground truth data set.

[0044] The data set generator offers the advantage of providing an effective technical means to generate the ground truth data set according to the invention.

[0045] According to a third aspect of the invention, the ground truth data set generated according to the method described is used to train specific models of the virtual assistant.

[0046] This offers the advantage that the virtual assistant's specific models, which are intended to process user queries, can be trained using a ground-truth dataset that is as error-free and comprehensive as possible. The ground-truth dataset preferably has an "optimized dataset size" to avoid the problem of overfitting the specific models.

[0047] Another advantage of using this approach is that only a few variations of an utterance are stored in the ground truth as a base utterance (no overfitting). This base utterance is then provided by the dataset generator for testing purposes or new model generation.

[0048] In the following, preferred embodiments of the present invention are explained with reference to the accompanying figures: Fig. 1: shows the inventive method for generating the ground-truth dataset. Fig. 2: shows the use of the inventively generated ground-truth dataset to generate new, specific functions for the virtual assistant.

[0049] Numerous features of the present invention are explained in detail below using preferred embodiments. The present disclosure is not limited to the specifically mentioned feature combinations. Rather, the features mentioned here can be combined in any desired way to form embodiments of the invention, unless expressly excluded below.

[0050] At this point, some terms of the invention will be explained once again.

[0051] Intents and entities are terms commonly used by experts in the field of virtual assistants, particularly virtual voice assistants. Ultimately, this refers to a hierarchical structure of classifications within the ground-truth dataset. An entity can, in principle, be assigned to multiple intents. The virtual voice assistant finds the correct entity by first analyzing and categorizing the request with regard to its intent. Another term in this context is that of domain: a domain is a collection of intents for a specific use case. For example, there is a domain for TV, smart home, weather, or to-do list. An entity always consists of a type and a value: "What's the weather like in New York?" Entity type: LOCATION; Entity value: New York; The entity type is defined GLOBALLY across all domains and is therefore uniform.

[0052] For the purposes of this invention, an editor, also referred to as an annotator, is understood to mean both a virtual editor and a real, i.e., human, editor, unless specifically reference is made to one of the two alternatives. For the purposes of this invention, the terms "entity" and "entity" are to be considered synonyms. Both terms are familiar to those skilled in the field of virtual voice assistants.

[0053] Fig. 1 shows the inventive method for generating the ground truth data set.

[0054] From a history 10 of user requests to the virtual assistant, requests, in particular audio elements (utterances) and / or transcribed text, are forwarded to an editing service 12. In particular, the requests are a pre-filtered selection of a total set of requests in the history 10. This editing service 12 is carried out by editors 14. The editors categorize or edit the requests submitted to them. In this case, Fig. 1 that both real editors 16 and virtual editors 18 edit the requests. These editors 14, 16 preferably form an editor group, whereby, in particular, several editor groups can edit the requests.

[0055] In step 20, the groups of editors analyze each individual request.

[0056] In step 22, the results from step 20 are quantified using at least one or any combination of the following metrics: Intent_Level_Score; Intent_Scoring_Metric; Entity_Level_Score; Team / Annotator Confidence.

[0057] The individual metrics are described in detail below.

[0058] Calculating the metrics allows us to calculate values ​​that can be compared to previously defined thresholds. These thresholds are chosen so that new queries are only added to the ground-truth dataset if there is a high probability that these queries will "improve" the existing ground-truth dataset.

[0059] These results are then transferred to the decision process 26 in step 24. The algorithm analyzes the results for each individual query and decides for each query whether it will be discarded in step 28 or added to the ground-truth dataset 32 ​​in step 30. If the value calculated for a query, i.e., the calculated metric, exceeds the predefined threshold, the query is added to the ground-truth dataset 32 ​​in step 30.

[0060] In addition, Fig. 1 a computer unit 34 on which the method can be carried out, in particular the computer unit 34 is a data set generator 34. The computer unit 34 has the corresponding channels for data transfer, a processor unit on which the method is implemented as an algorithm and, if appropriate, a memory module for the ground truth data set 32. The computer unit 34 can be provided by a server.

[0061] Fig. 2 : shows the use of the ground-truth dataset generated according to the invention to create new, specific functions for the virtual assistant. This enables virtual assistants to be continuously improved and better respond to user queries.

[0062] In step 50, a data set from the ground truth data set 32 ​​is passed to a learning service 52 for learning.

[0063] This dataset can be adapted again in step 53. For example, in step 53, entities can be expanded, supplemented, and / or replaced with synonyms and / or existing word lists. Filler words such as "thank you" and "please" can also be added. Another option is to grammatically restructure the query sentences: "What's the weather like in Darmstadt?", "What's the weather like in Darmstadt?". Furthermore, different utterances can be combined.

[0064] In step 54, supervised learning begins based on the dataset, specifically based on the adapted dataset. In step 56, there is another opportunity to consider the latest changes to the ground-truth dataset 32 ​​for supervised learning. As a result, the supervised learning produces a latest machine learning model based on the latest ground-truth dataset 32. The machine learning model provides the specific functions that enable the virtual assistant to answer user queries.

[0065] In step 58, this latest machine learning model is tested in a given test scenario to determine how reliably the latest machine learning model answers user queries.

[0066] In step 60, another machine learning model, in particular the machine learning model currently implemented on the virtual voice assistant, is transferred from a database 62 of machine learning models to the learning service 52 and is also tested there in the predetermined test scenario in step 64 with regard to how reliably this machine learning model answers user queries.

[0067] The test results for the latest machine learning model and for the other machine learning model are passed in step 66 to a decision algorithm 68 implemented on the learning service 52. If the test results show that the latest machine learning model is worse than the other machine learning model, the latest machine learning model is discarded in step 70. If the test results show that the latest machine learning model is better than the other machine learning model, the latest machine learning model is transferred to the database 62 in step 72 and can be implemented as a model to be used by the virtual assistant. The virtual assistant now has a machine learning model based on the ground truth data set 32 ​​generated according to the invention and can therefore also answer the latest user queries.

[0068] The metrics relevant to the invention are described in detail below: It may happen that the editors disagree on the classification of intents, entities—especially including the entity values—and whether the correct answer was selected. However, it has proven advantageous to only adopt those editor classifications in the ground truth for which there is at least a certain degree of agreement. For this reason, it is expedient to develop one or more metrics that allow the degree of agreement between the annotators to be quantified. These metrics can, in particular, be calculated at a granularity down to the smallest audio element and / or text element ("single utterance").

[0069] It should also be considered that agreement between editors can change over time, for example, over a period of one week. Both the Fleiss Kappa and the Krippendorf Alpha are suitable KPIs (key performance indicators) for assessing this. This allows you to observe whether the agreement between editors changes. This can happen, for example, when new editors are onboarded and improve over time. An adapted version of the Fleiss Kappa is described below.

[0070] In general, the audio elements to be added to the ground truth must exceed a first and / or a second independent threshold. The first threshold is an intent-inter-annotator-agreement-score (TH Intent), where the intent-inter-annotator-agreement-score is abbreviated as the "intent score." The second threshold is an entity-entity-value (TH Entity), where the entity-entity-value (TH Entity) is referred to as the "entity score." It is advantageous to implement the thresholds in a way that allows them to be modified via APIs (application programming interfaces). It is also useful to be able to flexibly specify the number of annotators, also known as editors. In particular, the number of virtual editors can also be changed via APIs.For example, the number of 8 annotators has proven to be useful.

[0071] The calculation of the intent score TS Intent , which is compared with the predefined threshold TH Intent , is explained below.

[0072] The calculation of the "entity score" TS Entity , which is compared with the predefined threshold TH Entity , is explained below.

[0073] An audio signal is preferentially added to ground truth when eight annotators have assessed the audio signal and when TSintent > THintent and / or TS Entity > TH Entity applies.

[0074] First, a metric is described that allows for quantifying agreement among editors. This metric is preferably a continuous scale between 0 (no agreement) and 1 (complete agreement) for a majority of editors, while also taking NA (no classification) values ​​into account.

[0075] This will be illustrated using examples. A "dominant / more likely" intent is indicated by a higher degree of agreement between the editors regarding this intent. For simplicity, four editors are used here instead of the preferred eight. Table 1 Editor 1 Editor 2 Editor 3 Editor 4 Score Intent Annotation 1 A A B C Partial agreement Intent Annotation 2 A A B B No match

[0076] Table 1 shows that Intent Annotation 1 is assumed to have a higher match than Intent Annotation 2, where it is not clear whether it is Intent A or Intent B.

[0077] Missing annotations "NA-values" result in a lower agreement than matching annotations, but have a higher score than an amputation of another intent - this is shown in Table 2. Table 2 Editor 1 Editor 2 Editor 3 Editor 4 Score Intent Annotation 1 A A A A Complete match Intent Annotation 2 A A A N / A Very high agreement Intent Annotation 3 A A A B accordance

[0078] The editor agreement metric can be calculated as follows: Normalize the agreement ("Normalized agreement score (row)") ∘ Row = {Vote e}; ∘ e is an editor (e = 1, ..., n editors ); ∘ Votum e is an intent i (i = 1, ..., n intents ); Example result 1: {Editor1: "Intent A", Editor2: "Intent A", Editor3: "Intent B", Editor4: "Intent C", Editor5: "Intent A"}; Conversion to a count table ∘ CT i :=sum(Vote e == i); ∘ applied to example result 1, this yields example result 1.1: {Intent A = 3, Intent B = 1, Intent C = 1} Identify the most frequent intent ∘ max intent := argmax(CT i ) ∘ applied to example result 1.1, this yields example result 1.2: Intent A; calculation of the "share value" PPS ∘ PPS : = CT max_intent / n Editor ∘ So: 3 / 5 = 0.6; Calculating matching intent pairs AP per intent i: ∘ AP i = CT i * CT i − 1 / 2 ; o So: Editor1 - Editor2 = A; Editor1 - Editor3 = / ; Editor1 - Editor4 = / ; Editor1 - Editor5 = A; Editor2 - Editor3 = / ; ▪ So: A=3; B= 0; C= 0; o So: APa, = 3*(3-1) / 2 = 3; AP B = 1*(1 -1) / 2 = 0; APc = 1*(1-1) / 2 = 0; Calculating the sum of the matching intent pairs AP per intent i: o So: AP SUM = sum(APi) = AP A + APs + APc = 3; Calculating the maximum pairing score for n editors: ∘ AP MAX = n Editoren * n Editoren − 1 / 2 ; ∘ So: n editors = 5 → AP MAX = 10; Calculation of the "agreing pairs score" APS ∘ If not sum(APi) == 0; then ▪ APS : = AP SUM / AP MAX = 3 / 10 = 0,3 ; Calculation (total) "Intent Scores" TS Intent ∘ TS Intent : = APS + PPS * 0,5 o So: (0.3 + 0.6) * 0.5 = 0.45;

[0079] The TS Intent metric returns a value between 0 and 1 and can be compared to the TH Intent threshold as explained above.

[0080] Additional metrics can be calculated for quantification.

[0081] One of these additional metrics is the so-called "Intent_Scoring_Metric", which indicates a normalized match value and is described below: ∘ Series = {Vote e}; ∘ e is an editor (e = 1, ..., n editors ); ∘ Vote e is an intent i (i = 1, ..., n intents ); Example result 2: {Editor1: "Intent A", Editor2: "Intent A", Editor3: "Intent B", Editor4: "Intent C", Editor5: "Intent A"};

[0082] A global metric for the "trust between editors" is calculated: Calculate separately the "mean annotation confidence" with respect to the intent, the entity, and the entity value: p(Intent global ) = mean(n max_intent / n editors ) calculate over >1000 utterances → e.g. p(Intent global ) = 3 / 5; p(Entity global ) = mean(n max_Entity / n editors ) calculate over >1000 utterances; p(Entity_value global ) = mean(n max_Entity_value / n editors ) calculate over >1000 utterances;

[0083] For a specific audio element, calculate the expected number of editor matches for an intent: Calculate the expected number of agreeing editors when n editors rate the audio element, assuming that the number of agreeing annotators follows a binomial distribution → E(X=N)=5*p(Intent global ); e.g., 4 out of 5 editors agree regarding the audio element or the text element: E(X=N)=5*p 3 / 5 = 3; The expected number of agreeing editors is calculated separately for Entity and Entity Value according to their individual binomial distributions.

[0084] A metric specific to the audio element is calculated for the agreement between editors (percent pairwise editor agreement): Calculate matching intent pairs per intent: ∘ AP i = CT i * CT i − 1 / 2 ; ∘ So: Editor1 - Editor2 = A; Editor1 - Editor3 = / ; Editor1 - Editor4 = / ; Editor1 - Editor5 = A; Editor2 - Editor3 = / ; ▪ So: A=3; B= 0; C= 0; ∘ So: APa, = 3*(3-1) / 2 = 3; AP B = 1*(1-1) / 2 = 0; APc = 1 *(1-1) / 2 = 0; Calculate the sum of the observed pairwise agreements ∘ AP SUM = sum AP i = AP A + AP B + AP C = 3 ; Calculate the maximum number of matching editor pairs: ∘ max_pairs = n editoren * n editoren − 1 / / 2 o e.g. a total of 5 editors ==> max_pairs = 5*(5-1) / 2 = 10; Calculate the percentage of pairwise agreement between the editors o pct_agreement = sum_AP / max_pairs, so e.g. pct_agreement = 3 / 10 = 0.3; o

[0085] Calculate the overall confidence threshold for the annotated audio element of interest The expected number of agreeing editors (as obtained from the binomial distribution) is multiplied by the degree of disagreement between editors regarding the specific audio element. Threshold = E(X=N) binomial * (1-pct_agreement); The higher the expected value of agreeing editors (if there are X voting editors), the higher the threshold; The lower the pct_agreement, the higher the threshold; e.g., observed number of agreeing editors = 3; total number of voting annotators = 5 ==>Threshold = E(X=5) * (1-0.3) = 3*0.7 = 2.1; Since the number of agreeing annotators is 3 ==> 3 > 2.1 => the utterance is added to the ground truth. (In Example 1 there is a low p(INTENTglobal) = 3 / 5 score - for a higher global intent score see Example 2.

[0086] Example 2: same statement with same editor note: e.g. {Editor1: "Intent A", Editor2: "Intent A", Editor3: "Intent B", Editor4: "Intent C", Editor5: "Intent A"}; But a higher global annotator confidence p(INTENT global ) = 5 / 6 score; → Same pct_agreement, but different expected number of approving editors: E X = 5 = N * p INTENT global = 5 * 6 / 7 ; A different threshold is calculated: Threshold = E(X=5) * (1-0.3) = 4.3*0.7= 3.0 --> 3 !> 3 --> the audio element is not stored in the ground truth;

[0087] Additional metrics can be calculated for quantification.

[0088] One of these additional metrics is the so-called "Entity_Level_Score", which is described below: Entity Agreement: Final - Selected metric for the entity level "inter-annotator-agreement": "Entity and entity value agreement for a single utterance (based on an entropy score)." Example:

[0089] Audio signal: "Switch TV to the next channel" editor Entity Type Entity Value A Device TV A Direction next B station TV C Device TV D Device TV D Direction next E Direction on next Entity = Entity Type + Entity Value 1.) For each token, calculate the frequency of each entity per token: e.g., token-TV: f(device) = 3 / 4; f(channel) = 1 / 4; token-next: f(direction) = 3 / 3 = 1; token-on: f(direction) = 1 / 1 = 1; 2.) For each token, calculate the cross-entropy across all entities. The cross-entropy for tokens without annotations is 0: cross-entropy-switch = 0; cross − entropy − TV = − sum f Gerät * log ( f Gerät + f Station * log f Station = − sum 3 / 4 * log 3 / 4 + 1 / 4 * log 1 / 4 = 0,24 ; cross-entropy-auf = − sum f auf * log f auf = − sum [ 1 * log 1 = 0 ; cross-entropy-sender = 0; 3.) Calculate the sum of all cross-entropies for each token and divide by the number of tokens: 4.) Convert the entropy score into an agreement score between 0 and 1: entity_token_agreement_score = 1 - avg_entropy = 1 - 0.05 = 0.95; 5.) Calculate the entropy of the number of tokens per entity selected by the editors: For each entity, calculate the frequency of the number of tokens per entity: ∘ e.g., entity-device: ∘ entity-device: f(device: 1 token = TV) = 3 / 3 = 1; ∘ entity-direction: f(direction: 1 token = next) = 2 / 3, f(direction: 2 tokens = on next) = 1 / 3; ∘ entity-Station: f(Station: 1 Token=TV) = 1 / 1 = 1; 6.) Calculate the cross_entropy for each entity: 7.) Calculate the total token_count entropy: 8.) Calculate the value for the token_count agreement: 9.) Calculate the total score for entity-entity-value agreement between annotators: TS entity = entity_token_agreement_score + token_number_agreement / 2 = 0,95 + 0,72 / 2 = 0,84 ; 10.) Transfer utterance to ground truth if (TS entity > TH entity ), especially if TS Intent > THntent also holds.

[0090] Additional metrics can be calculated for quantification.

[0091] One of these additional metrics is the so-called "scoring_metric": "normalized agreement score (based on a single utterance)." This is a metric for measuring the agreement between the editors for the entities.

[0092] Consider each audio element, each utterance, as a topic voted on by the editors. Input: (audio element / utterance, key-value pairs) ∘ e.g. utterance: "What will the weather be like in New York tomorrow?"; ∘ e.g. key-value pairs: [Editor1: {(nuance_CALENDARX, "tomorrow"), (location, "New York")}, Editor2: {("tomorrow", nuance_CALENDARX)}, ..., Editor5: {("tomorrow", nuance_CALENDARX), ("weather", weather_condition)}] ∘ Editor index: e = 1, ..., n editors; ∘ KVP ej : (key ej , val ej ) (key-value pair j of editor e); ∘ vals := {val ej} (all values ​​as specified by the editors for this utterance); ∘ keys := {key ej} (all values ​​specified by the editors for this utterance); 1. Tokenize the utterance using the values ​​provided by the editors, e.g., "What will the weather be like in New York tomorrow?" "What", "is", "tomorrow", "the", "weather", "in", "New York"; tokens := tokenize(Utterance, Values); tokeni (i-th token in Utterance); 2. Filter tokens that have not been annotated tokens := intersect(tokens, vals); 3. Add key-value pairs to editors for tokens that they have not annotated with the "no_entity" key: for e in range(n editor ): ∘ missing_tokens := intersect({val ej}, tokens); ∘ KVP em := ("no_entity", missing_tokens m ) form = 1,..., len (missing_tokens); e.g., For example, the key-value pair (no_entity, "Weather") must be valid for all editors except Editor 5, who (probably) incorrectly identified "Weather" as an entity; 4. Calculate normalized_agreement_score for each token and average score := 0; score += normalized_agreement_score({val ej}) for j = 1,...,len(vals); score / = len(tokens); 5. Return score return score; . Handling annotation service flags

[0093] For example, for missing values, incorrect grammar, etc., editors can set a flag (clicking the checkbox). While the checkbox is currently treated as a Boolean value (TRUE, FALSE, NULL), all flag values ​​must be defined as continuous values ​​between 0 and 1 for the utterances that are transferred to the ground truth.

[0094] The continuous value corresponds to the ratio of set flags to the true number of annotators who annotated the utterance. If number_of_flag == 0: then the returned value is set to NULL (since the complete absence of flags could either mean that all annotators left the flag set to FALSE because they were unsure whether to set it, or all annotators deliberately deselected it). Otherwise: pct_flag_selected = number_of_flags_TRUE / annotators return pct_flag_selected → the resulting flag value available in the ground truth may be 0 <pct_flag_selected<=1 oder NULL sein. Handling missing values:

[0095] Some annotations may have been overlooked by annotators. However, it can also happen—especially with entity annotation—that an annotator intentionally didn't assign an entity to a token. A simple solution is to calculate the normalized agreement score twice and take the average. Once by removing the missing values, resulting in a smaller number of annotators, which slightly increases the score. And a second time, replacing the missing values ​​accordingly. In the case of the intent annotation, this would result in a "NO_INTENT_ANNOTATED" intent; in the case of the entity annotation, this would result in key-value pairs for the missing annotators, where each key is "no_entity." Assign intents:

[0096] The inter-annotator agreement score for intents is designed so that for values ​​greater than 0, there is a "dominant" intent—an intent that most annotators assign to the respective utterance. This intent is considered consistent if the score exceeds a defined threshold and can therefore be assigned to the utterance.

Claims

1. A method for generating a ground truth data set for a virtual assistant, wherein the elements of the ground truth data set comprise possible user queries to the virtual assistant, wherein the method can be executed at least partially on a computer unit and comprises the following steps: • Providing queries from users to the computer unit, wherein said queries are such as are or were intended for the virtual assistant; • Analysing the queries with regard to their suitability for inclusion in the ground truth data; wherein the analysis is performed by at least two virtual editors, wherein each of the virtual editors prepares its own analysis result for each query; • Implementing means for evaluating the results of analysis on the computer unit, wherein the means for evaluating calculate a value TS that represents a probability that the query is suitable for the ground truth data set, wherein the calculation is carried out based on the results of analysis of the at least two virtual editors for the respective query; • Comparing the calculated value TS with a predefined threshold value TH; • Adding the query to the ground truth data set if the calculated value TS is greater than the threshold TH, characterized in that the virtual editors are implemented by an artificial intelligence which was trained based on the input of human editors, in particular a virtual editor was only trained based on the inputs of a single human editor.

2. The method according to Claim 1, characterized in that the queries are taken from a history of real user queries which have been stored in a database.

3. The method according to one of the preceding claims, characterized in that means for filtering queries are implemented on the computer unit and / or a server, wherein the queries provided are a filtered subset of a total set of real user queries.

4. The method according to Claim 3, characterized in that the means for filtering filter out user queries that have the following tags: poor quality of the audio signal, length of the audio signal too long or too short, and / or no response from the user.

5. The method according to any one of the preceding claims, characterized in that the queries are based on text input or voice input.

6. The method according to any one of the preceding claims, characterized in that the threshold value and / or a number of the virtual editors are changed by correspondingly configured APIs.

7. The method according to any one of the preceding claims, characterized in that the queries are additionally analysed by human editors with regard to suitability for inclusion in the ground truth data.

8. The method according to any one of the preceding claims, characterized in that an artificial intelligence of the virtual editors is trained with various models and / or various parameters.

9. The method according to any one of the preceding claims, characterized in that the virtual editors are updated regularly, particularly when a new ground truth data set has been generated.

10. The method according to any one of the preceding claims, characterized in that the ground truth data set generated is used for training specific models of the virtual assistant.