Machine learning systems for digital assistants

Abstract

To provide a method and program for training a machine learning system for a digital assistant.SOLUTION: A machine learning system 310 is based on an encoder-decoder sequence-to-sequence neural network architecture 346 that is trained to map input sequence data 344 to output sequence data 348. The input sequence data relates to an initial query, and the output sequence data represents canonical data representation for the query. A training method includes the steps of: generating a training dataset for the machine learning system on the basis of an original query data sample; and clustering vector representation of the query data sample to generate a pair of the original query and the canonical query for use in training the machine learning system.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] Field of Disclosure The present technology is in the field of machine learning systems used with digital assistants. For example, the present technology can be used to realize voice assistants and text-based digital conversational agents. A particular example relates to a method for training a machine learning system to map an initial query to a processed form. Another example relates to a trained machine learning system used in realizing a digital assistant. [Background technology]

[0002] background Digital assistants, such as those that provide natural language interfaces to humans, are rapidly becoming a part of everyday life for many people. For example, recent advances in computing have made many long-desired voice-controlled applications feasible. Improved statistical models, including practical frameworks for effective neural network architectures, have significantly improved the accuracy and reliability of previous natural language processing systems. This has been coupled with the rise of wide-area computer networks that offer a variety of modular services accessible simply through application programming interfaces. Voice assistants, digital conversational agents (so-called "chatbots"), and automated text response systems are often the preferred means for connecting with automated computer systems. Voice-controlled devices have become prevalent in the home, where they are often used to ask questions, play music, and set reminders, and they are also beginning to gain a foothold in the automotive sector. Because speech is a natural way of human communication, voice control offers many advantages over traditional user interfaces (such as keyboards and mice, or more recently, capacitive touchscreens). By controlling devices using speech, users can perform multiple tasks, such as safely operating a motor vehicle or cooking a meal.

[0003] Despite these advances, users often complain that current dialogue systems lack human-level responsiveness and intelligence. Natural language processing typically involves a complex processing pipeline, and errors at any stage can derail successful machine interpretation. Many of these challenges are not immediately apparent to adult humans, who can process speech and text without conscious thought using cortical and subcortical structures. However, engineers in the field quickly become aware of the gap between human capabilities and state-of-the-art machine processing.

[0004] One challenge in implementing a successful digital assistant is the inherent information noise of the environment. For example, neural networks are notoriously weak, especially when presented with patterns that were not present in their training data. However, the real world is rife with inherent uncertainties, which often lead to different error rates between controlled academic environments and real-world home, industrial, or vehicular environments. In particular, digital assistants often struggle to cope with overlapping conversations, ill-formed queries, interruptions, or digressions, which account for a high proportion of real-world voice and text input. For example, radio or television broadcasts can easily scramble voice queries, and ill-formed sentences can confuse many chatbots.

[0005] Therefore, it is desirable to improve the capabilities of digital assistants to provide more robust interaction devices, and further to provide improved natural language interfaces that can accommodate noisy text queries. Summary of the Invention

[0006] overview According to one aspect of the disclosure, there is a method of training a machine learning system for use with a digital assistant, the method comprising: obtaining training data including query data samples; obtaining vector representations of the query data samples; clustering the vector representations; determining canonical queries and corresponding query groups based on the clustered vector representations; generating paired data samples based on selected canonical queries and selections from the query groups corresponding to each selected canonical query; and training an encoder-decoder sequence-to-sequence neural network architecture using the paired data samples, wherein the selections from the query groups in the paired data samples are provided as input sequence data and the corresponding canonical queries are provided as output sequence data, and the trained encoder-decoder sequence-to-sequence neural network architecture, in conjunction with the digital assistant, is usable to map data representing an initial query to data representing revised queries, and the data representing the revised queries is further processed to provide a response to the initial query.

[0007] The method of the above aspect results in a trained machine learning system for processing queries that improves performance by mapping ill-formed and possibly noisy initial queries to canonical, i.e., representative, queries. This output canonical query can then be fed into a query processing pipeline to facilitate computation of a response. The method generates effective training data for the machine learning system by converting query data samples into corresponding vector representations and then clustering these vector representations in a vector space. The clusters generated by the clustering group queries, allowing one member in each group to be selected as the canonical query. This then generates, for example, for each group, a set of queries paired with the group's canonical query. This set of queries and the canonical query for each group are then used, respectively, to generate source and target data samples for the machine learning system, allowing the system to learn how to map ill-formed queries with more variation to more regularly formed queries that match commonly used requests.

[0008] In a particular example, obtaining the training data comprises obtaining text data pairs representing a query and a response, and obtaining vector representations of the query data samples comprises converting the text data pairs into corresponding vector representations. The use of textual query data samples means that the present technology is applicable to many different forms of dialogue and conversational agents, including text-based chatbots and voice systems, including speech-to-text systems.

[0009] In the case of text, the method may further comprise performing named entity extraction on the text data and replacing the text data of tagged named entities with named entity type tags. Performing named entity extraction to replace apparent token values ​​(e.g., "Toronto") with their corresponding generic named entity type tags (e.g., <location>) may help to map multiple different query data samples, for example collected from different geographic regions, to common or shared expressions that may facilitate clustering.

[0010] In certain cases, the first portion of the text data represents the output of a speech-to-text process performed on audio data of a voice query, the second portion of the text data includes data used in providing the response to the voice query, and the trained encoder-decoder sequence-to-sequence neural machine learning model, in conjunction with the digital assistant, is usable to map data representing an initial voice query to data representing a modified voice query. The present disclosure is particularly advantageous in voice applications where input queries are likely to be malformed due to inherent noise present both in the real world and in voice-to-text systems.

[0011] In a particular example, the given vector representation includes at least a response vector representation, the response vector representation being a vector representation of the data representing a response to the query, the response vector representation being paired with data representing a corresponding query, and clustering the vector representations includes clustering the response vector representations based on a distance between the response vector representations in a vector space, and for a given cluster in the clustered vector representations, a canonical query is selected based on the frequency of the query data paired with the response vector representation in the given cluster. By initially clustering based on responses rather than preceding queries, more effective grouping of potentially diverse input queries can be performed, facilitating subsequent selection of a canonical query for the group as the most frequent representation that provides a response. Selecting a canonical query based on frequency can also improve the accuracy of downstream query processing and responses, because many downstream machine learning systems perform better on data that occurs more frequently (e.g., because it occurs more frequently in a training corpus, which is easier to drive parameter learning).

[0012] In particular examples, clustering the vector representations comprises applying a hierarchical clustering technique to iteratively combine separate clusters, which can improve the quality of the clusters, for example, by first forming tight clusters around related data points and then combining these tight clusters as part of an iterative process.

[0013] In some cases, obtaining a vector representation comprises obtaining a text representation of at least the response to the query, generating an embedding vector for the text representation, and projecting the embedding vector into a lower-dimensional vector space to output the vector representation. Testing has shown that compressing the vector representation leads to more efficient clustering (in the lower-dimensional vector space) and helps remove surface noise that can result in low-quality clusters. In some cases, generating an embedding vector for the text representation comprises tokenizing the text representation and applying a transformer neural network architecture to the tokenized text representation to generate the embedding vector. Thus, an encoder neural network architecture similar to, or even a replica of, the encoder portion of an encoder-decoder sequence-to-sequence neural network architecture can be used to generate useful sentence-level embeddings, for example, representing queries and their responses. Using an encoder neural network architecture in this manner can result in a vector representation that is more useful than, for example, the average word embeddings in the text representation. In that case, the "sentence embeddings" from the initial encoder can form the input to the dimensionality reduction stage.

[0014] In some cases, projecting the embedding vector into a lower-dimensional vector space comprises selecting a subset of principal components, the principal components being determined after principal component analysis of the query data sample. Thus, principal component analysis provides a useful tool for performing dimensionality reduction, removing components that do not significantly contribute to the variation of the original query data sample.

[0015] In one example, clustering the vector representations includes performing a first stage of clustering based on vector representations of responses to queries and a second stage of clustering based on vector representations of the queries preceding the responses. This two-stage clustering results in accurate and useful query clusters for canonical query selection. The first clustering of responses can group queries with similar functions or desired results. The second stage of clustering of queries themselves then generates query groups that can be easily mapped onto canonical queries. For example, without the second stage, the group of queries would be large. By clustering queries, source samples and target samples can be closer, making training more effective.

[0016] In some cases, the method may include filtering the clustered vector representations before generating the paired data samples. This may include, for a given cluster, determining the centroid of the given cluster in the clustered vector representation and deallocating vector representations of queries for the given cluster that are greater than or equal to a predefined distance from the centroid. Additionally or alternatively, it may also include, for a given cluster, determining the size of the given cluster and deallocating vector representations of queries for the given cluster in response to the size falling below a predefined threshold. In either case, the deallocated vector representations may then be reassigned to the nearest cluster. This filtering may at least partially implement the second stage of clustering described above. These filtering heuristics have been found to generate query groups of just the right size, which improves performance during inference, for example by generating query groups of a size and composition that result in useful combinations of queries and canonical representations in training data.

[0017] In a particular example, generating paired data samples includes filtering the generated paired data samples. This may include one or more of: removing paired data samples having canonical queries whose named entity tags do not match the named entity tags in the corresponding selection from the query group; and removing paired data samples based on a comparison of the semantic distance metric of the canonical queries with the semantic distance metric of the corresponding selection from the query group. Filtering (i.e., post-processing) the generated dataset of source queries and canonical queries can improve the quality of the dataset and, consequently, improve training. Note that production implementations may include datasets with over one million training samples; at this scale, statistical noise and suboptimal source-target pairs may still exist despite the above operations. Therefore, filtering helps remove such pairs from the dataset before training.

[0018] In certain cases, the method further comprises optimizing one or more of the following metrics: one or more clustering distance thresholds, one or more cluster size thresholds, and one or more frequency thresholds for canonical query selection. These parameters may be optimized per domain, implementation, and / or application. Optimization for a specific use case can improve performance for that use case. Manual or automatic optimization techniques may be applied.

[0019] According to another aspect of the present disclosure, there is provided a method for processing queries for a digital assistant, the method comprising: receiving initial data representing a query; generating input sequence data from the initial data representing the query; and mapping the input sequence data to output sequence data using an encoder-decoder sequence-to-sequence neural network architecture, the output sequence data representing a canonical data representation of the query; the method further comprising processing the canonical data representation to obtain a response to the query; and providing the response to the query. Thus, a trained neural network architecture, e.g., an architecture trained according to the first aspect or another method described above, may be used to pre-process incoming queries to map them to canonical data representations that are more likely to elicit a successful response. Thus, the method may be considered a form of pre-filtering or noise reduction that improves digital assistants, such as chatbots or voice assistants.

[0020] In some cases, the encoder-decoder sequence-to-sequence neural network architecture is trained on paired data samples representing a canonical query and a corresponding initial query, where the paired data samples are generated by clustering vector representations of the query data samples, the vector representations including data representing the query and data representing a response to the query. In this case, the neural network architecture may be trained according to the training method described above to obtain the benefits described above.

[0021] In some cases, processing the canonical data representation to determine a response to the query comprises one or more of the following steps: selecting one of a top-K set of output sequences provided in the output sequence data, parsing data derived from at least the output sequence data with a language model and selecting an output sequence based on scoring applied by the language model, and parsing the initial data representing the query and data derived from at least the output sequence data with a named entity extractor and selecting an output sequence based on a comparison between named entities in the parsed data. These sets of post-processing operations, which may be applied individually or (preferably) together, can further improve the output selected for query processing, for example by ensuring that elements of the original input query are preserved to provide an appropriate response and / or by selecting one of a set of possible outputs generated by a neural network architecture that maximizes the likelihood of responding with an accurate and appropriate query response.

[0022] According to another aspect of the present disclosure, a machine learning system for a digital assistant includes a query interface for receiving initial data representing a query and outputting input sequence data derived from the initial data; and an encoder-decoder sequence-to-sequence neural network architecture for mapping the input sequence data to output sequence data, the output sequence data representing a canonical data representation of the query. The machine learning system further includes an output interface for providing the output sequence data for use in providing a response to the query. The query interface may include a speech-to-text interface. The machine learning system can be provided in a local client (and in certain cases, embedded) implementation and / or a distributed implementation of the digital assistant to improve the assistance provided and enable more natural interactions. The machine learning system may be configured to apply related aspects of any of the above methods.

[0023] In some cases, the machine learning system includes an output sequence filter for receiving a plurality of output sequences from the encoder-decoder sequence-to-sequence neural network architecture and selecting one of the plurality of output sequences based on at least one natural language processing metric, and the output sequence filter may provide the post-processing to provide the benefits.

[0024] In yet another aspect, a non-transitory computer-readable storage medium is provided that stores instructions that, when executed by at least one processor, cause the at least one processor to: obtain training data including query data samples; obtain vector representations of the query data samples; cluster the vector representations; determine canonical queries and corresponding query groups based on the clustered vector representations; generate paired data samples based on selected canonical queries and selections from the query groups corresponding to each selected canonical query; and train an encoder-decoder sequence-to-sequence neural network architecture using the paired data samples, wherein the selections from the query groups in the paired data samples are provided as input sequence data and the corresponding canonical queries are provided as output sequence data, and the trained encoder-decoder sequence-to-sequence neural machine learning model is usable to map data representing an initial query to data representing revised queries in a query interface, and the data representing the revised queries is further processed to provide a response to the initial query. In certain cases, the query data sample relates to a voice query. For example, the at least one processor may comprise a microprocessor or an embedded processor of a voice-controlled device. In other examples, a non-transitory computer-readable storage medium may be provided that stores instructions that, when executed by at least one processor, cause the at least one processor to perform any one of the methods described above. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a schematic diagram illustrating a query processor for a digital assistant according to an example. [Figure 2] FIG. 1 is a schematic diagram illustrating a voice assistant according to an example. [Figure 3] FIG. 1 is a schematic diagram illustrating an example of a neural network preprocessor for use with a digital assistant. [Figure 4] FIG. 1 is a flow diagram illustrating a method for training a machine learning system for use with a digital assistant according to an example. [Figure 5] FIG. 1 is a schematic diagram illustrating an example of a process flow for generating training data for a machine learning system. [Figure 6A] FIG. 5 is a schematic diagram illustrating an example of data processed according to the method of FIG. 4. [Figure 6B] FIG. 5 is a schematic diagram illustrating an example of data processed according to the method of FIG. 4. [Figure 6C] FIG. 5 is a schematic diagram illustrating an example of data processed according to the method of FIG. 4. [Figure 7A] FIG. 1 is a schematic diagram illustrating various aspects of a computing device adapted to process a set of instructions stored on a non-transitory computer-readable storage medium according to an example. [Figure 7B] FIG. 1 is a schematic diagram illustrating various aspects of a computing device adapted to process a set of instructions stored on a non-transitory computer-readable storage medium according to an example. DETAILED DESCRIPTION OF THE INVENTION

[0026] Detailed Description Introduction Various examples of the present technology that demonstrate various interesting aspects are described below. In general, the examples allow the described aspects to be used in any combination. All statements herein that describe principles, aspects, and examples are intended to encompass both structural and functional equivalents thereof. Furthermore, such equivalents are intended to encompass both currently known equivalents and equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0027] It should be noted that, as used herein, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. References throughout this specification to "one," "an," "particular," "various," "in some cases," "examples," or similar phrases mean that the particular aspect, feature, structure, or characteristic described in connection with a particular example is included in at least one embodiment of the disclosed invention. Thus, appearances of the phrases "in some cases," "in at least one example," "in one example," "in a particular instance," and similar phrases throughout this specification may, but do not necessarily, all refer to the same or similar embodiments. Furthermore, the aspects and examples described herein are merely illustrative and should not be construed as limiting the scope or spirit of the claims, as would be understood by one of ordinary skill in the art. The disclosed embodiments may be made or used in virtually any example that incorporates any novel aspects described herein. Furthermore, to the extent that the terms "including," "includes," "having," "has," "with," or variations thereof are used in any of the detailed description and claims, such terms are intended to be inclusive, similar to the term "comprising." In examples showing multiple similar elements, even though separate reference numerals are used, some such examples may address a single element serving the functions of those multiple similar elements.

[0028] Certain examples described herein provide a neural network-based data preprocessor for a digital assistant that improves the digital assistant's performance by mapping "noisy" input data to qualified input data for practical processing. Certain examples describe how training data can be generated that allows an encoder-decoder sequence-to-sequence neural network architecture (such as one based on a state-of-the-art neural network translation system) to be adapted to operate as this neural network-based data preprocessor, i.e., via training using the training data. Certain examples can take an existing collection of input-output pairs for a digital assistant, representing, for example, queries to the assistant and corresponding responses, and use these to generate additional training data sets for input preprocessing. These examples can cluster vector representations of data in the input-output pairs to generate new source-target pairs that teach the neural network-based data preprocessor how to map real-world inputs (sources) to preferred formats (targets).

[0029] Digital Assistant Example 1 illustrates a first example 100 of a system that can be used to implement one or more digital assistants. Although a distributed digital assistant system is shown in this example 100, implementations are not limited to distributed processing and may instead use exclusively local hardware processing.

[0030] FIG. 1 illustrates multiple client devices 110 that provide a query interface to a digital assistant. The client devices 110 may comprise a variety of computing devices, including embedded computing devices, e.g., processing resources within electronic circuitry located within a non-computing device. In FIG. 1, the client devices 110 illustratively include a personal computing device 112, a wearable device 114, a motor vehicle 116, and a home assistant 118. These examples are not limiting. For example, the client devices may alternatively include “smart” appliances such as refrigerators and televisions, security appliances, restaurant ordering stations, vending machines, desktop and office computing devices, cameras, and robotic devices. The personal computing device 112 may comprise a cellular communication device and / or a wireless networking device. The personal computing device 112 may comprise a smartphone, tablet, laptop, or desktop device. The wearable device 114 in FIG. 1 is shown as a so-called “smart” watch, but may alternatively comprise devices such as head-worn user interface devices, earphones, wearable fitness trackers, and “smart” clothing with embedded circuitry. The motor vehicle 116 is shown as a car, for example, the client device may be embedded in a vehicle control system, although other vehicle types and formats may be used, for example, an aircraft or watercraft. The vehicle may be manually controlled and / or may have autonomous capabilities (e.g., an aerial "drone"). The home assistant 118 may include audio devices for use in the home.

[0031] The query interface provided by the client device 110 allows the user to query the digital assistant. In certain examples, the query interface may comprise a voice query interface. For example, the user may speak to the motor vehicle 116 or the home assistant 118, and the utterances may comprise voice queries, such as requests for information or requests to execute specific commands. In other examples, the query interface may comprise a text interface, such as a messaging application on the personal computing device 112. In this case, the user may enter text via an on-screen keyboard or another user input device. In either case, the query may be represented in the form of text data by the client device 110, for example, as typed and prompted or via a speech-to-text system. If the client device 110 comprises a voice control device, it may further comprise one or more microphones and an audio preprocessor.

[0032] In the example 100 of FIG. 1 , client device 110 is communicatively coupled to server device 120 via one or more networks 130. The one or more networks 130 may comprise one or more local and / or wide area networks that may be implemented using various physical technologies (e.g., wired technologies such as Ethernet and / or wireless technologies such as the Wi-Fi (IEEE 802.11) standard and cellular communication technologies). In certain cases, the one or more networks 130 may comprise a combination of one or more private and public networks, such as the Internet. Client device 110 and server device 120 may communicate over the networks using various technologies and communication paths. Server device 120 may comprise a computing device having one or more central processing units and / or graphics processing units and memory resources.

[0033] 1, server device 120 includes network interface 122 and query processor 124. Network interface 122 may include a communications interface and an implementation of one or more communications protocols (e.g., in a multi-layer communications stack). Network interface 122 is configured to receive data from client device 110 over network 130. In one implementation, network interface 122 may include a wired or wireless physical interface and one or more communications protocols that provide a method for receiving requests in a predefined format. In some cases, network interface 122 may include an application layer interface operating on the Internet Protocol Suite. In this case, the application layer interface may be configured to receive communications directed toward a specific Internet Protocol address that identifies server device 120, with routing based on a route name or web address being performed by one or more proxies and / or communications (e.g., “web”) servers.

[0034] 1 is configured to receive query data 140 from one or more of the client devices 110. The query data 140 may comprise text data representing a query submitted by a user, for example, by speech or text input after speech-to-text processing. In other cases, the query data 140 may comprise audio data or audio features that are converted to text data after receipt by the query processor 124. The query data 140 may also have associated metadata, such as conversation state, user location information, and other user profile information. The query processor 124 is configured to receive the query data 140 and compute response data 142 to provide a response to the query. The response data 142 may comprise text data and / or data derived from a response representation used to generate one or more of a text response, an audio response (e.g., via a text-to-speech system), and a visual response (e.g., via one or more display devices). For example, a user may request information such as "How's traffic on Route 48?", "What's the weather like today?", or "Where can I get the best vegan tacos?", and the response data 142 may be a formatted response providing information such as "Traffic is light on Route 48 right now," "It's sunny now, but there's a 50% chance of rain," or "Shelia's Snacks on X Street, three blocks away." The user may be a human or a machine (e.g., the latter may correspond to machine-to-machine communication using text or artificial voice). Note that the query processor 124 may alternatively be implemented within the client device 110 that receives the initial response (i.e., implemented locally) rather than being located within a remote server device. While various implementations may divide local and remote processing functions differently, all types may use the following example preprocessor.

[0035] In the particular example described herein, the query processor 124 includes a neural network preprocessor for processing query data prior to operation of the query processor 124. Accordingly, this application is not directed to configurations downstream of the query processor 124. The query processor 124 may include any known or future pipeline for converting a query (e.g., in text format or other form) into a response (e.g., in text format or other form) to be provided to a user in one or more of a set of configured modalities. It is also possible for the preprocessor to be tightly integrated with the query processor.

[0036] FIG. 2 illustrates an example 200 of a digital voice assistant 210 that may implement one of the client devices 110 in FIG. 1. The digital voice assistant 210 is configured to receive audio input data 220 and generate a response 230 in one or more configured modalities. In the example of FIG. 2, the digital voice assistant 210 may be configured to provide a response to a query as one or more of an audio output 232 (e.g., a voice response), a text response 234, and / or a visual response 236 (such as one or more frames of video). Various modalities may be provided by implementations of the digital voice assistant 210. In the example 200 of FIG. 2, the digital voice assistant 210 includes an audio capture device 242, a speech-to-text system 244, a neural network preprocessor 246, a query processor 248, and a response interface 250.

[0037] The audio capture device 242 may include one or more microphones configured to generate audio data comprising time-series measurements, such as time-series pressure fluctuation measurements and / or time-series frequency measurements. For example, one or more channels of pulse-code modulation (PCM) data may be captured at a predefined sampling rate (e.g., 8 kHz or 44.1 kHz), with each sample represented by a predefined number of bits (e.g., 8, 16, or 24 bits per sample, each sample comprising an integer or floating-point value). The audio data may be processed after capture, for example, by filtering in one or more of the time and frequency domains, by applying beamforming and noise reduction, and / or by filtering and normalization. In some cases, the audio data may be converted into time-series measurements in the frequency domain, for example, by performing a fast Fourier transform to create one or more frames of spectrogram data. In certain cases, a filter bank may be applied to determine the values ​​of one or more frequency-domain features, such as Mel-frequency cepstral coefficients. The audio data described herein may comprise any measurements obtained along an audio processing pipeline.

[0038] The speech-to-text system 244 receives audio data and translates it into text data output. For example, the speech-to-text system 244 may comprise an automatic transcription system based on one or more of a hidden Markov model and a neural network architecture (e.g., convolutional and recurrent neural network encoders). In this example, the speech-to-text system 244 provides the text data output to the neural network preprocessor 246. The neural network preprocessor 246 comprises a neural network architecture for processing the text data received from the speech-to-text system 244 prior to invocation of the query processor 248. For example, the neural network preprocessor 246 may convert a first sequence of text data received from the speech-to-text system 244 into a second sequence of text data suitable for providing to the query processor 248. The neural network preprocessor 246 may operate similarly to a machine translation system, except that both sequences of text data are preferably in the same language. The neural network preprocessor 246 is trained to map sequences of text data into a more common or "canonical" form. For example, the neural network preprocessor 246 may operate in inference mode to map noisy or ill-formed input text data into well-formed or standardized text data. The canonical form may correspond to a more frequent, clean, and / or expected version of the original query. For example, if a user asks about the weather with a television in the background, the text data output by the speech-to-text system 244 may be similar to "Here...on the channel...um...how's the weather...later...", which may be mapped by the neural network preprocessor 246 to "How's the weather?"In this way, the neural network preprocessor 246 maps the query data into canonical query data that can be more accurately and efficiently processed by the query processor 248.

[0039] 2, query processor 248 may operate similarly to query processor 124 of FIG. 1. Query processor 248 receives mapped text data from neural network preprocessor 246 to generate data usable to derive a response to the original query. For example, this may comprise text data provided to response interface 250 for rendering into configured and / or available modalities. Query processor 248 may perform local processing to provide a response as shown in FIG. 1 and / or make network requests to obtain data for providing the response. Query processor 248 may output the response as pre-rendered data (e.g., audio and / or video samples), or response interface 250 may comprise a text modality system (such as a text-to-speech or text-to-visual system) or other vectorized information representation system (such as Musical Instrument Digital Interface (MIDI) music, Scalable Vector Graphics (SVG) graphics, or Flash animation) to provide response 230.

[0040] Examples of machine learning systems FIG. 3 illustrates an example 300 of a machine learning system 310, which may be used to provide the neural network preprocessor 246 of FIG. 2 or form part of the query processor 124 of FIG. 1. The machine learning system 310 may form part of or be used in conjunction with a digital assistant. The machine learning system 310 is configured to map initial query data 320 to modified query data 330 in an inference mode. The machine learning system 310 includes a query interface 342, an encoder-decoder sequence-to-sequence neural network architecture 346, and an output interface 350. The query interface 342 is configured to receive the initial query data 320 and provide input sequence data 344 derived from the initial query data 320. In some cases, the initial query data 320 comprises text data, and the query interface 342 is configured to parse the text data to provide input sequence data 344 in the form of a sequence of numbers. The query interface 342 may comprise a tokenizer and / or embedding layer such that the input sequence data 344 comprises a sequence of integers (e.g., representing token indices) or vectors (e.g., representing embedding vectors). In certain cases, tokenization and / or embedding may form part of the neural network architecture 346 and thus may be omitted by the query interface 342. The query interface 342 may comprise an internal interface (e.g., an application programming interface or a function interface) or may comprise a remote interface (e.g., a network interface in a distributed system).

[0041] The encoder-decoder sequence-to-sequence neural network architecture 346 is configured to map input sequence data 344 to output sequence data 348. The neural network architecture is referred to as "sequence-to-sequence" because it maps an input sequence of tokens (which may be of variable length) to an output sequence of tokens (which may also be of variable length, possibly different from the input sequence length). Furthermore, the neural network architecture is referred to as an encoder-decoder architecture because it includes a first set of neural network layers that operate to "encode" the input sequence as a fixed-size vector, e.g., using neural attention layers, and then "decode" this fixed-size vector, e.g., again using neural attention layers, to generate an output sequence. The input sequence data 344 may have a sequence length that is different from or the same as the sequence length of the output sequence data 348. The sequence length may vary between inputs (and the output sequence length may also vary). In many cases, the sequence length of output sequence data 348 may be less than the sequence length of input sequence data 344, although control over the length of output sequence data 348 may be provided by neural network architecture 346 itself. For example, neural network architecture 346 may be trained using special control tokens such as <start> and <stop> to represent the beginning and end of a sequence. Thus, output values ​​forming output sequence data 348 may be read from neural network architecture 346 until a <stop> value is output. In certain cases, neural network architecture 346 may output multiple possible output sequences, such as a set of top K sequences (K is a configurable parameter).

[0042] In general, output sequence data 348 represents a canonical data representation of the query. As shown in FIG. 3, it may be in a similar format to input sequence data 344, and may comprise, for example, a sequence of integer token indexes and / or embedding vectors. Output interface 350 is configured to process output sequence data 350 for use in providing a response to the query. In example 300 of FIG. 3, output interface 350 receives output sequence data 348 and generates modified query data 330. For example, output interface 350 may convert output sequence data 348 back into text data, for example, by replacing token indexes with corresponding text tokens and / or by mapping embedding vectors to selections from a dictionary of tokens.

[0043] In some cases, the encoder-decoder sequence-to-sequence neural network architecture 346 may comprise a neural machine translation system trained (e.g., from scratch or fine-tuned using pre-trained parameters) with a specific training set that allows it to learn a same-language text mapping. The neural network architecture 346 may be based on a so-called Transformer architecture (such as one derived from the initial design presented in the paper "Attention Is All You Need" by Vaswani et al., published on the arXiv in June 2017, which is incorporated herein by reference). This Transformer architecture uses attention layers applied to sequence data through successive encoding and decoding stages to provide a sequence output. Other encoder-decoder sequence-to-sequence neural network architectures 346 (such as those based on recurrent neural network encoders and / or decoders) may also be used. Implementations may use open-source libraries and toolkits, such as PyTorch, TensorFlow, and / or the Open Neural Machine Translation Toolkit, to realize the neural network architecture 346.

[0044] In some cases, query interface 342 may implement a speech-to-text system, such as speech-to-text system 244 in Figure 2. In this case, query data 320 may initially be received as audio data, as shown in Figure 2. In other cases, query interface 342 may form part of network interface 122 in Figure 1, which is adapted to receive query data from a remote client device. Various configurations are possible.

[0045] In a particular example, one or more of the neural network architecture 346 and the output interface 350 may include an output sequence filter. The output sequence filter may be configured to receive multiple output sequences from the encoder-decoder sequence-to-sequence neural network architecture 346 and select one of the multiple output sequences based on at least one natural language processing metric. For example, the neural network architecture 346 may provide multiple possible sequences as output (e.g., by a configured post-processing stage). In some cases, a natural language metric representing the similarity between an input sequence represented in the query data 320 or the input sequence data 344 and an output sequence may be used. In this case, the output sequence that is most similar to the input sequence may be selected. In other cases, a natural language metric representing a score output by a language model may be used, which represents the level of confidence of the language model (i.e., how easy the output sequence is to parse). In this case, the output sequence with the highest language model score may be selected. A composite metric function that outputs a combined score may also be defined, and the output sequence is selected based on the highest combined score. Further details of post-processing that may be performed by the output sequence filter are provided below.

[0046] How to train a machine learning system 4 illustrates an example method 400 for training a machine learning system for use with a digital assistant. For example, the machine learning system may comprise machine learning system 310 of FIG. 3 or may form part of one or more of query processor 124 of FIG. 1 or neural network preprocessor 246 of FIG. 3. Method 400 may be embodied as computer program code executed by one or more processors. The computer program code may form a set of instructions stored in a non-transitory computer-readable medium.

[0047] At block 405, training data is obtained. The training data comprises query data samples. The query data samples may comprise data derived from text data representing at least a query. In some cases, each query data sample may also comprise data derived from text data representing a response to a query associated with the query data sample. Each query data sample may comprise text data, such as a string of text representing a query submitted by a user of client device 110 in FIG. 1 . In other cases, the query data sample may comprise audio data representing a query or data according to another input modality. The training data may be collected from real-world queries captured by the digital assistant and / or public datasets characterizing queries and associated responses.

[0048] In block 410, a vector representation of the query data sample is obtained. This may comprise obtaining a vector encoding of one or more of the text data representing the query and the text data representing the corresponding response. The vector encoding may be generated by a neural network encoder. While the examples herein are primarily based on text-to-vector mapping, in other examples, direct audio-to-vector mapping may also be used. The neural network encoder may comprise a neural network architecture separate from the machine learning system (e.g., a system different from the neural network architecture 346) or may comprise the encoder portion of the machine learning system (e.g., the encoder portion of the encoder-decoder neural network architecture 346). In some cases, the neural network encoder may comprise a pre-trained sentence encoder (such as the universal sentence encoder described in the paper of the same name by Kale et al., published in the Archives in March 2018 and incorporated herein by reference, or the sentence-BERT encoder described below). The form of the vector encoding for block 410 may vary depending on the implementation and may not form part of the inference procedure, for example, it may be used only to create training data. The vector representation may comprise a predefined number of floating-point elements (eg, 256 to 1024) that are normalized within a predefined range (eg, 0 to 1 or −1 to 1).

[0049] In block 415, the vector representations obtained in block 410 are clustered. This may comprise clustering the vector representations in a vector space, i.e., identifying clusters based on one or more clustering functions. The clustering function may be provided by a machine learning library such as sci-kit learn. Note that clustering is typically performed in a vector space with hundreds of dimensions (e.g., corresponding to hundreds of vector elements), which is typically difficult to visualize in a two- or three-dimensional world. In some cases, a hierarchical clustering function may be used to iteratively define clusters. In some cases, the vector representations may comprise a vector representation of query data and a vector representation of accompanying response data, in which case the vector representation of the accompanying response data may be clustered in a first stage, and the vector representation of query data may be clustered in a second stage (e.g., a second stage of clustering within groups defined by the first stage clustering) using the results of the first stage clustering. The clustering may comprise evaluating a distance metric in the vector space, such as cosine similarity distance, and grouping adjacent points in the vector space based on this distance metric.

[0050] In block 420, canonical queries and corresponding query groups are determined based on the clustered vector representations. This may involve selecting clusters of vector representations related to various queries, e.g., determined in block 415, and determining one representative query data sample for each cluster. For example, the representative (canonical) query data sample may comprise a query data sample that is more frequent in the cluster and / or may have a vector representation that is closest to the aggregate vector representation of the cluster (e.g., the average vector representation of the group). In these cases, the query data sample in each cluster or group is not the selected canonical query data sample from the corresponding query group. The corresponding query group may or may not include a canonical query data sample (this does not have a significant impact on training). Preferably, the canonical query data sample is excluded.

[0051] At block 425, paired data samples are generated based on the selected canonical queries and selections from the query group corresponding to each selected canonical query. For example, for each cluster, a canonical query (i.e., a data representation of the query in text or sequence format) may be set as target or ground truth output data, and then each query in this cluster may be paired with this canonical query as source input data. Thus, at block 425, a new set of training data is generated using pairs of source data and target data, with each pair forming a data sample.

[0052] Then, in block 430, the paired data samples from block 425 are used to train an encoder-decoder sequence-to-sequence neural network architecture. Here, a selection from the query group in the paired data samples is provided as input sequence data, and the corresponding canonical query is provided as output sequence data. Thus, the encoder-decoder sequence-to-sequence neural network architecture is trained to map source data to target data. The training in block 430 may use a typical transformer or neural machine translation system training procedure. Hyperparameters for training may be determined based on recommended public values ​​and / or through experimentation. The encoder-decoder sequence-to-sequence neural network architecture may comprise, among other things, the encoder-decoder sequence-to-sequence neural network architecture 346 of FIG. 3 or may form part of the query processor 124 of FIG. 1 or the neural network preprocessor 246 of FIG. 2. The encoder-decoder sequence-to-sequence neural network architecture may be based on one or more of a one-way recurrent neural network architecture, a two-way recurrent neural network architecture, a transformer neural network architecture, and a convolutional neural network architecture. Further details of example neural network architectures are provided in the paper by Klein et al., "The OpenNMT Neural Machine Translation Toolkit: 2020 Edition" (AMTA, 2020), which is incorporated herein by reference.

[0053] The trained encoder-decoder sequence-to-sequence neural network architecture can be used in conjunction with a digital assistant to map data representing an initial query to data representing a revised query, e.g., as described with reference to Figures 2 and 3. The data representing the revised query output by the neural network architecture may be further processed to provide a response to the initial query, e.g., as described with reference to Figures 1 and 2. By providing the source and target data samples generated from the clusters in block 425, the neural network architecture is enabled to map a sequence of non-standard tokens to a standard or canonical sequence of tokens that reflects a qualifying input for successful and accurate query processing.

[0054] Process Flow Example Figure 5 illustrates an example of a process flow that may result from one implementation of the training method 400 of Figure 4. In this example, an initial training data set 510 comprises M pairs of text data, with a first portion 512 of each pair comprising query data and a second portion 514 of each pair comprising response data. This text data may be provided in a string format (e.g., a sequence of character encodings) or in a derived data format adapted to a particular processing pipeline (e.g., a pre-tokenized list of integers, etc.).

[0055] Through a first process, such as block 410 in FIG. 4, the text data pair 512, 514 is converted into a corresponding vector pair 522, 524. This may involve tokenizing the text representation in the data pair 512, 514 and applying a transformer or recurrent neural network architecture to the tokenized text representation to generate an embedding vector. Generating the embedding vector may involve applying one of the encoders described above or a sentence encoder, such as the sentence-BERT encoder described in the paper "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks" by Reimer et al., published in the Archives in August 2019 (which is incorporated herein by reference). Known public sentence encoder parameters may be used without separate training (e.g., pre-trained parameters based on a large dataset). While FIG. 5 shows vector pairs 522, 524, in certain implementations, they may be combined into a single vector representation, for example, by concatenating either the text data pair 512, 514 or the resulting vector encoding. Performing clustering on masked or limited dimensions of larger vectors, or performing two-stage clustering on separate vector portions, may be functionally equivalent, or at least provide similar usable groupings for generating suitable training data. In a particular test example, vector pair 522, 524 was generated as a 768-dimensional vector. In a particular implementation, vector pair 522, 524 may be cached to save time on data retrieval.

[0056] In certain variations, the vector pair 522, 524 (or a single concatenated vector) may be further processed before clustering. For example, in certain implementations, a dimensionality reduction preprocessor may be applied. The dimensionality reduction preprocessor may include an operation of projecting the embedding vector into a lower-dimensional vector space. For example, in some cases, principal component analysis (PCA) may be performed on the initial vector pair 522, 524 to obtain a set of linear variation components, which may be used to decompose the vector representation. In some cases, a subset of linear variation components may be selected that represent a specific, predefined amount of variation in the data represented by the initial vector pair 522, 524. The vector representation may be reduced via PCA as a single concatenated vector or separately. For example, the data formed by the query vector may be processed by a PCA fitting function, and a subset of linear variation components that represent 95% of the variation may be selected, with the elements of the transformed output representing various amounts of each of the subset of linear variation components. This operation may help reduce the dimensionality from a high-dimensional representation (e.g., 768 elements) to a low-dimensional representation (e.g., a few hundred elements), which may improve memory efficiency.

[0057] In the next stage of the process, which may result from applying block 415 in FIG. 4, the vector representations 522, 524 are clustered. FIG. 5 shows a two-dimensional schematic diagram of clustering in n-dimensional space. As noted above, n may be equal to several hundred. In some cases, a first stage of clustering based on the response vectors 524 is performed. For example, this may comprise clustering the response vectors 524. Two example clusters 532 and 534 are shown in FIG. 5. In some cases, the clustering may comprise iteratively combining separate clusters using a hierarchical clustering technique; for example, smaller clusters may be defined based on a first predefined distance threshold, and these smaller clusters may then be clustered themselves or combined with other points based on a second predefined distance threshold. These distance thresholds may comprise hyperparameters that are optimized manually and / or via an automated optimization procedure.

[0058] In one example, query samples (e.g., 512) may first be grouped based on a first specified distance criterion or threshold between points representing response vectors 524. This may result in a first set of clusters. The remaining points representing response vectors 524 may then be clustered using a second specified distance criterion or threshold. The second distance criterion may be larger than the first distance criterion, but the second distance criterion may be a lower value to group nearly identical responses. The hierarchical clustering function may use cosine similarity distance as the distance criterion for clustering. In this example, initial tight clustering ensures that only very similar responses are grouped together. However, this may place other similar responses into separate clusters. Therefore, this clustering operation may be performed multiple times to combine similar response clusters that are initially separate. In some cases, text responses may be clustered using agglomerative clustering, and then the corresponding text queries may be clustered using the same approach. In one set of examples, various levels of clustering may be applied to various query vectors 522 and response vectors 524. For example, the initial group of query vectors may be based on clusters of response vectors (e.g., after one or more rounds of clustering in the response space), and clustering may then be performed within (or between) the initial groups in the query vector space.

[0059] 6A illustrates an example of queries grouped based on response clusters. FIG. 6A shows three clusters 601, 602, and 603 of query data (e.g., 512) for each cluster as a list of text data. In this example, the three clusters 601-603 may result from an initial stage of clustering. In subsequent iterations of clustering, clusters 601 and 603 may be merged (e.g., based on the distance between the centroids or means of each cluster and a second predefined distance threshold).

[0060] Returning to FIG. 5, clustering results in groups of queries, such as group 540 of query data. These may be similar to the groups shown in FIG. 6A (or the result of combining two groups 602 and groups 601 and 603). These groups of queries may be groups in one or more of the original data space (e.g., based on 510) or vector space. As part of block 420 in FIG. 4, groups of queries corresponding to response clusters (e.g., 532, 534) may be further processed to further refine group membership. For example, each response cluster may be obtained in turn to identify response points or vectors in these clusters, and then the query indexes corresponding to these response points or vectors (e.g., response vector i from dataset 510 and corresponding query i) are grouped according to the response cluster. In certain cases, the query vectors within each group may themselves be clustered. This may be used to select one of the groups as the canonical query for that group, for example, based on its distance to the query cluster's centroid.

[0061] In certain cases, for example, the clustered vector representations resulting from block 415 may be filtered as part of block 420 in FIG. 4. This may include determining the centroid of each cluster of query vectors 522 and deallocating vector representations of queries in a given cluster that are greater than or equal to a predefined distance from the centroid. Additionally or alternatively, this may include determining the size of each cluster of query vectors 522 and deallocating vector representations of queries in a given cluster in response to the size falling below a predefined threshold. These filtering operations may then be followed by a reallocation operation, whereby previously deallocated vector representations are reallocated to the closest cluster. This filtering may therefore be considered a second stage of clustering based on query vectors 522 as opposed to response vectors 524.

[0062] 5 also illustrates the process of selecting a canonical query for each group or cluster of queries. In FIG. 5, the canonical query is selected as the top query 542 in an ordered list 544 of queries within each group. In one example, this ordered list may be based on frequency of occurrence in the original dataset 510. In another example, this ordered list may be based on distance from the centroid of the query cluster. In yet another example, multiple metrics may be combined and evaluated. The top query 542 may be the query that occurs most frequently and / or is closest to the centroid of the query cluster.

[0063] 5 shows how the canonical query 546 (selected as the top query 542) is set as the target data sample, and the remaining queries in the group or cluster are set as source data samples 548. This may comprise selecting original text data inputs from the dataset 510 based on their corresponding clustered vector representations. The bottom of FIG. 5 shows how each source data sample 552 in the group is paired with a target data sample 554 to create multiple target-source data samples 550, which form a training set for training the neural network architecture in block 430 of FIG. 4.

[0064] In one variation, method 400 of FIG. 4 and / or the process shown in FIG. 5 further comprise filtering the generated paired data samples, such as target-source data samples 550, before training the machine learning system. This filtering may also be applied in inference mode to discard mapped output sequences that do not meet specified criteria. Having a filtering stage can improve the quality of the training set for the machine learning system and thus improve mapping during inference. When performed as a post-processing stage after inference, it also relaxes constraints on the machine learning system, for example, and may occasionally introduce errors and noise in the mapping output sequence, which can then be processed by the query processing pipeline.

[0065] In some cases, the filtering includes one or more of: removing paired data samples having canonical queries whose named entity tags do not match the named entity tags in the corresponding selection from the query group; and removing paired data samples based on a comparison of the semantic distance metric of the canonical queries with the semantic distance metric of the corresponding selection from the query group. In a first case, method 400 may include performing named entity extraction on one or more of query data 512 and response data 514, e.g., as part of blocks 405 or 410. This may be performed by applying one or more functions from available machine learning processing libraries, such as the spaCy library managed by ExplosionAI, Inc. or the Stanford Named Entity Extractor provided by Stanford University. Named entity extraction is a known syntactic analysis function that can be applied to unstructured text data to locate tokens (e.g., characters, words, or groups of words) within the text data that are associated with entities or things that have consistent referents, such as people, places, and times. In certain cases, named entity extraction is performed on both the query data 512 and the response data 514 before vectorization and clustering. In this case, method 400 may comprise replacing the tagged named entity text data with named entity type tags, such as replacing the detected names of people, places, and times with the defined tags <person>, <place>, and <time>, respectively. This helps cluster similar queries with different named entities. For example, the text strings "What's the weather like in Toronto?" and "What's the weather like in San Diego?" may both be converted to the token sequence "[<place>, of, the, weather, is, how, is,?]" and thus mapped to a common point in the vector space.

[0066] In certain cases, when named entity extraction and tag substitution are performed as described above, the original text tokens may be stored in association with (e.g., together with or indexed by) the processed text portions, such as pairs 512 and 514. After generation of dataset 550 for training the machine learning system, the original text tokens may be rewritten into the data sample. For example, after named entity parsing, query data 512 may comprise [["How's the weather in <location>?"], "Toronto"], where clustering is performed on the first list element, and then within the text tokens in one of target data 552 and source data 554, the original token "Toronto" is reinserted into the set of tokens in place of <location>, i.e., again becoming ["How's the weather in "Toronto"?"]. This is performed for both source data sample 512 and target data sample 514, with the same substitution being performed in both source data sample 512 and target data sample 514, i.e., "Toronto" is inserted in place of the <location> tag. This can also be useful, for example, to enrich the training data to improve accuracy, because source query samples with different named entities but a common processed form may be mapped to target query samples with further different named entities, and then different versions may be created using different versions of the original named entities. For example, all queries in data 601 and 603 may be mapped to one target data sample to initially generate 16 source-target pairs, and then all original <date> and <location> named entities corresponding to different dates and locations, where there may be more than 16 entities, may be inserted into different copies of the 16 source-target pairs to generate more than 16 training samples.

[0067] Additional post-processing filters may also compare the named entities in the source data sample 512 and the target data sample 514 for a given pair and use the results of this comparison to determine whether the pair should be removed. For example, if the named entity tags in the source and target text do not match, the given pair may be deleted. This ensures that any named entity tags needed to respond to the query are present in both the source and target data samples. For example, if <date> and / or <location> are present in the source query 552, they must also be present in the target query 554 to remain in the training data 550.

[0068] If a check for semantic distance is performed, it may include comparing a distance metric in a vector space between the source data sample and the target data sample and removing data sample pairs that are greater than a predefined threshold. It may also include generating language parsing metrics for both the source and target data samples and then removing data samples based on a comparison of these metrics. In yet another case, further tests may be applied to one or more of the source and target data samples, and pairs may be removed as data samples if one or more of the source and target data samples have test metrics that are below or above a predefined threshold. For example, a data sample may be removed if the confidence score of the language model provided for the target data sample is below a predefined threshold, e.g., indicating that the target data sample is difficult to parse or does not reflect well-formed text. The language model may be the language model used by one or more of the speech-to-text and text-to-speech systems associated with the digital assistant, for example, to parse voice queries and render voice responses.

[0069] Here, reference is made to post-processing that may be applied in inference mode, which may be applied to select an output sequence for query processing from a set of possible or candidate output sequences provided by the machine learning system. In some cases, a machine learning system, e.g., in the form of an encoder-decoder sequence-to-sequence neural network architecture, may have the option of outputting a set of possible or candidate output sequences as part of its standard operation. For example, the number of possible or candidate output sequences provided may be a hyperparameter of the machine learning system, corresponding to, e.g., the top-K possible output sequences. For post-processing, one or more of the filters described below may be applied.

[0070] In the first case, filtering may be performed based on a confidence or prediction score output by the machine learning system as part of its standard operation. For example, this may comprise a probability value indicating a confidence level for each possible output sequence in other comparative applications (e.g., the confidence in each of a set of translations when used in machine translation). This confidence or prediction score may then be used as a metric for selecting an output sequence for query processing. In some cases, this may be combined with a cosine similarity score generated by comparing the vector representations of the source and target data samples. A final score may be generated using a composition function, and the highest-scoring output sequence may be selected as the output of the machine learning system. In some cases, the highest-scoring output sequence may also be compared to a threshold to determine whether to use it as an output or return an error, for example, causing the digital assistant to request the user to repeat their query.

[0071] In a second case of post-processing, a named entity tag matching operation may be performed. This may be similar to the named entity tag matching operation described above applied to the training dataset 550. In this case, the named entity tag matching operation may comprise eliminating possible or candidate output sequences whose named entity tags do not match the named entity tags associated with the input sequence data (e.g., extracted from the text form of the query data 320). For example, named entity extraction may be performed on both the text form of the query data 320 and the text form of each of the possible or candidate output sequences. Preferably, the machine learning system is trained using the original named entity values ​​(e.g., “Toronto,” “12:15 PM,” “Mike”) and configured to generate an output that passes these to the output sequence. Named entity tag matching may require a match between tag types (e.g., <location>, <time>) and tag values ​​(e.g., “Toronto,” “12:15 PM”) in both the input text data and the output text data of the machine learning system. If there is no match for a given candidate output sequence, e.g., even the higher or highest scoring output sequence, then that output sequence may be removed from the list of possibilities for query processing. In addition to, or instead of, named entity matching, noun phrase matching may also be applied.

[0072] In the third case, the proposed output sequence may undergo part-of-speech validation before being passed on for query processing. This may be applied after the composite scoring and named entity matching described above. This post-processing check may be applied to ensure that any output passed on for query processing is a grammatically correct or accurate sentence.

[0073] 6B illustrates data 610 resulting from the application of part-of-speech (POS) validation. This validation is performed on a text version of the candidate output sequence, shown as query field 612. In this validation, named entity extraction is performed as described above to identify the query start ( <boq>) tag and query end ( <eoq>) tags are added. The output from this process is shown as tagged query 614. Part-of-speech tagging is then applied and the speech tags for each token are stored in an output part-of-speech list shown as 616 in FIG. 6B. Part-of-speech tagging may be performed using known natural language processing libraries such as spaCy mentioned above or the Natural Language Toolkit (NLTK) provided by Stanford University. As can be seen in FIG. 6B, the standard part-of-speech tagger uses existing named entities and custom tags ( <boq>, <Location>, <Date>, <eoq>) itself (e.g., effectively skipping part-of-speech conversion for existing tags). The part-of-speech list may then be parsed, and the output of the parsing is used to determine whether to keep the query 612 and pass it on for query processing, or whether to reject the query and select a different output or return an error.

[0074] In a third case, a parse tree-like object may be constructed from a database of queries that resulted in successful responses, e.g., query data 512 in FIG. 5. In this case, the samples in query data 512 may be converted into part-of-speech lists similar to 616 above. Thus, the set of part-of-speech lists for "successful" queries may be used as a comparison database. The part-of-speech list 616 generated for the proposed output query 612 may then be compared to this comparison data (e.g., the constructed parse tree may be traversed) to find matches (e.g., <eoq>6B , the entire part-of-speech list has been traversed (e.g., the traversed field data 620 matches the entire part-of-speech list 616), and the match has been located in the comparison database (e.g., as indicated by the identifier of the matched sequence shown in 622), so the verification is successful, as indicated by Boolean variable 618. Thus, in this case, query 612 may be determined as the preferred “translated” output query for query processing, such as being sent by query processor 124 or query processor 248 for further processing and used to generate response 142 or 230 in FIG. 1 or 2 .

[0075] As mentioned above, the approaches described herein may be adapted for use in one or more of filtering the training set 550 and filtering the possible output sequences generated by the trained machine learning system. For example, an approach described above with respect to one of training data post-processing or output sequence post-processing may also be applied to the other. In some cases, filtering the training set 550 or filtering the possible output sequences generated by the trained machine learning system may use a combination weighting function of multiple different metrics. These different metrics may be based on any combination of the criteria described above. In this case, the combination weighting function may generate a normalized combination metric value, which may be compared to a predefined threshold to determine whether to retain the training data sample and / or select an output sequence for further processing from a set of candidate output sequences. The weights of the combination weighting function may be predefined or learned hyperparameters (which may be 1 in certain cases). Post-processing is any operation that serves to (further) improve the quality of the training data 550 and the output sequences used for query processing, for example, in addition to or in synergy with the query translation or modification performed by the machine learning system described herein.

[0076] Query Preprocessing Example FIG. 6C illustrates several examples of query mapping that may be performed using a machine learning system configured and trained as described herein. The left side illustrates possible query data inputs 632, 642, and 652, which may relate to the initial query data 320 received at the query interface 342 (or the partially processed query data at said interface). The right side illustrates possible revised query data outputs 634, 644, and 654, which may relate to the revised query data 330 output in FIG. 3 (or the partially processed revised query data at the output interface 350). Note that these examples are, of course, simplified for ease of explanation, and the actual data format may differ from that shown in the implementation. In FIG. 6C, named entity extraction is performed, with named entities shown in angle brackets <...> (again, the tagging format may differ between implementations based on the parsing library utilized). In a first example, the initial input query 632, "My wife and I are going to <Seattle> on <Wednesday>. My wife asked me to check the weather before packing our suitcases." is "translated" into the canonical query 634, "What is the weather like in <Seattle> on <Wednesday>?" This canonical query proves to be well-formed and easy to process to generate a response. Similarly, in a second example, the initial input query 642, "I'm going to <San Francisco> <tomorrow>. Will it rain?" is "translated" into the canonical query 644, "Will it rain <tomorrow> in <San Francisco>?" Finally, the initial input query 652, "What's the weather like in <Seattle> <tomorrow>?" with some overlapping or confusing utterances, is mapped to the canonical query 654, "What's the weather like in <Seattle> <tomorrow>?"

[0077] While the examples described herein may be used in any form of dialog or conversational system (e.g., text chatbots used online and via short message service), they are particularly suited to voice assistants and voice query interfaces where experimentation and data collection demonstrate that unclear and incompetent speech is the norm rather than the exception (people naturally imagine that they always produce competent speech, but experiments with real-world speech recordings show that this is not in fact true and reflects a built-in positivity bias).

[0078] In Figure 6C, a third example 652, 654 is annotated to illustrate an optional named entity extraction process. In this example, input query data 652 in text format is parsed, and the tokens "Seattle" and "tomorrow" are detected as named entities with respective tag types <location> 662 and <date> 664. In this case, the original token values ​​are stored as variables 672, 674 in association with the tag types 662 and 664. Figure 6C shows the results 684 of named entity extraction performed on output 654, which may illustrate how tag types 662, 664 and tag values ​​672, 674 may be matched as part of a post-processing validation check.

[0079] In certain variations, the machine learning system may be trained on data samples that use named entity tags rather than named entity values ​​(e.g., query data such as 684 rather than 652). In this case, the machine learning system may be trained to maintain named entity tags from input to output (e.g., as a result of filtered training data in which named entity tags in the source and target match). However, this approach may not be compatible with the use of large, pre-trained neural network architectures (e.g., available pre-trained monolingual “language models” or neural translation systems) that would not initially be trained using named entity tags. When large, pre-trained neural network architectures are used (e.g., they may be preferable for more natural mappings), the training data (e.g., 550) may consist of actual named entity values ​​(e.g., as in 632-654), and named entity extraction may be performed as parallel pre- and post-processing operations. In certain cases, it may be preferable to apply so-called transfer learning training after using a pre-trained neural network architecture, where the pre-trained parameters are fine-tuned to a specific use case (in this case, query mapping). This may allow the linguistic principles embodied in pre-trained models to be preserved, and specific use cases to be quickly learned "on-top." This would be preferable to training neural network architectures from scratch, which could speed up training and improve mapping.

[0080] Server Device Example 7A and 7B illustrate an example server device 710 that may be used to implement certain methods and machine learning systems described herein. FIG. 7A illustrates an example server device 710 that may implement server device 120 in FIG. 1 or machine learning system 310 (e.g., if implemented remotely). Server device 710 may comprise a rack-mounted server blade multiprocessor server system. It may comprise multiple networked computer processors that execute software in parallel. FIG. 7B illustrates certain exemplary components of server device 710. Server device 710 comprises a multi-core cluster of computer processor (CPU) cores 712 and a multi-core cluster of graphics processor (GPU) cores 714. These processors are connected to a random access memory (RAM) device 718 via a board-level interconnect 716 for program code and data storage. Server system 710 also includes a network interface 720 that allows these processors to access the Internet. By executing instructions stored in the RAM device via interface 716, CPU 712 and GPU 714 may perform steps of methods described herein. In certain cases, the client devices described herein may have a similar general component arrangement, but in certain cases may have fewer computing resources and may not have a dedicated graphics processor 714. The client devices may have one or more CPUs that perform the steps of the methods described herein in a similar manner as the server device 710. In certain cases, the server device 710 may be used to perform training of the neural network architecture to determine sets of parameters that are then distributed to local instantiations of the neural network architecture (e.g., in the client device).

[0081] Example hyperparameters and test configurations As described above, the machine learning system may be configured with multiple hyperparameters. The values ​​of the hyperparameters may be set based on industry best practices, experimentation, and / or automatic optimization. Methods for training additional hyperparameters may include one or more of a clustering distance threshold (e.g., a cosine distance threshold), a cluster size threshold, and a frequency (e.g., a target data sample count) threshold for canonical query selection. In certain cases, different hyperparameters may be determined for different domains of operation. For example, one set of hyperparameters may be selected for the automotive domain, another set may be selected for a medical domain chatbot, and a third set may be selected for a home voice assistant.

[0082] The following are some examples of hyperparameters that may commonly be selected and / or optimized as hyperparameters of a machine learning model: word embedding sizes for input and output, encoder type (which may be selected as one of a recurrent neural network, a bidirectional neural network, a graph neural network (such as a gated graph neural network), a Transformer neural network, a convolutional neural network, and a mean neural network), decoder type (which may be selected as one of a recurrent neural network, a Transformer neural network, and a convolutional neural network), multiple encoding layers, multiple decoding layers, size of any recurrent neural network hidden state, parameter initialization (e.g., whether it is performed and / or whether it is performed with a uniform distribution with support or with Glorot initialization with a Xavier uniform distribution for Transformer architectures), batch size (e.g., maximum batch size for training), optimization method (e.g., one of stochastic gradient descent, ADA and variants, ADMA and variants), dropout (e.g., whether dropout is applied and the probability of dropout). These hyperparameters may be configured for different domains and applications through experimentation, or default settings may be used initially.

[0083] Other hyperparameters that can be varied in experiments to fine-tune performance include, among others, the learning rate, learning rate decay options, learning rate decay method (e.g., NOAM or RSQRT, if used), warm-up step for custom decay methods, whether shared weights are used (e.g., in the encoder and decoder and / or for both input and output word embeddings), floating-point bit depth (e.g., 16-bit, 32-bit, or 64-bit), fixed-point representation (e.g., 8-bit or 16-bit), recurrent neural network type (such as LSTM, GRU, or SRU), noise settings, and whether to train and copy attention layers.

[0084] The example shown in Figure 6C is based on a test implementation trained using weather domain data on 1 million training samples (e.g., source-target pairs) over 60,000 runs. Initial queries and responses were obtained from a weather dataset with 1.8 million query and response pairs (provided as text). During training, 90% of this dataset was used for training, 5% for validation, and the remaining 5% for testing. Training with 500,000 to 1 million samples was found to provide adequate results. Other tests were performed using a broader dataset with 11.8 million queries and their responses (in text format) and their domains. In the specific example, domains with fewer than 1,000 samples were removed. Other dataset filtering may include removing short responses with common, unhelpful patterns, removing responses indicating errors or invalid queries, and removing custom domains such as dictionary or translation requests. In testing, a process of grouping query transcriptions based on clustered text responses is first performed (e.g., clustered vector representations of responses with exact or very close matches), and then the vector representations of the queries are clustered. The latter clustering may be based on one or more of the domain, written response clusters, and query content.

[0085] In testing, the quality of a machine learning system may be evaluated using accuracy and perplexity metrics (e.g., similar to those used in neural machine translation). For example, these metrics may be used to compare training and validation performance to capture the model's learning ability and avoid overfitting. Note that while perplexity can be a useful metric, accuracy may not be as relevant for machine translation tasks where there is only one translation (or slightly modified version) of a source sentence in the target language.

[0086] Examples of alternative methods The following is a set of exemplary items referencing various aspects of the specific example described, all of which are non-limiting. In one aspect, a method for training a model for rewriting transcriptions comprises projecting original transcriptions and their transcribed response embedding vectors into a lower-dimensional space, grouping the transcriptions into clusters based on exact matches of the transcribed responses, and assigning the transcriptions as target transcriptions for the groups. In a variation on this case, the method further comprises grouping the remaining ungrouped responses into clusters with nearly identical responses. This grouping operation may be repeated multiple times. In one variation, the method comprises removing transcriptions farther from the cluster centroid than a certain threshold, subdividing clusters smaller than a certain threshold, and assigning individual transcriptions from the previous step to their nearest clusters. In another variation, the method includes generating pairs by assigning transcriptions in each cluster having a frequency greater than a certain threshold as target transcriptions and assigning the remaining transcriptions in the cluster as source transcriptions, removing pairs in which the target transcription does not have all entity tags available in the source transcription, and removing pairs with a semantic distance greater than a threshold. In yet another variation, the method includes determining a part-of-speech sequence of the target transcription, and assigning the transcription as the target transcription is conditioned on the part-of-speech sequence matching that of a successfully interpreted transcription. In another aspect, a method for rewriting a transcription includes receiving a source transcription, calculating an embedding vector for it, mapping the embedding vector to a group, and rewriting the transcription into a target transcription for the group.This aspect may include calculating a model prediction score from a neural machine translation model and calculating a similarity score as the cosine distance between the source transcription and the generated transcription, wherein the mapping is conditioned on a weighted average of the model prediction score and the similarity score being above a threshold.

[0087] Summary and Benefits Example A particular example described herein presents a method for constructing a training set for a machine learning system configured to improve the operation of a digital assistant by filtering input queries to make them easier to process and answer. In a particular example, the training set is generated based on pairs of transcriptions representing input (source) queries and desired output (target) queries. This allows neural network architectures (such as those traditionally used for sequence-to-sequence mapping as part of machine translation) to be adapted as query preprocessors (e.g., elaborated through further training using the proposed training set). The described method transforms what is initially a problem associated with unsupervised data (e.g., how to manage unpredictable input variations) into a supervised learning problem that can be solved by training a machine learning system. The trained machine learning system is then suitable for use as a specialized preprocessing module in a query processing pipeline that maps infrequent forms of input (e.g., text derived from noisy voice queries or poorly typed text) to frequent forms (which may be considered "ideal," standard, or canonical forms). Thus, the trained machine learning system described herein can convert irregular text queries into learned canonical query patterns. Because the mapping is performed using a trained neural network architecture, the mapping may not necessarily resemble the training data. For example, the training target examples may not necessarily be generated. However, the generated output sequences may resemble the training target examples, and the machine learning architecture may be able to robustly manage unseen data.

[0088] In these examples, training data containing pairs of invalid and valid queries is generally considered unavailable or difficult to collect. These examples provide a method for constructing a training set of such queries. This method enables constructing a training set with high-quality mappings and training a high-quality preprocessor. A machine learning system can then pick up the mapping patterns between invalid and valid queries and provide reasonable transformations. Because the distance between source and target data samples is generally small, machine learning systems can be trained quickly and effectively (e.g., neural machine translation systems generally provide strong, high-quality mappings between source and target because many token embeddings can be maintained due to the narrower problem than machine translation in general). Because the quality of a machine learning system is directly related to the quality of the training dataset, and any noise or bias introduced into the data affects this quality, this method enables high-quality machine learning systems with high-quality datasets.

[0089] The specific examples described herein provide improvements over the comparative examples. In some cases, query texts may be grouped based on exact matches of text responses. However, this comparison approach is heavily influenced by the quality of the text responses and any named entity extraction performed (e.g., to replace values ​​with tag types as described). It also misses many invalid / valid pairs because even slight differences between seemingly identical written responses do not allow similar queries to be grouped to form training data. In another comparative example, the semantic distance between pairs of queries may be compared (e.g., n for n queries). 2 pairs), which may then be filtered based on a semantic distance threshold (e.g., only pairs with a similarity score above a predefined threshold are retained). In this case, the most frequent of the pairs in the dataset may be selected as the target, and the other of the pair is set as the source. Here, as in the first comparative example, the pairing is highly dependent on the semantic distance metric used, and this approach is impractical for large datasets (e.g., 1 million data samples would require 10 million samples to evaluate). 12 This approach can also result in datasets with a large amount of variation in the target data samples, making it difficult to successfully train a robust machine learning system.

[0090] It should be noted that although particular examples have been described herein, various combinations of various components from the various examples may be possible. While prominent features are shown to better explain the examples, it will be apparent that certain features may be added, modified, or omitted without changing the functional aspects of these described examples.

[0091] Various examples are methods that employ either human or machine behavior, or a combination of human and machine behavior. Method examples are complete regardless of where most of the constituent steps occur. Some examples are one or more non-transitory computer-readable media arranged to store such instructions for the methods described herein. Any machine that holds a non-transitory computer-readable medium with any of the necessary code may implement an example. Some examples may be implemented as a physical device such as a semiconductor chip, as a hardware description language representation of the logical or functional behavior of such a device, or as one or more non-transitory computer-readable media arranged to store such hardware description language representations. The statements herein describing principles, aspects, and embodiments encompass both structural and functional equivalents thereof.

[0092] Those skilled in the art will recognize many possible modifications and variations. These modifications and variations include any relevant combination of the disclosed features. The statements herein setting forth principles, aspects, and examples also encompass both structural and functional equivalents thereof. Accordingly, the scope of the present disclosure is not intended to be limited to the examples shown and described herein. Rather, the scope and spirit of the present invention is embodied by the appended claims.< / eoq> < / eoq> < / boq> < / eoq> < / boq>

Claims

1. A method executed by one or more processors of a computer to train a machine learning system for use with a digital assistant, comprising: obtaining training data including query data samples; obtaining a vector representation of the query data samples; clustering the vector representations; selecting a canonical query based on the clustered vector representations and determining a corresponding query group, wherein selecting the canonical query includes selecting a most frequent query data sample from among query data samples of each cluster of the clustered vector representations as the canonical query, and determining the corresponding query group includes determining a group of vector representations for each cluster of the clustered vector representations, the method further comprising: generating paired data samples based on selected canonical queries and selections from the query group corresponding to each selected canonical query; training an encoder-decoder sequence-to-sequence neural network architecture using the paired data samples, wherein the selection from the query group in the paired data samples is provided as input sequence data and the corresponding canonical query is provided as output sequence data; The method, wherein the trained encoder-decoder sequence-to-sequence neural network architecture, in conjunction with the digital assistant, can be used to map data representing an initial query to data representing a revised query, and the data representing the revised query is further processed to provide a response to the initial query.

2. obtaining training data comprises obtaining pairs of text data representing queries and responses; The method of claim 1 , wherein obtaining vector representations of the query data samples comprises converting the text data pairs into corresponding vector representations.

3. performing named entity extraction on the text data; and replacing the text data of the tagged named entities with named entity type tags.

4. 4. The method of claim 2 or claim 3, wherein a first portion of text data represents the output of a speech-to-text process performed on audio data of a voice query, and a second portion of text data includes data used in providing the response to the voice query, and wherein the trained encoder-decoder sequence-to-sequence neural network architecture, together with the digital assistant, is usable to map data representing an initial voice query to data representing a revised voice query.

5. the given vector representation includes at least a response vector representation, the response vector representation being a vector representation of the data representing a response to a query, the response vector representation being paired with data representing a corresponding query; The step of clustering the vector representations includes: clustering the response vector representations based on distances between the response vector representations in a vector space; 4. The method of claim 1, wherein for a given cluster in the clustered vector representations, a canonical query is selected based on a frequency of query data paired with response vector representations in the given cluster.

6. The step of clustering the vector representations includes: The method of claim 5 , comprising applying a hierarchical clustering technique to iteratively combine separate clusters.

7. The step of obtaining the vector representation comprises: obtaining a textual representation of at least a response to the query; generating an embedding vector for the text representation; and projecting the embedding vectors into a lower dimensional vector space to output the vector representation.

8. generating an embedding vector for the text representation, tokenizing the textual representation; and applying a transformer neural network architecture to the tokenized text representation to generate the embedding vector.

9. 9. The method of claim 7, wherein projecting the embedding vectors into a lower-dimensional vector space comprises selecting a subset of principal components, the principal components being determined after principal component analysis of a query data sample.

10. The step of clustering the vector representations includes: performing a first stage of clustering based on the vector representation of the response to the query; performing a second stage of clustering based on a vector representation of the query preceding the response; the second stage of clustering includes clustering within the clusters defined in the first stage of clustering; 10. The method according to claim 1, wherein selecting the canonical query comprises selecting, as the canonical query, a most frequent query data sample from among query data samples in each of the clusters generated in the second stage of clustering.

11. The method of any one of claims 1 to 10, comprising filtering the clustered vector representations before generating the paired data samples.

12. The filtering step may include, for a given cluster: determining the centroid of the given cluster in the clustered vector representation; and deallocating vector representations of queries of the given cluster that are greater than or equal to a predefined distance from the centroid.

13. The filtering step may include, for a given cluster: determining the size of the given cluster; and deallocating the vector representation of the query for the given cluster in response to the size falling below a predefined threshold.

14. The filtering step includes: The method of claim 12 or claim 13, further comprising the step of reassigning vector representations of deallocated queries to the nearest cluster.

15. The step of generating paired data samples comprises: A method according to any preceding claim, comprising filtering the generated paired data samples.

16. The filtering step is removing paired data samples having canonical queries whose named entity tags do not match the named entity tags in the corresponding selection from the query group; and removing paired data samples based on a comparison of the semantic distance metric of the canonical query with the semantic distance metric of the corresponding selection from the query group; 16. The method of claim 15, comprising one or more of:

17. The following metrics: one or more clustering distance thresholds; one or more cluster size thresholds; and one or more frequency thresholds for the selection of canonical queries; 17. The method of claim 1, comprising optimizing one or more of:

18. A method for processing queries to a digital assistant, executed by one or more processors of a computer, comprising: receiving initial data representing a query; generating input sequence data from the initial data representing the query; and mapping the input sequence data to output sequence data using an encoder-decoder sequence-to-sequence neural network architecture, the output sequence data representing a canonical data representation of the query, the encoder-decoder sequence-to-sequence neural network architecture having been trained by a machine learning process using training data including query data samples, the machine learning process using a group of vector representations of each cluster of clusters of the clustered vector representations as input sequence data, the method further comprising: processing the canonical data representation to determine a response to the query; and providing the response to the query.

19. 20. The method of claim 18, wherein the encoder-decoder sequence-to-sequence neural network architecture is trained on paired data samples representing a canonical query and a corresponding initial query, the paired data samples being generated by clustering vector representations of query data samples that include data representing a query and data representing a response to the query.

20. processing the canonical data representation to determine a response to the query, comprising: selecting one of a top K set of output sequences provided in the output sequence data; parsing data derived from at least the output sequence data with a language model and selecting an output sequence based on scoring applied by the language model; and parsing the initial data representing the query and data derived from at least the output sequence data using a named entity extractor and selecting an output sequence based on a comparison between named entities in the parsed data; 20. The method of claim 18 or 19, comprising one or more of:

21. 1. A machine learning system for a digital assistant, comprising: a query interface for receiving initial data representing a query and outputting input sequence data derived from said initial data; an encoder-decoder sequence-to-sequence neural network architecture for mapping the input sequence data to output sequence data, the encoder-decoder sequence-to-sequence neural network architecture being trained by a machine learning process using training data including query data samples, the machine learning process using a group of vector representations of each cluster of the clustered vector representations as input sequence data; The output sequence data represents a canonical data representation of the query, and the machine learning system further comprises: and an output interface for providing the output sequence data for use in providing a response to the query.

22. 22. The machine learning system of claim 21, wherein the query interface comprises a speech-to-text interface.

23. 23. The machine learning system of claim 21 or claim 22, comprising an output sequence filter for receiving a plurality of output sequences from the encoder-decoder sequence-to-sequence neural network architecture and selecting one of the plurality of output sequences based on at least one natural language processing metric.

24. A program which, when executed by at least one processor, causes the at least one processor to perform the method of any one of claims 1 to 17.

25. 25. The program of claim 24, wherein the query data sample relates to a voice query.

26. A program which, when executed by at least one processor, causes the at least one processor to perform the method of any one of claims 18 to 20.

27. 27. The program of claim 26, wherein the initial data is related to a voice query.

Images