Guided enhancement of data sets for machine learning models

Abstract

Techniques are disclosed for enhancing a dataset for training a machine learning model and for generating predictions from the trained machine learning model. The techniques may increase the number and diversity of examples within an initial sentence training dataset by extracting a subset of words from an existing sentence training dataset. The techniques may preserve scarce sample data with few samples by training a data generation model using generic data obtained from a generic data source.

Description

[0001] Incorporation by Reference; Disclaimer

[0002] The following applications are hereby incorporated by reference: U.S. Non-Provisional Patent Application No. 17 / 903,798, filed on September 6, 2022; U.S. Provisional Patent Application No. 63 / 352,110, filed on June 14, 2022, and U.S. Non-Provisional Application No. 17 / 903,796, filed on September 6, 2022. Applicants hereby revoke any disclaimer of claim scope in the parent application(s) or its prosecution history, and give notice that the claims in this application may be broader than any claim in the parent application(s). Technical Field

[0003] The present disclosure relates to machine learning. More specifically, the present disclosure relates to enhancing data used to train machine learning models. Background Art

[0004] Machine learning produces trained models that generate output predictions based on sample data sets. The accuracy of the predictions depends on the quantity and quality of the sample data sets. Machine learning models trained using sample data sets are likely to be inaccurate if there are only a few example data points in one or more categories, or if there is a lack of variation between the data points contained in a particular category. Thus, the accuracy of machine learning models can be improved by obtaining sample data sets with a large number of diverse examples. However, in some cases, obtaining sample data sets of sufficient size and diversity can be challenging.

[0005] The approaches described in this Background section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A functional flow diagram is illustrated according to one or more embodiments.

[0007] Figure 2 A system block diagram is illustrated in accordance with one or more embodiments.

[0008] Figure 3A and 3B Illustrated is a set of example operations for improving the diversity of examples within an existing dataset for training a machine learning model in accordance with one or more embodiments.

[0009] Figure 4 A block diagram illustrating a computer system in accordance with one or more embodiments is shown. DETAILED DESCRIPTION

[0010] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding. One or more embodiments may be practiced without these specific details. The features described in one or more embodiments may be combined with the features described in different embodiments. In some examples, well-known structures and devices are described with reference to block diagrams in order to avoid unnecessary confusion of the present invention.

[0011] In the figures of the accompanying drawings, embodiments are illustrated by way of example only and not limitation. It should be noted that references to "an" or "one" embodiment in the present disclosure do not necessarily refer to the same embodiment, and they mean at least one.

[0012] This detailed description section includes the following subsections:

[0013] A. General Overview

[0014] B. Data enhancement process

[0015] C. Data Augmentation System Architecture

[0016] D. System Operation

[0017] E. Example Embodiments

[0018] F. Computer Networks and Cloud Networks

[0019] G. Other aspects; expansion

[0020] H. Hardware Overview

[0021] A. General Overview

[0022] Machine learning models can be trained and validated using a combination of sample data points and synthetic data points. Sample data points include information representing actual events, observations, situations, etc. Sample data points can be input into a machine learning algorithm to train the model to make predictions. Synthetic data points include information that is artificially created and not generated by actual events. Synthetic data is typically created from sample data points or portions thereof using a synthetic data generation model. In an example, a synthetic data generation model can generate sentences by modifying words provided as sample data points.

[0023] One or more embodiments generate training data that can be used to train or validate a machine learning model using a synthetic data generation model so that the synthetic data generation model can generate synthetic data points. In an example scenario, the system generates training data to train the synthetic data generation model to generate sentences. The system obtains a text corpus containing sentences and extracts sentence pairs so that the first sentence in the sentence pair satisfies a similarity threshold relative to the second sentence in the sentence pair. Then, the system extracts a subset of words from the first sentence so that the subset of words does not include all words in the first sentence. The system generates multiple training instances for training the synthetic data generation model based on (a) a model input including the second sentence and a subset of words extracted from the first sentence and (b) a model output (e.g., a target) including the first sentence. The training instance is used to train the synthetic data generation model so that the synthetic data generation model generates the first sentence when the second sentence and a subset of words from the first sentence are given as input.

[0024] In one or more embodiments, as described above, extracting a subset of words from the first sentence includes removing stop words, such as common articles, conjunctions, verbs, and prepositions (e.g., the, is, in, for, where, when, to, at, etc.). Extracting the subset of words may also include removing one or more other words from the target. For example, one or more words may be randomly removed from the sentence. In addition, the remaining words may be reordered or shuffled.

[0025] One or more embodiments optimize machine learning in few-shot situations. Few-shot situations as referred to herein include situations where data points in some or all categories of a training data set are scarce. Due to the lack of examples that can be used to train and verify models for specific tasks, the scarcity of data points makes it difficult to determine an accurate machine learning model. According to one or more aspects of the present disclosure, by using non-specific data points (e.g., non-task-specific data points) obtained from a general data source to train data to generate a model, sample data points for tasks specific to the machine learning model can be retained. The general data source can be, for example, a publicly available data source, including data that may be related to the intended task of the classifier model but is not specific to the purpose of the model. For example, an example machine learning model can be a chatbot for a customer of an insurance provider, and task-specific data can be for insurance customer service. Task-specific data points can be obtained from confidential logs of interactions between insurance customer service representatives and insurance customers. General data points can be obtained from a database containing information from online FAQs and Q&A (Q&A) from various online service providers. Thus, in this example, the general data points may be related to the task of responding to user queries and may have similar characters regarding, for example, sentence length and sentence type (such as commands, questions, and responses), but not specific to the task of responding to insurance customer queries.

[0026] One or more embodiments use generic data points to train a data generation model, and use few or no task-specific data points for training. One or more other embodiments use only generic data points to train a data generation model. Once trained, the data generation model can be prompted with task-specific data points to synthesize additional task-specific data points, which are used for training and / or validation of a machine learning model.

[0027] According to aspects of the present disclosure, a data generation model generates synthetic example data points that are similar to task-specific data. Data generation can be quasi-controlled via guide words, which are part of the input to the data generation model. The model is trained to generate new examples containing given guide words. For example, a data generation model can be trained to reconstruct a given utterance or natural language string that expresses a general category of intent (e.g., ordering pizza) in another utterance of the same or similar intent and the guide words that appear in the utterance. After the data generation model is trained, it can be used to generate new example data points, which can then be added to the training set and / or validation set to enhance model training or model selection, respectively.

[0028] One or more embodiments described in the specification and / or recited in the claims may not be included in this general summary.

[0029] B. Data enhancement process

[0030] Figure 1 A functional block diagram illustrating an example processing flow 100 for augmenting machine learning training data in accordance with one or more embodiments is shown. The example processing flow 100 includes a general data source 101, a task-specific data source 103, a machine learning system 105, and a production system 107. The general data source 101 may be an information repository that stores general or generic data 110 that may be relevant to an intended task of a desired machine learning model but is not specific to the intended task. In one or more embodiments, the general data source 101 is one or more public or commercially available data repositories. For example, the general data source 101 may be a machine learning library such as COMMONCRAWL, KAGGLE AND UCI MACHINE LEARNING REPOSITORY. The task-specific data source 103 may be one or more information repositories that store task-specific data 112 for the intended task of the desired machine learning model. In one or more embodiments, the task-specific data source 103 is a non-public data repository. For example, the task-specific data source 103 may be a confidential or proprietary data repository, such as a database of transcribed conversations between customers and service representatives maintained by a confidential information system. In another example, the task-specific data source 103 may be one or more private surveillance information logs generated by a security system.

[0031] The machine learning system 105 may be one or more computing systems configured to process data from the data sources 101 and 103 to construct a production model 108 for processing by the production system 107. The production system 107 may be one or more computing systems configured to generate predictions 109 based on the production data 106 using the production model 108 determined by the machine learning system 105. For example, the production model 108 may be an intent classifier for a chatbot, a digital assistant, or an automated voice response system. The production data 106 may be a natural language query, such as an utterance, from a user to the production system 107. The prediction 109 may be a classification of the query intent for determining a response to the query.

[0032] The example processing flow 100 includes a machine learning system 105 that constructs a generalized training data set 113 using general data 110 obtained from a general data source 101. In addition, the machine learning system 105 constructs a custom training data set 115 using task-specific data 112 obtained from a task-specific data source 103. The task-specific data 112 can be specifically for the purpose of the desired machine learning model. According to aspects of the present disclosure herein, the task-specific data 112 can be few-sample data. Few-sample data as referred to herein includes task-specific data 112 having only a few examples in one or more categories. In some embodiments, the few-sample sample data set predicted by the machine learning model can have five or fewer data points per category. In some other examples, the few-sample sample data set can have ten or fewer data points per category. In some other examples, the few-sample sample data set can have less than 1,000 total data points.

[0033] The example processing flow 100 may also include a training data module 119 that constructs a training input 121 for a training data set generator model 127. The training data module 119 may generate the training input 121 using a general training data set 113 including the general data 110. For example, the training data module 119 may extract similar sentence pairs from the general training data set 113 using a sentence similarity function. For each sentence pair, the training data module 119 may designate the first sentence as a "primer" and the second sentence as a "target", and may create a set of guide words based on the target. Determining the set of guide words may also include removing one or more other words from the target. For example, the training data module 119 may randomly remove words from the target. Creating the set of guide words may also include shuffling the order of the remaining words.

[0034] Using guide words, the training data module 119 can construct the training input 121 using a predefined pattern processed by the data generator training module 125. For example, the training data module 119 can construct the training input using the following format: <introduction>|<guide word 1><guide word 2>...<guide word k>. The output corresponding to this training input is the target. Thus, the data generator training module 125 is trained to construct the target based on the introduction (i.e., similar sentences) and the guide words (i.e., words that appear in the target). The introduction "guides" the generator to generate similar sentences, and the guide words constrain the generated content. A desired number (N) of such pairs are constructed and used as a generator training set (e.g., N=20).

[0035] The example processing flow 100 also includes a data generator training module 125 that uses the training input 121 determined by the training data module 119 to generate a data set generation model 127. In some examples, the data set generation machine learning model 127 may include one or both of a supervised machine learning algorithm and an unsupervised machine learning algorithm. In addition, in some examples, the data set generation machine learning model 127 may be a machine learning model suitable for various aspects of natural language processing (NLP). For example, the data set generation machine learning model 127 may be a "sequence to sequence" model ("Seq2Seq") that is trained to receive a labeled first sequence and generate a different second sequence that is consistent with the label applied to the first sequence. For example, the Seq2Seq model may be a T5 model that takes a sequence as input and generates a sequence as output.

[0036] The dataset generation model 127 can generate synthetic data points 129 (e.g., new training examples) based on the task-specific data 112 in the custom training dataset 115. The synthetic data points 129 can be combined with the task-specific data 112 to construct an enhanced training dataset 131. The dataset generation model 127 can generate new synthetic data points 129 by selecting primer examples from the task-specific data 112. For example, the dataset generation model 127 can randomly select primers and guide words from the task-specific data 112. In a specific non-limiting example, the dataset generation model 127 constructs a mapping from words used in the task-specific data to the number of times each word is used (relative to each class). Then, when generating examples of a class, the dataset generation model 127 samples the guide words in proportion to the guide word frequency defined by the mapping corresponding to the class. The number of guide words that can be sampled makes the generated examples have a length similar to that of the examples in the task-specific data.

[0037] In addition, the example processing flow 100 includes a machine learning training module 133, which determines the production model 108 by training the machine learning algorithm using the enhanced training data set 131 in a supervised or unsupervised machine learning algorithm. In some embodiments, the machine learning algorithm can be a classification algorithm, such as K-nearest neighbor, naive Bayes, random forest, support vector machine, and logistic regression. The training algorithm can also be a regression algorithm, such as linear regression, support vector regression, decision tree / random forest, and Gaussian process regression.

[0038] C. Data Augmentation System Architecture

[0039] Figure 2 A system 200 is illustrated according to one or more embodiments. Figure 2As shown in FIG. 1 , system 200 includes machine learning system 105, client device 203, data repository 205, and external resource 209. Machine learning system 105 may be the same or similar to the system described above. In one or more embodiments, system 200 may include Figure 2 More or fewer components may be used than shown. Figure 2 The components shown in may be local to each other, or may be remote from each other. Figure 2 The components shown in the figure can be implemented in software and / or hardware. Each component can be distributed on multiple applications and / or machines. Multiple components can be combined into one application and / or machine. The operations described with respect to one component can be performed by another component instead.

[0040] The client device 203 may be a web browser, a mobile application, or other software application communicatively coupled to a network (e.g., via a computing device). The client device 203 may interact with other elements of the system 200 directly or via a cloud service using one or more communication protocols (such as HTTP and / or other communication protocols of the Internet Protocol (IP) suite). In some examples, the client device 203 is configured to receive and / or generate data items stored in the data repository 205. The client device 203 may transmit a target data item to the machine learning system 105 for analysis. In some examples, the client device 203 may send an instruction to the machine learning system 105 that initiates a process of generating a synthetic data point (e.g., synthetic data point 129) from task-specific data points (e.g., task-specific data 112) using one or more machine learning models (e.g., data set generation model 127). The client device 203 may send an instruction to the machine learning system 105 to analyze the target data item.

[0041] The client device 203 may also include a user input / output device configured to render a graphical user interface (GUI) generated by the machine learning system 105. The GUI may present an interface through which a user triggers the execution of a computing transaction to generate and / or analyze data items. In some examples, the GUI may include features that enable a user to view training data, classify training data, instruct the machine learning system 105 to perform processing to enhance or otherwise increase the number of examples in the training data set, and other features of the embodiments described herein. In addition, the client device 203 may be configured to enable a user to provide user feedback on the accuracy of the machine learning system 105 analysis via the GUI. That is, a user may use the GUI to label the analysis generated by the machine learning system 105 as accurate or inaccurate. In some examples, using the GUI, a user may cause operations (e.g., loss function analysis) to be performed that measure the accuracy of the analysis generated by the machine learning system 105. These latter features enable a user to label or otherwise "grade" the data analyzed by the machine learning system 105 so that the machine learning system 105 can update its training.

[0042] In one or more embodiments, the data repository 205 can be any type of storage unit and / or device for storing data (e.g., a file system, a database, a collection of tables, or any other storage mechanism). In addition, the data repositories 122 can each include multiple different storage units and / or devices. The multiple different storage units and / or devices may or may not be of the same type or located at the same physical site. In addition, the data repository 122 can be implemented or executed on the same computing system as the machine learning application 213. Additionally or alternatively, the data repository 122 can be implemented or executed on a computing system separate from the machine learning application 213. The data repository 122 can be communicatively coupled to the machine learning application 213 via a direct connection or via a network.

[0043] Some embodiments of the machine learning system 105 are implemented on one or more digital devices. The term "digital device" generally refers to any hardware device that includes a processor. A digital device may refer to a physical device that executes an application or a virtual machine. Examples of digital devices include computers, tablets, laptops, desktops, netbooks, servers, web servers, network policy servers, proxy servers, general-purpose machines, function-specific hardware devices, hardware routers, hardware switches, hardware firewalls, hardware firewalls, hardware network address translators (NATs), hardware load balancers, hosts, televisions, content receivers, set-top boxes, printers, mobile phones, smart phones, personal digital assistants ("PDAs"), wireless receivers and / or transmitters, base stations, communication management devices, routers, switches, controllers, access points, and / or client devices.

[0044] Figure 2 The machine learning system 105 shown in includes a storage device 211 and a machine learning application 213. The storage device 211 may include a magnetic storage device, a flash drive, a compact disk ROM, a digital versatile disk (DVD) optical storage technology, or a suitable other fixed, non-transitory computer-readable storage device. The storage device 211 may also store program instructions (e.g., code) and operational data for the operation of the machine learning system 105. In addition, the storage device 211 may store a general training data set 113, a custom training data set 115, a training input 121, and an enhanced training data set 131, all of which may be the same or similar to those described above. It will be appreciated that some or all of the information used by the machine learning system may be stored remotely, such as in a data repository 205.

[0045] The machine learning application 213 of the system 200 can be configured to train a data generation model (e.g., a data set generation model 127) that generates synthetic training data (e.g., synthetic data points 129) based on an initial data set (e.g., general data 110). For example, the data generation model can generate a new natural language processing data set based on a limited (e.g., containing a small number of examples) initial data set, and use the new natural language processing data set to further train a separate machine learning model (e.g., a production model 108), which is configured to interpret human-generated natural language (e.g., a chatbot). The new natural language processing data set generated by the machine learning application 213 and used to train the chatbot can improve the accuracy and operational efficiency of the chatbot's operation. The machine learning application 213 may include a training data module 119, a data generator training module 125, a data set generation model 127, and a machine learning training module 133, all of which may be the same or similar to those described above and are further described in detail below. In addition, the machine learning system 105 may include a front-end interface 217 and an action interface 220.

[0046] The front-end interface 217 manages the interaction between the client device 203 and the machine learning application 213. In one or more embodiments, the front-end interface 217 refers to hardware and / or software configured to facilitate communication between a user and the client device 203 and / or the machine learning application 213. In some embodiments, the front-end interface 217 is a presentation layer in a multi-layer application. The front-end interface 217 can process requests received from the client and translate the results from other application layers into a format that the client can understand or process. For example, the client device 203 can submit a request to the machine learning application 213 via the front-end interface 217 to perform various functions, such as labeling training data and / or analyzing target data. In some examples, the client device 203 can submit a request to the machine learning application 213 via the front-end interface 217 to view a graphical user interface associated with a natural language processing analysis. In some other examples, the front-end interface 217 can receive user input to reorder various interface elements.

[0047] Front-end interface 217 refers to hardware and / or software that can be configured to render user interface elements and receive input via user interface elements. For example, front-end interface 217 can generate web pages and / or other graphical user interface (GUI) objects. Client applications such as web browsers can access and render interactive displays according to the protocols of the Internet Protocol (IP) suite. Additionally or alternatively, front-end interface 217 can provide other types of user interfaces, including hardware and / or software configured to facilitate communication between users and applications. Example interfaces include, but are not limited to, GUI, web interface, command line interface (CLI), tactile interface, and voice command interface. Example user interface elements include, but are not limited to, check boxes, radio buttons, drop-down lists, list boxes, buttons, toggle switches, text fields, date and time selectors, command lines, sliders, pages, and forms.

[0048] In one or more embodiments, different components of the front-end interface 217 are specified in different languages. The behavior of the user interface elements is specified in a dynamic programming language such as JavaScript. The content of the user interface elements is specified in a markup language such as Hypertext Markup Language (HTML) or XML User Interface Language (XUL). The layout of the user interface elements is specified in a style sheet language such as Cascading Style Sheets (CSS). Alternatively, the front-end interface 217 is specified in one or more other languages ​​such as Java, C, or C++.

[0049] Action interface 220 may include an API, CLI, or other interface for calling functions to perform actions. One or more of these functions may be provided by a cloud service or other application, which may be external to machine learning application 213. For example, one or more components of machine learning application 213 may call an API to access information stored in a data repository (e.g., data repository 222) to serve as a training corpus for machine learning application 213. It will be appreciated that the actions performed may vary from implementation to implementation.

[0050] In some embodiments, machine learning application 213 can access external resources 226, such as cloud services. Example cloud services can include, but are not limited to, social media platforms, email services, short messaging services, enterprise management systems, and other cloud applications. Additional embodiments and / or examples related to computer networks are described below in Section 6 entitled "Computer Networks and Cloud Networks".

[0051] In some examples, external resources 226 may include external machine learning models 230 trained using a training dataset generated by machine learning application 213. In one example, the training dataset generated by machine learning application 213 may be used to train a user-facing natural language processing application, such as a chatbot (for instant text messaging) or an interactive voice recognition (IVR) system.

[0052] The action interface 220 can be used as an API endpoint for calling cloud services. For example, the action interface 220 can generate an outbound request that conforms to a protocol that can be ingested by an external resource. The action interface 220 can process and translate inbound requests to allow further processing by other components of the machine learning application 213. The action interface 220 can store, negotiate and / or otherwise manage authentication information for accessing external resources. Example authentication information may include, but is not limited to, digital certificates, cryptographic keys, usernames, and passwords. The action interface 220 can include authentication information in a request to call a function provided by an external resource.

[0053] In one or more embodiments, the data repository 222 can be any type of storage unit and / or device for storing data (e.g., a file system, a database, a collection of tables, or any other storage mechanism). In addition, the data repositories 222 can each include multiple different storage units and / or devices. The multiple different storage units and / or devices may or may not be of the same type or located at the same physical site. In addition, the data repository 222 can be implemented or executed on the same computing system as the machine learning application 213. Additionally or alternatively, the data repository 222 can be implemented or executed on a computing system separate from the machine learning application 213. The data repository 222 can be communicatively coupled to the machine learning application 213 via a direct connection or via a network.

[0054] Information related to target data items and training data may be implemented across any component within system 200. It should be appreciated that this information may be stored in data repository 222 for purposes of clarity and explanation.

[0055] In one or more embodiments, the system 200 is implemented on one or more digital devices. The term "digital device" generally refers to any hardware device including a processor. A digital device may refer to a physical device that executes an application or a virtual machine. Examples of digital devices include computers, tablet computers, laptop computers, desktop computers, netbooks, servers, web servers, network policy servers, proxy servers, general machines, function-specific hardware devices, hardware routers, hardware switches, hardware firewalls, hardware firewalls, hardware network address translators (NATs), hardware load balancers, hosts, televisions, content receivers, set-top boxes, printers, mobile phones, smart phones, personal digital assistants ("PDAs"), wireless receivers and / or transmitters, base stations, communication management devices, routers, switches, controllers, access points, and / or client devices.

[0056] Figure 2 The components shown in may be local to each other, or may be remote from each other. Figure 2 The components shown in the figure can be implemented in software and / or hardware. Each component can be distributed on multiple applications and / or machines. Multiple components can be combined into one application and / or machine. The operation described with respect to one component can be performed by another component instead.

[0057] D. System Operation

[0058] Figure 3A and 3B The flowcharts in the drawings illustrate the functionality and operation of systems, devices, processes, and computer program products according to various embodiments of the present disclosure. Figure 3A and 3B Each block in the flowchart may represent a module, a segment, or a portion of program instructions, which includes one or more computer executable instructions for implementing the functions and operations shown. In some embodiments, the functions and / or operations shown in a specific block of the flowchart may not be in order. Figure 3A and 3B . For example, two boxes shown in succession may be executed substantially simultaneously, or may sometimes be executed in reverse order, depending on the functions involved. In addition, in some embodiments, the boxes of the flowchart may be rearranged in a different order. In addition, in some embodiments, the flowchart may include fewer boxes or more boxes. It should also be noted that each box in the flowchart and the combination of boxes in the flowchart may be implemented by a dedicated hardware-based system or a combination of dedicated hardware and computer instructions that performs the specified function or action.

[0059] 3 illustrates a process 300 for increasing the diversity of examples in a training data set according to one or more embodiments. The training data set generated according to the process 300 can be used to train other machine learning models, such as chatbots and automatic voice response systems. When trained using models generated according to the process 300, other machine learning models show higher accuracy in predicting natural language received from users.

[0060] At box 303, the process 300 obtains task-specific data points (e.g., task-specific data 112), including information for a desired machine learning model. For example, for a customer service chatbot for a particular service department, the task-specific data points may be examples obtained from a confidential or proprietary log of interactions between customers and service representatives of that service department. In some embodiments, the task-specific data may include, such as a training data set with a small number of examples (e.g., fewer than 30 or fewer than 5), or classes within a training data set with a small number of examples. For example, the task-specific data may include sentences that have too few examples to effectively train a machine learning model so that it generates accurate, relevant, and / or grammatically correct predictions based on input data (e.g., text communications received from a person).

[0061] At block 305, process 300 obtains general data points (e.g., general data 110). A general data source (e.g., general data source 101) can provide general data to a machine learning system (e.g., machine learning system 105). As described above, an example general source can be a public database or a commercially available database. The general data points can be relevant or irrelevant to the purpose of the desired machine learning model. For example, the general data points can be included in public FAQs and Q&As, which contain lists of questions and answers, such as basic information of users of an online service.

[0062] At box 307, the process 300 constructs data (e.g., training input 121) to train a data generation model (e.g., data set generation model 127). According to aspects of the present disclosure, one or more embodiments use only the general data points obtained at box 305 or substantially only the general data points to train the data generation module without or substantially without task-specific data. Constructing the training generation model may include determining a desired number (N) of synthetic data points (e.g., synthetic data points 129) to be generated at box 309. In one or more embodiments, a user of a client device (e.g., client device 203) may transmit the number (N) of synthetic data points. For example, for each class label of data contained in a task-specific training data set (e.g., custom training data set 115), the desired number (N) of synthetic data points may be 20.

[0063] Constructing the training generative model may also include extracting a number (N) pairs of similar sentences (S1, S2) from general data points that meet the similarity criteria at box 311. Sentence similarity or semantic text similarity is a measure of the degree of similarity between two pieces of text or the degree to which they express the same meaning. Related tasks include paraphrase or repetition identification, search and matching applications. Techniques for determining text similarity may include word vector dot products, pairwise classification, and deep neural networks. Conventional sentence similarity functions (such as SENTEVAL in BERT) may be used to determine similarity.

[0064] Constructing the training generation model may also include, for each pair determined at box 311, designating the first sentence (S1) as an introduction sentence at box 317, and designating the second sentence (S2) as a target sentence at box 317. Constructing the data generation model may also include constructing a certain number (K) of guide words at box 327 according to the target specified at box 321. Constructing the guide words may include removing stop words from the target at box 329. There are many benefits to removing stop words from a sentence. For example, by removing stop words, the system improves the diversity of new examples generated because the system has greater flexibility when generating examples by reordering words. More specifically, one aspect of the diversity of the training data set is the order of the words used in the example sentences. By providing the system with a set of words associated with a specific sentence but without stop words, the system can generate one, two or more different examples using the same set of input words but a combination of any number of different stop words in each different, newly generated and grammatically correct example sentence.

[0065] In some examples, the system can remove stop words by, for example, first identifying the stop words within the vector representation of the sentence. The system can identify stop words by applying search conditions, filtering, matching or other NLP techniques to the vector representation of the sentence. Then, the system can use the applied labels to apply filters and / or search conditions. In other examples, the system can simply use a text matching system (e.g., character recognition, neural networks, classifiers, and other machine learning models) to identify the stop words in the target sentence (or corresponding vector). Once identified, the system can remove the stop words by deleting them from the sentence.

[0066] Constructing guide words can also include removing a subset of the remaining words in the target using the removal probability (P) at box 331. In addition to removing stop words, the system also extracts a subset of words from the sentence (or, equivalently, extracts a subset of tokens from the vector corresponding to the sentence). The extracted word subset includes a complete set of words less than the initial sentence set received from the data set generation model. The system can use any of a variety of techniques to select the word subset to be extracted. In one example, the system applies a random selection function to each word (or equivalently, a token) to determine whether to select the words in the sentence as a subset. In some examples, the system can bias the selection of words based on part of speech, word length, type of content words (e.g., based on the topic associated with the word) or other criteria. Removing a subset of words can have many of the same benefits as removing stop words as described above. In particular, when the system generates additional examples for the training data set, removing a subset of content words will increase the number of possible combinations of the remaining words.

[0067] Constructing guide words can also include changing the order of the remaining words from box 331 at box 333. Then, the system can change the order of the extracted words relative to the order in which the extracted words appear in the corresponding sentences received in the initial training sentence set. The system can apply any technology to the extracted word subset to change the order of words. In some examples, the order of words is randomized by applying a randomization function. The example in which the order of the remaining words is randomized may be particularly advantageous for improving the training of the model by separating the order of words from the predictive analysis of the model. In other words, randomizing the words in the sentence can improve the analysis flexibility because the model is not trained to recognize the specific order required for certain input words. In other examples, the order of words is changed by applying a system function. For example, the first word is moved to one of the beginning or end of the sentence and / or the second word is moved to one of the beginning, end or middle of the sentence. In some examples, a single word is moved from a first position to a second position different from the first position. In other examples, two, three or more words are moved from their corresponding positions to corresponding different positions. In another example, a randomly selected number of words are moved from their respective first positions to different second positions. This reordering (or "shuffling") process may be performed on each sentence of the initial set of sentences (or the remainder of the initially received sentences).

[0068] Continue to Figure 3B, as indicated by the off-page connector "A", at box 337, the process 300 constructs a training input (e.g., training input 121) using the training data constructed at box 307. More specifically, using a predefined pattern, a training data module (e.g., training data module 119) can construct a training input based on the primer and target specified at boxes 317 and 321 and the guide words constructed at box 327. For example, the training data module can construct a target training input using the following format: <primer>|<guide word 1><guide word 2>...<guide word K>. The training data module generates sentences similar to the primer, and the guide words constrain the generation of the target training input.

[0069] At box 339, process 300 trains the data generator model to generate targets using primers and guide words. In other words, the data generator model is trained to construct targets based on primers (i.e., similar sentences) and guide words (i.e., words that appear in the targets). For example, process 300 uses primers and guide words as inputs and uses targets as outputs to train the data generator model. In one or more embodiments, the data generator model is a sequence to sequence ("Seq2Seq") model. Seq2Seq can be, for example, an open source text to text transfer transformer ("T5") model.

[0070] At box 341, the process generates synthetic data points for a training data set (e.g., an enhanced training data set 131) based on task-specific data (e.g., task-specific data 112) using a data generator model. The data generator model constructs synthetic data points (targets) based on the task-specific data (primers) and guide words that appear in the target. In one or more embodiments, the primers are randomly selected task-specific examples. The guide words are randomly sampled from the task-specific data based on their frequency in the task-specific data. For example, the process 300 can construct a mapping from words used in the task-specific data to the number of times each word is used (e.g., relative to each class). Then, when generating examples, the process 300 selects guide words in proportion to the guide word frequency defined by the mapping (e.g., corresponding to the class). The number of sampled guide words is such that the generated examples will have a length similar to that of the examples in the task-specific data.

[0071] At box 345, the process 300 uses the task-specific data points obtained at box 303 and the synthetic data generated at box 341 to train a machine learning module (e.g., production model 108). In one or more embodiments, the machine learning module is an intent classifier that is configured to classify a natural language query (e.g., text or speech) from a user to determine a response to the query. For example, the machine learning module (e.g., machine learning training module 133) may include one or both of a supervised machine learning algorithm and an unsupervised machine learning algorithm. In some embodiments, the machine learning algorithm may be a classification algorithm such as K-nearest neighbor, naive Bayes, random forest, support vector machine, and logistic regression. The training algorithm may also be a regression algorithm such as linear regression, support vector regression, decision tree / random forest, and Gaussian process regression. During training, the weights of the machine learning algorithm are optimized by minimizing the loss, as defined by the loss function. Using the process 300 to increase the diversity of examples in the training data set has an additional efficiency advantage because the diverse samples are based on an already existing training data set. The additional work (computational or other work) required to obtain, filter, and classify a completely new and different data set is avoided.

[0072] E. Example Embodiments

[0073] For the sake of clarity, detailed examples are described below. The components and / or operations described below should be understood as a specific example, which may not be applicable to certain embodiments. Therefore, the components and / or operations described below should not be interpreted as limiting the scope of any claim.

[0074] This example trains a data generator model (e.g., dataset generator model 127) to generate new training examples (e.g., synthetic data 129) that are similar to task-specific data, as described above. The new training examples can augment the training data (e.g., training datasets 113 and 115) to generate an augmented training dataset (e.g., augmented training dataset 131) for model training data and / or validation data.

[0075] The training data generator model includes determining an enhanced training set. More specifically, the system (e.g., the machine learning system 105 executing the training data module 119) determines a certain number of training examples (N) of the enhanced training set from a text corpus (C) using a sentence similarity function (S). The text corpus (C) may include publicly available intent classification data sets or documents on various topics. For example, if the task of the machine learning model to be trained is a chatbot that responds to customer service questions, the corpus (C) may include non-task-specific questions from FAQ pages in a public FAQ web crawl. The sentence similarity function (S) may be an algorithm executed by the system that measures the similarity between pairs of sentences in the corpus (C) to identify and extract N pairs of similar sentences. Some embodiments of the similarity function (S) identify sentences as similar based on common attributes. For example, the similarity function S may determine that two sentences are similar based on the fact that the two sentences are included in the same FAQ. In each pair, the system designates the first sentence (S1) as the "introduction" and the second sentence (S2) as the "target". In this example, the similarity function (S) may identify the following example sentence pairs from the same customer service FAQ as similar: "How do I obtain a refund?" and "What is the telephone number for customer service?" The first sentence "How do I obtain a refund?" may be designated as the primer, and the second sentence "What is the telephone number for customer service?" may be designated as the target.

[0076] For each of the N pairs extracted from the corpus (C), the system determines a set of guide words from the target sentence. Determining the guide words includes removing stop words from the target sentence. Stop words include those words used in phrases or sentences that perform grammatical functions but do not directly affect the meaning or content of the sentence. Examples of stop words include, but are not limited to, conjunctions (e.g., "and", "but"), articles (e.g., "a", "the"), prepositions (e.g., "by", "at", "with"), and some linking verbs and auxiliary verbs (e.g., "is", "are", "be"), etc. In this example, the system can remove the words "is", "the", and "for" from the target sentence, leaving "telephone number customer service".

[0077] Determining the guide word may also include randomly removing one or more remaining words in the target sentence using a probability (P). Some embodiments apply a random selection function to each word to determine whether to select a word in the sentence as a subset. For example, using a random number generator with 20%, the system may make a random determination for each word of "telephone number customer service". Assuming that for this example, the system determines to remove the word "service" and determines not to remove other words, the remaining words in the target sentence (S2) are: "telephone number customer".

[0078] Using the remaining words of the target sentence (S2), the system can determine the guide word by reordering. Some embodiments can use a randomization function to randomize the order of words. In this example, the system can randomly reorder "telephone number customer" to "number telephone customer".

[0079] The guide words are part of the input to the data generator model, and the model is trained to generate new samples containing the guide words. More specifically, using the guide words determined for each of the N pairs, the system can determine N training inputs using the following format: "<guide>|<guide word 1><guide word 2>...<guide word k>". The guides "guide" the generator to generate similar sentences, and the guide words help constrain the generated content. Using this example, one training input can be:<How do Iobtain a refund> | <number> <telephone> <service>The output corresponding to this training input is the target. Thus, the data generator model is trained to construct the target from the primer (i.e., similar sentences) and the guide word (i.e., the word that appears in the target), so that the data generator model is trained to determine task-specific training examples.

[0080] Although this example details a single training input of a primer and a guide word, it should be understood that the system constructs N such pairs to determine a generator training set. The system (e.g., executing generator training module 125) trains a Seq2Seq (sequence to sequence) model using the generator training set (e.g., training input 121). For example, the generator training set can fine-tune a T5 pre-trained sequence to sequence model to generate new training examples (e.g., synthetic data 129).

[0081] Using the trained data generator model, the system (e.g., executing the dataset generator module 127) generates new training examples using task-specific data points. Some embodiments randomly select primers and guide words based on the frequency of primers and guide words in the task-specific data (e.g., the task-specific data 112). For example, the system constructs a mapping from words used in the task-specific data to the number of times each word is used. Then, when generating examples, the system samples guide words in proportion to the guide word frequency defined by the mapping. After generating the desired number of training examples, the system can add the training examples to a task-specific training set or validation set, as described above. The system (e.g., executing the machine learning training module 133) can use the new training examples to train one or more machine learning models (e.g., the production model 108).

[0082] F. Computer Networks and Cloud Networks

[0083] In one or more embodiments, a computer network provides connectivity between a collection of nodes. The nodes may be local to each other and / or remote from each other. The nodes are connected by a collection of links. Examples of links include coaxial cables, unshielded twisted pairs, copper cables, optical fibers, and virtual links.

[0084] A subset of nodes implements a computer network. Examples of such nodes include switches, routers, firewalls, and network address translators (NATs). Another subset of nodes uses a computer network. Such nodes (also referred to as "hosts") can execute client processes and / or server processes. A client process makes a request for a computing service (such as the execution of a particular application and / or the storage of a particular amount of data). The server process responds by performing the requested service and / or returning corresponding data.

[0085] A computer network may be a physical network, comprising physical nodes connected by physical links. A physical node is any digital device. A physical node may be a function-specific hardware device, such as a hardware switch, a hardware router, a hardware firewall, and a hardware NAT. Additionally or alternatively, a physical node may be a general-purpose machine configured to execute various virtual machines and / or applications that execute corresponding functions. A physical link is a physical medium that connects two or more physical nodes. Examples of links include coaxial cables, unshielded twisted cables, copper cables, and optical fibers.

[0086] A computer network may be an overlay network. An overlay network is a logical network implemented on top of another network (such as a physical network). Each node in the overlay network corresponds to a corresponding node in the underlying network. Therefore, each node in the overlay network is associated with both an overlay address (addressing the overlay node) and an underlying address (addressing the underlying node that implements the overlay node). An overlay node may be a digital device and / or a software process (such as a virtual machine, an application instance, or a thread). The link connecting the overlay nodes is implemented as a tunnel through the underlying network. The overlay nodes at either end of the tunnel treat the underlying multi-hop path between them as a single logical link. Tunneling is performed by encapsulation and decapsulation.

[0087] In one or more embodiments, the client can be located locally and / or remote from the computer network. The client can access the computer network through other computer networks, such as a private network or the Internet. The client can transmit the request to the computer network using a communication protocol, such as a hypertext transfer protocol (HTTP). The request is transmitted through an interface such as a client interface, such as a web browser, a program interface, or an application programming interface (API).

[0088] In one or more embodiments, a computer network provides a connection between a client and network resources. Network resources include hardware and / or software configured to execute a server process. Examples of network resources include processors, data storage devices, virtual machines, containers, and / or software applications. Network resources are shared between multiple clients. Clients request computing services from a computer network independently of each other. Network resources are dynamically allocated to requests and / or clients on demand. The network resources allocated to each request and / or client can be expanded or reduced based on, for example, (a) computing services requested by a specific client, (b) aggregate computing services requested by a specific tenant, and / or (c) the requested aggregate computing services of the computer network. Such a computer network may be referred to as a "cloud network."

[0089] In one or more embodiments, a service provider provides a cloud network to one or more end users. The cloud network can implement various service models, including but not limited to software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS). In SaaS, a service provider provides end users with the ability to use the service provider's applications that are being executed on network resources. In PaaS, a service provider provides end users with the ability to deploy customized applications on network resources. Custom applications can be created using programming languages, libraries, services, and tools supported by the service provider. In IaaS, a service provider provides end users with the ability to supply processing, storage, network, and other basic computing resources provided by network resources. Any arbitrary application, including an operating system, can be deployed on network resources.

[0090] In one or more embodiments, the computer network can implement various deployment models, including but not limited to private cloud, public cloud and hybrid cloud. In a private cloud, network resources are supplied to a specific group of one or more entities for exclusive use (the term "entity" as used herein refers to an enterprise, organization, individual or other entity). Network resources can be local to and / or remote from the premises of a specific entity group. In a public cloud, cloud resources are supplied to multiple entities (also referred to as "tenants" or "customers") that are independent of each other. The computer network and its network resources are accessed by clients corresponding to different tenants. Such a computer network can be referred to as a "multi-tenant computer network". Several tenants can use the same specific network resources at different times and / or at the same time. Network resources can be local to and / or remote from the premises of a tenant. In a hybrid cloud, the computer network includes a private cloud and a public cloud. The interface between the private cloud and the public cloud allows portability of data and applications. Data stored at a private cloud and data stored at a public cloud can be exchanged through an interface. Applications implemented at a private cloud and applications implemented at a public cloud may have dependencies on each other. Calls from an application at the private cloud to an application at the public cloud (and vice versa) may be performed through the interface.

[0091] In one or more embodiments, the tenants of a multi-tenant computer network are independent of each other. For example, the business or operation of one tenant can be separated from the business or operation of another tenant. Different tenants may have different network requirements for the computer network. Examples of network requirements include processing speed, data storage capacity, security requirements, performance requirements, throughput requirements, latency requirements, elasticity requirements, quality of service (QoS) requirements, tenant isolation and / or consistency. The same computer network may need to implement different network requirements required by different tenants.

[0092] In one or more embodiments, in a multi-tenant computer network, tenant isolation is implemented to ensure that applications and / or data of different tenants are not shared with each other. Various tenant isolation methods can be used.

[0093] In one or more embodiments, each tenant is associated with a tenant ID. Each network resource of the multi-tenant computer network is marked with a tenant ID. Only when the tenant and the specific network resource are associated with the same tenant ID, the tenant is allowed to access the specific network resource.

[0094] In one or more embodiments, each tenant is associated with a tenant ID. Each application implemented by the computer network is tagged with the tenant ID. Additionally or alternatively, each data structure and / or data set stored by the computer network is tagged with the tenant ID. A tenant is allowed to access a particular application, data structure, and / or data set only if the tenant and the particular application, data structure, and / or data set are associated with the same tenant ID.

[0095] As an example, each database implemented by a multi-tenant computer network can be marked with a tenant ID. Only the tenant associated with the corresponding tenant ID can access the data of a particular database. As another example, each entry in a database implemented by a multi-tenant computer network can be marked with a tenant ID. Only the tenant associated with the corresponding tenant ID can access the data of a particular entry. It should be understood that a database can be shared by multiple tenants.

[0096] In one or more embodiments, the subscription list indicates which tenants have access to which applications. For each application, a list of tenant IDs of tenants authorized to access the application is stored. A tenant is allowed to access a specific application only if the tenant ID of the tenant is included in the subscription list corresponding to the specific application.

[0097] In one or more embodiments, network resources corresponding to different tenants (such as digital devices, virtual machines, application instances, and threads) are isolated to tenant-specific overlay networks maintained by a multi-tenant computer network. As an example, a data packet from any source device in a tenant overlay network can be sent only to other devices within the same tenant overlay network. The encapsulation tunnel is used to prohibit any transmission from a source device on a tenant overlay network to devices in other tenant overlay networks. Specifically, a data packet received from a source device is encapsulated in an outer data packet. The outer data packet is sent from a first encapsulation tunnel endpoint (communicating with a source device in a tenant overlay network) to a second encapsulation tunnel endpoint (communicating with a destination device in a tenant overlay network). The second encapsulation tunnel endpoint decapsulates the outer data packet to obtain the original data packet sent by the source device. The original data packet is sent from the second encapsulation tunnel endpoint to a destination device in the same specific overlay network.

[0098] G. Other aspects; expansion

[0099] Embodiments are directed to a system having one or more devices comprising a hardware processor and configured to perform any of the operations described herein and / or in any of the following claims.

[0100] In one or more embodiments, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more hardware processors, cause performance of any of the operations described herein and / or in any of the claims.

[0101] Any combination of the features and functions described herein may be used according to one or more embodiments. In the foregoing description, various embodiments have been described with reference to many specific details, which may vary from one implementation to another. Therefore, the description and drawings should be considered in an illustrative rather than a restrictive sense. The sole and exclusive indication of the scope of the invention, and what the applicants intend to be the scope of the invention, is the literal and equivalent scope of the set of claims generated by this application, in the specific form in which such claims are generated, including any subsequent corrections.

[0102] H. Hardware Overview

[0103] According to one or more embodiments, the technology described herein is implemented by one or more special-purpose computing devices. The special-purpose computing device can be hard-wired to perform the technology, or can include digital electronic devices such as one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or network processing units (NPUs) that are permanently programmed to perform the technology, or can include one or more general-purpose hardware processors programmed to perform the technology according to program instructions in firmware, memory, other storage devices, or combinations. These special-purpose computing devices can also combine custom hard-wired logic, ASICs, FPGAs, or NPUs with custom programming to implement the technology. The special-purpose computing device can be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hard-wired and / or program logic to implement the technology.

[0104] For example, Figure 4 4 is a block diagram illustrating a computer system 400 on which one or more embodiments of the present invention may be implemented. Computer system 400 may be the same or similar to the computing systems previously described herein (e.g., machine learning system 105). Computer system 400 includes a bus 402 or other communication mechanism for communicating information and a hardware processor 404 coupled to bus 402 for processing information. Hardware processor 404 may be, for example, a general-purpose microprocessor.

[0105] Computer system 400 also includes a main memory 406, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 402 for storing information and instructions to be executed by processor 404. Main memory 406 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by processor 404. When these instructions are stored in a non-transitory storage medium accessible to processor 404, these instructions cause computer system 400 to become a special-purpose machine customized to perform the operations specified in the instructions.

[0106] Computer system 400 also includes a read only memory (ROM) 408 or other static storage device coupled to bus 402 for storing static information and instructions for processor 404. A storage device 410, such as a magnetic or optical disk, is provided and coupled to bus 402 for storing information and instructions.

[0107] The computer system 400 may be coupled via the bus 402 to a display 412, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 414, including alphanumeric and other keys, is coupled to the bus 402 for communicating information and command selections to the processor 404. Another type of user input device is a cursor control 416, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 404 and for controlling cursor movement on the display 412. Such input devices typically have two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), to allow the device to specify a position in a plane.

[0108] Computer system 400 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, makes computer system 400 a special purpose machine or programs computer system 400 to be a special purpose machine. According to one or more embodiments, the techniques herein are performed by computer system 400 in response to processor 404 executing one or more sequences of one or more instructions contained in main memory 406. These instructions may be read into main memory 406 from another storage medium, such as storage device 410. Execution of the sequences of instructions contained in main memory 406 causes processor 404 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0109] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 410. Volatile media include dynamic memory, such as main memory 406. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with hole patterns, RAM, PROM and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cassette, content addressable memory (CAM) and ternary content addressable memory (TCAM).

[0110] Storage media are distinct from but may be used in conjunction with transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires comprising bus 402. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0111] Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 404 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 400 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on bus 402. Bus 402 carries the data to main memory 406, from which processor 404 retrieves and executes the instructions. The instructions received by main memory 406 may optionally be stored on storage device 410 before or after execution by processor 404.

[0112] Computer system 400 also includes a communication interface 418 coupled to bus 402. Communication interface 418 provides bidirectional data communication coupled to network link 420, wherein network link 420 is connected to local network 422. For example, communication interface 418 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides data communication connection to a telephone line of a corresponding type. As another example, communication interface 418 can be a LAN card that provides data communication connection to a compatible local area network (LAN). Wireless links can also be implemented. In any such implementation, communication interface 418 sends and receives electrical signals, electromagnetic signals, or optical signals that carry digital data streams representing various types of information.

[0113] The network link 420 typically provides data communication to other data devices through one or more networks. For example, the network link 420 can provide a connection to a host computer 424 or to data equipment operated by an Internet Service Provider (ISP) 426 through a local network 422. The ISP 426, in turn, provides data communication services through a global packet data communication network now commonly referred to as the "Internet" 428. Both the local network 422 and the Internet 428 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 420 and through the communication interface 418 are example forms of transmission media that carry the digital data to or from the computer system 400.

[0114] Computer system 400 can send messages and receive data, including program code, through the network(s), network link 420, and communication interface 418. In the Internet example, server 430 can transmit the requested code for an application program through Internet 428, ISP 426, local network 422, and communication interface 418.

[0115] The received code may be executed by processor 404 as it is received, and / or stored in storage device 410 or other non-volatile storage for later execution.

[0116] In the foregoing specification, embodiments have been described with reference to numerous specific details that vary from implementation to implementation. Accordingly, the specification and drawings should be regarded as illustrative rather than restrictive. The sole and exclusive indicator of the scope of the invention, and what the applicants intend as the scope of the invention, is the literal and equivalent range of the set of claims issuing from this application, in the specific form in which such claims issue, including any subsequent corrections.< / service> < / telephone> < / number>

Claims

1. One or more non-transitory computer-readable media storing program instructions that, when executed by one or more hardware processors, cause operations to be performed, the operations comprising: Construct a set of training inputs for training the synthetic data generation model, Wherein constructing the set of training inputs comprises: Extracting a plurality of pairs of sentences satisfying a similarity criterion from a first training data set, each pair of the plurality of pairs of sentences comprising a first sentence and a second sentence; For each pair of the plurality of pairs of sentences: extracting a first subset of words from the first sentence, the first subset not including one or more words included in the first sentence; and A first training instance is generated based on the set of training inputs, the first training instance comprising: (a) a model input comprising the second sentence and the subset of words from the first sentence, and (b) a model output comprising the first sentence.

2. The medium of claim 1, wherein the operations further comprise: The synthetic data generation model is trained using the set of training inputs.

3. The medium of claim 2, wherein the operations further comprise: Synthetic data is generated using the trained synthetic data generation model.

4. The medium of claim 3, wherein: The first training data set includes general data; The second training dataset includes task-specific data; as well as Generating synthetic data includes applying the trained synthetic data generation model to sentences included in the second training data set.

5. The medium of claim 4, wherein: The trained synthetic data generation model is trained to construct the synthetic data from an introduction sentence and a plurality of guide words; The introduction causes the trained synthetic data generation model to generate similar sentences; as well as The plurality of guide words constrain the generation of the similar sentences relative to the introduction.

6. The medium of claim 5, wherein the synthetic data comprises a second set of training data for training a classifier.

7. The medium of claim 6, wherein the synthetic data comprises validation data for model selection of a classifier.

8. The medium of claim 7, wherein: The classifier is task-specific; The first training data set is not specific to the task; The second training data set is for the task.

9. The medium of claim 8, wherein the content of the first training set does not include any content of the second training set.

10. A method comprising the operations as claimed in any one of claims 1 to 9.

11. A system comprising a device having at least one hardware processor, the system being configured to perform the operations of any one of claims 1-9.

12. A system comprising means for performing the operations of any of claims 1-9.

Images