Variant Mismatch Attack (VIA) as a Simple and Effective Adversarial Attack Method

Abstract

A simple and effective adversarial attack methodology system and non-transitory computer-readable memory are provided for generating adversarial samples useful for adversarial training of machine learning models. [Solution] The method includes obtaining a set of input samples (the set of samples does not have corresponding labels) for attacking a machine learning model, modifying samples from the set of samples in a utility-preserving manner to generate pairs of modified samples, attacking the machine learning model with the pairs of modified samples to generate pairs of predictions for the pairs of modified samples, comparing the pairs of predictions to determine whether the pairs of predictions are the same or different, and adding the pairs of modified samples to a set of adversarial samples depending on whether the pairs of predictions are different.

Description

[Technical Field]

[0001] Priority claim This application is a nonprovisional patent application of and claims the benefit of U.S. Provisional Application No. 63 / 185,531, filed May 7, 2021, which is incorporated herein by reference in its entirety for all purposes.

[0002] Field The present disclosure relates to adversarial attacks, and in particular to techniques for using variant mismatch attacks (VIA) as a simple and effective adversarial attack method to generate adversarial samples useful for adversarial training of machine learning models. [Background technology]

[0003] background With the rapid development of machine learning techniques over the past two decades, machine learning has expanded to real-world problems such as facial recognition, fake news detection, and gene sequence analysis, to name a few. However, all machine learning models have proven time and time again to be imperfect. In fact, they are highly vulnerable to adversarial examples carefully crafted to cause the trained model to fail. Adversarial examples that can fool a model can have serious adverse effects on applications that handle sensitive information, such as health analysis, financial analysis, and personal shopping behavior analysis. For example, adversarial examples crafted to create a model that redacts personal information could leak sensitive information or cause catastrophic damage to applications like autonomous driving.

[0004] Therefore, defending against such adversarial attacks before bringing a model into production is an important and challenging topic in machine learning research. Previous research has found that adversarial training, i.e., mixing adversarial examples with the original training dataset, is an effective way to defend models against adversarial attacks. Naturally, how to automatically generate adversarial examples for adversarial training is a key problem in defending against such attacks. Summary of the Invention

[0005] overview

[0001] The present disclosure relates generally to adversarial attacks. More specifically, the techniques described herein relate to using VIA as a simple and effective adversarial attack method for generating adversarial samples useful for adversarial training of machine learning models. Various inventive embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions by one or more processors, and the like.

[0006] In various embodiments, a computer-implemented method is provided, the method including: obtaining a set of input samples for attacking a machine learning model, the set of samples having no corresponding labels; modifying samples from the set of samples in a utility-preserving manner to generate modified sample pairs, the utility-preserving manner including modifying the samples without changing their original meanings or expected labels, the modified sample pairs sharing the same expected labels; and attacking the machine learning model with the modified sample pairs, the attacking including (i) modifying the modified samples while maintaining the same expected labels unknown to the machine learning model. The method includes (i) inputting pairs of predictions into a machine learning model, and (ii) generating pairs of predictions for the pairs of correction samples using the machine learning model; comparing the pairs of predictions to determine whether the pairs of predictions are the same or different; and, in response to the pairs of predictions being different, adding the pairs of correction samples to a set of adversarial samples.

[0007] In some embodiments, the method further includes training the machine learning model or a different machine learning model using the set of adversarial examples, where the training further includes annotating each sample in the set of adversarial samples with an expected label to obtain annotated adversarial samples; feeding the annotated adversarial samples to the machine learning model or a different machine learning model; and learning a plurality of model parameters using the annotated adversarial examples based on maximizing or minimizing an objective function.

[0008] In some embodiments, the method further includes, prior to adding the pair of modified samples to the set of pairs of adversarial samples in response to the pair of predictions being dissimilar, determining individually whether each modified sample of the pair of modified samples is a true adversarial sample, and restricting the addition of modified samples to the set of adversarial samples to only those modified samples that are deemed to be true adversarial samples.

[0009] In some embodiments, the machine learning model is part of a chatbot system. In some embodiments, the machine learning model is a named entity recognition model, an intent recognition model, or an image recognition model.

[0010] In some embodiments, the modifying step comprises adding noise to the samples.

[0011] In some embodiments, the modifying step includes selecting one or more characters or words from the sample and performing an insert, delete, or replace operation on the one or more characters or words.

[0012] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein.

[0013] In some embodiments, a computer program product is provided, the product tangibly embodied in a non-transitory machine-readable storage medium and including instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0014] The above, together with further features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a simplified block diagram of an environment incorporating a chatbot system according to various embodiments. [Figure 2] FIG. 1 is a simplified block diagram of a computing system implementing a master bot according to various embodiments. [Figure 3] FIG. 1 is a simplified block diagram of a computing system implementing a skillbot according to various embodiments. [Figure 4A] 1 is a pair of adversarial samples of an image according to various embodiments. [Figure 4B] 1 is a pair of adversarial samples of an image according to various embodiments. [Figure 5] FIG. 1 is a simplified block diagram illustrating a computing environment for implementing a variant mismatch attack in accordance with various embodiments. [Figure 6] 1 is a process flow for a variant mismatch attack in accordance with various embodiments. [Figure 7]1 is a simplified diagram of a distributed system for implementing various embodiments. [Figure 8] FIG. 1 is a simplified block diagram of one or more components of a system environment whereby services provided by one or more components of an embodiment system may be offered as cloud services, according to various embodiments. [Figure 9] FIG. 1 is a diagram of an exemplary computer system that may be used to implement various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0016] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular invention embodiments. It will be apparent, however, that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0017] introduction Recently, new algorithms have been proposed to address the problem of how to automatically generate adversarial examples. For example, Goodfellow et al. J Goodfellow, Jonathon Shlens, and Christian Szegedy.Explaining and harnessing adversarial examples.In Proceedings of the 2015 International Conference on Learning Representation), a fast gradient signature method (FGSM)-based white-box algorithm, the authors claim that by adding small perturbations to the panda images, the attack algorithm successfully fools GoogLeNet (Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. Technical report, arXiv preprint arXiv:1409.4842, 2014a) into misclassifying the adversarial panda image as a gibbon. Biggio et al. (B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Smdi'c, P. Laskov, G. Giacinto, and F. Roli, "Evasion attacks against machine-learning at test time," in Joint European Conference on Machine-learning and Knowledge Discovery in Databases, Springer, 2013, pp. 387-402) is also a gradient-based attack method, in which the authors generate adversarial examples for linear classifiers, support vector machines (SVMs), and neural networks by manipulating the gradient of the discriminant function. In another work, Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2017. Deep Text Classification Can be Fooled. arXiv preprint arXiv:1704.08006(2017)), the authors extend the idea of ​​FGSM to text data. By defining three different actions (insertion, deletion, and modification) and manipulating the magnitude of the cost gradient, they successfully generate adversarial examples of text to fool DNN-based classifiers.

[0018] Compared to white-box attacks, black-box attacks appear more attractive because they are model-independent and do not assume prior knowledge of the victim model's internal architecture and parameters. Black-box models only require access to the victim model's inputs and outputs. The victim model can be as complex as a multi-layer deep neural network or as simple as a linear regression model. In Jin et al. (Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Textfool: Fool your model with natural adversarial text. http: / / groups.csail.mit.edu / medg / ftp / psz-papers / 2019%20Di%20Jin.pdf, 2019), the authors use a black-box attack method to attack NLP models by carefully selecting synonyms for some selected keywords, and by applying several other techniques such as filtering stop words and calculating semantic similarities between texts. The authors claim that they outperform state-of-the-art techniques in terms of success rate and perturbation rate. Gao et al. (Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. arXiv preprint arXiv:1801.04354) uses a similar idea to Jin et al., but in a different way by selecting important words and transforming tokens.Jia et al.'s (Robin Jia and Percy Liang. 2017. Adversarial Examples for Evaluating Reading Comprehension Systems. In Proc. Of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017). Copenhagen, Denmark, 2021-2031) is an alternative work for attacking reading comprehension systems. They trick the victim model by adding nonsensical sentences to the end of the paragraph, maintaining the same meaning for the paragraph and the answer to the question.

[0019] While these existing attack methods can generate adversarial samples, they typically require labeled data as input for the attack algorithm, which is difficult to obtain in real-world applications. To address these and other issues, the present disclosure introduces an adversarial attack technique that releases the labeling constraint and utilizes unlabeled data as input, thereby dramatically increasing the number of adversarial samples that can be generated. The proposed algorithm and adversarial attack technique for variant disagreement (VI) are based on a simple but effective logic: it assumes that a given input has only one correct prediction; that is, if a model generates different predictions, at least one of them must be incorrect. By applying VI, it is possible to remove the constraint of requiring labeled data to create adversarial samples useful for training machine learning models.

[0020] An exemplary embodiment of the present disclosure is directed to a computer-implemented process, the process including: obtaining a set of input samples for attacking a machine learning model, the set of samples having no corresponding labels; modifying examples from the set of samples in a utility-preserving manner to generate modified sample pairs, the utility-preserving manner including modifying the samples without changing their original meanings or expected labels, the modified sample pairs sharing the same expected labels; attacking the machine learning model with the modified sample pairs, the attacking including: (i) inputting the modified sample pairs into the machine learning model while maintaining the same expected labels unknown to the machine learning model; and (ii) generating, by the machine learning model, pairs of predictions for the modified sample pairs; comparing the pairs of predictions to determine whether the pairs of predictions are the same or different; and, in response to the pairs of predictions being different, adding the modified sample pairs to a set of adversarial samples.

[0021] Bot System The adversarial attack techniques described herein for generating adversarial samples can be used as an effective method for defending models against adversarial attacks. In various embodiments, the models may be implemented as part of a digital assistant, although it should be understood that the techniques described herein are applicable to other contexts, such as image recognition. A digital assistant is an artificial intelligence-driven interface that assists users in accomplishing various tasks through natural language conversation. For each digital assistant, customers may assemble one or more skills. A skill (also described herein as a chatbot, bot, or skillbot) is an individual bot focused on a specific type of task, such as tracking inventory, submitting a time card, or creating an expense report. When an end user engages with a digital assistant, the digital assistant evaluates the end-user input and routes a round-trip conversation with the appropriate chatbot. The digital assistant may be made available to end users through various channels, such as Facebook Messenger, SKYPE MOBILE Messenger, or Short Message Service (SMS). Channels carry chats from end users on various messaging platforms to the digital assistant and its various chatbots. Channels may also support user-agent escalation, event-initiated conversations, and testing.

[0022] Intents enable a chatbot to understand what a user wants it to do. Intents consist of a sequence of typical user requests and statements, also called utterances (e.g., getting an account balance, making a purchase, etc.). As used herein, utterance or message may refer to a set of words (e.g., one or more sentences) exchanged during a conversation with a chatbot. An intent may be created by providing a name for some user action (e.g., ordering a pizza) and compiling a set of utterances that are commonly associated with actual user statements or action triggers. Because chatbot recognition is inferred from these intents, each intent may be created from a robust and diverse dataset (1-24 utterances) so that the chatbot may interpret ambiguous user input. A rich set of utterances enables the chatbot to understand what a user wants when it receives messages like "Forget this order!" or "Cancel delivery!"—messages that mean the same thing but are expressed differently. Collectively, intents and their associated utterances create a training corpus for the chatbot. By training a model using a corpus, customers may essentially turn the model into a reference tool for fitting end-user input into a single intent. Customers can improve the chatbot's cognitive clarity through repeated intent testing and intent training.

[0023] A bot (also called a skill, chatbot, chatterbot, or talkbot) is a computer program that can conduct a conversation with an end user. Bots can respond to natural language messages (e.g., questions or comments) typically through a messaging application using natural language messages. A business may use one or more bot systems to communicate with end users through messaging applications. The messaging application, sometimes called a channel, may be an end user's preferred messaging application that the end user already has installed and is familiar with. Thus, end users do not need to download and install a new application to chat with a bot system. Messaging applications may include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, SMS), virtual private assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), mobile and web app extensions that extend native or hybrid / responsive mobile apps or web applications with chat capabilities, or voice-based input (such as Siri, Cortana, Google Voice, or another device or app with an interface that uses voice input for interaction).

[0024] In some examples, a bot system may be associated with a uniform resource identifier (URI). The URI may identify the bot system using a string of characters. The URI may be used as a webhook for one or more messaging application systems. The URI may include, for example, a uniform resource locator (URL) or a uniform resource name (URN). The bot system may be designed to receive a message (e.g., a HyperText Transfer Protocol (HTTP) post call message) from the messaging application system. The HTTP post call message may be directed to the URI from the messaging application system. In some embodiments, the message may be different from an HTTP post call message. For example, the bot system may receive a message via short message service (SMS). While the description herein may refer to a communication received by the bot system as a message, it should be understood that the message may be an HTTP post call message, an SMS message, or any other type of communication between two systems.

[0025] End users may interact with bot systems through conversational interactions (sometimes called conversational user interfaces (UIs)), just like human-to-human interactions. In some cases, an interaction may involve the end user saying "hello" to the bot, and the bot responding "hi" and asking the end user how things are going. In some cases, an interaction may also be a transactional interaction with a banking bot, such as transferring money from one account to another, an informational interaction with an HR bot, such as checking a vacation balance, or an interaction with a retail bot, such as discussing the return of a purchased item or requesting technical support.

[0026] In some embodiments, the bot system may intelligently handle end-user interactions without interaction with an administrator or developer of the bot system. For example, an end-user may send one or more messages to the bot system to achieve a desired goal. The bot system may send a message to the end user. The message may include specific content such as text, emojis, audio, images, video, or another method of conveying a message. In some embodiments, the bot system may convert the content into a standardized format (e.g., a Representational State Transfer (REST) ​​call to an enterprise service with appropriate parameters) and generate a natural language response. The bot system may also prompt the end user for additional input parameters or request other additional information. In some embodiments, the bot system may also initiate communication with the end user rather than passively responding to the end user's utterance. Various techniques are described herein for identifying explicit invocations of the bot system and determining input for the invoked bot system. In certain embodiments, explicit invocation analysis is performed by the master bot based on detection of a call name in the utterance. In response to detection of the call name, the utterance may be refined for input to a skill bot associated with the call name.

[0027] A conversation with a bot may follow a specific conversational flow that includes multiple states. The flow may define what happens next based on the input. In some embodiments, a state machine that includes user-defined states (e.g., end-user intents) and actions to perform in those states or actions from state to state may be used to implement a bot system. A conversation may follow different paths based on end-user input, which may affect the decisions the bot makes for the flow. For example, at each state, based on the end-user input or utterance, the bot may determine the end-user's intent to determine the next appropriate action to take. When used in connection with an utterance herein, the term "intent" refers to the intent of the user who provided the utterance. For example, a user may intend to engage with a bot in a conversation to order a pizza, and thus the user's intent may be expressed by the utterance "order a pizza." A user's intent may be directed toward a specific task the user wants the chatbot to perform on their behalf. As such, an utterance may be stated as a question, command, request, etc. that reflects the user's intent. An intent may include a goal the end user wants to achieve.

[0028] In connection with configuring a chatbot, the term “intent” is used herein to refer to configuration information for mapping a user utterance to a specific task / action or category of task / action that the chatbot can perform. To distinguish between the intent of an utterance (i.e., a user's intent) and the intent of a chatbot, the latter may be referred to herein as a “bot intent.” A bot intent may include a set of one or more utterances associated with the intent. For example, an intent to order a pizza may include various permutations of utterances expressing a desire to order a pizza. These related utterances may be used to train the chatbot's intent classifier, enabling the intent classifier to subsequently determine whether an input utterance from a user matches the pizza-ordering intent. A bot intent may be associated with one or more dialog flows for initiating a conversation with a user in a particular state. For example, the initial message for a pizza-ordering intent may be the question, “What kind of pizza would you like?” In addition to the associated utterances, a bot intent may further include named entities associated with the intent. For example, an order pizza intent may include variables or parameters used to perform the task of ordering a pizza, e.g., topping 1, topping 2, type of pizza, size of pizza, quantity of pizza, etc. The values ​​of the entities are typically obtained by conversing with the user.

[0029] 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to certain embodiments. The environment 100 comprises a Digital Assistant Builder Platform (DABP) 102, which enables users of the DABP 102 to create and deploy digital assistant or chatbot systems. The DABP 102 is used to build one or more digital assistant (or DA) or chatbot systems. For example, as shown in FIG. 1, a user 104 representing a particular business may use the DABP 102 to create and deploy a digital assistant 106 for users of the particular business. For example, the DABP 102 may be used by a bank to create one or more digital assistants for use by the bank's customers. The same DABP 102 platform may be used by multiple businesses to create digital assistants. As another example, a restaurant (e.g., a pizza place) owner may use the DABP 102 to create and deploy a digital assistant that allows customers of the restaurant to order food (e.g., order pizza).

[0030] For purposes of this disclosure, a "digital assistant" is an entity that assists a user of the digital assistant in accomplishing various tasks through natural language conversation. A digital assistant may be implemented using software alone (e.g., a digital assistant is a digital entity implemented using programs, codes, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant may be embodied or implemented in various physical systems or devices, such as a computer, a mobile phone, a watch, an appliance, a vehicle, etc. A digital assistant may also be referred to as a chatbot system. Thus, for purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0031] Digital assistants, such as digital assistant 106 built using DABP 102, can be used to perform various tasks through natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user provides one or more user inputs 110 to the digital assistant 106 and obtains responses 112 from the digital assistant 106. A conversation can include one or more of the inputs 110 and responses 112. Through these conversations, the user can request one or more tasks to be performed by the digital assistant, and in response, the digital assistant is configured to perform the user-requested tasks and respond to the user with appropriate responses.

[0032] User input 110 is generally in the form of natural language and is referred to as an utterance. User utterance 110 can be in text form, such as when a user types a sentence, a question, a text fragment, or a single word and provides it as input to the digital assistant 106. In some embodiments, user utterance 110 can be in voice input or conversational form, such as when a user says or speaks something that is provided as input to the digital assistant 106. The utterance is typically in the language spoken by the user 108. For example, the utterance may be in English or some other language. If the utterance is conversational, the conversational input is converted into textual speech in that particular language, and the textual speech is then processed by the digital assistant 106. Various speech-to-text processing techniques may be used to convert speech or voice input into textual speech, which is then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion may be performed by the digital assistant 106 itself.

[0033] The utterance, which may be a text utterance or a conversational utterance, may be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the aforementioned types, etc. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing of the utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, which may include identifying one or more intents and one or more entities corresponding to the utterance. Upon understanding the meaning of the utterance, the digital assistant 106 may perform one or more actions or behaviors according to the understood meaning or intent. For the purposes of this disclosure, the utterance is assumed to be a text utterance provided directly by a user 108 of the digital assistant 106 or the result of converting an input conversational utterance into text form. However, this is not intended to be limiting or restrictive in any way.

[0034] For example, user 108's input may request that a pizza be ordered by providing an utterance such as "I would like to order a pizza." Upon receiving such an utterance, digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. The appropriate action may include, for example, responding to the user with a question requesting user input about the type of pizza the user wants to order, the size of the pizza, any toppings on the pizza, etc. The responses provided by digital assistant 106 may be in natural language format, typically in the same language as the input utterance. As part of generating these responses, digital assistant 106 may perform natural language generation (NLG). For a user ordering a pizza, through a conversation between the user and digital assistant 106, the digital assistant may guide the user to provide all the information needed to order the pizza, and then at the end of the conversation, the pizza may be ordered. Digital assistant 106 may end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0035] At a conceptual level, digital assistant 106 performs various processes in response to utterances received from a user. In some embodiments, this processing includes a series or pipeline of processing steps, including, for example, understanding the meaning of the input utterance (sometimes referred to as natural language understanding (NLU)), determining an action to perform in response to the utterance, causing the action to be performed if appropriate, generating a response to be output to the user in response to the user's utterance, and outputting the response to the user. NLU processing includes parsing the received input utterance to understand the structure and meaning of the utterance, and refining and restructuring the utterance to develop a more understandable form (e.g., logical form) or structure for the utterance. Generating a response may include using NLG techniques.

[0036] NLU processing performed by a digital assistant, such as digital assistant 106, may include various NLP-related processes such as syntactic analysis (e.g., tokenization, lemmatization, identifying part-of-speech tags for sentences, identifying named entities in sentences, generating dependency trees to represent sentence structure, dividing sentences into clauses, parsing individual clauses, analyzing anaphora, performing chunking, etc.). In particular embodiments, the NLU processing, or portions thereof, is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may use other resources to perform portions of the NLU processing. For example, the syntax and structure of an input spoken sentence may be identified by processing the sentence using a syntactic parser, a part-of-speech tagger, and / or a named entity recognizer. In one embodiment, for English, a syntactic parser, a part-of-speech tagger, and a named entity recognizer, such as those provided by the Stanford Natural Language Processing (NLP) Group, are used to parse sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0037] Although the various samples provided in this disclosure show utterances in English, this is merely an example. In certain embodiments, the digital assistant 106 is also capable of handling utterances in languages ​​other than English. The digital assistant 106 may perform processing of different languages. The NLU core server may provide subsystems (e.g., components that implement NLU functionality) that are configured to:

[0038] 1 can be available or accessible to user 108 through a variety of different channels, such as, but not limited to, through a particular application, through a social media platform, various messaging services and applications, and other applications or channels. A single digital assistant can have several channels configured for it, thereby running on and being accessed by different services simultaneously.

[0039] Digital assistant or chatbot systems typically include or are associated with one or more skills. In particular embodiments, these skills are individual chatbots (referred to as skillbots) that are configured to interact with users to perform specific types of tasks, such as tracking inventory, submitting timecards, creating expense reports, ordering food, checking bank accounts, making reservations, purchasing widgets, etc. For example, for the embodiment shown in FIG. 1 , the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For purposes of this disclosure, the term "skill" is used synonymously with the term "skillbot."

[0040] Each skill associated with a digital assistant helps a user of the digital assistant complete a task through a conversation with the user, which may include a combination of text or voice input provided by the user and responses provided by the skill bot. These responses may be in the form of text or voice messages to the user and / or may use simple user interface elements (e.g., selection lists) presented to the user for selection.

[0041] There are various ways in which skills or skillbots can be associated with or added to a digital assistant. In some cases, skillbots may be developed by a company and then added to a digital assistant using DABP 102. In other cases, skillbots may be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In yet another example, DABP 102 provides an online digital store (referred to as a "skill store") that offers multiple skills aimed at a wide range of tasks. Skills offered by the skill store may also expose various cloud services. To add a skill to a digital assistant being created using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skill, and indicate that the selected skill be added to the digital assistant created using DABP 102. Skills from the skill store can be added to a digital assistant either as is or in modified form (e.g., a user of DABP102 may select and clone a particular skillbot provided by the skill store, customize or modify the selected skillbot, and then add the modified skillbot to a digital assistant created using DABP102).

[0042] A variety of different architectures may be used to implement a digital assistant or chatbot system. For example, in a particular embodiment, the DABP 102 is Digital assistants created and deployed using may be implemented using a masterbot / child (or sub)bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a masterbot that interacts with one or more childbots, which are skillbots. For example, in the embodiment shown in FIG. 1 , the digital assistant 106 comprises a masterbot 114 and skillbots 116-1, 116-2, etc., that are childbots of the masterbot 114. In certain embodiments, the digital assistant 106 itself is considered to act as a masterbot.

[0043] A digital assistant implemented according to the master-child bot architecture allows users of the digital assistant to interact with multiple skills through a unified user interface, i.e., through a master bot. When a user engages with the digital assistant, user input is received by the master bot. The master bot then performs processing to determine the meaning of the user input utterance. The master bot then determines whether the task requested by the user in the utterance can be handled by the master bot itself. If not, the master bot selects an appropriate skill bot to handle the user request and routes the conversation to the selected skill bot. This allows users to interact with the digital assistant through a common, single interface and still provide the ability to use several skill bots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the digital assistant's master bot may interface with skill bots having specific functions, such as a CRM bot for performing functions related to customer relationship management (CRM), an ERP bot for performing functions related to enterprise resource planning (ERP), an HCM bot for performing functions related to human capital management (HCM), etc. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a common master bot interface, and behind the scenes multiple skill bots are provided to handle user requests.

[0044] In certain embodiments, in a masterbot / childbot infrastructure, the masterbot is configured to recognize an available list of skillbots. The masterbot may have access to metadata identifying various available skillbots and, for each skillbot, the skillbot's capabilities, including tasks that can be performed by the skillbot. Upon receiving a user request in the form of an utterance, the masterbot is configured to identify or predict a specific skillbot from multiple available skillbots that can best accommodate or process the user request. The masterbot then routes the utterance (or portion of the utterance) to that specific skillbot for further handling. Control thus flows from the masterbot to the skillbot. The masterbot may support multiple input and output channels. In certain embodiments, routing may be performed utilizing processing performed by one or more available skillbots. For example, as discussed below, a skillbot may be trained to infer the intent of an utterance and determine whether the inferred intent matches the intent with which the skillbot is configured. Thus, routing performed by the masterbot may include the skillbot communicating to the masterbot an indication of whether the skillbot is configured with an appropriate intent to handle the utterance.

[0045] The embodiment of FIG. 1 illustrates a digital assistant 106 with a masterbot 114 and skillbots 116-1, 116-2, and 116-3, which is not intended to be limiting. A digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems The systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in implementations using a combination of software and hardware.

[0046] DABP 102 provides infrastructure and various services and features that enable a user of DABP 102 to create a digital assistant, including one or more skillbots associated with the digital assistant. In some cases, a skillbot may be created by cloning an existing skillbot, for example, by cloning a skillbot provided by a skill store. As described above, DABP 102 provides a skill store or skill catalog that offers multiple skillbots for performing various tasks. A user of DABP 102 may clone a skillbot from the skill store. Modifications or customizations may be made to the cloned skillbot, as needed. In other examples, a user of DABP 102 created a skillbot from scratch using tools and services provided by DABP 102. As described above, a skill store or skill catalog provided by DABP 102 may offer multiple skillbots for performing various tasks.

[0047] In certain embodiments, at a high level, creating or customizing a skillbot involves: (1) configuring settings for a new skill bot; (2) configuring one or more intents for a skillbot; (3) configuring one or more entities for one or more intents; (4) training the skill bot; (5) Creating a dialogue flow for the skill bot; (6) adding custom components to the skill bot as needed; (7) testing and deploying the skill bot; Includes: Each of the above steps is briefly described below.

[0048] (1) Configuring Settings for a New Skillbot—Various settings may be configured for a skillbot. For example, a skillbot designer may specify one or more call names for the skillbot to be created. These call names may then be used by a user of the digital assistant to explicitly call the skillbot. For example, a user may enter a call name in a user utterance to explicitly call the corresponding skillbot.

[0049] (2) Configuring One or More Intents and Associated Example Utterances for the Skillbot—A skillbot designer specifies one or more intents (also called bot intents) for the skillbot being created. The skillbot is then trained based on these specified intents. These intents represent categories or classes that the skillbot is trained to infer input utterances. Upon receiving an utterance, the trained skillbot infers the intent for the utterance, and the inferred intent is selected from a set of predefined intents used to train the skillbot. The skillbot then performs an appropriate action in response to the utterance based on the inferred intent for the utterance. In some cases, the intent for the skillbot represents a task that the skillbot can perform for a user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skillbot trained for banking, the intents specified for the skillbot may be "CheckBalance," "TransferMoney," "Depo," and so on. It may also include "sitCheck".

[0050] For each intent defined for a skillbot, the skillbot designer may provide one or more example utterances that express and demonstrate the intent. These example utterances are intended to represent utterances that a user may input to the skillbot for the intent. For example, for the CheckBalance intent, example utterances may include "How much is my savings balance?", "How much is in my checking account?", "How much is in my account?", etc. Thus, various permutations of typical user utterances may be specified as example utterances for the intent.

[0051] The intents and their associated example utterances are used as training data to train the skillbot. A variety of different training techniques may be used. This training results in a predictive model configured to execute the utterance as input and output an intent inferred for the utterance by the predictive model. In some cases, the input utterance is provided to an intent analysis engine, which is configured to predict or infer an intent for the input utterance using the trained model. The skillbot may then perform one or more actions based on the inferred intent.

[0052] (3) Configuring entities for one or more intents of the skill bot—In some cases, additional context may be required to enable the skill bot to respond appropriately to a user utterance. For example, there may be situations where a user input utterance resolves to the same intent within the skill bot. For example, in the above example, the utterances "How much is my savings account balance?" and "How much is in my checking account?" both resolve to the same CheckBalance intent, but these utterances are different requests asking about different things. To disambiguate such requests, one or more entities are added to the intent. Using the banking skill bot example, an entity called AccountType that defines values ​​called "checking" and "savings" may enable the skill bot to parse the user request and respond appropriately. In the above example, the utterances fall into the same intent, but the values ​​associated with the AccountType entity are different for the two utterances. This allows the skill bot to potentially perform different actions for the two utterances, even though they fall into the same intent. One or more entities may be specified for a particular intent configured for the skill bot. Entities are therefore used to add context to the intent itself: they help to more completely describe the intent and enable the skill bot to complete the user request.

[0053] In certain embodiments, there are two types of entities: (a) built-in entities provided by DABP 102 and (2) custom entities that can be specified by a skill bot designer. Built-in entities are generic entities that can be used in a wide variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, recurring period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for a banking skill, an AccountType entity may be defined by the skill bot designer that enables various banking transactions by checking user input for keywords such as check, savings, credit card, etc.

[0054] (4) Training a SkillBot—The SkillBot is configured to receive user input in the form of speech parsing, otherwise process the received input, and identify or select an intent associated with the received user input. As described above, the SkillBot The skillbot needs to be trained for this. In particular embodiments, the skillbot is trained based on intents configured for the skillbot and example utterances associated with the intents (collectively, training data), so that the skillbot can fit a user input utterance into one of its configured intents. In particular embodiments, the skillbot uses a trained predictive model using the training data, allowing the skillbot to identify what a user is saying (or, in some cases, intending to say). DABP 102 provides a variety of different training techniques that a skillbot designer may use to train the skillbot, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In particular embodiments, a portion (e.g., 80%) of the training data is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes referred to as a trained skillbot) may be used to handle and respond to user utterances. In some cases, a user utterance may be a question that requires only a single answer and does not require further conversation. To handle such situations, a Q&A (Question and Answer) intent may be defined for a skill bot. This allows the skill bot to output a response to a user request without needing to update the dialog definition. A Q&A intent is created in the same manner as a regular intent. The dialog flow for a Q&A intent may differ from the dialog flow for a regular intent.

[0055] (5) Creating a Dialog Flow for the Skill Bot—The dialog flow specified for the skill bot describes how the skill bot will react as various intents for the skill bot are resolved in response to received user input. The dialog flow defines the behaviors or actions that the skill bot will perform, for example, how the skill bot will respond to user utterances, how the skill bot will prompt the user for input, and how the skill bot will return data. The dialog flow is like a flowchart that the skill bot follows. Skill bot designers specify the dialog flow using a language like Markdown. In certain embodiments, a version of YAML called OBotML may be used to specify the dialog flow for the skill bot. The dialog flow definition for the skill bot acts as a model for the conversation itself, allowing the skill bot designer to orchestrate the interaction between the skill bot and the user that the skill bot serves.

[0056] In particular embodiments, the dialog flow definition for a skill bot may include: (a) the context section, (b) Default transition section, (c) Status section; It includes the following three sections:

[0057] Context Section - The skill bot designer may define variables used in the conversation flow in the context section. Other variables that may be named in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, user variables that allow the skill bot to recognize and preserve user preferences, etc.

[0058] Default Transition Section - Transitions for a skill bot can be defined in the dialog flow state section or the default transition section. Transitions defined in the default transition section act as fallbacks and are triggered if there are no applicable transitions defined in a state or if the conditions required to trigger a state transition are not met. The default transition section can be used to define routing that allows the skill bot to gracefully handle unexpected user actions.

[0059] State Section - A dialog flow and its associated behaviors are defined as a series of temporary states that manage the logic within the dialog flow. Each state node in a dialog flow definition names a component that provides the functionality needed at that point in the dialog. Thus, states are built around components. A state contains component-specific characteristics and defines a transition to another state that is initiated after the component executes.

[0060] Special case scenarios may be handled using the state section. For example, you may want to offer a user the option to temporarily leave a first skill they are currently working on to do something in a second skill within the digital assistant. For example, if a user is engaged in a conversation with a shopping skill (e.g., the user has made some choices about a purchase), the user may want to jump to a banking skill (e.g., the user may want to confirm they have enough money for the purchase), and then return to the shopping skill to complete the user's order. To address this, an action in the first skill may be configured to initiate an interaction with a second, different skill within the same digital assistant, and then return to the original flow.

[0061] (6) Adding Custom Components to the Skillbot—As described above, a state specified in the dialog flow for a skillbot names a component that provides the functionality needed to respond to that state. The component enables the skillbot to perform its function. In certain embodiments, DABP 102 provides a set of pre-configured components to perform a wide range of functions. A skillbot designer can select one or more of these pre-configured components and associate them with states in the skillbot's dialog flow. A skillbot designer can create custom or new components using tools provided by DABP 102 and associate the custom components with one or more states in the skillbot's dialog flow.

[0062] (7) Testing and Deploying Skillbots—DABP 102 provides several features that allow skillbot designers to test the skillbots being developed, which can then be deployed and included within a digital assistant.

[0063] While the above description describes how to create a skillbot, similar techniques may also be used to create a digital assistant (or masterbot). At the masterbot or digital assistant level, built-in system intents may be configured for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the masterbot) can handle without invoking a skillbot associated with the digital assistant. Examples of system intents defined for the masterbot are: (1) Exit: applied when a user signals a desire to end the current conversation or context within the digital assistant; (2) Help: applied when a user asks for assistance or direction; and (3) Unanalyzed Intent: applied to user input that does not match well with the Exit and Help intents. The digital assistant also stores information about one or more skillbots associated with the digital assistant. This information allows the masterbot to select a specific skillbot to handle an utterance.

[0064] At the master bot or digital assistant level, when a user enters a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and associated conversation. The digital assistant determines this using a routing model, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model is used to determine whether a conversation corresponding to a user input utterance should be routed to a specific skill for handling, handled by the digital assistant or MasterBot itself according to the built-in system intent, or treated as a separate state within the current conversation flow.

[0065] In certain embodiments, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its call name. If a call name is present in the user input, it is treated as an explicit call to the skill bot corresponding to the call name. In such a scenario, the digital assistant may route the user input to the explicitly called skill bot for further handling. In the absence of a specific or explicit call, in certain embodiments, the digital assistant evaluates the received user input utterance and calculates a confidence score for the system intent and the skill bot associated with the digital assistant. The calculated score for the skill bot or system intent represents the likelihood that the user input is indicative of a task the skill bot is configured to perform or represents a system intent. Any system intent or skill bot with an associated calculated confidence score that exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further handling of the user input utterance. In certain embodiments, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to an intent model for each skill), and a confidence score is determined for each intent. Generally, any intent with a confidence score above a threshold (e.g., 70%) is treated as a candidate intent. If a particular skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed by the master bot itself according to the selected system intent.

[0066] FIG. 2 is a simplified block diagram of a MasterBot (MB) system 200 according to certain embodiments. The MB system 200 may be implemented in software only, hardware only, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multiple-intention subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skillbot invoker 240, and a data store 250. The MB system 200 shown in FIG. 2 is only one example of an arrangement of components within a MasterBot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the MB system 200 may have more or fewer systems or components than those shown in FIG. 2, may combine two or more subsystems, or may have a different configuration or arrangement of subsystems.

[0067] The pre-processing subsystem 210 receives the utterance "A" 202 from the user and processes the utterance through a language detector 212 and a language parser 214. As noted above, the utterance may be provided in a variety of ways, including audio or text. The utterance 202 may be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 may include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 may convert the audio to text using a speech-to-text converter (not shown) that inserts punctuation, such as commas, semicolons, periods, etc., into the resulting text.

[0068] The language detector 212 detects the language of the utterance 202 based on the text of the utterance 202. The way the utterance 202 is treated is language dependent, as each language has its own grammar and semantics. Differences between languages ​​are taken into account when analyzing the syntax and structure of the utterance.

[0069] The language parser 214 parses the utterance 202 to extract part-of-speech (POS) tag portions for individual linguistic units (e.g., words) within the utterance 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (VB), etc. The language parser 214 may also tokenize the linguistic units of the utterance 202 (e.g., to convert each word into a separate token) and lemmatize the words. A lemma is the primary form of a set of words as represented in a dictionary (e.g., "run" is a lemma for run, runs, ran, running, etc.). Another type of preprocessing that the language parser 214 may perform includes chunking complex expressions, e.g., combining "credit" and "card" into a single expression "credit_card." The language parser 214 may also identify relationships between words within the utterance 202. For example, in some embodiments, language parser 214 generates a dependency tree that indicates which parts of the utterance (e.g., particular nouns) are direct objects, which parts of the utterance are prepositions, etc. The results of the processing performed by language parser 214 form extracted information 205, which is provided as input to MIS 220 along with utterance 202 itself.

[0070] As noted above, utterance 202 may include multiple sentences. For purposes of detecting multiple intents and explicit invocations, utterance 202 may be treated as a single unit even if it includes multiple sentences. However, in particular embodiments, preprocessing may be performed, for example, by preprocessing subsystem 210, to identify a single sentence among multiple sentences for multiple intent and explicit invocation analysis. In general, the results produced by MIS 220 and EIS 230 are substantially the same regardless of whether utterance 202 is processed at the individual sentence level or as a single unit including multiple sentences.

[0071] The MIS 220 determines whether the utterance 202 expresses multiple intents. While the MIS 220 may detect the presence of multiple intents in the utterance 202, the processing performed by the MIS 220 does not include determining whether the intent of the utterance 202 matches any intent configured for the bot. Instead, the processing to determine whether the intent of the utterance 202 matches the intent of the bot may be performed by the intent classifier 242 of the MB system 200 or by an intent classifier of a skill bot (e.g., as shown in the embodiment of FIG. 3). The processing performed by the MIS 220 assumes the existence of a bot (e.g., a specific skill bot or the master bot itself) that can handle the utterance 202. To that end, the processing performed by the MIS 220 does not require knowledge of what bots exist in the chatbot system (e.g., the identities of skill bots registered by the master bot) or what intents have been configured for a particular bot.

[0072] To determine that utterance 202 contains multiple intents, MIS 220 applies one or more rules from set of rules 252 in data store 250. The rules applied to utterance 202 depend on the language of utterance 202 and may include a sentence pattern that indicates the presence of multiple intents. For example, the sentence pattern may include a coordinating conjunction (e.g., a conjunction) that joins two parts of a sentence, both parts corresponding to separate intents. If utterance 202 matches the sentence pattern, it may be inferred that utterance 202 expresses multiple intents. Note that an utterance with multiple intents does not necessarily have different intents (e.g., intents directed to different bots or intents directed to different intents within the same bot). Instead, the utterance may have separate instances of the same intent, for example, "order a pizza using payment account X, then order a pizza using payment account Y."

[0073] As part of determining that utterance 202 expresses multiple intents, MIS 220 also determines which portions of utterance 202 are associated with each intent. For each intent expressed in the multiple-intent utterance, MIS 220 constructs a new utterance for separate processing in place of the original utterance, e.g., utterance “B” 206 and utterance “C” 208 as shown in FIG. 2 . Thus, original utterance 202 may be split into two or more separate utterances that are handled one at a time. MIS 220 determines which of the two or more utterances should be handled first using extracted information 205 and / or from an analysis of utterance 202 itself. For example, MIS 220 may determine that utterance 202 includes an indicator word indicating that a particular intent should be handled first. The newly formed utterance corresponding to this particular intent (e.g., utterance 206 or one of utterances 208) will be sent first for further processing by EIS 230. After the conversation initiated by the first utterance has ended (or been temporarily discontinued), the next highest priority utterance (e.g., the other of utterance 206 or utterance 208) may then be sent to EIS 230 for processing.

[0074] The EIS 230 determines whether the utterance it receives (e.g., utterance 206 or utterance 208) includes a skillbot's invocation name. In particular embodiments, each skillbot in the chatbot system is assigned a unique invocation name that distinguishes the skillbot from other skillbots in the chatbot system. A list of invocation names may be maintained as part of the skillbot information 254 in the data store 250. An utterance is considered to be an explicit invocation when the utterance includes words that match the invocation name. If the bot is not explicitly invoked, the utterance received by the EIS 230 is considered an implicit invocation utterance 234 and is input to the masterbot's intent classifier (e.g., intent classifier 242) to determine which bot to use to handle the utterance. In some cases, the intent classifier 242 will determine that the masterbot should handle the implicit invocation utterance. In another example, the intent classifier 242 will determine which skillbot to route the utterance to for handling.

[0075] The explicit call functionality provided by EIS 230 has several advantages. It may reduce the amount of processing that the masterbot needs to perform. For example, when there is an explicit call, the masterbot may not need to perform an intent classification analysis (e.g., using intent classifier 242) or may need to perform a reduced intent classification analysis to select a skillbot. Thus, explicit call analysis may enable the selection of a particular skillbot without relying on intent classification analysis.

[0076] Additionally, there may be situations where functionality overlaps between multiple skill bots. This may occur, for example, when the intents handled by two skill bots overlap or are very close to each other. In such situations, it may be difficult for the master bot to identify which of multiple skill bots to select based solely on intent classification analysis. In such scenarios, an explicit call clarifies the specific skill bot to be used.

[0077] In addition to determining that an utterance is an explicit invocation, the EIS 230 is responsible for determining whether any portion of the utterance should be used as input to the explicitly invoked skillbot. In particular, the EIS 230 may determine whether a portion of the utterance is associated with an invocation. The EIS 230 may perform this determination by analyzing the utterance and / or by analyzing the extracted information 205. Instead of sending the entire utterance received by the EIS 230, the EIS 230 may send the portion of the utterance that is not associated with the invocation to the invoked skillbot. In some cases, the input to the invoked skillbot may be easily formed by removing any portion of the utterance that is associated with the invocation. For example, " "I'd like to order a pizza using PizzaBot" can be shortened to "I'd like to order a pizza" because "using PizzaBot" is related to the invocation of PizzaBot but unrelated to the processing to be performed by PizzaBot. In some cases, EIS230 may reformat the portion to be sent to the invoked bot, for example to form a complete sentence. Thus, EIS230 not only determines that there is an explicit invocation, but also determines what to send to the skillbot if there is an explicit invocation. In some cases, there may be no text to enter into the bot being invoked. For example, if the utterance was "PizzaBot," EIS230 may determine that PizzaBot is being invoked, but there is no text to be processed by PizzaBot. In such a scenario, EIS230 may indicate to the skillbot invoker240 that there is nothing to send.

[0078] The skillbot invoker 240 invokes a skillbot in various manners. For example, the skillbot invoker 240 may invoke a bot in response to receiving an indication 235 that a particular skillbot has been selected as a result of an explicit invoke. The indication 235 may be sent by the EIS 230 along with input for the explicitly invoked skillbot. In this scenario, the skillbot invoker 240 will direct control of the conversation to the explicitly invoked skillbot. The explicitly invoked skillbot will determine an appropriate response to the input from the EIS 230 by treating the input as a standalone utterance. For example, the response may be to perform a particular action or to start a new conversation in a particular state, but the initial state of the new conversation depends on the input sent from the EIS 230.

[0079] Another manner in which the skillbot invoker 240 may invoke a skillbot is through implicit invocation using an intent classifier 242. The intent classifier 242 may be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skillbot is configured to perform. The intent classifier 242 is trained in different classes, one class for each skillbot. For example, each time a new skillbot is registered with a masterbot, a list of example utterances associated with the new skillbot may be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skillbot may perform. Parameters created as a result of this training (e.g., a set of values ​​for parameters of a machine learning model) may be stored as part of the skillbot information 254.

[0080] In particular embodiments, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. The machine learning model may be trained by inputting at least a subset of utterances from example utterances associated with various skill bots, to generate, as an output of the machine learning model, an inference about which bot is appropriate for handling any particular training utterance. For each training utterance, an indication of the appropriate bot to use for that training utterance may be provided as ground truth information. The behavior of the machine learning model may be adapted (e.g., by backpropagation) to minimize the discrepancy between the generated inference and the ground truth information.

[0081] In certain embodiments, the intent classifier 242 determines, for each skillbot registered with the masterbot, a confidence score indicating the likelihood that the skillbot can handle the utterance (e.g., the implicit invoke utterance 234 received from the EIS 230). The intent classifier 242 may also determine a confidence score for each configured system-level intent (e.g., help, quit). If a particular confidence score meets one or more conditions, the skillbot invoker 240 will invoke the bot associated with the particular confidence score. For example, a threshold confidence score value may need to be met. Thus, the output 245 of the intent classifier 242 is either an identification of the system intent or an identification of a particular skill bot. In some embodiments, in addition to meeting the threshold confidence score value, the confidence score must exceed the next highest confidence score by a certain win rate. Imposing such a condition may allow routing to a particular skill bot if the confidence scores of multiple skill bots each exceed the threshold confidence score value.

[0082] After identifying the bot based on the evaluation of the confidence score, the skillbot invoker 240 hands over processing to the identified bot. If it is a system intent, the identified bot is the master bot. Otherwise, the identified bot is the skill bot. Furthermore, the skillbot invoker 240 will determine what to provide as input 247 to the identified bot. As mentioned above, in the case of an explicit invoke, the input 247 may be based on a portion of the utterance that is not associated with the invoke, or there may be no input 247 (e.g., an empty string). In the case of an implicit invoke, the input 247 may be the entire utterance.

[0083] The data store 250 comprises one or more computing devices that store data used by various subsystems of the masterbot system 200. As described above, the data store 250 includes rules 252 and skillbot information 254. The rules 252 include, for example, rules for determining, by the MIS 220, if an utterance expresses multiple intents and how to split an utterance expressing multiple intents. The rules 252 further include rules for determining, by the EIS 230, which portions of an utterance that explicitly invokes a skillbot should be sent to the skillbot. The skillbot information 254 includes the call names of skillbots in the chatbot system, for example, a list of the call names of all skillbots registered to a particular masterbot. The skillbot information 254 may also include information, for example, machine learning model parameters, used by the intent classifier 242 to determine a confidence score for each skillbot in the chatbot system.

[0084] 3 is a simplified block diagram of a Skillbot system 300 according to certain embodiments. Skillbot system 300 is a computing system that may be implemented in software only, hardware only, or a combination of hardware and software. In certain embodiments, such as the embodiment shown in FIG. 1, Skillbot system 300 may be used to implement one or more Skillbots within a digital assistant.

[0085] Skillbot system 300 includes MIS 310, intent classifier 320, and conversation manager 330. MIS 310 is similar to MIS 220 of FIG. 2 and provides similar functionality, including being operable to use rules 352 in data store 350 to determine (1) whether an utterance expresses multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In particular embodiments, the rules applied by MIS 310 to detect multiple intents and to split the utterance are the same as the rules applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. Extracted information 304 is similar to extracted information 205 in FIG. 1 and can be generated using language parser 214 or a language parser local to Skillbot system 300.

[0086] The intent classifier 320 may be trained in a manner similar to the intent classifier 242 discussed above in connection with the embodiment of FIG. 2 and described in further detail herein. For example, in particular embodiments, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 may be trained to generate at least a subset of example utterances associated with a particular skill bot. The set is used as training utterances to train a particular skill bot, and the ground truth for each training utterance may be the particular bot intent associated with the training utterance.

[0087] The utterance 302 may be received directly from a user or may be provided by a masterbot. If the utterance 302 is provided by a masterbot, for example, as a result of processing by the MIS 220 and the EIS 230 in the embodiment shown in FIG. 2 , the MIS 310 may be bypassed, thereby avoiding repetition of processing already performed by the MIS 220. However, if the utterance 302 is received directly from a user, for example, during a conversation that occurs after routing to a skillbot, the MIS 310 may process the utterance 302 to determine whether the utterance 302 represents multiple intents. If so, the MIS 310 applies one or more rules to split the utterance 302 into separate utterances for each intent, e.g., utterance “D” 306 and utterance “E” 308. If the utterance 302 does not represent multiple intents, the MIS 310 forwards the utterance 302 to the intent classifier 320 for intent classification and does not split the utterance 302.

[0088] The intent classifier 320 is configured to match a received utterance (e.g., utterance 306 or 308) to an intent associated with the skillbot system 300. As described above, a skillbot may be configured with one or more intents, each including at least one example utterance associated with the intent and used to train the classifier. In the embodiment of FIG. 2, the intent classifier 242 of the masterbot system 200 is trained to determine a confidence score for each individual skillbot and a confidence score for the system intent. Similarly, the intent classifier 320 may be trained to determine a confidence score for each intent associated with the skillbot system 300. While the classification performed by the intent classifier 242 is at the bot level, the classification performed by the intent classifier 320 is at the intent level and is therefore more fine-grained. The intent classifier 320 has access to intent information 354. For each intent associated with the skillbot system 300, the intent information 354 includes a list of utterances that represent and illustrate the meaning of the intent and are typically associated with tasks that can be performed by that intent. The intention information 354 may further include parameters created as a result of training based on this utterance list.

[0089] The conversation manager 330 receives, as an output of the intent classifier 320, an indication 322 of the particular intent identified by the intent classifier 320 when it best matches the utterance input to the intent classifier 320. In some cases, the intent classifier 320 may not be able to determine a match. For example, if the utterance is directed to a system intent or the intent of a different skill bot, the confidence score calculated by the intent classifier 320 may fall below a threshold confidence score value. When this occurs, the skill bot system 300 may refer the utterance to a master bot for handling, e.g., routing to a different skill bot. However, if the intent classifier 320 is successful in identifying the intent within the skill bot, the conversation manager 330 will initiate a conversation with the user.

[0090] A conversation initiated by the conversation manager 330 is specific to the intent identified by the intent classifier 320. For example, the conversation manager 330 may be implemented using a state machine configured to execute a dialog flow for the identified intent. The state machine may include a default starting state (e.g., for when the intent is invoked without additional input) and one or more additional states, each having associated therewith an action to be performed by the skill bot (e.g., complete a purchase transaction) and / or a dialog to be displayed to the user (e.g., question, response). Thus, the conversation manager 330 may determine an action / dialog 335 upon receiving an instruction 322 identifying the intent and respond to subsequent utterances received during the conversation. Additional actions or dialogue may be determined depending on the context.

[0091] Data store 350 comprises one or more computing devices that store data used by various subsystems of skillbot system 300. As shown in Figure 3, data store 350 includes rules 352 and intent information 354. In certain embodiments, data store 350 may be integrated with a masterbot or digital assistant data store, such as data store 250 of Figure 2.

[0092] Variant Mismatch Attack (VIA) As described herein, the idea behind VIA is simple logic: assume that a given input has only one correct prediction, i.e., if the model makes two different predictions, at least one of them must be wrong. This simple logic can be extended to treat an input and its utility-preserving variants as the same input set, and given that the variants are generated in a utility-preserving way, the model can be expected to be consistent for all examples in the input set.

[0093] For example, suppose a trained supervised machine learning model M is obtained and then M is fed a new sample x1 with a unique expected prediction. M generates an output y1 for this x1. Subsequently, x1 is modified, with the constraint that its original "meaning" cannot be changed, to create a new sample x2. In other words, x2 is a utility-preserving variant of x1. Given this structure, the corresponding expected prediction for x2 should be exactly the same as the prediction for x1. Then, x2 is passed to M and an output y2 is received. Now, if we observe that the two predictions y1 and y2 are indeed different, then at least one of the predictions should be wrong! Here, the difference between x1 and x2 is the "deviation," and the potential difference between y1 and y2 is the "disagreement." Essentially, a discrepancy output for a utility-preserving variant of the input fits nicely into the adversarial attack paradigm. A formal explanation of this logic is provided in Algorithm 1 below. input: M: A pre-trained supervised machine learning model subject to adversarial attacks. X: A set of input samples without corresponding labels. output: A set of pairs of potentially adversarial examples Steps: 1.Initialize output list:adversarial_examples=[] 2.For each x i in X: 3. modified_pairs=modify_without_changing_expected_label(xi) 4. For x i_modified1 ,x i_modified2 in modified_pairs: 5. / / x i_modified1 and x i_modified2 share the same expected label,which 6. / / is unknown to the model M 7.y i_modified1 =M(x i_modified1 ) 8. y i_modified2 =M(x i_modified2 ) 9. if y i_modified1 !=(does not equal)y i_modified2 : 10. Add(x i_modified1 ,x i_modified2 )into adversarial_examples 11.Return adversarial_examples Algorithm 1 is a basic version of the VI algorithm, although it should be understood that several different variants may actually be used.

[0094] In some embodiments, the VI algorithm (e.g., Algorithm 1) is run on all samples in modified_pairs from the set of input samples to test for mismatches. In an alternative embodiment, the VI algorithm (e.g., Algorithm 1) is stopped as soon as a potentially adversarial sample pair is found, which may reduce running costs but may also result in good adversarial sample pairs being missed. This option may be particularly beneficial when dealing with large attack sets of input samples. As will be appreciated, none of the techniques using the VI algorithm for adversarial attacks require that the set of input samples have associated labels. In machine learning, data labeling is the process of identifying raw data (images, text files, videos, etc.) and adding (annotating) one or more meaningful, informative labels that provide context so that a machine learning model can learn from it. For example, labels may be ground truth indicating whether a photo contains a bird or a car, which words were uttered in an audio recording, or whether an X-ray contains a tumor. Data labeling is typically required for a variety of use cases, including computer vision, natural language processing, and speech recognition. However, not requiring labels, as with the VI algorithm, allows for finding an unlimited number of potentially adversarial examples as long as more input is provided, which is particularly easy to satisfy for text data.

[0095] In the output list (a set of pairs of potential adversarial examples), each element is a tuple containing two candidate potential adversarial examples. Although it cannot be guaranteed that both elements will fool the victim model M, the VI algorithm determines that at least one of the elements will cause the victim model M to give an incorrect prediction. Also, in most cases, especially for natural language or image recognition tasks, it is easy for users to manually verify whether any element in each of the pairs is a true adversarial example.

[0096] Two related techniques are LIME (Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “Why should I trust you?: Explaining the predictions of any classifier.” In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM. 2016, pp. 1135-1144) and Anchors (Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: High-precision model agnostic explanations. In: AAAI Conference on Artificial Intelligence). Both techniques first perturb the original input, and then either build a local surrogate linear model (LIME) on top of it, or build some rules (Anchors) to explain the model in a black-box manner. The VI algorithm described herein (e.g., Algorithm 1) shares a similar idea of ​​generating pairs of potential adversarial sample candidates ("modified_pairs" in Algorithm 1), except that the VI algorithm tests whether the output of the potential candidates differs from the original input, and then tries to filter out all actual adversarial samples that are completely different from LIME or Anchor. Thus, the VI algorithm frees the constraint of requiring pre-labeled inputs, which can significantly improve the search power of adversarial attack algorithms.

[0097] With respect to Algorithm 1, the modify_without_changing_expected_label function on line 3 shows how the original input sample is modified in a utility-preserving manner. The steps of this modification technique may differ for different applications of the VI algorithm, particularly for applications across various modalities such as visual, text, image processing, etc. This is also where users may modify the VI algorithm in several intuitive ways to create utility-preserving variants. A detailed description of such an approach is provided herein with respect to Examples 1 and 2.

[0098] The systems and methods implemented in various embodiments may be better understood with reference to the following examples.

[0099] Example 1 - Attacking an Image Recognition Model The dataset used in this experiment is the MNIST database of handwritten digits (Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. "Gradient-based learning applied to document recognition." Proceedings of the IEEE, 86(11):2278-2324, November 1998), which contains 60,000 training samples and 10,000 testing samples. All samples are 28x28 pixel grayscale images of a single handwritten digit between 0 and 9.

[0100] A convolutional neural network (CNN) was trained for this image recognition task, and the goal was to find a way to automatically detect some potentially adversarial examples that could fool the CNN model. The architecture of the CNN is as follows:

[0101] A single convolutional layer has 64 filters of size (4,4), followed by a max pooling layer of size (3,3). The output of the pooling layer is then flattened and input to a fully connected layer of size 100, which is then mapped to a 10-dimensional output representing the 10 digits from 0 to 9.

[0102] Since we need some initial samples for the attack, for convenience, 20% of the training data (12,000 samples) are randomly selected for the attack. The remaining 48,000 samples form the actual training dataset. Note that it is not necessary to use the corresponding 12,000 labels associated with the attack inputs.

[0103] After training, the CNN model achieved relatively good performance on the test dataset with an average macro precision of 0.989, recall of 0.989, and F1 score of 0.989.

[0104] A relatively simple yet effective approach to implement the modification function -modify_without_changing_expected_label was adopted for this image recognition task. The approach involved adding normal noise with mean 0 and standard deviation 0.05 to the samples in the attack dataset. Since the ultimate goal is to fool the victim model, it is not a problem to receive contaminated samples that may potentially contain negative pixel values ​​after modification, as long as they can cause the victim model to make incorrect predictions.

[0105] The VI algorithm (Algorithm 2) used in Example 1 is described below, and differs slightly from the basic Algorithm 1 above. Algorithm 2: input: M: the trained CNN model under attack; X: a set of input samples that do not have corresponding labels; output: A subset of all potentially adversarial examples Steps: 1.Initialize output list:adversarial_examples=[] 2.For each x i in X: 3. noise1=random.normal(mean=0,standard deviation=0.05, 4. size=(28,28)) 5. noise2=random.normal(mean=0,standard deviation=0.05, 6. size=(28,28)) 7. x modified1 =x i +noise1 8. x modified2 =x i +noise2 9. / / x i_modified1 and x i_modified2 share the same expected label,which 10. / / is unknown to the model M 11. y i_modified1 =M(x i_modified1 ) 12. y i_modified2 =M(x i_modified2 ) 13. if y i_modified1 !=(does not equal)y i_modified2 : 14. Add(x i_modified1 ,x i_modified2 )into adversarial_examples 15. break 16.Return adversarial_examples If Algorithm 2 above is run continuously on a set of input samples, an unlimited number of pairs of potentially adversarial samples may be obtained (lines 3 and 4 provide a random function, which will return different values ​​on each run), and for each pair, at least one candidate is a true adversarial sample.

[0106] Table 1 shows that we run Algorithm 2 only 10 times on the set of input samples.

[0107] [Table 1]

[0108] In each of the above pairs, there was at least one true adversarial sample, so a total of at least 487 adversarial samples were obtained in the 10 runs of the set of input samples. Two adversarial sample pairs from Run 10 are shown in Figure 4A. As mentioned in the previous section, after this pair of potential adversarial samples is found, the expected prediction is zero, so it is possible to identify (e.g., by user reviews) that Candidate 1 is the true adversarial sample. Figure 4B shows two more adversarial sample pairs. In this example, the expected prediction should be 3, so it can be observed that both Candidate 1 and Candidate 2 are true adversarial samples.

[0109] Example 2 - Attacking a Named Entity Recognition Model Attacking a supervised model with more general labels is much more effective than attacking a supervised model with more specific labels (e.g., the CNN model in Example 1). The reason is that general labels are much easier to find than specific labels. For example, in named entity recognition, general entities include "people," "places," "organizations," etc., which are very likely to appear in general text. Therefore, it is much easier to collect data to use as input for the attack.

[0110] In Experiment 2, a subset of the Common Crawl Corpus dataset (http: / / commoncrawl.org / the-data / get-started / ) containing 50K randomly selected sentences was used as the attack input. Some samples from this dataset are shown below. Sample A: Do I need a skip permit to Fitzrovia? Sample B: The carbonated drinks look delicious. This is my first time visiting your blog. You have a great recipe. Sample C: Pay particular attention to the Wired article above, it's very comprehensive.

[0111] A Bidirectional Encoder Representations from Transformers (BERT)-based named entity recognition (NER) model is trained, which serves as the victim model. The goal is to detect adversarial samples that can fool this pre-trained model. The entities used in this experiment include "person," "location," "organization," "currency," "time or date," and "number." To implement the modification function - modify_without_changing_expected_label, a relatively simple but effective approach was taken to introduce typos into words. The approach is to first randomly select some words from the original utterance and then apply one of three modifications (insertion, deletion, and substitution) to the words. The complete algorithm is shown below in Algorithm 3. An alternative way to implement the above correction function is by using synonyms.

[0112] The VI algorithm (Algorithm 3) used in Example 2 is described below, and differs slightly from the basic Algorithm 1 above. Algorithm 3: input: M: The trained NER model to be attacked. X: The set of input samples that do not have corresponding labels. S: A threshold used to determine whether two strings are semantically similar. output: A subset of all potentially adversarial examples Steps: 1.Initialize output list:adversarial_examples=[] 2.For each x i in X: 3. pivot=x i 4. candidate words=RandomPickWordsFrom(pivot) 5. For each word wj in candidate words:6. variant_pairs=[] 7. operations=SelectTypoTypes() 8. cadidates=ApplyTypoOperations(wj,operations) 9. For each candidate c k In candidates: 10. Create pivot_variant by Replace w j with c k in pivot 11. Add(pivot,pivot_variant)to variant_pairs 12. inconsistent_pair_list=[] 13. For each pair p m in variant_pairs 14.If the two elements in it are inconsistent 15. Add pair pm into inconsistent_pair_list 16. if inconsistent_pair_list is not empty 17. sort inconsistent_pair_list by semantic similarity 18. Add inconsistent_pair_list[0]to adversarial_examples 19. Break 20. else 21. If variant_pairs is not empty 22. Sort variant_pairs by semantic similarity 23. p1,p2=variant_pairs[0] 24. pivot=p2 26.Return adversarial_examples In Algorithm 3, for each original sample (pivot), first, several candidate words are randomly selected. Second, a correction operation (e.g., a typo operation) is applied to the candidate word to obtain a corrected word, and the corrected word is used to replace the original word in the original sample to obtain a corrected sample (pivot variant). Third, the pivot variant and the pivot are compared to see if they are inconsistent. If so, the pivot variant is added to an auxiliary list. Fourth, after comparing the pair (pivot variant and pivot), the auxiliary list is sorted by string similarity, and the pivot variant with the highest similarity score is returned or identified. If the auxiliary list is empty after comparing the pair (pivot variant and pivot), the pivot is updated to the one with the highest similarity score, and Algorithm 3 returns to step 2 to correct the next candidate word and obtain a new corrected sample, and then repeats the process of steps 3 and 4.

[0113] As an example, assume the original sample (pivot) is "Edison is an employee of Oracle," and "Edison" and "employee" are selected as two candidate words. Then, a correction function (e.g., typo) is applied to the word "Edison," correcting the word "Edison" to ("Edisson" and "Ediso"). The pivot variants are then "Edisson is an employee of Oracle" and "Ediso is an employee of Oracle." Then, "Edison is an employee of Oracle" (pivot) is compared with "Edisson is an employee of Oracle" (pivot variant) and "Ediso is an employee of Oracle" (pivot variant) to see if the pivot variant is inconsistent with the pivot. If so, the pivot variant is put into an auxiliary list. Case 1: Assume that both of the two pivot variants are inconsistent with the pivot. In this case, the auxiliary list is sorted based on the similarity between the pivot variant and the pivot. If "Edisson is an employee of Oracle" has a higher similarity, then "Edison is an employee of Oracle" and "Edisson is an employee of Oracle" can be directly added to the return list as a pair of adversarial samples.Case 2: If none of the pivot variants contradict the pivot, the pivot is updated to "Edisson is an employee of Oracle" since it has a higher similarity score.Then, a correction function (e.g., typo) is applied to the second candidate word "Oracle" and the above process is repeated.

[0114] One important detail of Algorithm 3 is line 14, where two candidates are defined as inconsistent if and only if they simultaneously have different entities and different contexts. The entities associated with each candidate are given by the victim model M, and the context is a separate part of everything except the entities.

[0115] In Example 2, to improve efficiency, Algorithm 3 did not consider different orders of the candidate words selected in line 4, but instead used their natural order. When the size of the attack input is medium or small, it is possible to first obtain all possible perturbations of the candidate words by applying a heuristic method to the first-rank words in each input, and Algorithm 3 can be applied for each possible order. There were a total of 26,773 potentially adversarial sample pairs found by Algorithm 3, which means that at least 26,773 adversarial samples were identified.

[0116] Two of the adversarial sample pairs found by the algorithm are shown in Table 2 below.

[0117] [Table 2]

[0118] [Table 3]

[0119] As shown, in both of these two pairs, there is a small difference between candidate 1 and candidate 2, and the corresponding semantic meaning remains unchanged. This is enough for the victim model to make an incorrect prediction: in pair 1, candidate 2 misrecognizes "ned" as the entity "PERSON", and in pair 2, candidate 1 misrecognizes "us" as "LOCATION".

[0120] conclusion As a result, these examples demonstrate that VI is conceptually well-suited for adversarial attacks, in the sense that this technique can be easily applied to find potential adversarial pairs without requiring pre-labeled data as attack input. Experiments on both image recognition and named entity recognition were conducted to demonstrate the usefulness and effectiveness of applying VI to adversarial attacks.

[0121] Systems and techniques for variant mismatch attacks 5 is a block diagram illustrating a computing environment 500 for implementing a variant mismatch attack according to various embodiments. As shown, the computing environment 500 includes a training system 505, a variant mismatch attack system 510, and a model deployment system 515 interconnected via one or more communication networks, e.g., as part of a chatbot system described in detail herein.

[0122] The training system 505 comprises a resource manager 520, a model store 525, a training processing system 530, a training framework 535, and a file sharing system 540 with a model cache 545. The resource manager 520, the training processing system 530, and the training framework 535 configure one or more models (e.g., machine learning language models) to perform a task using one or more training techniques. The one or more training techniques include Word2vec (a shallow neural network model that generates static word embeddings), The pre-training techniques may include, but are not limited to, BERT (a transformer-encoder-based autoencoder language model for creating word embeddings, and language modeling- and encoder-based methods using fine-tuning), ELMo (a long short-term memory (LSTM)-based model for creating word embeddings and language modeling using an encoder-based method), GPT (a transformer-decoder-based autoregressive language model for creating word embeddings and language modeling using a fine-tuning-based method), XLNet (an autoregressive and denoising autoencoder-based model for language modeling), or any combination thereof. The algorithms and hyperparameters for one or more pre-training techniques may be identified by a user (e.g., a developer) and used to configure the model. Additionally or alternatively, the algorithms and hyperparameters may be obtained by the resource manager 520 and the training processing system 530 in an automated manner, such as by data search, evaluation, optimization, etc.

[0123] The machine learning model may be part of a chatbot system, such as a named entity recognition model, an intent recognition model, or an image recognition model. In some cases, the machine learning model is a convolutional neural network ("CNN"), e.g., a start neural network, a residual neural network ("Resnet"), or a recurrent neural network, e.g., a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model, or another variant of a deep neural network ("DNN") (e.g., a multi-label n-binary DNN classifier or a multi-class DNN classifier for single-intent classification). The machine learning model may be any other suitable machine learning model trained for natural language processing, such as a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more of such techniques, e.g., a CNN-HMM or an MCNN (multiscale convolutional neural network). The chatbot system may use the same or different types of machine learning models to recognize one or more entities in an utterance, to determine the likelihood that an utterance represents a task that a particular skillbot is configured to perform, to predict intent from utterances for a first type of skillbot, and to predict intent from utterances for a second type of skillbot. Still other types of machine learning models may be implemented in other examples consistent with this disclosure.

[0124] The resource manager 520, the training processing system 530, and the training framework 535 work in combination to train the model (e.g., pre-training, fine-tuning, training, etc.). For example, the resource manager 520 may obtain datasets for training, testing, and validating the model, the training processing system 530 may implement a multi-node network for hosting and training the model, and the training framework 535 may perform steps for training the model on the multi-node network using sets of training data, test data, and validation data that may perform the steps for training. The goal of training is to train the model to help learn model parameters that can be used in one or more processing tasks (e.g., one or more natural language processing tasks).

[0125] The resource manager 520 may take sample data, split the sample data into a subset of samples for training (e.g., 90%) and a subset of samples for testing and / or validation (e.g., 10%), preprocess the subset of samples, optionally augment the subset of samples, and possibly annotate the subset of samples with labels. The split may be random (e.g., 90 / 10% or 70 / 30%). 0%, or splitting may be performed by more complex validation techniques such as K-Fold Cross-Validation, Leave-one-out Cross-Validation, Leave-one-group-out Cross-Validation, Nested Cross-Validation, etc. to minimize sampling bias and overfitting.

[0126] The process of training a machine learning model may include selecting hyperparameters for the machine learning model and performing iterations of inputting samples from a subset of samples into the machine learning model to find a set of model parameters (e.g., weights and / or biases) that maximizes or minimizes the machine learning model's objective function (e.g., minimize a cost function, such as a loss function or error function). Hyperparameters are settings that can be tuned or optimized to control the behavior of a machine learning model. Most models explicitly define hyperparameters that control various features of the model, such as memory and execution cost. However, additional hyperparameters may be defined to adapt the machine learning model to specific scenarios. For example, hyperparameters may include the number of hidden units of the model, the model's learning rate, the convolution kernel width, or the number of kernels of the model.

[0127] The training step performed by the training processing system 530 and the training framework 535 may include repeated training and validation runs until the model is sufficiently trained for use in the inference phase. For example, in the case of a supervised learning-based model, the goal of training is to learn a function “h()” (sometimes called a hypothesis function) that maps a training input space X to a target value space Y, h:X→Y, such that h(x) is a good predictor of the corresponding value of y. A variety of different techniques may be used to learn this hypothesis function. In some techniques, as part of deriving the hypothesis function, an objective function may be defined that measures the difference between the ground truth value of an input and the predicted value of that input. As part of training, techniques such as backpropagation, random feedback, direct feedback alignment (DFA), indirect feedback alignment (IFA), Hebbian learning, etc. are used to maximize or minimize this objective function (e.g., minimize a cost function).

[0128] The training technique may depend on the type of model being trained. For example, there are various types of supervised learning models, such as various types of neural network models, support vector machine (SVM) models, etc. A variety of different training techniques may be used. For example, as described above, a loss or cost function may be defined for the model, and backpropagation techniques may be used to minimize this loss or cost function. The resource manager 520, training processing system 530, and training framework 535 may perform training to build and train the neural network model. A neural network or neural network model represents a computational model influenced by the way neurons and networks of neurons function in the human brain. A neural network includes multiple nodes arranged in layers. Each node receives inputs from several other nodes or external sources and calculates an output. Each input to a node has an associated weight that is assigned based on the relative importance of that input to other inputs. A node applies a function (activation function) to the weighted sum of its inputs and a bias input to generate an output. The activation function may introduce nonlinearity into the node's output. The layers of a neural network may comprise an input layer containing one or more input nodes, an output layer containing one or more output nodes, and zero or more hidden layers sandwiched between the input and output layers, each hidden layer containing one or more nodes. The outputs of nodes in one layer may be provided as or connected as inputs to nodes in another layer. The outputs of the output layer represent the results or predictions made by the neural network processing. A neural network may have one or It may be implemented using code running on multiple processors, the code implementing the nodes, the connections between the nodes, the functions performed by the nodes, and the processing flow through the nodes. Neural networks may be trained, for example, using backpropagation training techniques, in which weights associated with inputs to nodes in the neural network are manipulated with the goal of maximizing or minimizing an objective function associated with the outputs provided by the output layer of the neural network.

[0129] Once the set of model parameters that minimizes the cost function is identified, the model is trained and can be validated using a subset of samples (a test or validation dataset). The validation process involves iterative operations of tuning hyperparameters by inputting aspect samples from the subset of samples into the model using validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, nested cross-validation, etc., to ultimately find an optimal set of hyperparameters. Once the optimal set of hyperparameters is obtained, a reserved test set of samples from the subset of samples is input into the model to obtain outputs (e.g., classes), which are evaluated against ground truth using correlation techniques such as the Bland-Altman method and Spearman's rank correlation coefficient to calculate performance metrics such as error, accuracy, precision, recall, receiver operating characteristic curves (ROC), etc. Once the model is tested and validated, the training framework 535 outputs the trained model for inference or prediction during the inference or runtime phase based on real-time or inference data points.

[0130] It should be understood that other training / validation mechanisms are contemplated and may be implemented within computing environment 500. For example, a model may be trained and hyperparameters may be tuned based on samples from a subset of samples, and samples from the subset of samples may only be used to test and evaluate the model's performance. Furthermore, however, the training mechanisms described herein focus on training new models. These training mechanisms may also be utilized to fine-tune existing models trained from another dataset. For example, in some cases, a model may have been pre-trained using samples from a dataset in one domain. In such cases, the model may be used for transfer learning and retrained / validated using samples from another domain.

[0131] After a model is trained, it may then be stored in model store 525, and the model may be retrieved by one or more users for fine-tuning and / or making inferences or predictions during the inference or runtime phase based on real-time or inferred data points. For example, a customer may retrieve and use a pre-trained model to identify named entities in text. Additionally or alternatively, the model may be stored in model cache 545 and made available to various tenants in a distributed environment, such as file sharing system 540. The models in model cache 545 may be retrieved or accessed via file sharing system 540 by one or more tenants for fine-tuning and / or making inferences or predictions during the inference or runtime phase based on real-time or inferred data points.

[0132] The variant mismatch attack system 510 comprises a variant mismatch algorithm store 550, a variant mismatch framework 555, an adversarial example store 560, and an adversarial training framework 565 for adversarial training of a model 570 on adversarial examples 575. The variant mismatch algorithm store 550 stores various algorithms (e.g., algorithms 560, 565, 565) for identifying adversarial examples 575 used in adversarial training. The variant mismatch framework 555 stores algorithms 570 (e.g., algorithm 1, algorithm 2, algorithm 3, etc.) from the variant mismatch algorithm store 550. One or more algorithms from the variant mismatch algorithm store 550 are selected to run on the variant mismatch framework 555 using pre-trained machine learning models 570 (e.g., models from the model store 525 or the model cache 545). The one or more algorithms may be selected based on the type of original data available for training, testing, or validating the model, the type of modify_without_changing_expected_label function used for the original data and / or the model, the type of model that is adversarial and trained to defend against adversarial attacks, or a combination thereof. Each algorithm is configured to take as input the pre-trained machine learning model 570 (e.g., a model from the model store 525 or the model cache 545) and a set of input samples 580 without corresponding labels (e.g., a set or subset of data for training, testing, or validating the model obtained by the resource manager 520 that does not have corresponding labels).

[0133] The algorithm and variant disagreement framework 555 generates pairs or sets of modified samples, such as x1 and x2, by executing the modify_without_changing_expected_label function on various samples from the set of input samples 580. The modify_without_changing_expected_label function modifies the original input samples in a utility-preserving manner, as described in detail herein. The implementation of this function may differ for different applications depending on the algorithm selected, particularly for applications across various modalities such as visual and text. The algorithm and variant disagreement framework 555 executes the pre-trained machine learning model 570 to take sample x1 as input and generate an output y1 for x1, and to take modified sample x2 as input and generate an output y2 for x2. The algorithm and variant disagreement framework 555 then determines whether the two predictions y1 and y2 are actually different, and if so, determines that at least one of the predictions is incorrect. The difference between x1 and x2 is the "deviation," and the potential difference between y1 and y2 is the "disagreement." If the pair of predictions are different, the adversarial sample pair 575 is added to the set of adversarial sample pairs stored in the adversarial sample store 560. If the pair of predictions are the same, the sample pair is discarded because both samples are deemed adversarial. Optionally, a user may review the sample pairs and prediction pairs before adding them to the set of adversarial sample pairs, individually determine whether each modified sample in a modified sample pair is a true adversarial sample 575, and restrict the addition of modified samples to the set of adversarial sample pairs to only those modified samples deemed by the user to be true adversarial samples 575.

[0134] The adversarial training framework 565 performs a fine-tuning or adversarial training process on the pre-trained machine learning model 570. The fine-tuning or adversarial training process involves retraining the pre-trained machine learning model 570 using adversarial samples 575 from the adversarial sample store 560. As a result of the fine-tuning or adversarial training process, the model parameters of the original pre-trained language model 570 are updated to take into account the adversarial data and the characteristics of the task of interest to the user in model execution. The adversarial training framework 565 may obtain a set of adversarial samples for training, testing, and validating the pre-trained machine learning model 570 and perform steps to fine-tune the pre-trained machine learning model 570 using the sets of training data, test data, and validation data.

[0135] The fine-tuning step performed by the adversarial training framework 565 is based on a pre-trained The machine learning model 570 may include repeatedly performing training and validation until it is sufficiently trained for use in the inference phase. For example, in the case of a supervised learning-based model, the goal of training is to learn a function “h()” (sometimes referred to as a hypothesis function) that maps a training input space X to a target value space Y, h:X→Y, where h(x) is a good prediction for the corresponding value of y. Various techniques may be used to learn this hypothesis function. In some techniques, as part of deriving the hypothesis function, an objective function may be defined that measures the difference between ground truth values ​​for an input and predicted values ​​for that input. As part of training, this objective function is maximized or minimized using techniques such as backpropagation, random feedback, direct feedback alignment (DFA), indirect feedback alignment (IFA), Hebbian learning, etc. As described with respect to the training system 505, the training technique may depend on the type of model being trained. For example, there are various types of supervised learning models, such as various types of neural network models, support vector machine (SVM) models, etc. Various training techniques may be used depending on the type of pre-trained model being fine-tuned.

[0136] Once a set of model parameters that minimizes the cost function is identified, the pre-trained machine learning model 570 may be retrained or fine-tuned and validated using a subset of samples (a test or validation dataset). The validation process involves iteratively inputting samples from the subset of samples into the pre-trained machine learning model 570 to adjust hyperparameters and ultimately find an optimal set of hyperparameters using validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, nested cross-validation, etc. Once the optimal set of hyperparameters is obtained, a reserved test set of samples from the subset of samples may be input into the model to obtain an output (e.g., predicted class), which is evaluated against aspects of the ground truth using correlation techniques such as the Bland-Altman method and Spearman's rank correlation coefficient to calculate performance metrics such as error, accuracy, precision, recall, receiver operating characteristic curve (ROC), etc.

[0137] Once the pre-trained machine learning model 570 is tested and validated, the adversarial training framework 565 outputs a model for making inferences or aspect predictions during the inference or runtime phase based on real-time or inference data points. For example, after the pre-trained machine learning model 570 is trained, the pre-trained machine learning model 570 is fine-tuned, or the pre-trained machine learning model 570 is exposed to adversarial samples, the model may then be deployed using the model deployment system 515. The model may be deployed to a container-based system 585, such as Kubernetes, with a health monitoring service 590. The model in the container-based system 585 may be used to make inferences or predictions via the inference framework 590 during the inference or runtime phase based on real-time or inference data points. The health monitoring service 595 may monitor the health of the model on the container-based system 585 and may trigger retraining or fine-tuning of the model as needed (e.g., upon drift, adversarial attack detection, or new domain provisioning). Although the models are described as being deployed in a container-based system, it should be understood that any other type of system may be used to deploy the models in the runtime phase without departing from the spirit of this disclosure.

[0138] 6 is a flowchart illustrating a process 600 for a variant mismatch attack according to various embodiments. The process illustrated in FIG. 6 may be implemented in various systems, hardware, or The method illustrated in FIG. 6 and described below may be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the combination. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method illustrated in FIG. 6 and described below is intended to be illustrative and not limiting. While FIG. 6 depicts various processing steps occurring in a particular order or sequence, this is not intended to be limiting. In certain alternative embodiments, steps may be performed in some different order, or some steps may be performed in parallel. In certain embodiments, such as those illustrated in FIGS. 1-3 and 5, the process illustrated in FIG. 6 may be performed by a computing environment (e.g., pre-processing subsystem 210 or environment 500) to generate a set of potentially adversarial sample pairs for training one or more models (e.g., intent classifier 242 or 320 or model 570).

[0139] In step 605, a set of input samples is obtained to challenge the machine learning model. The set of samples does not have corresponding labels. In other words, the set of samples is not tagged with meaningful, informative labels to provide context, so the machine learning model cannot learn from meaningful, informative labels (e.g., cannot infer functionality from labeled samples). In some cases, the machine learning model is part of a chatbot system. In a particular example, the machine learning model is a named entity recognition model.

[0140] In step 610, samples from the set of samples are modified in a utility-preserving manner to generate pairs of modified samples. Modification in a utility-preserving manner means modifying the samples without changing the original meaning or expected label of the samples. Pairs of modified samples share the same expected label. In some cases, modification includes adding noise to the samples. In other examples, modification includes selecting one or more characters or words from the samples and performing an insertion, deletion, or substitution operation on the one or more characters or words. Table 3 provides examples of utility-preserving transformations that may be used to modify samples in a utility-preserving manner.

[0141] [Table 4]

[0142] TIFF2026035651000006.tif142159

[0143] In step 615, the machine learning model is attacked using pairs of modified samples. This attack involves (i) inputting pairs of modified samples into the machine learning model while maintaining the same expected labels unknown to the machine learning model, and (ii) generating pairs of predictions for the pairs of modified samples by the machine learning model.

[0144] In step 620, the pair of predictions is compared to determine whether the pair of predictions is the same or different. As used herein, "same" means that the two predictions must be exactly the same. For example, in the case of NER, suppose two correction samples are obtained: (1) Edison buys a new book, and (2) Edison buys a new book. For correction sample (1), the model recognizes "Edison" as a name entity, but for correction sample (2), the model does not recognize "Edison" as a name entity; they are different. Therefore, a pair of predictions is the same if and only if the model recognizes "Edison" as a name entity for both of the two correction samples of this sample.

[0145] In response to the pair of predictions being different, the pair of modified samples is added to the set of adversarial samples in step 625. Optionally, a user reviews the pair of modified samples before adding them to the set of adversarial samples, and individually determines whether each modified sample of the pair of modified samples is a true adversarial sample, and the user excludes the modified sample from the set of adversarial samples from the set of modified samples that are considered to be true adversarial samples. It may be possible to restrict samples to only be added to the set of adversarial samples. In some cases, pairs of correction samples are discarded in response to pairs of predictions being the same, since neither of the samples is considered adversarial.

[0146] In optional step 630, the machine learning model (or a different machine learning model) is trained using the set of adversarial sample pairs. The training includes annotating each sample in the set of adversarial samples with an expected label to obtain annotated adversarial samples, feeding the annotated adversarial samples to the machine learning model or a different machine learning model, and learning multiple model parameters using the annotated adversarial samples based on maximizing or minimizing an objective function.

[0147] Exemplary System 7 shows a simplified diagram of a distributed system 700. In the depicted example, the distributed system 700 includes one or more client computing devices 702, 704, 706, and 708 coupled to a server 712 via one or more communication networks 710. The client computing devices 702, 704, 706, and 708 may be configured to execute one or more applications.

[0148] In various examples, server 712 may be adapted to run one or more services or software applications that enable one or more embodiments described in this disclosure. In particular examples, server 712 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some examples, these services may be provided to users of client computing devices 702, 704, 706, and / or 708 as web-based or cloud services, such as under a software-as-a-service (SaaS) model. User-operated client computing devices 702, 704, 706, and / or 708 may utilize the services provided by these components by interacting with server 712 utilizing one or more client applications.

[0149] 7, server 712 may include one or more components 718, 720, and 722 that implement the functions performed by server 712. These components may include software components that may be executed by one or more processors, hardware components, or a combination thereof. It should be understood that a variety of different system configurations are possible that differ from distributed system 700. The example shown in FIG. 7 is therefore one example of a distributed system for implementing an exemplary system and is not intended to be limiting.

[0150] A user may use client computing devices 702, 704, 706, and / or 708 to run one or more applications, models, or chatbots, which may then generate one or more events or models that may be implemented or services provided in accordance with the teachings of this disclosure. The client devices may provide an interface that allows a user of the client device to interact with the client device. The client devices may also output information to the user via this interface. Although FIG. 7 shows only four client computing devices, any number of client computing devices may be supported.

[0151] Client devices include portable handheld devices, personal computers, and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and other types of computing systems. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems such as Google Chrome™ OS), including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android™, BlackBerry®, Palm OS®, etc.). Portable handheld devices may include mobile phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), and the like. Wearable devices may include Google Glass® head-mounted displays and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft Windows Mobile® with or without a Kinect® gesture input device, and other devices). Client devices may include Xbox® game consoles, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc. Client devices may be capable of running a variety of different applications, such as various internet-related apps, communication applications (e.g., email applications, short message service (SMS) applications), and may use a variety of communication protocols.

[0152] Network 710 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk®, etc. By way of example only, network 710 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suite), Bluetooth®, and / or another wireless protocol), and / or any combination of these and / or other networks.

[0153] Servers 712 may be comprised of one or more general-purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable arrangement and / or combination. Servers 712 may also include another computing architecture that includes virtualization, such as one or more virtual machines running a virtual operating system, or one or more flexible pools of logical storage that may be virtualized to maintain the server's virtual storage. In various examples, servers 712 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0154] The computing system of server 712 may run one or more operating systems, including any of those described above, as well as any commercially available server operating system. Server 712 may also run any of a variety of additional server and / or middle-tier applications, including an HTTP (Hypertext Transport Protocol) server, an FTP (File Transfer Protocol) server, a CGI (Common Gateway Interface) server, a JAVA server, a database server, etc. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), etc.

[0155] In some implementations, server 712 may include one or more applications for parsing and consolidating data feeds and / or event updates received from users of client computing devices 702, 704, 706, and 708. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party information sources and continuous data streams, which may include real-time events associated with sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 712 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 702, 704, 706, and 708.

[0156] The distributed system 700 may include one or more data repositories 714, 716. These data repositories may, in particular examples, be used to store data and other information. For example, one or more of the data repositories 714, 716 may be used to store information, such as information related to chatbot performance or generated models used by the server 712 when performing various functions according to various embodiments. The data repositories 714, 716 may reside in various locations. For example, the data repository used by the server 712 may be local to the server 712 or remote from the server 712 and may communicate with the server 712 via a network-based or dedicated connection. The data repositories 714, 716 may be of different types. In particular examples, the data repository used by the server 712 may be a database, e.g., a relational database such as a database provided by Oracle Corporation® or another vendor. One or more of these databases may be adapted to allow data to be stored in, updated, and retrieved or retrievable from the database in response to SQL-formatted commands.

[0157] In particular examples, one or more of the data repositories 714, 716 may be used by an application to store application data. The data repositories used by the application may be of various types, such as, for example, a key-value store repository, an object store repository, or a general storage repository backed by a file system.

[0158] In certain examples, the functionality described in this disclosure may be provided as a service via a cloud environment. Figure 8 illustrates an example where various services are provided as cloud services, according to certain examples. 8 is a simplified block diagram of a cloud-based system environment that may be provided. In the example shown in FIG. 8, cloud infrastructure system 802 may provide one or more cloud services that may be requested by users using one or more client computing devices 804, 806, and 808. Cloud infrastructure system 802 may comprise one or more computers and / or servers that may include those described above for server 712. The computers in cloud infrastructure system 802 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0159] Network 810 may facilitate communication and exchange of data between clients 804, 806, and 808 and cloud infrastructure system 802. Network 810 may include one or more networks. The networks may be of the same type or different types. Network 810 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0160] The example shown in Figure 8 is merely one example of a cloud infrastructure system and is not limiting. It should be understood that in some alternative examples, cloud infrastructure system 802 may have more or fewer components than those shown in Figure 8, may combine two or more components, or may have a different configuration or arrangement of components. For example, while Figure 8 shows three client computing devices, any number of client computing devices may be supported in alternative examples.

[0161] The term cloud service is generally used to refer to services made available to users on demand over a communications network such as the Internet by a service provider's system (e.g., cloud infrastructure system 802). Typically, in a public cloud environment, the servers and systems that comprise the cloud service provider's system are different from a customer's own on-premises servers and systems. The cloud service provider's systems are managed by the cloud service provider. Customers may therefore use cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system hosts applications, and users may order and use the applications on demand over the Internet without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services are offered by Oracle Corporation® of Redwood Shores, California, such as middleware services, database services, Java cloud services, etc.

[0162] In particular examples, cloud infrastructure system 802 may provide one or more cloud services using various models, such as under a software-as-a-service (SaaS) model, a platform-as-a-service (PaaS) model, an infrastructure-as-a-service (IaaS) model, etc., including hybrid service models. Cloud infrastructure system 802 may include a set of applications, middleware, databases, and other resources that enable the provision of various cloud services.

[0163] The SaaS model allows applications or software to be delivered to customers as a service over a communications network such as the Internet, without the customer having to purchase hardware or software for the underlying application. For example, the SaaS model may be used to provide customer access to on-demand applications hosted by cloud infrastructure system 802. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, etc.

[0164] The IaaS model is commonly used to provide flexible computing and storage capabilities by offering infrastructure resources (e.g., servers, storage, hardware, and networking resources) as cloud services to customers. Various IaaS services are offered by Oracle Corporation.

[0165] The PaaS model is commonly used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without the customer having to procure, build, or maintain such resources. Examples of PaaS services offered by Oracle Corporation® include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0166] Cloud services are generally provided on an on-demand self-service basis, a subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by cloud infrastructure system 802 via a subscription order. Cloud infrastructure system 802 then performs processing to provide the services requested in the customer's subscription order. For example, a user may use utterances to request the cloud infrastructure system to perform a particular action (e.g., intent), as described above, and / or provide a service to a chatbot system as described herein. Cloud infrastructure system 802 may be configured to provide one or even more cloud services.

[0167] Cloud infrastructure system 802 may provide cloud services through different deployment models. In a public cloud model, cloud infrastructure system 802 may be owned by a third-party cloud service provider, and cloud services may be offered to any public customer, which may be an individual or a business. In another particular example, under a private cloud model, cloud infrastructure system 802 may be operated within an organization (e.g., within a corporate organization), and services may be offered to customers within the organization. For example, customers may be various departments of a company, such as a human resources department, a payroll department, etc., or even individuals within the company. In another particular example, under a community cloud model, cloud infrastructure system 802 and the offered services may be shared by several organizations within an associated community. Various other models, such as hybrids of the above models, may also be used.

[0168] Client computing devices 804, 806, and 808 may be of different types (such as client computing devices 702, 704, 706, and 708 shown in FIG. 7 ) and may be capable of running one or more client applications. Users may use the client devices to interact with cloud infrastructure system 802, such as requesting services provided by cloud infrastructure system 802. For example, users may use client devices to request information or actions from a chatbot, as described in this disclosure.

[0169] In some examples, the processing performed by cloud infrastructure system 802 to provide services may include training and deploying models. This analysis may include using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis may be performed by one or more processors, possibly processing the data in parallel, such as running simulations using the data. For example, big data analysis may be performed by cloud infrastructure system 802 to generate and train one or more models for a chatbot system. The data used in this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0170] 8, cloud infrastructure system 802 may include infrastructure resources 830 utilized to facilitate the provision of various cloud services offered by cloud infrastructure system 802. Infrastructure resources 830 may include, for example, processing resources, storage or memory resources, networking resources, etc. In a particular example, a storage virtual machine available to provide storage requested by an application may be part of cloud infrastructure system 802. In another example, the storage virtual machine may be part of a different system.

[0171] In particular examples, to facilitate efficient provisioning of these resources to support various cloud services offered by cloud infrastructure system 802 for different customers, resources may be bundled into sets of resources or resource modules (also referred to as “pods”). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In particular examples, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for database services, a second set of pods may be provisioned for Java services, and so on, which may include a different combination of resources than the pods in the first set of pods. For some services, the resources allocated to provision the services may be shared among the services.

[0172] Cloud infrastructure system 802 may itself use services 832 internally that are shared by different components of cloud infrastructure system 802 and that facilitate the provision of services by cloud infrastructure system 802. These internal shared services include security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services enabling cloud support, email services, and notification services. , file transfer services, etc., but are not limited to these.

[0173] Cloud infrastructure system 802 may comprise multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in FIG. 8 , the subsystems may include a user interface subsystem 812 that allows users or customers of cloud infrastructure system 802 to interact with cloud infrastructure system 802. User interface subsystem 812 may include a variety of different interfaces, such as a web interface 814, an online store interface 816, where cloud services offered by cloud infrastructure system 802 are advertised and available for purchase by consumers, and another interface 818. For example, a customer may use a client device to request one or more services offered by cloud infrastructure system 802 (service request 834) using one or more of interfaces 814, 816, and 818. For example, a customer may access an online store, browse cloud services offered by cloud infrastructure system 802, and place a subscription order for one or more services offered by cloud infrastructure system 802 to which the customer wishes to subscribe. The service request may include information identifying the customer and one or more services to which the customer wishes to subscribe. For example, the customer may place a subscription order for services provided by cloud infrastructure system 802. As part of the order, the customer may provide information identifying the chatbot system to which the service will be provided and, optionally, one or more authentication information for the chatbot system.

[0174] 8 , cloud infrastructure system 802 may include an order management subsystem (OMS) 820 configured to process new orders. As part of this processing, OMS 820 may be configured to create an account for the customer if not already completed, receive billing and / or accounting information from the customer that is used to bill the customer for providing the requested services to the customer, verify the customer information, and, once verified, book the customer's order, and coordinate various workflows to prepare the supply order.

[0175] Upon proper validation, the OMS 820 may then invoke an order fulfillment subsystem (OPS) 824 configured to provision resources for the order, including processing, memory, and networking resources. Provisioning may include allocating resources to the order and configuring the resources to facilitate the service requested by the customer's order. The manner in which resources are provided to the order and the type of resources provided may depend on the type of cloud service ordered by the customer. For example, according to one workflow, the OPS 824 may be configured to determine the specific cloud service being requested and identify the number of pods that may be pre-configured for that specific cloud service. The number of pods allocated to the order may depend on the size / amount / level / scope of the service requested. For example, the number of pods allocated may be determined based on the number of users supported by the service, the duration for which the service is requested, etc. The allocated pods may then be customized for the particular requesting customer to provide the requested service.

[0176] In certain examples, the setup phase processing may be performed by cloud infrastructure system 802 as part of the provisioning process, as described above. Cloud infrastructure system 802 generates an application ID and A storage virtual machine for an application may be selected from among storage virtual machines provided by the infrastructure system 802 itself or from among storage virtual machines provided by another system other than the cloud infrastructure system 802.

[0177] Cloud infrastructure system 802 may send a response or notification 844 to the requesting customer to indicate when the requested service is available for use. In some cases, information (e.g., a link) may be sent to the customer that enables the customer to begin using and utilizing the benefits of the requested service. In a particular example, for a customer requesting a service, the response may include a chatbot system ID generated by cloud infrastructure system 802 and information identifying the chatbot system selected by cloud infrastructure system 802 for the chatbot system corresponding to the chatbot system ID.

[0178] Cloud infrastructure system 802 may provide services to multiple customers. For each customer, cloud infrastructure system 802 is responsible for managing information regarding one or more subscription orders received from the customer, maintaining customer data associated with the orders, and providing the requested services to the customer. Cloud infrastructure system 802 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected about the amount of storage used, the amount of data transferred, the number of users, the amount of system uptime and system downtime, etc. This usage information may be used to bill the customer. Billing may occur, for example, on a monthly cycle.

[0179] Cloud infrastructure system 802 may provide services to multiple customers in parallel. Cloud infrastructure system 802 may store information for these customers, possibly including proprietary information. In a particular example, cloud infrastructure system 802 includes an identity management subsystem (IMS) 828 configured to manage customer information and provide separation of management information so that information associated with one customer cannot be accessed by another customer. IMS 828 may be configured to provide various security-related services, such as identity services like information access management, authentication, and authorization services, services for managing customer identities and roles, and related functions.

[0180] 9 illustrates an example of a computer system 900. In some examples, the computer system 900 may be used to implement any of the digital assistant or chatbot systems in a distributed environment, as well as the various servers and computer systems described above. As shown in FIG. 9, the computer system 900 includes various subsystems, including a processing subsystem 904 that communicates with many other subsystems via a bus subsystem 902. These other subsystems may include a processing acceleration unit 906, an I / O subsystem 908, a storage subsystem 918, and a communications subsystem 924. The storage subsystem 918 may include non-transitory computer-readable storage media, including a storage medium 922 and a system memory 910.

[0181] Bus subsystem 902 provides a mechanism that allows the various components and subsystems of computer system 900 to communicate with each other as intended. While bus subsystem 902 is shown schematically as a single bus, alternative examples of the bus subsystem may utilize multiple buses. Bus subsystem 902 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, Such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, for example.

[0182] The processing subsystem 904 controls the operation of the computer system 900 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 900 may be organized into one or more processing units 932, 934, etc. The processing units may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or another combination of cores and processors. In some examples, the processing subsystem 904 may include one or more special-purpose coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units of the processing subsystem 904 may be implemented using customized circuitry, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0183] In some examples, processing units within processing subsystem 904 may execute instructions stored in system memory 910 or on computer-readable storage medium 922. In various examples, the processing units may execute various program or code instructions and may maintain multiple simultaneously executing programs or processes. At any time, some or all of the program code being executed may reside in system memory 910 and / or in computer-readable storage medium 922, potentially including one or more storage devices. With appropriate programming, processing subsystem 904 may provide the various functions described above. If computer system 900 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0184] In particular examples, a processing acceleration unit 906 may optionally be provided to accelerate the overall processing performed by the computer system 900, to perform customized processing, or to offload portions of the processing performed by the processing subsystem 904.

[0185] I / O subsystem 908 may include devices and mechanisms for inputting information into computer system 900 and / or outputting information from or through computer system 900. In general, use of the term input device is intended to include all possible types of input devices and mechanisms for inputting information into computer system 900. User interface input devices may include, for example, a keyboard, a pointing device such as a mouse or trackball, a touchpad or touchscreen integrated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keypad, an audio input device with a voice command recognition system, a microphone, and another type of input device. User interface input devices may include motion-sensing and / or video input devices such as a Microsoft Kinect® motion sensor, a Microsoft Xbox® 360 game controller, and devices that provide an interface for receiving input using gestures and voice commands, which allow a user to control and interact with the input device. A gesture recognition device may also be included. The user interface input device may also include a Google Glass® device that detects a user's eye activity (e.g., "blinking" when taking a photo and / or making a menu selection) and translates eye gestures as input to an input device (e.g., Google Glass®). It may also include an eye gesture recognition device such as a Glass® blink detector. In addition, the user interface input device may include a voice recognition sensing device that allows a user to interact with a voice recognition system (e.g., the Siri® navigator) by voice commands.

[0186] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser range finders, and eye-tracking devices. In addition, user interface input devices may include medical imaging input devices such as, for example, computed tomography, magnetic resonance imaging, positional emission tomography, and medical ultrasound equipment. User interface input devices may also include audio input devices such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0187] In general, use of the term output device(s) is intended to include all possible types of devices and mechanisms for outputting information from computer system 900 to a user or to another computer. User interface output devices may include non-visual displays such as a display subsystem, indicator lights, or audio output devices. Display subsystems may be flat-panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs) or plasma displays, projection devices, touch screens, etc. For example, user interface output devices include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, voice output devices, and modems.

[0188] The storage subsystem 918 provides a repository or data store for storing information and data used by the computer system 900. The storage subsystem 918 provides a tangible, non-transitory computer-readable storage medium for storing the basic programming and data structures that provide some example functionality. The storage subsystem 918 may store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 904, provide the functionality described above. The software may be executed by one or more processing units of the processing subsystem 904. The storage subsystem 918 may also provide authentication in accordance with the teachings of the present disclosure.

[0189] The storage subsystem 918 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 9, the storage subsystem 918 includes a system memory 910 and a computer-readable storage medium 922. The system memory 910 may include a number of memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution, and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some implementations, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computer system 900, such as during start-up, may typically be stored in ROM. The RAM is typically used by the processing subsystem 904. The system memory 910 may include a number of different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0190] 9, system memory 910 may load running application programs 912, program data 914, and operating system 916, which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc. By way of example, operating system 916 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.) and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, Palm® OS operating systems, etc.

[0191] The computer-readable storage medium 922 may store programming and data structures that provide some example functionality. The computer-readable medium 922 may provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 900. Software (programs, code modules, instructions) that, when executed by the processing subsystem 904, provide the functionality described above may be stored in the storage subsystem 918. By way of example, the computer-readable storage medium 922 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, an optical disk drive, such as a CD-ROM, a DVD, a Blu-Ray® disk, or another optical media. The computer-readable storage medium 922 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disk, a digital video tape, etc. The computer-readable storage medium 922 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid-state ROM, etc., SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0192] In particular examples, storage subsystem 918 may also include a computer-readable storage medium reader 920, which may be further connected to a computer-readable storage medium 922. Reader 920 may be configured to receive and read data from a memory device such as, for example, a disk, a flash drive, or the like.

[0193] In certain examples, computer system 900 may support virtualization techniques, including, but not limited to, virtualization of processing and memory resources. For example, computer system 900 may provide support for running one or more virtual machines. In certain examples, computer system 900 may execute a program such as a hypervisor that facilitates configuration and management of virtual machines. Each virtual machine may have allocated memory, computational (e.g., processors, cores), I / O, and network resources. Each virtual machine typically runs independently from other virtual machines. Each virtual machine typically runs its own operating system, which may be the same or different from the operating system run by another virtual machine executed by computer system 900. Thus, multiple operating systems may potentially be executed by computer system 900 simultaneously.

[0194] The communications subsystem 924 provides an interface to other computer systems and networks. The communications subsystem 924 serves as an interface for receiving data from the computer system 900 and transmitting data to other systems. For example, the communications subsystem 924 enables the computer system 900 to send and receive information to one or more client devices by establishing a communications channel to the client device over the Internet. For example, if the computer system 900 is used to implement the bot system 120 shown in FIG. 1, the communications subsystem may be used to communicate with a chatbot system selected for the application.

[0195] The communications subsystem 924 may support both wired and / or wireless communication protocols. In particular examples, the communications subsystem 924 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies), e.g., 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.XX family of standards, or another mobile communications technology, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some examples, the communications subsystem 924 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0196] The communications subsystem 924 may send and receive data in various formats. In some examples, in addition to other formats, the communications subsystem 924 may receive incoming communications in the form of structured and / or unstructured data feeds 926, event streams 928, event updates 930, etc. For example, the communications subsystem 924 may be configured to receive (or send) data feeds 926 in real time from users of a social media network and / or another communications service, such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0197] In particular examples, the communications subsystem 924 may be configured to receive data in the form of a continuous data stream, which may include an event stream 928 of real-time events and / or event updates 930, which may be continuous or unlimited in nature with no apparent end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0198] The communications subsystem 924 may also be configured to communicate data from the computer system 900 to another computer system or network. The data may be transmitted in various formats, such as structured and / or unstructured data feeds 926, event streams 928, event updates 930, etc., to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 900. It may be communicated in a variety of different formats.

[0199] Computer system 900 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system. Because the nature of computers and networks is constantly changing, the description of computer system 900 shown in FIG. 9 is intended only as a specific example. Many alternative configurations are possible, having more or fewer components than the system shown in FIG. 9. It should be understood that there are alternative aspects and / or methods for implementing the various examples, based on the disclosure and teachings provided herein.

[0200] While specific examples have been described, various modifications, substitutions, alternative configurations, and equivalents are possible. The examples are not limited to operation in any particular data processing environment, but rather operate freely in multiple data processing environments. Additionally, while specific examples have been described using a particular sequence of transactions and steps, those skilled in the art will recognize that this is not intended to be limiting. While some flowcharts describe operations as sequential processes, many of the operations may be performed in parallel or simultaneously. Furthermore, the order of operations may be rearranged. Processes may include additional steps not included in the figures. Various features and aspects of the above-described examples may be used individually or in combination.

[0201] Furthermore, while particular examples have been described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are possible. Particular examples may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor, or on different processors in any combination.

[0202] Where a device, system, component, or module is described as being configured to perform particular operations or functions, such configuring may be achieved, for example, by designing electronic circuitry to perform the operations, by programming a programmable electronic circuitry (such as a microprocessor), by executing computer instructions or code, or a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof, to perform such operations. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0203] Specific details are provided in this disclosure to provide a thorough understanding of the examples. However, the examples may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the examples. This description provides illustrative examples only and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the foregoing description of the examples will provide one skilled in the art with possible descriptions for implementing various examples. Various changes may be made in the function and arrangement of elements.

[0204] The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, deductions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope as set forth in the appended claims. Accordingly, although specific examples have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0205] While aspects of the disclosure have been described in the foregoing specification with reference to specific examples thereof, those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above-described disclosure may be used individually or in any combination. Moreover, the examples may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are therefore to be regarded as illustrative rather than restrictive.

[0206] In the foregoing description, the methods have been described in a particular order for purposes of explanation. It should be understood that, in alternative examples, the methods may be performed in an order different from that described. It should also be understood that the methods described above may be performed by hardware components or embodied in sequences of machine-executable instructions, which may be used to cause a machine, such as a general-purpose or special-purpose processor or logic circuitry programmed with the instructions, to perform the methods. These machine-executable instructions may be stored on one or more machine-readable media, such as a CD-ROM or another type of optical disk, a floppy diskette, a ROM, a RAM, an EPROM, an EEPROM, a magnetic or optical card, a flash memory, or another type of machine-readable medium suitable for storing electronic instructions. Alternatively, the methods may be performed by a combination of hardware and software.

[0207] Where an element is described as being configured to perform a particular operation, such configuration is achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or by any combination thereof.

[0208] While illustrative examples of applications have been described in detail herein, it is to be understood that the inventive concepts may be variously embodied and used in other manners, and that the appended claims are intended to be construed to include such variations except insofar as limited by the prior art.

Claims

1. 1. A computer-implemented method comprising: The method comprises: obtaining a set of input samples for attacking a machine learning model, the set of samples having no corresponding labels; The method comprises: further comprising modifying samples from the set of samples in a utility-preserving manner to generate pairs of modified samples, the utility-preserving manner comprising modifying the samples without changing their original meanings or expected labels, wherein the pairs of modified samples share the same expected labels; The method comprises: The method further includes attacking the machine learning model using the modified sample pairs, the attacking step comprising: inputting the modified sample pairs into the machine learning model while keeping the same expected labels unknown to the machine learning model; and the machine learning model generating a pair of predictions for the pair of modified samples; The method comprises: further comprising comparing the pairs of predictions to determine whether the pairs of predictions are the same or different; The method comprises: adding the pair of correction samples to a set of adversarial samples in response to the pair of predictions being different. The way computers run.

2. 2. The method of claim 1 , further comprising: training the machine learning model or a different machine learning model using the set of adversarial examples, the training comprising: annotating each sample in the set of adversarial samples with an expected label to obtain annotated adversarial samples; feeding the annotated adversarial samples to the machine learning model or the different machine learning model; and learning a plurality of model parameters using the annotated adversarial samples based on maximizing or minimizing an objective function.

3. 2. The method of claim 1, further comprising: prior to adding the pair of modified samples to the set of adversarial sample pairs in response to the pair of predictions being different, determining individually whether each modified sample of the pair of modified samples is a true adversarial sample; and limiting the addition of the modified samples to the set of adversarial samples to only those modified samples that are deemed to be true adversarial samples.

4. The method of claim 1 , wherein the machine learning model is part of a chatbot system.

5. The method of claim 1 , wherein the machine learning model is a named entity recognition model, an intent recognition model, or an image recognition model.

6. The method of claim 1 , wherein the modifying step comprises adding noise to the samples.

7. 2. The method of claim 1, wherein the modifying step comprises the steps of: selecting one or more characters or words from the sample; and performing an insert, delete, or replace operation on the one or more characters or words.

8. one or more processors; a memory coupled to the one or more processors; the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions including instructions that, when executed by the one or more processors, cause the one or more processors to perform an operation; The process comprises: obtaining a set of input samples for attacking a machine learning model, the set of samples having no corresponding labels; The process comprises: further comprising modifying samples from the set of samples in a utility-preserving manner to generate pairs of modified samples, the utility-preserving manner comprising modifying the samples without changing their original meanings or expected labels, the pairs of modified samples sharing the same expected labels; The process comprises: The method further includes attacking the machine learning model using the modified sample pairs, the attacking step comprising: inputting the modified sample pairs into the machine learning model while keeping the same expected labels unknown to the machine learning model; and the machine learning model generating a pair of predictions for the pair of modified samples; The process comprises: further comprising comparing the pairs of predictions to determine whether the pairs of predictions are the same or different; The process comprises: and further comprising adding the pair of correction samples to a set of adversarial samples in response to the pair of predictions being different. system.

9. 10. The system of claim 8, wherein the processing further comprises training the machine learning model or a different machine learning model using the set of adversarial samples, the training comprising: annotating each sample in the set of adversarial samples with an expected label to obtain annotated adversarial samples; feeding the annotated adversarial samples to the machine learning model or the different machine learning model; and learning a plurality of model parameters using the annotated adversarial samples based on maximizing or minimizing an objective function.

10. 10. The system of claim 8, wherein the processing further comprises: determining individually whether each modified sample of the modified sample pair is a true adversarial sample before adding the modified sample pair to the set of adversarial sample pairs in response to the pair of predictions being different; and limiting the addition of the modified samples to the set of adversarial samples to only those modified samples deemed to be true adversarial samples.

11. The system of claim 8 , wherein the machine learning model is part of a chatbot system.

12. The system of claim 8 , wherein the machine learning model is a named entity recognition model, an intent recognition model, or an image recognition model.

13. The system of claim 8 , wherein the modifying step includes adding noise to the samples.

14. 9. The system of claim 8, wherein the modifying step comprises the steps of: selecting one or more characters or words from the sample; and performing an insert, delete, or replace operation on the one or more characters or words.

15. a non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions including instructions that, when executed by the one or more processors, cause the one or more processors to perform an operation; The process comprises: obtaining a set of input samples for attacking a machine learning model, the set of samples having no corresponding labels; The process comprises: further comprising modifying samples from the set of samples in a utility-preserving manner to generate pairs of modified samples, the utility-preserving manner comprising modifying the samples without changing their original meanings or expected labels, the pairs of modified samples sharing the same expected labels; The process comprises: The method further includes attacking the machine learning model using the modified sample pairs, the attacking step comprising: inputting the modified sample pairs into the machine learning model while keeping the same expected labels unknown to the machine learning model; and the machine learning model generating a pair of predictions for the pair of modified samples; The process comprises: further comprising comparing the pairs of predictions to determine whether the pairs of predictions are the same or different; The process comprises: and further comprising adding the pair of correction samples to a set of adversarial samples in response to the pair of predictions being different. Non-transitory computer-readable memory.

16. 16. The non-transitory computer-readable memory of claim 15, wherein the process further comprises training the machine learning model or a different machine learning model using the set of adversarial samples, the training comprising: annotating each sample in the set of adversarial samples with an expected label to obtain annotated adversarial samples; feeding the annotated adversarial samples to the machine learning model or the different machine learning model; and learning a plurality of model parameters using the annotated adversarial samples based on maximizing or minimizing an objective function.

17. 16. The non-transitory computer-readable memory of claim 15, wherein the processing further comprises: determining individually whether each modified sample of the modified sample pair is a true adversarial sample before adding the modified sample pair to the set of adversarial sample pairs in response to the pair of predictions being different; and limiting the adding of the modified samples to the set of adversarial samples to only the adding of the modified samples to the set of adversarial samples that are deemed to be true adversarial samples.

18. The non-transitory computer-readable memory of claim 15 , wherein the machine learning model is a named entity recognition model, an intent recognition model, or an image recognition model.

19. The non-transitory computer-readable memory of claim 15 , wherein the modifying step comprises adding noise to the samples.

20. 16. The non-transitory computer-readable memory of claim 15, wherein the modifying step comprises: selecting one or more characters or words from the sample; and performing an insert, delete, or replace operation on the one or more characters or words.

Images