Method and system for safely deploying artificial intelligence model

Abstract

To provide a method and a system for safely deploying an artificial intelligence (AI) model using a model maintenance system.SOLUTION: In a model maintenance system including a model evaluation engine and a firewall, a method identifies one or more defects of an AI model and protects the AI model from an attack by external data on the basis of the identification. The model evaluation engine receives sample input data and sample output data related to the AI model. The representative sample input data includes tabular format data, biometric data, an audio file, a face image, an object image, and the like. The representative sample output data includes deduction of whether or not the sample input data is fraudulent, a label and / or classification indicating the audio file and / or a person related to the face image, an object detected in the object image and the place thereof, and the like.SELECTED DRAWING: Figure 2

Description

Related Applications

[0001] This application is a divisional application of Japanese Patent Application No. 2023-5335844, which claims priority to U.S. Provisional Patent Application No. 63 / 153667, filed on February 25, 2021. [Technical Field]

[0002] The disclosed embodiments relate generally to artificial intelligence (AI), and more particularly, but not exclusively, to methods and systems for securely deploying AI models. [Background technology]

[0003] Artificial intelligence (AI) is becoming widespread across many industries. However, the benefits of automation can easily hide inherent vulnerabilities in AI, and current AI development practices often expose organizations to systemic risk. While AI research is advancing and making great strides, the integrity and reliability of AI technology are being left behind. "AI attacks" are emerging in the market as a new type of data attack. Unlike traditional cyberattacks caused by code bugs or human error, AI attacks are enabled by inherent limitations in the underlying AI algorithms that cannot be easily detected or corrected. AI vulnerabilities mean that data science teams spend valuable development cycles coming up with ad-hoc solutions to address the myriad vulnerabilities associated with AI use, rather than investing efforts in developing an organization's core AI capabilities.

[0004] AI technology has been attacked in early ways, such as email spam, financial fraud, or fake account creation. However, beyond these examples, recent AI adoption, automated attacks, and industry practices make AI integrity an even more pressing issue. For example, AI is rapidly spreading within industries other than major consumer technology companies. Protecting against spam and click fraud, once a concern unique to large internet service providers, is now a concern for all businesses, such as banks, insurance companies, and government agencies. Furthermore, methods for attacking AI systems are rapidly evolving. Fraudsters are now executing algorithmic attacks on AI. These attacks are becoming automated, allowing fraudsters to counter defense updates much more quickly. These attacks can be used not only to fool AI models but also to steal sensitive user data or information about AI systems. Furthermore, new trends in the AI ​​industry include many developers and researchers making their state-of-the-art "pre-trained" models and datasets publicly available, and many companies relying on crowdsourcing to collect and label data. Such trends make it significantly easier for fraudsters and adversaries to spread malware models or contaminate the data used for model development. Summary of the Invention [Problem to be solved by the invention]

[0005] In view of the above, there is a need for improved systems and methods for secure deployment of AI models that overcome the shortcomings of existing solutions. [Means for solving the problem]

[0006] According to a first aspect disclosed herein, a method for securely deploying an artificial intelligence (AI) model is set forth, the method comprising: identifying one or more defects in the AI ​​model; protecting the AI ​​model from being attacked by external data based on the identifying step; Includes.

[0007] In some embodiments of the disclosed method, the identifying step includes simulating one or more attacks on the AI ​​model with simulated attack data, wherein the simulating includes: simulating multiple parallel attacks and identifying flaws in the AI ​​model against those attacks based on a combinatorial algorithm; attacking at least one alternative model configured to replicate the AI ​​model; Contains at least one of the following:

[0008] In some embodiments of the disclosed method, the method comprises: filtering or modifying input data provided to the alternative model; labeling the input data; training the replacement model using the labeled input data such that the replacement model is an improved version of the AI ​​model; Further includes:

[0009] In some embodiments of the disclosed method, the identifying step comprises: establishing one or more baselines of model performance and training data profile; comparing performance of the AI ​​model to one or more alternative models based on the establishing step and selecting a model from the AI ​​model and the alternative models for production deployment; Including, the protecting step includes blocking one or more users associated with the external data based on a behavioral pattern of the users' queries to the AI ​​model; or The protecting step includes using a maximum entropy method to detect attacks on the AI ​​model, the maximum entropy method including generating multiple detectors and aggregating their responses.

[0010] In some embodiments of the disclosed method, the external data includes tabular data related to user transaction history, account history, account data, user data, transaction information, or combinations thereof.

[0011] In some embodiments of the disclosed method, the identifying step is based on sample input data including one or more data points and sample output data including labels assigned to the data points based on the AI ​​model, each label including a determination of whether the data point is fraudulent or not fraudulent; or The identifying step is based on fast sub-module sampling to explore the state space.

[0012] In some embodiments of the disclosed method, the protecting step includes generating an alert level based on the identifying step to indicate whether the external data includes one or more attacks that circumvent the AI ​​model.

[0013] In some embodiments of the disclosed method, the method further includes generating a probability that the external data is fraudulent, wherein generating the probability includes using one or more machine learning (ML) frameworks for binary classification, wherein the ML frameworks include boosted trees, neural networks, or a combination thereof.

[0014] In some embodiments of the disclosed method, the protecting step comprises: receiving the external data from a data feed related to a commercial transaction, the external data including tabular data defining risk attributes of entities; converting the tabular data into an input format; identifying manipulation of one or more elements in the tabular data to generate a false positive result; rejecting the tabular data as a manipulated data set; Includes.

[0015] In some embodiments of the disclosed method, the protecting step comprises: receiving the external data including the tabular data; detecting manipulation of the tabulated data to identify false positive results; determining that the false positive result will occur within the AI ​​model; alerting the AI ​​model and flagging the tabular dataset for human review; Includes.

[0016] In some embodiments of the disclosed method, the external data includes voice data, and the AI ​​model includes a voice authentication model for detecting whether the external data is classified as being that of a specified person, and the identifying step, the protecting step, or both, further comprises: training a binary classification machine learning model to generate an output label indicating whether the audio data is fake or not; using a convolutional neural network-based architecture for said detection; Extracting various acoustic features suitable for said detection; or Generating synthetic voice data from authentic voice data using data augmentation Based on.

[0017] In some embodiments of the disclosed method, the identifying step is based on sample input data including one or more sample audio files and sample output data including labels assigned to the sample audio files based on the AI ​​model, each label indicating an individual, and the identifying step includes identifying at least one of the sample audio files that includes synthesized audio data and is accepted by the AI ​​model as being the designated individual; or The identifying step includes generating a redundancy audio file that includes both authentic audio data of a first person and added noise, the noise being added to the authentic audio data such that the AI ​​model is configured to recognize the selected sample audio file as being of a second person different from the first person.

[0018] In some embodiments of the disclosed method, the protecting step includes detecting whether a voice presented in the external data is synthetic.

[0019] In some embodiments of the disclosed method, the external data includes at least one image file, and the AI ​​model includes a facial recognition model for detecting whether the image file shows a face of a specified person, and the identifying step, the protecting step, or both, comprises: Neural network based face embedding, Compressing face images into lower dimensional embeddings, Categorizing lower dimensional embeddings for face recognition, Attacking the AI ​​model by adding noise to facial images; or A neural network that uses a generative adversarial network to extract specific facial images from noise Based on.

[0020] In some embodiments of the disclosed method, the identifying step is based on sample input data including one or more sample image files of faces and sample output data including labels assigned to the sample image files based on the AI ​​model, each label indicating an individual, and the identifying step includes identifying at least one of the sample image files that has been misclassified by the AI ​​model; or The identifying step includes generating one or more photorealistic images at least in part based on one or more images of the person of interest.

[0021] In some embodiments of the disclosed method, the external data further comprises a label attached to the image, and the protecting step comprises indicating whether the label is correctly attached to the image; or The protecting step includes indicating whether the faces in the image file are real or machine-generated.

[0022] In some embodiments of the disclosed method, the external data includes at least one image, and the AI ​​model includes an object detection model for classifying at least one object in the image.

[0023] In some embodiments of the disclosed method, the identifying step is based on sample input data including one or more sample image files and sample output data including one or more objects and their locations within each of the sample image files, and the identifying step includes identifying at least one of the sample image files that is misclassified by the AI ​​model; or The identifying step includes generating manipulated images that are misclassified by the AI ​​model.

[0024] In some embodiments of the disclosed method, the protecting step includes indicating whether the image has been manipulated to fool the AI ​​model via digital adversarial attacks, physical adversarial attacks, or a combination thereof.

[0025] According to another aspect disclosed herein, a system for securely deploying an artificial intelligence (AI) model is described, the system comprising: at least one hardware processor; a non-transitory computer-readable medium storing instructions; wherein the set of instructions, when executed by the at least one hardware processor, cause the at least one hardware processor to identifying one or more defects in the AI ​​model; protecting the AI ​​model from being attacked by external data based on the identifying step; The operation includes:

[0026] Additional features of the present invention will become more readily apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a diagram illustrating an exemplary embodiment of a model maintenance system for securely deploying an AI model within an AI operating environment. [Figure 2] 2 is a flowchart illustrating an exemplary embodiment of a method for securely deploying an AI model using the model integrity system of FIG. 1. [Figure 3] 2 is a schematic diagram illustrating an alternative exemplary embodiment of the system of FIG. 1, the system including a model evaluation engine for receiving sample input data. [Figure 4] 2 is a diagram illustrating another alternative exemplary embodiment of the system of FIG. 1, the system including a firewall for receiving external data. [Figure 5] 2 is a diagram illustrating another alternative exemplary embodiment of the system of FIG. 1, the system including a fraud detection engine for receiving external data. [Figure 6] 4 is a diagram illustrating an alternative exemplary embodiment of the system of FIG. 3, where sample input data includes tabular data. [Figure 7] 5 is a diagram illustrating an alternative exemplary embodiment of the system of FIG. 4, where the external data includes tabular data. [Figure 8] 6 is a diagram illustrating an alternative exemplary embodiment of the system of FIG. 5, where the external data includes tabular data. [Figure 9] 4 is a diagram illustrating another alternative exemplary embodiment of the system of FIG. 3, where the sample input data includes audio data. [Figure 10]5 is a diagram illustrating another alternative exemplary embodiment of the system of FIG. 4, where the external data includes audio data. [Figure 11] 4 is a diagram illustrating another alternative exemplary embodiment of the system of FIG. 3, where the sample input data includes one or more images for facial recognition. [Figure 12] 5 is a diagram illustrating another alternative exemplary embodiment of the system of FIG. 4, where the external data includes one or more images for facial recognition. [Figure 13] 4 is a diagram illustrating another alternative exemplary embodiment of the system of FIG. 3, where the sample input data includes one or more images for object detection. [Figure 14] 5 is a diagram illustrating another alternative exemplary embodiment of the system of FIG. 4, where the external data includes one or more images for object detection. [Figure 15] 2 is a diagram illustrating exemplary data processing for the model maintenance system of FIG. 1. [Figure 16] 2 is a diagram illustrating an exemplary training of the model maintenance system of FIG. 1; [Figure 17] 2 is a schematic diagram illustrating another alternative exemplary embodiment of the system of FIG. 1, the system including a disinfectant machine. [Figure 18] 2 is a diagram illustrating an exemplary embodiment of a software architecture for implementing the model maintenance system of FIG. 1. [Figure 19] 2 is a diagram illustrating an exemplary embodiment of a machine for implementing the model maintenance system of FIG. 1.

[0028] It should be noted that the figures are not drawn to scale, and that for purposes of illustration, elements of similar structure or function are generally represented by similar numerals throughout all figures. It should also be noted that the figures are intended only to facilitate the description of the preferred embodiments. The figures do not show every aspect of the described embodiments and do not limit the scope of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0029] Machine learning (ML) algorithms have been used to defend against attacks on commercial AI platforms. In some cases, machine learning algorithms are better than the human review and rule-based methods initially used by organizations. However, existing machine learning techniques have limited capacity to defend AI models due to a lack of flexibility and comprehensiveness, a lack of large training datasets, and a lack of awareness of connections within data and data feeds. More specifically, existing ML models suffer from the drawback of being highly sensitive to fundamental changes in real-world data distributions and / or data concept drift. Another drawback is that existing ML models suffer from being highly sensitive to data errors, which are inevitable in highly complex systems. The lack of research in these areas, as mentioned above, makes commercial AI systems vulnerable to exploitation.

[0030] Solutions to existing security threats to commercial AI systems are needed that involve understanding how fraud detection systems require a near-continuous set of tests to patch exploits that may be discovered through experimentation or partial knowledge of commercial AI systems. Unfortunately, most conventional AI systems are not designed to prevent fraud and often cannot be flexible enough to catch and block other exploits as they occur. Many AI system exploits center around small perturbations across multiple independent data feeds rather than single data feed exploits. Data feeds typically have multiple interconnections that help detect sophisticated manipulation.

[0031] Because currently available systems and methods for deploying AI models cannot detect vulnerabilities in AI models and cannot effectively defend against AI attacks, improved systems and methods for securely deploying AI models that can overcome the above-mentioned shortcomings would prove desirable and may provide the foundation for a wide range of applications, such as active defense of commercial AI from attacks, fraud detection for financial and / or e-commerce companies (or protecting their fraud-detection AI models), biometric authentication, object detection for security systems, and / or autonomous vehicles.

[0032] 1 is a schematic diagram of a model maintenance system 200 for securely deploying an AI model 300 within an AI operating environment 100. The AI ​​model 300 may include one or more computer-implemented mathematical algorithms that are trained using data and / or human expert input to replicate the decisions based on that information that an expert would make from the same information. Exemplary AI models 300 may include, but are not limited to, expert systems, case-based reasoning, behavior-based artificial intelligence, evolutionary computing, classifiers, statistical models, probabilistic models, neural networks, decision trees, hidden Markov models, support vector machines, fuzzy logic, Bayesian classifiers, or any combination thereof.

[0033] Model integrity system 200 is shown as comprising model evaluation engine 220 and firewall 240. Model evaluation engine 220 may be configured to identify one or more flaws (and / or vulnerabilities) in AI model 300. Stated slightly differently, model evaluation engine 220 may determine data that can attack AI model 300. Attacking AI model 300 may include tricking AI model 300. Stated slightly differently, an attack may include tricking AI model 300 into making an incorrect decision, tricking the AI ​​model into recognizing fraudulent data as non-fraudulent data, tricking synthetic (or forged or altered) data into recognizing authentic data, or a combination thereof. An attack may include data configured to attack AI model 300. In one embodiment, model evaluation engine 220 may output a report summarizing vulnerabilities in AI model 300.

[0034] The firewall 240 can protect the AI ​​model 300 from being tricked by the external data 400 based on flaws identified by the model evaluation engine 220. The external data 400 can include any data that would be input to the AI ​​model 300 if the firewall 240 were not established. Stated slightly differently, the firewall 240 can patch loopholes identified by the model evaluation engine 220, creating an additional layer of security between the external data 400 and the AI ​​model 300. In some embodiments, the firewall 240 can generate an alert upon detecting an attack in the external data 400. In various embodiments, the firewall 240 can have the ability to flag the external data 400 as erroneous or suboptimal in some way. Additionally and / or alternatively, the firewall 240 can flag errors in the external data 400 by automatically profiling the training data on which the AI ​​model 300 was trained. Thus, any external data 400 that appears to be outside the distribution of training data can be identified. Additionally and / or alternatively, firewall 240 can flag external data 400 that is unlikely to cause AI model 300 to provide corrective predictions. Additionally and / or alternatively, firewall 240 can provide the user with suggestions for corrective actions to improve the performance of AI model 300 in response to a profile of external data 400.

[0035] In some embodiments, the model maintenance system 200 may be driven at least in part by an application programming interface (API) and inserted into the data path of external data 400 in front of the AI ​​model 300. The model maintenance system 200 may return and / or output clean, exploit-free data to the AI ​​model 300. In various embodiments, the AI ​​model 300 may remain unchanged and / or unchanged. Advantageously, the model maintenance system 200 can protect the AI ​​model 300 without extensive updates to the AI ​​model 300.

[0036] 1 depicts the model evaluation engine 220 and the firewall 240 as separate units for illustrative purposes only, the model evaluation engine 220 and the firewall 240 may be at least partially integrated and / or combined without limitation. For example, the model evaluation engine 220 and the firewall 240 may each be implemented in computer hardware, firmware, and / or software. Thus, the model evaluation engine 220 and the firewall 240 may be implemented as coded instructions stored on one or more computer systems. The coded instructions associated with the model evaluation engine 220 and the firewall 240 may be separate and / or integrated programs, and the model evaluation engine 220 and the firewall 240 are not necessarily implemented in separate hardware.

[0037] 2, an exemplary method 700 for securely deploying an AI model 300 is shown. One or more defects in the AI ​​model 300 may be identified in step 710. In various embodiments, the model evaluation engine 220 (shown in FIG. 1) may perform step 710.

[0038] In step 720, AI model 300 may be protected from attacks by external data 400 (shown in FIG. 1 ) based on the identification in step 710. In various embodiments, firewall 240 (shown in FIG. 2 ) may perform step 720. The protection provided by firewall 240 may advantageously be customized for AI model 300 and therefore effective. Compared to existing vulnerability assessment tools, model evaluation engine 220 may provide a more comprehensive measurement of ML model vulnerabilities, and its comprehensive suite of stress tests can test many more edge cases and identify many more weaknesses in AI model 300.

[0039] 3, a representative diagram illustrating the operation of the model evaluation engine 220 is shown. In some embodiments, the model evaluation engine 220 can determine simulated attack data 222. The simulated attack data 222 can include any data that can attack the AI ​​model 300. The model evaluation engine 220 can output the simulated attack data 222 to a human operator (e.g., an engineer and / or a data scientist) and / or a computer system.

[0040] The model evaluation engine 220 is shown as receiving sample input data 320 and sample output data 340 associated with the AI ​​model 300. The sample input data 320 may include any data input to the AI ​​model 300 during trial and / or actual use of the AI ​​model 300. Exemplary sample input data 320 may include tabular data, biometric data, audio files, facial images, object images, etc. The sample output data 340 may include data including decisions of the AI ​​model 300 based on the sample input data 320. Exemplary sample output data 340 may include an inference as to whether the sample input data 320 is fraudulent, a label (and / or classification) indicating a person associated with the audio file and / or facial image, an object (and its location) detected within the object image, etc.

[0041] Additionally and / or alternatively, the model evaluation engine 220 may have access to the AI ​​model 300 so as to interact directly with the AI ​​model 300 for a complete analysis, including, for example, inputting specific data into the AI ​​model 300 to expose vulnerabilities in the AI ​​model 300. For example, the model evaluation engine 220 may have black-box and / or API access to the AI ​​model 300. Black-box access may include access that allows the model evaluation engine 220 to inspect the functionality (inputs and outputs) of the AI ​​model 300 without accessing its internal structure or mechanisms.

[0042] In various embodiments, the model evaluation engine 220 may determine the simulated attack data 222 based at least in part on the sample input data 320 and the sample output data 340. In one embodiment, the simulated attack data 222 may include at least a portion of the sample input data 320. Stated slightly differently, the model evaluation engine 220 may identify sample input data 320 that can attack and / or has attacked the AI ​​model 300. In another embodiment, the model evaluation engine 220 may generate simulated attack data 222 that is not part of the sample input data 320.

[0043] In one embodiment, the model evaluation engine 220 can simulate one or more attacks on the AI ​​model 300 using the simulated attack data 222. Thus, the model evaluation engine 220 can verify that the simulated attack data 222 can successfully attack the AI ​​model 300. In one example, the model evaluation engine 220 can input the simulated attack data 222 to the AI ​​model 300 and analyze whether and / or how the simulated attack data 222 attacks the AI ​​model 300 based on the output of the AI ​​model 300. In another example, the model evaluation engine 220 can input the simulated attack data 222 to a proxy model (not shown) that represents the AI ​​model 300 and analyze whether and / or how the simulated attack data 222 attacks the AI ​​model 300 based on the output of the proxy model. In another embodiment, the model evaluation engine 220 can determine the simulated attack data 222 without necessarily inputting it to the AI ​​model 300 or the proxy model.

[0044] In various embodiments, the model evaluation engine 220 can use parallel attacks for the combination algorithm. For example, some attacks may involve combination algorithms that are nondeterministic polynomial-time (NP) hard. The model evaluation engine 220 can solve the combination algorithm problem using parallelization. In one embodiment, the model evaluation engine 220 can run tests in which multiple feature values ​​can be actively changed simultaneously for a sample of multiple data points (e.g., in the simulated attack data 222 and / or the sample input data 320) to estimate the impact of encountering such changes in production (e.g., in interactions with external data 400).

[0045] In various embodiments, as described above, the model evaluation engine 220 can use a surrogate model for the attack. Stated slightly differently, the model evaluation engine 220 can at least partially replicate the AI ​​model 300 that is subject to simulated attack.

[0046] In various embodiments, the surrogate model can be used as a corrective lens. Stated slightly differently, the model evaluation engine 220 can improve and / or customize the AI ​​model 300. For example, the model maintenance system 200 can filter and additionally label the input data fed to and / or wrapped around the AI ​​model 300 and use those labels to train a model that may be better than the AI ​​model 300. In other words, the model maintenance system 200 provides logical filters to organize the input data and notify the user. Additionally and / or alternatively, the model maintenance system 200 can modify the input data and notify the user.

[0047] In various embodiments, the model evaluation engine 220 can use reference model detection. In various embodiments, the model evaluation engine 220 can establish a baseline of model performance and training data profile (range, type, etc.). Using the baseline, the model maintenance system 200 and / or an operator can compare the performance of multiple AI models 300 and determine the best model for production deployment.

[0048] As mentioned above, fraudsters often attempt to fool AI model 300. Thus, fraudsters and developers play a constantly evolving game of cat-and-mouse. To put it a little differently, a fraud detection company creates an AI model 300; the fraudster is initially thwarted but eventually evaded. The fraud detection company retrains the AI ​​model 300 to catch new types of fraud; the fraudster is again thwarted and must work again to evade. The model evaluation engine 220 may advantageously be the only solution for AI model evaluation that discovers vulnerabilities in the AI ​​model 300 before the fraudster does, enabling the AI ​​model 300 to defend against future types of attacks.

[0049] 4, further details of the AI ​​operating environment 100 are shown. The firewall 240 can protect the AI ​​model 300 from attacks by external data 400. For example, the firewall 240 can determine whether the external data 400 has been manipulated to deceive the AI ​​model 300.

[0050] In various embodiments, firewall 240 can detect similar data requests to defend AI model 300. Stated slightly differently, firewall 240 can block malicious users based on behavioral patterns in their queries to AI model 300.

[0051] In various embodiments, firewall 240 can use maximum entropy techniques to detect attacks. Stated slightly differently, firewall 240 can create multiple detectors and aggregate their responses.

[0052] In various embodiments, the firewall 240 can use a machine learning model as the detection engine 260 (shown in FIG. 5).

[0053] 5, model integrity system 200 is shown to include a fraud detection engine 260. Fraud detection engine 260 is capable of receiving external data 400 and determining whether external data 400 contains an attack. In some embodiments, fraud detection engine 260 may include an end-to-end pluggable engine in the form of an API that can ingest external data 400, automatically apply various signature manipulation techniques, and output a conclusion that external data 400 is fraudulent or non-fraudulent.

[0054] By including fraud detection engine 260, model maintenance system 200 may optionally include model evaluation engine 220 or firewall 240. However, model maintenance system 200 may include both, one, or neither model evaluation engine 220 and firewall 240. In various embodiments, fraud detection engine 260 may function to detect fraud as a stand-alone system without the need for AI model 300. However, in various embodiments, the process for developing fraud detection engine 260 may be based at least in part on the interaction of model maintenance system 200 and / or any component thereof with AI model 300.

[0055] Fraud detection based on tabular data The model security system 200 can be applied in various fields, a typical field being transaction fraud, such as conducting a transaction using fraudulent information. The model security system 200 can be widely used by both financial institutions and e-commerce companies in credit and debit card payments, bank account payments, electronic payments, and mobile wallet payments.

[0056] Another typical area is account takeover fraud, for example, when a fraudster maliciously accesses another user's savings account. The model security system 200 can be used extensively by both financial institutions and e-commerce companies.

[0057] Yet another representative area is check fraud, similar to transaction fraud but with additional information. Model security system 200 may be fed with, for example, images of the check in question and / or transaction history records.

[0058] The techniques used by the model maintenance system 200 can also be applied to general tabular data problems.

[0059] 6, sample input data 320 is shown as including tabular data. Tabular data may include information organized into rows and / or columns. Typical tabular data may be in the form of numbers, letters, characters, symbols, etc. Accordingly, AI model 300 may include a fraud detection model for detecting fraud in external data 400 (shown in FIG. 7).

[0060] In one embodiment, the sample input data 320 may include one or more data points. Each data point may include a set of tabulated data related to a user, an account, a transaction, or a combination thereof. The sample output data 340 may include labels assigned to the data points based on the AI ​​model 300. For example, each label may include a determination of whether the data point is fraudulent or not. Additionally and / or alternatively, the model evaluation engine 220 may access the AI ​​model 300. In one embodiment, the model evaluation engine 220 may output a report summarizing vulnerabilities in the AI ​​model 300.

[0061] In various embodiments, the model evaluation engine 220 may include an automated engine that acts as an impostor. The model evaluation engine 220 can simulate attacks that an impostor might attempt and discover vulnerabilities before the impostor does so.

[0062] In various embodiments, since the AI ​​model 300 is based on tabular data, the model evaluation engine 220 can use various techniques to simulate attacks by knowing feature dependencies across different columns.

[0063] In various embodiments, the model evaluation engine 220 can use algorithmic attacks on tabular data. In one embodiment, the model evaluation engine 220 can use fast submodule sampling to explore the state space.

[0064] In various embodiments, the model evaluation engine 220 can apply novel methods to handle feature dependencies.

[0065] In various embodiments, the model evaluation engine 220 can apply novel methods to handle categorical features.

[0066] In various embodiments, the model evaluation engine 220 can use multiple algorithms to discover vulnerabilities in the AI ​​model 300. While adversarial ML has been studied, the application of ML to fraud detection by the model evaluation engine 220 is unique at least because the model evaluation engine 220 attacks an AI model 300 trained on tabular data that does not necessarily include images. Tabular data may present various complicating factors not yet addressed by existing approaches. Complicating factors include, for example, categorical variables and / or feature dependencies. At least because the model evaluation engine 220 addresses such complicating factors, the model evaluation engine 220 is novel.

[0067] 7, external data 400 may include tabular data. In one embodiment, firewall 240 may indicate whether external data 400 includes any attacks identified by model evaluation engine 220. Accordingly, firewall 240 may generate alerts at selected levels based on customized levels of alerts. For example, customized levels of alerts may be determined based on severity and / or user preferences.

[0068] In various embodiments, firewall 240 can patch loopholes identified by model evaluation engine 220, creating an additional layer of security in front of AI model 300. Stated slightly differently, external data 400 may first pass through firewall 240 before passing through AI model 300. If firewall 240 detects suspicious data, such as an attack, in external data 400, firewall 240 may flag the suspicious data. Firewall 240 may alert engineers and / or data scientists to the dynamically evolving behavior of fraudsters to keep fraud detection up to date.

[0069] 8, fraud detection engine 260 is shown receiving external data 400 including tabulated data. Exemplary external data 400 may include user transaction history, user data, account history, account data, transaction information, or combinations thereof. In one embodiment, fraud detection engine 260 may output a probability that external data 400 is fraudulent.

[0070] In various embodiments, the fraud detection engine 260 can use an ML framework for binary classification. Exemplary ML frameworks can include boosted trees, neural networks, etc. The binary classification model can be trained with a variety of raw and / or processed features. The binary classification model can be exposed via an API and invoked by sending the requested data to the binary classification model. The underlying binary classification model can use deep learning techniques to achieve more accurate results. The API can be clearly designed to be simple and easy to use.

[0071] In various embodiments, the fraud detection engine 260 may use feature design and / or feature selection. In one embodiment, the fraud detection engine 260 may include features based on the user data to select which user data is relevant for detecting fraud.

[0072] Voice Authentication Voice authentication is a form of biometric authentication system that compares an individual's input voice with that person's registered voice to authenticate and verify the voice similarity. Voice authentication can be performed by extracting a voiceprint (e.g., a waveform that captures several key characteristics of the voice) and measuring the similarity using various acoustic statistical methods.

[0073] Voice authentication can be used in places such as call centers, consumer banks, and user authentication for smart home devices. The hands-free nature of voice authentication and its ease of integration (e.g., it can be done on a phone without the need for a camera) make it useful.

[0074] However, voice authentication is vulnerable to various forms of attack. For example, an impostor can impersonate the voice of a person they wish to be authenticated as. Imitation can take the form of (1) a talented voice actor impersonating the victim, or (2) recording the victim's voice and playing it in front of the voice authentication system (a replay attack). However, impersonation is difficult because (1) is a manual process that requires finding suitable personnel, and (2) voice authentication requires dynamic conversation or specific phrases, making it difficult to use recorded voices.

[0075] Recently, voice fraud has increasingly made use of voice synthesis. Voice synthesis is a technology that generates a fake voice from a recording of someone's voice. Recent advances in technology allow fraudsters to generate a realistic-sounding voice from just a few minutes of a victim's recording. Using the fake voice, the fraudster can fool voice authentication systems or even people by pretending to be the victim. For example, a fraudster could trick a colleague into transferring a large amount of money to the fraudster's bank account.

[0076] The model maintenance system 200 (shown in FIG. 1) is based on the technique for detecting synthetic speech as described above.

[0077] The model integrity system 200 can be applied in various fields. A typical application is deceiving call center authentication. Traditionally, call centers used knowledge-based authentication, where operators asked callers various questions (e.g., date of birth). Recently, some call centers have transitioned to using AI models 300 for integrated authentication. Voice authentication saves call time by automatically authenticating callers using their voiceprints as they speak. However, AI models 300 can be deceived using real-time voice conversion models that convert the speaker's voice into the voice of the person the speaker wishes to be recognized as.

[0078] Another typical area is deceiving voice authentication. For example, some banks use AI models 300 to authenticate users to deposit accounts, transfer money, and purchase financial products. AI models 300 can be deceived with synthetic voices.

[0079] Yet another typical area is calling coworkers: synthesized voices can be used, for example, to trick people into wire-wiping large sums of money to hackers.

[0080] All of the above areas may be protected by firewall 240 (shown in FIG. 1) of model security system 200.

[0081] 9, sample input data 320 is shown to include audio data. The audio data may include digital audio data in any suitable file format. Exemplary file formats may include uncompressed audio formats (e.g., Waveform Audio File Format (WAV), Audio Interchange File Format (AIFF), Au File Format, or raw headerless pulse code modulation (PCM)), lossless compressed formats (e.g., Free Lossless Audio Codec (FLAC), Monkey Sound (filename extension .ape), WavPack (filename extension .wv), True Audio (TTA), Adaptive Transform Acoustic Coding (ATRAC) highly lossless, Moving Picture Experts Group (MPEG)-4, and Shortened Noise Cancelling (SHN)), and / or lossy compressed formats such as Opus, MP3, Vorbis, Musepack, Advanced Audio Coding (AAC), ATRAC, and Windows Media Audio Lossy (WMA Lossy)). Thus, the AI ​​model 300 may include a voice recognition model for detecting whether external data 400 (shown in FIG. 10) is classified and / or labeled as belonging to a particular person.

[0082] In one embodiment, the sample input data 320 may include one or more sample audio files. The sample output data 340 may include labels assigned to the sample audio files based on the AI ​​model 300. For example, the labels may indicate a person (e.g., a user), and the AI ​​model 300 accepts the sample audio files as belonging to that person. Additionally and / or alternatively, the model evaluation engine 220 may have access to the AI ​​model 300.

[0083] In one embodiment, the model evaluation engine 220 can identify at least one of the sample audio files that includes synthetic audio data. Stated slightly differently, the simulated attack data 222 can include sample audio files that include synthetic audio data.

[0084] Additionally and / or alternatively, the model evaluation engine 220 may generate at least one red team audio file. The red team audio file may include a selected sample audio file to which noise has been added. The AI ​​model 300 recognizes the selected sample audio file as being that of a first person. Noise is added to the selected sample audio so that the AI ​​model 300 recognizes the red team audio file as being that of a second person different from the first person. Stated slightly differently, the simulated attack data 222 may include a red team audio file. Additionally and / or alternatively, the red team audio file may be generated by the model evaluation engine 220 without being based on a sample audio file.

[0085] In various embodiments, the model evaluation engine 220 may generate synthetic speech and / or add noise to the speech data to trick the AI ​​model 300 into misclassifying person A's voice as person B's voice. In some embodiments, the noise may be imperceptible to the AI ​​model 300.

[0086] 10, external data 400 may include audio data. In one embodiment, external data 400 may include one or more audio files, either synthetic or authentic. In various embodiments, synthetic may include fully synthetic and / or at least partially synthetic. Non-synthetic may include authentic, real, legitimate, or unedited, or untampered with, unaltered, or a combination thereof. In one embodiment, firewall 240 may indicate whether a voice presented in external data 400 is synthetic.

[0087] Additionally and / or alternatively, the external data 400 may further include identifications and / or labels assigned to each audio file. In one embodiment, the firewall 240 may indicate whether the identifications and / or labels have been correctly assigned to the audio files. Stated slightly differently, the firewall 240 may include a classifier that is not easily affected by any external data 400, including incorrect identifications of sounds within the audio files. The incorrect identifications may be adversarial and attack the AI ​​model 300.

[0088] In various embodiments, firewall 240 may include a speech synthesis detection model that uses various feature extraction methods from speech data and / or fine-tuned architectures specific to the speech synthesis detection model.

[0089] In various embodiments, firewall 240 may include a binary classification machine learning model trained to generate an output label indicating whether the audio data is deceptive or not.

[0090] In various embodiments, the firewall 240 may use a convolutional neural network-based architecture for the speech synthesis detection model.

[0091] In various embodiments, firewall 240 can perform feature extraction. When the audio data cannot be fed directly into the speech synthesis detection model, firewall 240 can extract various acoustic features suitable for speech synthesis detection.

[0092] In various embodiments, the firewall 240 may include a data augmentation technique that generates synthetic speech from real speech.

[0093] A voice synthesis detection model may erroneously pick up features or characteristics that are not authentic (or good) to learn from, and such features or characteristics are not necessarily spurious. For example, if an authentic audio file happens to have more female voices than male voices, the voice synthesis detection model may be trained to predict that the voice data is authentic when it is female. However, such a result is simply a coincidence of the data used for training. In various embodiments, firewall 240 may have a way to adjust the voice synthesis detection model so that it does not erroneously pick up such data distribution patterns.

[0094] In various embodiments, firewall 240 may include a detector that is trained on a substantial set of real and synthetic speech data. Additionally and / or alternatively, the detector may be trained on additional speech data generated using state-of-the-art synthetic speech generation techniques and / or improvements thereto (suitable for data augmentation in the particular instance of synthetic speech detection).

[0095] Additionally and / or alternatively, software may be deployed around the detector for deployment, which may be easy and seamless to install on a particular voice recognition system to put synthetic voice detection into production.

[0096] In various embodiments, the model integrity system 200 can detect fake voices over the phone with 95% detection accuracy and a 0.01% false positive rate, exceeding the performance of conventional systems.

[0097] Facial Recognition Facial recognition algorithms can be used to determine which people are present in a given photograph. Facial recognition algorithms can be an extension of face detection algorithms. Face detection algorithms can determine whether a person is present, if identity information is available, without verifying the identity of the individual. In some embodiments, firewall 240 can determine whether an image contains a human face.

[0098] Facial recognition algorithms may involve feature extraction and feature classification. For example, facial landmark analysis may extract features such as those around the nose, eyes, and cheeks (or any features that can distinguish a human face from other objects) and return the features as points.

[0099] Alternative feature extraction algorithms may include local binary patterns and eigenfaces, which return vectors or arrays as numerical descriptions of faces. Statistical methods such as histograms or principal component analysis can be used to separate feature vectors and distinguish multiple faces from each other accordingly. Other alternative approaches may include accelerated robust features (SURF) and scale invariant feature transform (SIFT).

[0100] Some computer vision methods can use deep learning models to generate feature vectors (or embeddings) from images. Advances have been made in separating feature vectors. Stated slightly differently, the better feature vectors belonging to different people can be separated, the more reliably the identity of the person to whom the vector belongs can be determined.

[0101] A cloud service to provide facial recognition capabilities can use the computer vision techniques described above. The model integrity system 200 can expose and protect against vulnerabilities in the AI ​​model 300 deployed by the cloud service. The model integrity system 200 can add imperceptible noise to cause the AI ​​model 300 to misidentify an individual without knowledge of what model is being used in the attack. The model integrity system 200 can deploy detectors that can determine whether images provided or to be provided to the AI ​​model 300 are clean or tampered.

[0102] The model security system 200 may be applied in a variety of fields. In various embodiments, at border crossings, the model security system 200 may identify individuals of interest crossing a secured boundary, for example, through an airport.

[0103] In various embodiments, in the case of bank personalization, the model security system 200 may identify an individual and match them with relevant banking information to allow for quick withdrawals and easy access to an individual's money.

[0104] In various embodiments, in the case of law enforcement, the model security system 200 can locate and identify individuals of interest in surveillance and traffic camera images.

[0105] In various embodiments, in the case of travel authorization, the model security system 200 can identify a person passing through a travel checkpoint and verify whether the person is trustworthy, and if so, the model security system 200 can expedite the travel checkpoint process.

[0106] In various embodiments, in the case of contactless checkout, the model security system 200 can identify individuals entering a store and automatically charge them for the items they purchase.

[0107] In various embodiments, in the case of a sporting event, the model maintenance system 200 can track players on the field and provide real-time tracking and information and / or statistics.

[0108] 11, sample input data 320 is shown including one or more facial images. The images may include computer-readable still images and / or videos in any suitable file format. Exemplary file formats may include Joint Photographic Experts Group (JPEG), Graphics Interchange Format (GIF), bitmap (BMP), Tagged Image File Format (TIFF), and / or Portable Network Graphics (PNG). Accordingly, AI model 300 may include a facial recognition model for detecting whether external data 400 (shown in FIG. 12) depicts a particular person's face.

[0109] In various embodiments, the sample input data 320 may include one or more sample image files. The sample output data 340 may include labels assigned to the sample image files based on the AI ​​model 300. For example, the labels may indicate a person (e.g., a user), and the AI ​​model 300 may accept the image and audio files as belonging to that person. Additionally and / or alternatively, the model evaluation engine 220 may have access to the AI ​​model 300.

[0110] In one embodiment, the model evaluation engine 220 can identify at least one sample image file that has been misclassified and / or incorrectly labeled by the AI ​​model 300. Misclassification can include misclassification to a particular label (e.g., a particular label selected or specified by an attacker) and / or misclassification to a label that is different from the original label. Stated slightly differently, the simulated attack data 222 can include sample image files that are misclassified.

[0111] Additionally and / or alternatively, the model evaluation engine 220 can obtain one or more images of the target person. The target person can be any selected person. A typical target person can include labels selected by an attacker from the sample output data 340. Stated slightly differently, the target person can include misclassified labels and / or the original labels of the misclassified sample image files. In various embodiments, the model evaluation engine 220 can generate one or more red team image files containing images of the target person. The red team image files can be synthetic. In one example, the red team image files can be at least partially photorealistic. In another example, the red team image files can include noisy images of the target person. In various embodiments, the model evaluation engine 220 can generate any type of image that can operate to fool the AI ​​model 300. While some generated images can have photorealistic elements, not all generated images are limited to photorealistic images. The AI ​​model 300 can recognize the red team image files as being of a person different from the target person. Stated slightly differently, the simulated attack data 222 can include the red team image files. Additionally and / or alternatively, the red team image files may be generated by the model evaluation engine 220 based neither on images of the target person nor on sample image files.

[0112] In various embodiments, the model evaluation engine 220 can attack an AI model 300 for detecting facial similarity. Stated slightly differently, the AI ​​model 300 can be given two photos of faces, A and B, and determine whether photos A and B are similar. Additionally and / or alternatively, the model evaluation engine 220 can attack an AI model 300 for detecting facial identity. Stated slightly differently, the AI ​​model 300 can be given a photo of a face and classify that face as a particular person. For both types of AI model 300, the model evaluation engine 220 can add small noises to fool the AI ​​model 300, even if the AI ​​model 300 is state-of-the-art.

[0113] In various embodiments, the model evaluation engine 220 can use algorithmic attacks to measure and evaluate the robustness of the AI ​​model 300 to state-of-the-art attacks.

[0114] In various embodiments, based on the evaluation by the model evaluation engine 220, the model evaluation engine 220 can generate a report that allows a user to easily assess how vulnerable the AI ​​model 300 is and understand which aspects of the AI ​​model 300 are most vulnerable. The report can be easy to understand and explainable.

[0115] In various embodiments, the model evaluation engine 220 can use algorithmic attacks to fool the AI ​​model 300, even if the AI ​​model 300 includes a state-of-the-art ML model.

[0116] 12 , external data 400 may include one or more images of faces. External data 400 may further include an identification and / or label assigned to each image. In one embodiment, firewall 240 may indicate whether the identification and / or label has been correctly assigned to the image. Stated slightly differently, firewall 240 may include a robust classifier that is not easily affected by any external data 400 that includes incorrect identification of images. The incorrect identification may be adversarial and attack AI model 300.

[0117] Additionally and / or alternatively, external data 400 may include one or more image files that are synthetic (or machine-generated or machine-manipulated) or real. Stated slightly differently, the image files may include photographs of real faces, or images of synthetic (or machine-generated) faces. External data 400 may further include a category to which the image file belongs. In one embodiment, the categories may include "real faces" and "machine-generated faces." In various embodiments, firewall 240 may indicate whether a face in an image file is real or machine-generated. For example, firewall 240 may include a robust detector for determining the correct category of an image file.

[0118] In various embodiments, firewall 240 can protect AI model 300 against flaws in AI model 300 identified by model evaluation engine 220.

[0119] In various embodiments, firewall 240 can receive external data 400 and output whether external data 400 is a legitimate facial image or an attacked facial image.

[0120] In various embodiments, firewall 240 may use a model specifically configured for facial and / or image recognition. The model may be trained on a large collection of legitimate facial images and challenged images. The challenged images may include authentic images that have been altered to challenge AI model 300. In one embodiment, the challenged images may be generated by model evaluation engine 220.

[0121] In various embodiments, firewall 240 may perform liveness detection. For example, if external data 400 is a still image of a face, firewall 240 may not recognize that face as real or may detect another person wearing a mask of that face (even though both such images may appear to the camera as a face). Firewall 240 can detect both of these types of attacks (i.e., presenting a photo of an individual and / or wearing a mask of an individual).

[0122] In various embodiments, firewall 240 can be a software platform deployed on-premise and / or in the cloud to act as a firewall layer before potentially hostile inputs are fed to AI model 300. Firewall 240 can be customized with deep learning models that act as detectors that can determine whether inputs are hostile or benign and provide a response to an operator. The detectors can function with high availability and low latency. API keys can be provided to operators to specify which detectors, classifications, and / or services they can access.

[0123] In various embodiments, the detectors of firewall 240 may be created using deep machine learning techniques.

[0124] In various embodiments, the model integrity system 200 may use various techniques to expose and / or attack the AI ​​model 300. Exemplary techniques may include open-source face recognition algorithms, neural network-based face embedding algorithms, etc.

[0125] In various embodiments, the model integrity system 200 can expose and / or attack methods for training the AI ​​model 300 by compressing facial images into effective low-dimensional embeddings.

[0126] In various embodiments, the model integrity system 200 can expose and / or attack methods for training the AI ​​model 300 by dividing their embeddings into multiple categories to act as a facial recognition algorithm.

[0127] In various embodiments, the model integrity system 200 can use a method to fool face recognition algorithms by adding small amounts of noise to images of faces.

[0128] In various embodiments, the model maintenance system 200 can use neural network-based methods to extract specific images of faces from noise (e.g., with a generative adversarial network).

[0129] In various embodiments, the model maintenance system 200 can use methods to make existing deep learning-based face recognition models more robust to spurious inputs.

[0130] Object detection Object detection can be used to identify objects in an image. A typical example of object detection can be gun detection, which is detecting whether a gun is present in an image.

[0131] Some object detection systems can use deep learning techniques. A machine learning model can be trained over a large number of images containing the object that the model is to detect. In contrast to other image classification techniques, an object detection model can specify the location within an image of the detected object.

[0132] Cloud services can provide object detection results, for example, a cloud service can provide an API to detect objects in images uploaded by users.

[0133] Like other ML models, ML models for object detection are vulnerable to attacks. Imperceptible, carefully crafted noise added to an image can fool the model into not detecting objects in the image. The model integrity system 200 can expose and protect against vulnerabilities in the AI ​​model 300. The AI ​​model 300 can be deployed via a cloud service.

[0134] Model integrity system 200 may be applied in a variety of fields. In various embodiments, in the case of an autonomous vehicle, object detection based on model integrity system 200 may be used to detect other vehicles, traffic signals, pedestrians, and / or other objects so that the vehicle can adapt its direction.

[0135] In various embodiments, in the case of gun detection and / or security, object detection based on the model security system 200 may be used to identify anomalies in a location, such as guns or bombs.

[0136] In various embodiments, in the case of people counting, object detection based on the model integrity system 200 can be used to estimate the number of people in a particular location.

[0137] In various embodiments, in the case of sports, object detection based on the model integrity system 200 may be used to track the ball and players.

[0138] 13, sample input data 320 is shown as including one or more images depicting one or more objects. The images may include computer-readable still images and / or video in any suitable file format. Accordingly, AI model 300 may include an object detection model for classifying objects in images included in external data 400 (shown in FIG. 14). Classifying may include detecting the object and / or its location in the image.

[0139] In various embodiments, sample input data 320 may include one or more sample image files. In various embodiments, sample output data 340 may represent one or more objects and their positions within each sample image file based on AI model 300. Additionally and / or alternatively, model evaluation engine 220 may have access to AI model 300.

[0140] In one embodiment, the model evaluation engine 220 may generate a report of the robustness and / or deficiencies of the AI ​​model 300. The report may be simple and easy for an operator to interpret. In various embodiments, the model evaluation engine 220 may run a set of pre-defined tests on the AI ​​model 300 and associated data. Each test may measure the vulnerability of the AI ​​model 300 along a particular axis, and the model evaluation engine 220 may generate a report based on the results of these tests.

[0141] Additionally and / or alternatively, the model evaluation engine 220 may identify at least one sample image file that has fooled or could fool the AI ​​model 300. Stated slightly differently, the simulated attack data 222 may include the sample image files identified by the model evaluation engine 220.

[0142] Additionally and / or alternatively, the model evaluation engine 220 can generate red team image files by manipulating the sample image files. Each red team image file can fool the AI ​​model 300, such that the AI ​​model 300 can correctly classify objects in the sample image files but cannot correctly classify objects in the red team image files. Stated slightly differently, the simulated attack data 222 can include such red team image files.

[0143] Additionally and / or alternatively, the red team image files may be generated by the model evaluation engine 220 without being based on sample image files. Stated slightly differently, the simulated attack data 222 may include red team image files.

[0144] In various embodiments, the model evaluation engine 220 may digitally and physically attack images and videos to trick the AI ​​model 300. Digital adversarial attacks are similar to attacks on facial recognition, in which the model evaluation engine 220 may add noise to images of an object to trick the AI ​​model 300 into classifying the object as a different object. The noise may be small. Physical adversarial attacks may include attacks in which people print special stickers and wear them, for example, to represent various objects (e.g., guns), preventing the AI ​​model 300 from correctly classifying the object.

[0145] 14 , the external data 400 may include one or more images depicting one or more objects. In various embodiments, the firewall 240 can protect the AI ​​model 300 against deficiencies in the AI ​​model 300 identified by the model evaluation engine 220.

[0146] Additionally and / or alternatively, external data 400 may include one or more image files. Firewall 240 may indicate whether an image file has been manipulated to fool AI model 300. In one embodiment, firewall 240 may generate a confidence number that indicates whether the image file is a legitimate (or actual or genuine) image or an image that has been manipulated to fool AI model 300 through digital and / or physical adversarial attacks.

[0147] In various embodiments, firewall 240 can use a model specifically configured for object detection. The model can be trained over a large population of legitimate and / or challenged images of objects. In one embodiment, the challenged images can be generated by model evaluation engine 220.

[0148] Representative Applications for Fraud Detection Based on Tabular Data AI attacks using tabular data can include:

[0149] 1) Fake reviews and purchases that artificially boost the ranking of a product or seller 2) Fake accounts created to take advantage of membership promotions / benefits 3) Fraudulent product listings that would convince commerce AI platforms to allow listings to be displayed and subsequently deceive buyers 4) Engagement attacks that generate artificial click, installation, and app engagement metrics 5) Fraudulent transactions using stolen credit cards 6) Fraudulent credit cards and bank accounts opened using stolen and / or false identities Tabular data, which accounts for 10–15% of all existing data, is used as the basis for authorization or access for many transactions. The model integrity system 200 and / or fraud detection engine 260 (shown in FIG. 8 ) can detect whether a particular transaction is fraudulent. External data 400 input to the fraud detection engine 260 may include a data vector representing a transaction. The fraud detection engine 260 may perform several steps. Data may be normalized. Feature selection may be performed to reduce the number of features. For example, one or more models, including deep neural networks and / or gradient-boosted decision trees, may be used to generate various predictions. The individual predictions may be aggregated to generate a final score. The output of the fraud detection engine 260 may include a fraud score, for example, ranging from 0 to 100, with a score of 100 meaning the transaction is likely fraudulent. Data issues such as preparation, cleaning, error detection, and missing value imputation are examples where AI environments are used to make large-scale decisions and exploit (or attack) weaknesses.

[0150] In various embodiments, neural networks may include techniques for mimicking human brain activity. Typical activities may include pattern recognition and / or passing input through various layers of simulated neural connections. Deep neural networks may include one or more networks with an input layer, an output layer, and at least one hidden layer between them. Each layer may perform a specific type of sorting and / or ordering in feature hierarchy processing. Deep learning may characterize the function of deep neural networks because deep learning may represent a form of machine learning, and the technique uses AI to classify and / or order information in a more complex way than a simple input / output protocol.

[0151] In various embodiments, gradient-boosted decision trees may include algorithm-based machines that combine individual decision trees using boosting techniques. Boosting may involve serially combining learning algorithms to achieve a strong learner from many sequentially trained weak learners. In the case of gradient-boosted decision tree algorithms, the weak learners may be decision trees. Each tree can minimize the error of the previous tree. Although trees in boosting are weak learners, connecting many trees in series, with each tree converging on the error of the previous tree, can make boosting a very efficient and accurate model. Unlike bagging, boosting does not involve bootstrap sampling.

[0152] The model integrity system 200 may address various aspects of system flexibility, including data augmentation and data modeling for training defensive data detectors, integration with recursive training whenever possible to enhance all data received by the AI ​​defender (including using techniques such as validating input data with other submitted data or data sources), and unique system optimization and learning local models to enable scalable deployment and accommodation of unique data sets and multiple data outputs.

[0153] Typical tabular data exploits may include:

[0154] In the case of bank loan approval, loan approval is a critical process for banking institutions. This process is used to approve or reject customer loan applications. Loan collection is a major factor in a bank's financial statements, and therefore approving the correct loan application is very important. An AI model 300 can be useful in making these decisions based on customer data provided in the application, such as expected earnings, collateral value, current liquidity, or age.

[0155] In a typical situation, AI model 300 rejects an application from an individual. The individual believes they want to be approved for a loan and therefore provides false information to the bank in order to trick AI model 300 into approving the application. To avoid getting caught for bank fraud, it is required that the false information be imperceptible to humans and still be relevant and credible when a potential expert reviews the application. This requires that exploits be imperceptible to human inspection, or at least slow to be detected by a human. Therefore, exploits must manipulate the underlying variables used (which humans typically do not catch if the application is flagged for human inspection). Exploits may require at least partial knowledge of AI model 300 to determine which features are less important to AI model 300. Exploits focus on changing the values ​​of features that are less important to AI model 300 over a range, so that human inspection of various features to detect fraud requires longer time. Accordingly, modifying more significant features will result in higher attack sensitivity, allowing attacks to be more easily detected by reviewers. It is also crucial that each feature in an adversarial attack preserves the relevance and authenticity of its original instance. This can be achieved by verifying that each modified feature preserves its inherent constraints and type. Loan applications that use more variables in the AI ​​model 300 are more susceptible to attack because reviewers are less likely to base their reviews on a combination of several tabular data. Attacks must also maintain global consistency, which is not necessary for pixel-by-pixel data manipulation. If the exploitation model does not satisfy data correlations, reviewers may more easily doubt the results. The exploitation model may include a machine learning model used to generate external data 400 to attack the AI ​​model 300. Therefore, the exploitation model must exist within varying noise factors (e.g., addresses cannot exist outside of a ZIP code, or professional titles exist outside of educational levels).Stated slightly differently, while fraud due to falsifying data correlations may be more easily detected by an inspector, fraud due to varying noise factors is not as easily detected. The model integrity system 200 can prevent such fraud by looking at the relationships between various varying noise factors to determine the legitimacy of a particular input sequence.

[0156] An embodiment of the model maintenance system 200 FIG. 15 illustrates a model maintenance system 200 that uses AI to analyze fraud threats to an AI model 300. The exemplary AI model 300 may be used for telemedicine consultations, e-commerce transactions, prescription systems, fund transfer systems, or other systems that can receive one or more variable tabular data sets. The model maintenance system 200 may also receive non-tabular data. Examples of non-tabular data may include handwriting samples, images, audio speech, and / or biometric inputs. The model maintenance system 200 may have one or more sets of operating machines that allow the model maintenance system 200 to interact with entities in a cloud-based environment and / or via hardware machines in the case of offline devices such as item scanners.

[0157] Within a cloud environment, the model maintenance system 200 may include an ingestion machine 202 that receives data from the external AI world via API calls and / or hard-wired connections. The model maintenance system 200 may deploy a customizable set of models to a data feed of external data 400. The data feed may be used by the AI ​​model 300. The model maintenance system 200 may test the external data 400 for compliance or conformance with expected threshold parameters. In various embodiments, external data 400 that is outside of the normally expected parameters may be directed to a classification machine 206. The classification machine 206 may use selected models to classify the data set into categories of erroneous data (or incorrect data) 210 and / or suspicious data 212. Because the suspicious data 212 may include data that the reviewer cannot identify, the suspicious data 212 may be passed to a display machine 216, allowing the reviewer to appropriately review the suspicious data 212 and determine the source and consequences of the suspicious data 212. A reviewer can more easily determine the validity of the specified data using the results for the specified data from the model integrity system 200. Incorrect data 210 can be returned to the intake machine 202 and rejected.

[0158] Additionally and / or alternatively, near-threshold data 214 may be sent to a testing machine 217. Near-threshold data 214 may include data that is close to acceptable and / or slightly above a set threshold. Testing machine 217 may identify patterns and / or other repeating data within near-threshold data 214. Each identified pattern may be a vulnerability exploitation pattern 218. Once testing machine 217 determines a pattern, testing machine 217 may generate data parameters and identify various thresholds for that pattern to better understand whether model integrity system 200 is vulnerable to this type of vulnerability exploitation. Testing machine 217 may pass the vulnerability exploitation pattern 218 to model evaluation engine 220.

[0159] Traditional red-team attacks are tests of computer systems and involve penetration testing and vulnerability exploitation assessments. While red-team attacks are the practice of finding possible attack vectors, penetration testing is the practice of actually attacking the system. Traditional vulnerability exploitation assessments involve analyzing software to expose coding flaws that can be exploited. Traditional vulnerability exploitation assessments primarily examine individual software, focusing more on coding flaws and less on system configuration and business processes, and lacking a broader consideration of the entire system. While the results of penetration testing actually involve the implementation of red-team attacks, the attack vectors provided by penetration testing are very narrow and often do not reveal much about the system as a whole.

[0160] In contrast to traditional red-team attack approaches, the model integrity system 200 can use weakness exploitation patterns 218 characteristic and behavior modeling (e.g., via the model evaluation engine 220), modeling the AI ​​model 300 from a systems perspective, creating attack plans through simulation, and / or issues related to executing red-team attacks. This disclosure specifies how the model evaluation engine 220 can detect and create attack plans that exploit multiple vulnerabilities across the AI ​​model 300 rather than isolated vulnerabilities.

[0161] According to this disclosure, model evaluation may be used in the context of a model evaluation engine 220 that can continuously test an AI model 300 to understand where the AI ​​model 300 is weakest. The AI ​​model 300 may be weakest at the level of near-threshold data 214. Thus, by performing continuous testing of near-threshold data 214, the model integrity system 200 may proactively improve its capabilities and act as a virtual data firewall against suspect data 212.

[0162] Additionally and / or alternatively, in a cloud environment, the model maintenance system 200 may have access to multiple models to review the received external data 400. The external data 400 may contain alterations that may cause the AI ​​model 300 to misinterpret and / or misclassify the external data 400.

[0163] Additionally and / or alternatively, the model maintenance system 200 may be used in a stand-alone mode as a hardware component of various devices. In this configuration, the model maintenance system 200 may be embedded in a mobile device, computer application, and / or hardware appliance that may be positioned between the external data 400 and the AI ​​model 300 as described above.

[0164] FIG. 16 illustrates the training and use of a model integrity system 200 according to various embodiments. The model integrity system 200 may be trained with machine learning algorithms, programs, and / or tools and used to perform operations related to fraud or attack detection of an AI model 300 (shown in FIG. 1). The model integrity system 200 may operate by building a model from training data 232 to make data-driven predictions or decisions represented as outputs and / or create a successful exploitation model 234. A representative decision may include determining the training data 232 to be a "false positive fraud result." While embodiments are presented with respect to a few machine learning tools, the principles presented herein may be applied to other machine learning tools, particularly when discussing the detection of suspicious data presented to the AI ​​model 300 and intended to fraudulently alter results.

[0165] The model evaluation engine 220 can receive training data 232 and generate exploitation models 230 that are tested by an AI testing machine 236. When the exploitation models 230 are tested, variations that produce a positive result for the AI ​​testing machine 236 can be evaluated to determine whether the positive result is a false positive or an intended result of the AI ​​testing machine 236. As new successful exploitation models 234 are discovered, the firewall 240 can be updated to defend against the successful exploitation models 234.

[0166] In some embodiments, one or more machine learning tools may be used, such as logistic regression (LR), naive Bayes, random forest (RF), neural network (NN), matrix factorization, and support vector machine (SVM) tools, to classify or score at least one data feed to the ML program to perform attacks on discrete sets of data in the data feed against the AI ​​model 300 via the exploitation model 230.

[0167] ML can be used to solve classification and regression problems. Classification problems aim to classify items into one of several category values ​​(e.g., is this object an apple or an orange?). Regression algorithms aim to quantify multiple items (e.g., by assigning a value that is a real number). Regression analysis can measure the strength of cause-and-effect relationships within a structured data set. Regression analysis can be more sophisticated when applied to fraud detection due to the number of variables and the size of the data set. Regression analysis can provide value by evaluating the predictive power of individual variables or combinations of variables as part of a larger fraud strategy. The model integrity system 200 can predict whether a new transaction is fraudulent or not. In some embodiments, the model integrity system 200 can be specific to a particular trained customer base, but can also be applied for general use. The model integrity system 200 can utilize classification and / or regression to identify fraud attacks at various points in the solution.

[0168] In FIG. 17 , another set of machine learning tools used in a variant of the solution can be in the form of a disinfectant machine 280. The disinfectant machine 280 can detect fraudulent and / or manipulated data by understanding dependencies between multiple types of data presented to the AI ​​model 300. A typical application can be a financial transaction based on multiple data inputs. The data inputs, including addresses, phone numbers, and / or other aspects of an individual's data input, can be characterized as a data vector representing the transaction. The data points in the vector are subtly interrelated. For example, an unresolved address can use the phone number and / or area code to help determine whether the address and its phone number change to flag a fraudulent submission. The disinfectant machine 280 can create a graph representing the dependencies between different data points associated with the transaction. The model integrity system 200 can compare external data 400 to the graph and flag the external data 400 depending on how many dependencies the external data 400 violates. The output of the comparison can include a score between 0 and 1 representing how clean the data is.

[0169] Additionally and / or alternatively, the disinfectant machine 280 may be used to analyze the data feeds and discover unusual or irregular aspects of the data feeds of the external data 400. Each feature of the external data 400 may be decomposed into a set of individual measurable characteristics of the observed phenomenon. A representative phenomenon may include a combination of near-threshold and / or threshold data feeds. Features may relate to explanatory variables used in statistical methods such as linear regression. Selecting informative, discriminative, and independent features is important for the effective operation of ML programs in pattern recognition, classification, and regression. Features may be of various types, such as numeric features, strings, and graphs.

[0170] Additionally and / or alternatively, the model maintenance system 200 can identify various tabular and / or non-tabular data weakness exploits. Tabular data weakness exploits can use one or more weighting methods and can be based on tabular data that is altered and / or corrupted such that the AI ​​model 300 may record erroneous results based on training.

[0171] Figure 18 is a block diagram illustrating a software architecture 500 that may be implemented in any one or more of the devices described above. It will be appreciated that Figure 18 is merely a non-limiting example of a software architecture, and that many other architectures may be implemented to enable the functionality described herein. In various embodiments, software architecture 500 is executed by hardware, such as machine 600 of Figure 19.

[0172] In this example architecture, software architecture 500 may be conceptualized as a stack of layers, with each layer providing a particular function. For example, software architecture 500 may include layers such as operating system 504, libraries 506, frameworks 508, and applications 510. In operation, applications 510 invoke API calls 512 through the software stack and receive messages 514 in response to API calls 512, which may be the same in some embodiments.

[0173] In various embodiments, operating system 504 manages hardware resources and provides common services. Operating system 504 includes, for example, kernel 520, services 522, and drivers 524. Kernel 520 acts as an abstraction layer between the hardware and other software layers, which are identical in some embodiments. For example, kernel 520 provides memory management, processor management (e.g., scheduling), device management, network connectivity, security settings, and other functions. Services 522 may provide other common services to other software layers. According to some embodiments, drivers 524 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 524 may include a display driver, a camera driver, a Bluetooth® or Bluetooth® Low Energy driver, a flash memory driver, a serial communications driver (e.g., a Universal Serial Bus (USB) driver), a Wi-Fi® driver, an audio driver, a power management driver, etc.

[0174] In some embodiments, libraries 506 provide a low-level common infrastructure utilized by applications 510. Libraries 506 may include system libraries 530 (e.g., the C standard library), which may provide functions such as memory allocation functions, string manipulation functions, and mathematical functions. Libraries 506 may also include API libraries 532, such as a media library (e.g., a library supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group 4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer 3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework used for 2D and 3D drawing in a graphics context on a display), a database library (e.g., SQLite, which provides various relational database functions), a web library (e.g., WebKit, which provides web browsing functions), etc. The library 506 may also include a wide variety of other libraries 534 to provide many other APIs to the application 510 .

[0175] According to some embodiments, framework 508 provides a high-level common infrastructure that applications 510 can utilize. For example, framework 508 may provide various graphical user interface (GUI) functionality, high-level resource management, high-level location services, etc. Framework 508 may provide a wide range of other APIs (some of which are specific to a particular operating system 504 or platform) that applications 510 can utilize.

[0176] In one embodiment, applications 510 include a home application 550, a contacts application 552, a browsing application 554, a book reading application 556, a location application 558, a media application 560, a messaging application 562, a game application 564, and a wide variety of other applications, such as third-party applications 566. According to some embodiments, applications 510 are programs that perform defined functions. Various programming languages ​​may be used to create one or more of applications 510, with various structures. Examples include object-oriented programming languages ​​(e.g., Object C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In particular examples, third-party applications 566 (e.g., applications developed by entities other than the vendor of a particular platform using an ANDROID® or IOS™ Software Development Kit (SDK)) may be mobile phone software running on a mobile phone operating system, such as IOS, ANDROID®, WINDOWS® Phone, or another mobile phone operating system. In this example, the third party application 566 may invoke API calls 512 provided by the operating system 504 to enable the functionality described herein.

[0177] FIG. 19 illustrates a schematic diagram of a machine 600 in the form of a computer system that executes a set of instructions to cause the machine 600 to perform any one or more of the methods described herein, according to an embodiment. Specifically, FIG. 19 illustrates a schematic diagram of the machine 600 in the form of an example computer system, within which instructions 616 (e.g., software, programs, applications, applets, apps, or other executable code) may be executed to cause the machine 600 to perform any one or more of the methods described herein. For example, the instructions 616 may cause the machine 600 to perform the method of FIG. 2. Additionally or alternatively, the instructions 616 may execute the instructions of FIGS. 1 and 3-18. The instructions 616 transform an unprogrammed, general-purpose machine 600 into a specialized machine 600 that is programmed to perform the described functions in the described manner. In other embodiments, the machine 600 may operate as a standalone device or may be coupled (e.g., networked) to other machines. When networked, the machine 600 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 600 may comprise, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a mobile phone, a smartphone, a handheld device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 616 that specify operations for the machine 600 to perform. Also, although only a single machine 600 is shown, the term "machine" should be considered to include a collection of machines 600 that individually or collectively execute instructions 616 to perform any one or more of the methods described herein.

[0178] Machine 600 may include multiple processors 610, memory 630, and I / O components 650 configured to communicate with each other, for example, via bus 602. In one embodiment, multiple processors 610 (e.g., central processing units (CPUs), reduced instruction set computing (RISC) processors, complex instruction set computing (CISC) processors, graphics processing units (GPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), radio frequency integrated circuits (RFICs), another processor, or any suitable combination thereof) may include, for example, processor 612 and processor 614 that execute instructions 616. The term "processor" is intended to include multi-core processors consisting of two or more independent processors (sometimes referred to as cores) that execute instructions 616 simultaneously. Although FIG. 6 shows multiple processors 610, machine 600 may include a single single-core processor 612, a single multiple-core processor 612 (e.g., a multi-core processor 612), multiple single-core processors 612, 614, multiple multiple-core processors 612, 614, or any combination thereof.

[0179] Memory 630 may include a main memory 632, a static memory 634, and a storage unit 636, each accessible to processor 610 via bus 602. Main memory 632, static memory 634, and storage unit 636 store instructions 616 that embody any one or more of the methods or functions described herein. Instructions 616 may reside, completely or partially, within main memory 632, static memory 634, storage unit 636, within at least one processor 610 (e.g., within a processor's cache memory), or any suitable combination thereof during execution by machine 600.

[0180] I / O components 650 may include a wide variety of components that receive input, provide output, generate output, transmit information, exchange information, capture measurements, etc. The specific I / O components 650 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It will be understood that I / O components 650 may include many other components not shown in FIG. 6 . I / O components 650 are grouped according to function solely to simplify the following description, and this grouping is in no way limiting. In various embodiments, I / O components 650 may include output components 652 and input components 654. Output components 652 may include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistive mechanisms), other signal generators, etc. Input components 654 may include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or another pointing instrument), tactile input components (e.g., physical buttons, a touchscreen that provides the location and / or force of a touch or touch gesture, or other tactile input component), audio input components (e.g., a microphone), etc.

[0181] In other embodiments, I / O component 650 may include a biometric component 656, a motion component 658, an environmental component 660, or a position component 662, among a wide range of other components. For example, biometric component 656 may include components for detecting facial expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measuring biometric signals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., voice identification, retinal identification, face identification, fingerprint identification, or electroencephalogram-based identification), etc. Motion component 658 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environmental components 660 may include, for example, an illuminance sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects the concentration of harmful gases for safety or to measure atmospheric pollutants), or other components that may provide an indication, measurement, or signal corresponding to the surrounding physical environment. The location components 662 may include a location sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0182] Communication may be achieved using a wide variety of technologies. I / O component 650 may include a communication component 664 operable to couple machine 600 to network 680 or device 670 via coupling 682 and coupling 672. For example, communication component 664 may include a network interface component or another suitable device to interface with network 680. In further examples, communication component 664 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, a Bluetooth component (e.g., Bluetooth low energy), a Wi-Fi component, and other communication components providing communication via other methods. Device 670 may be another machine or any of a wide variety of peripheral devices (e.g., coupled via USB).

[0183] The communication component 664 may also include a component that detects an identifier or is operable to detect an identifier. For example, the communication component 664 may include a radio frequency identification (RFID) tag reading component, an NFC smart tag detection component, an optical reading component (e.g., an optical sensor that detects one-dimensional barcodes such as the Universal Product Code (UPC) barcode, multidimensional barcodes such as QR Code, Aztec Code, DATA Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone that identifies tagged audio signals). Additionally, various information may be obtained via the communication component 664, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi signal triangulation, or location via detection of NFC beacon signals indicating a particular location.

[0184] Various memories (i.e., 630, 632, 634 and / or memory of processor 610) and / or storage unit 636 may store one or more sets of instructions 616 and data structures (e.g., software) that embody or are utilized by any one or more of the methods or functions described herein. These instructions (e.g., instructions 616), when executed by processor 610, cause various operations that implement the disclosed embodiments.

[0185] As used herein, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” mean the same thing and may be used interchangeably. These terms refer to a single or multiple storage devices and / or media (e.g., centralized or distributed databases and / or associated caches and servers) that store executable instructions and / or data. Accordingly, these terms should be understood to include, but are not limited to, solid-state memory and optical and magnetic media, and to include memory internal or external to a processor. Specific examples of machine storage medium, computer storage medium, and / or device storage medium include, by way of example, semiconductor memory devices, non-volatile memory including, for example, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), field programmable gate arrays (FPGA), and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage media," "computer storage media," and "device storage media" specifically do not include carrier waves, modulated data signals, and other such media (at least some of which are included in the term "signal media" described below).

[0186] In various embodiments, one or more portions of network 680 may be an ad-hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular network, a wireless network, a Wi-Fi network, another type of network, or a combination of two or more such networks. For example, network 680 or a portion of network 680 may include a wireless or cellular network, and coupling 682 may be a code division multiple access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or another type of cellular or wireless coupling. In this example, coupling 682 may implement any of a variety of data transfer technologies, such as single-carrier radio transmission technology (1xRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, Third Generation Partnership Project (3GPP®) including 3G, Fourth Generation Wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standards, other standards defined by various standards-setting bodies, other long-range protocols, or other data transfer technologies.

[0187] The instructions 616 may be transmitted or received over the network 680 using a transmission medium via a network interface device (e.g., a network interface component included in the communications component 664) and utilizing any one of several well-known transfer protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 616 may be transmitted or received using a transmission medium via a coupling 672 (e.g., a peer-to-peer coupling) to the device 670. The terms “transmission medium” and “signal medium” mean the same thing and may be used interchangeably in this disclosure. The terms “transmission medium” and “signal medium” should be understood to include any intangible medium that may store, encode, or carry the instructions 616 for execution by the machine 600, including digital or analog communications signals or other intangible media that enable the communication of such software. Accordingly, the terms “transmission medium” and “signal medium” should be understood to include any form of modulated data signal, carrier wave, etc. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.

[0188] The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure. These terms are defined to include both machine storage media and transmission media. Thus, these terms include both storage devices / media and carrier / modulated data signals.

[0189] Embodiments of this solution may be implemented using one or a combination of hardware, firmware, and software. Embodiments may also be implemented as instructions stored on a computer-readable storage device that can be read and executed by at least one processor to perform the operations described herein. A computer-readable storage device may include any non-storage information in a form readable by a machine (e.g., a computer). For example, a computer-readable storage device may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, cloud servers, or other storage devices and media. Some embodiments may include one or more processors and have instructions stored on a computer-readable storage device. The following description and the referenced drawings fully describe certain embodiments and enable those skilled in the art to implement them. Other embodiments may include structural, logical, electrical, process, and other changes. Portions and features of some embodiments may be included in or substituted for those of other embodiments. Embodiments specified in the embodiments include all valid equivalents of those embodiments.

[0190] It should be understood that the recitation of one or more method steps does not preclude the presence of additional or intermediate method steps between the stated steps. Similarly, it should be understood that the recitation of one or more components in a device or system does not preclude the presence of additional or intermediate components between the stated components.

[0191] The above description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are also referred to herein as "embodiments." Such embodiments may include elements in addition to those shown or described. However, the inventors also contemplate embodiments in which only the elements shown or described are provided. The inventors also contemplate embodiments using any combination or permutation of the elements (or one or more aspects thereof) shown or described with respect to a particular embodiment (or one or more aspects thereof) or with respect to any other embodiment (or one or more aspects thereof) shown or described herein.

[0192] In the event of a conflicting usage between this document and any document cited, the usage in this document will prevail.

[0193] As is common in patent documents, the English terms "a" or "an" are used herein to include one or more than one, independently of any other instance or use of "at least one" or "one or more." The English term "or" is used herein to refer to a non-exclusive or, such that, unless otherwise indicated, "A or B" includes "A but not B," "B but not A," and "A and B." The English terms "including" and "in which" are used herein as the plain English equivalents of the terms "comprising" and "wherein," respectively. Also, in the accompanying embodiments, the English terms "including" and "comprising" are open-ended, i.e., systems, devices, articles, compositions, formulations, or processes that include elements in addition to the elements listed after such terms are also considered to fall within the scope of the embodiments. Also, in the accompanying embodiments, the terms "first," "second," "third," etc. are used merely as labels and are not intended to impose numerical requirements on their objects.

[0194] Geometric terms such as "parallel," "perpendicular," "round," or "square" are not intended to require absolute mathematical precision unless the context indicates otherwise. On the contrary, such geometric terms allow for variations due to manufacture or equivalent functions. For example, if an element is described as round or approximately round, elements that are not exactly circular (e.g., elements that are slightly elliptical or multi-sided polygonal) are also included in this description.

[0195] The example methods described herein may be executed, at least in part, by a machine or computer. Some examples may include a computer-readable medium or machine-readable medium encoded with instructions operable to configure an electronic device to perform the methods described in the examples. Such method embodiments may include code, such as microcode, assembly language code, high-level language code, etc. Such code may include computer-readable instructions for performing various methods. The code may form part of a computer program product. Furthermore, in one example, the code may be stored, for example, during execution or at other times, on one or more volatile, persistent, or non-volatile tangible computer-readable media. Examples of these tangible computer-readable media may include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or sticks, random access memory (RAM), read-only memory (ROM), etc.

[0196] The above description is intended to be illustrative, not limiting. For example, the above-described examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments may be used, for example, by those of ordinary skill in the art after reviewing the above description. The Abstract is provided to comply with 37 CFR 1.72(b) and to allow the reader to quickly ascertain the nature of the present technical disclosure. The Abstract has been submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the embodiments. Additionally, the above Detailed Description may have organized the disclosure by grouping various features. This should not be construed as intending that any disclosed feature not present in an embodiment is essential to any embodiment. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Therefore, the following embodiments are recited in the Detailed Description as examples or embodiments, and it is contemplated that each embodiment stands on its own as a separate embodiment, and that such embodiments can be combined with each other in various combinations or permutations. The scope of the present invention should be determined with reference to the following embodiments, along with the full range of equivalents to which such embodiments are entitled.

[0197] While the disclosed embodiments are susceptible to various modifications and alternative forms, and specific examples thereof have been shown by way of example in the drawings and described in detail herein, it should be understood that the disclosed embodiments are not limited to the particular forms or methods disclosed, but rather, these embodiments include all modifications, equivalents, and alternatives.

[0198] The embodiments of the present invention are listed below.

[0199] (Embodiment 1) 1. A method for securely deploying an artificial intelligence (AI) model, comprising: identifying one or more defects in the AI ​​model; protecting the AI ​​model from being attacked by external data based on the identifying step; A method comprising:

[0200] (Embodiment 2) The identifying step includes simulating one or more attacks on the AI ​​model with simulated attack data, and the simulating includes: simulating multiple parallel attacks and identifying flaws in the AI ​​model against those attacks based on a combinatorial algorithm; attacking at least one alternative model configured to replicate the AI ​​model; 2. The method of embodiment 1, comprising at least one of:

[0201] (Embodiment 3) filtering or modifying input data provided to the alternative model; labeling the input data; training the replacement model using the labeled input data such that the replacement model is an improved version of the AI ​​model; 3. The method of embodiment 2, further comprising:

[0202] (Embodiment 4) The step of identifying includes: establishing one or more baselines of model performance and training data profile; comparing performance of the AI ​​model to one or more alternative models based on the establishing step and selecting a model from the AI ​​model and the alternative models for production deployment; Including, the protecting step includes blocking one or more users associated with the external data based on a behavioral pattern of the users' queries to the AI ​​model; or 4. The method of any one of claims 1 to 3, wherein the protecting step includes using a maximum entropy method to detect attacks on the AI ​​model, the maximum entropy method including generating a plurality of detectors and aggregating their responses.

[0203] (Embodiment 5) 5. The method of any one of embodiments 1 to 4, wherein the external data includes tabular data related to user transaction history, account history, account data, user data, transaction information, or a combination thereof.

[0204] (Embodiment 6) the identifying step is based on sample input data including one or more data points and sample output data including labels assigned to the data points based on the AI ​​model, each label including a determination of whether the data point is fraudulent or not fraudulent; or 6. The method of embodiment 5, wherein the identifying step is based on fast submodule sampling to explore the state space.

[0205] (Embodiment 7) 7. The method of claim 5 or 6, wherein the protecting step includes generating an alert level based on the identifying step to indicate whether the external data includes one or more attacks that deceive the AI ​​model.

[0206] (Embodiment 8) 8. The method of any one of claims 5 to 7, further comprising generating a probability that the external data is fraudulent, wherein generating the probability comprises using one or more machine learning (ML) frameworks for binary classification, wherein the ML frameworks comprise boosted trees, neural networks, or a combination thereof.

[0207] (Embodiment 9) The protecting step includes: receiving the external data from a data feed related to a commercial transaction, the external data including tabular data defining risk attributes of entities; converting the tabular data into an input format; identifying manipulation of one or more elements in the tabular data to generate a false positive result; rejecting the tabular data as a manipulated data set; 9. The method according to any one of embodiments 5 to 8, comprising:

[0208] (Embodiment 10) The protecting step includes: receiving the external data including the tabular data; detecting manipulation of the tabulated data to identify false positive results; determining that the false positive result will occur within the AI ​​model; alerting the AI ​​model and flagging the tabular dataset for human review; 10. The method according to any one of embodiments 5 to 9, comprising:

[0209] (Embodiment 11) the external data includes voice data, and the AI ​​model includes a voice authentication model for detecting whether the external data is classified as belonging to a specified person, and the identifying step, the protecting step, or both: training a binary classification machine learning model to generate an output label indicating whether the audio data is fake or not; using a convolutional neural network-based architecture for said detection; Extracting various acoustic features suitable for said detection; or Generating synthetic voice data from authentic voice data using data augmentation 11. The method according to any one of embodiments 1 to 10,

[0210] (Embodiment 12) the identifying step is based on sample input data including one or more sample audio files and sample output data including labels assigned to the sample audio files based on the AI ​​model, each label indicating an individual, and the identifying step includes identifying at least one of the sample audio files that includes synthesized audio data and is accepted by the AI ​​model as being the designated individual; or 12. The method of claim 11, wherein the identifying step includes generating a redundancy audio file that includes both authentic audio data of a first person and added noise, the noise being added to the authentic audio data such that the AI ​​model is configured to recognize the selected sample audio file as being of a second person different from the first person.

[0211] (Embodiment 13) 13. The method of claim 11 or 12, wherein the protecting step includes detecting whether a voice presented in the external data is synthetic.

[0212] (Embodiment 14) The external data includes at least one image file, and the AI ​​model includes a facial recognition model for detecting whether the image file shows a face of a specified person, and the identifying step, the protecting step, or both of them: Neural network based face embedding, Compressing face images into lower dimensional embeddings, Categorizing lower dimensional embeddings for face recognition, Attacking the AI ​​model by adding noise to facial images; or A neural network that uses a generative adversarial network to extract specific facial images from noise 14. The method according to any one of embodiments 1 to 13,

[0213] (Embodiment 15) the identifying step is based on sample input data including one or more sample image files of faces and sample output data including labels assigned to the sample image files based on the AI ​​model, each label indicating an individual, and the identifying step includes identifying at least one of the sample image files that has been misclassified by the AI ​​model; or 15. The method of embodiment 14, wherein the identifying step includes generating one or more photorealistic images at least in part based on one or more images of the person of interest.

[0214] (Embodiment 16) the external data further comprises a label attached to the image, and the protecting step comprises indicating whether the label is correctly attached to the image; or 16. The method of claim 14 or 15, wherein the protecting step includes indicating whether the faces in the image file are real or machine-generated.

[0215] (Embodiment 17) 17. A method as described in any one of claims 1 to 16, wherein the external data includes at least one image, and the AI ​​model includes an object detection model for classifying at least one object in the image.

[0216] (Embodiment 18) the identifying step is based on sample input data including one or more sample image files and sample output data including one or more objects and their locations within each of the sample image files, and the identifying step includes identifying at least one of the sample image files that is misclassified by the AI ​​model; or 18. The method of claim 17, wherein the identifying step includes generating manipulated images that are misclassified by the AI ​​model.

[0217] (Embodiment 19) 19. The method of claim 17 or 18, wherein the protecting step includes indicating whether the image has been manipulated to fool the AI ​​model by digital adversarial attacks, physical adversarial attacks, or a combination thereof.

[0218] (Embodiment 20) 1. A system for securely deploying artificial intelligence (AI) models, comprising: at least one hardware processor; a non-transitory computer-readable medium storing instructions; wherein the set of instructions, when executed by the at least one hardware processor, cause the at least one hardware processor to identifying one or more defects in the AI ​​model; protecting the AI ​​model from being attacked by external data based on the identifying step; A system that causes an operation including [Explanation of symbols]

[0219] 100 AI operating environment 200 Model Maintenance System 202 Intake Machine 206 Sorting Machine 216 Display Machine 217 Test Machine 220 Model Evaluation Engine 222 Simulated attack data 240 Firewall 260 Detection Engine 280 Disinfectant Machine 300 AI models 320 sample input data 340 sample output data 400 External Data

Claims

[Claim 1] 1. A method for securely deploying an artificial intelligence (AI) model, comprising: identifying one or more defects in the AI ​​model; protecting the AI ​​model from being attacked by external data based on the identifying step; A method comprising:

Images