DEVICE, SYSTEM, AND METHOD FOR PROTECTING MACHINE LEARNING, ARTIFICIAL INTELLIGENCE, AND DEEP LEARNING UNITS

Abstract

A system and method for protecting machine learning, artificial intelligence, and deep learning engines, the offline protection unit configured to analyze one or more characteristics of the protected engine and harden the protected engine offline against attacks by modifying operating characteristics or parameters of the protected engine to reduce vulnerability of the protected engine to attacks, the online protection unit configured to analyze at least one of (i) inputs directed as inputs to the protected engine and (ii) outputs generated by the protected engine and based on the analysis, dynamically harden the protected engine online against attacks by dynamically modifying operating characteristics or parameters of the protected engine to reduce vulnerability of the protected engine to attacks.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims priority to and the benefit of 63 / 176375, filed April 19, 2021, the entirety of which is incorporated herein by reference.

[0002] Field

[0002] Some embodiments relate to the field of electronic systems. [Background technology]

[0003]

[0003] Millions of people use electronic devices daily, such as desktop computers, laptop computers, smartphones, tablets, etc. Electronic devices are used to perform a variety of tasks, such as making purchases and transactions online, watching videos and consuming digital content, sending and receiving electronic mail (email) messages, engaging in online chats and instant messaging (IM), participating in audio and video conferences, playing games, etc.

[0004]

[0004] Some computer servers and applications can use machine learning (ML), deep learning (DL), artificial intelligence (AI), and other techniques to analyze data and derive analytics and insights. Summary of the Invention

[0005]

[0005] Some embodiments include devices and methods for protecting machine learning, artificial intelligence, and deep learning units or engines, and units or engines that perform reinforcement learning or use neural networks (NNs) or perform AI tasks.

[0006]

[0006] Systems and methods for protecting machine learning, artificial intelligence, and deep learning engines. The offline protection unit is configured to analyze one or more characteristics of the protected engine and harden the protected engine offline against attacks by modifying operating characteristics or parameters of the protected engine to reduce the vulnerability of the protected engine to attacks. The online protection unit is configured to analyze (i) at least one of inputs directed to an input of the protected engine and (ii) outputs produced by the protected engine and, based on the analysis, dynamically harden the protected engine online against attacks by dynamically modifying operating characteristics or parameters of the protected engine to reduce the vulnerability of the protected engine to attacks.

[0007]

[0007] Some embodiments may provide further and / or other benefits and / or advantages. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates a schematic block diagram of a system and its components according to some example embodiments. [Diagram 2] FIG. 2 illustrates a schematic block diagram of a system and its multi-layer architecture according to some exemplary embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009]

[0010] Applicant recognizes that various electronic and information processing systems employ machine learning (ML) units or engines, deep learning (DL) units or engines, and / or artificial intelligence (AI) units or engines to analyze or process data and generate output (e.g., results, predictions, estimates, quantifications, conclusions, insights, correlations, recommendations, actions (e.g., for robotic systems), etc.), or otherwise. Such ML / DL / AI units are sometimes used to accomplish a task when a completely deterministic and sufficient algorithm for the task is not available.

[0010]

[0011] Applicant recognizes that some ML / DL / AI systems may be subject to attacks, cyber-attacks, or compromise attempts by adversaries, competitors, hackers, or third parties. Some embodiments may provide devices and methods for protecting ML / DL / AI systems from such attacks, or for mitigating or curing such attacks, or for remedying or reversing or reducing or isolating damage caused (or attempted to cause) by such attacks.

[0011]

[0012] For example, applicants recognize that some ML / DL / AI systems may be subject to evasion attacks or adversarial example attacks. In such attacks, for example, data fed to an ML / DL / AI system may include "adversarial examples," such as (I) images of dogs labeled as cats, or (II) images of cats that have been manipulated by adding "noise" or subtle details that make them appear to the human eye as images of cats but that mislead or confuse the classification unit. Applicants recognize that in some situations, even minor alterations of only a few pixels in a digital image may cause confusion or misclassification in an ML / DL / AI system. Applicant recognizes that in such an evasion attack, the attacker does not attempt to infiltrate the ML / DL / AI system or change the way it operates; rather, an evasion attack injects, supplies or introduces into the ML / DL / AI system erroneous data, or misclassified data, or data that confuses the machine, or data augmented with digital "noise" or addenda, or data that is completely erroneous, or partially erroneous, or partially substituted, or otherwise altered or manipulated, that ultimately causes the ML / DL / AI system to produce outputs that are different from those expected by the designers of the ML / DL / AI system.Additionally, such evasion attacks are not limited to image analysis or computer vision systems, but may also be used in speech recognition systems, text analysis systems, video analysis / recognition systems, image analysis / recognition systems, audio analysis / recognition systems, tabular data analysis systems, LIDAR analysis systems, signal analysis systems, time series analysis systems, malware detection systems, financial and fintech systems (e.g., electronic title issuing systems, electronic mortgage issuing systems, electronic credit card issuing systems, electronic bank account opening systems, electronic brokerage or securities account opening systems), ML / AI systems that use (or apply processing to) real-time or near real-time data or data items or data streams. Applicant recognizes that similar may be done with respect to DL / AI systems (e.g., live or near live or real-time or near real-time video streams, or video frame streams, or stock price streams, or stock trading streams, automated trading systems, and autonomous vehicle driving systems, advanced driver assistance systems (ADAS), automated industrial systems that monitor or sense and analyze data), "black box" ML / DL / AI systems (e.g., hardware components in a vehicle that perform ML / DL / AI tasks therein based on input signals and generate output signals), and / or other systems.

[0012]

[0013] Applicants also recognize that some ML / DL / AI systems may be subject to theft attacks, including model theft attacks and / or dataset theft attacks. For example, an adversary or competitor may perform model extraction, model reconstruction, or model imitation based on the output generated by the ML / DL / AI system, may leverage or utilize the output from a competitor's ML / DL / AI system (e.g., trained on a proprietary or non-public dataset) to leverage or improve the performance, efficiency, or accuracy of an adversary's ML / DL / AI system (trained on a public dataset), and may generate inferences about the data used by the ML / DL / AI system (e.g., including extraction of personal or private data) based on queries sent to the ML / DL / AI system and analysis of the output generated by the ML / DL / AI system.

[0013]

[0014] Additionally, applicants recognize that some ML / DL / AI systems may be subject to reprogramming attacks, for example, where an attacker sends queries to and receives output from an existing ML / DL / AI system, leveraging or exploiting the attacker's ability to accomplish new tasks that the owner of the ML / DL / AI system did not intend to accomplish (or facilitate). For example, an object detection ML / DL / AI system may be used or exploited by an attacker to actually perform optical character recognition (OCR) of text, an image recognition ML / DL / AI system may be used or exploited by an attacker to actually generate medical diagnoses from images of people, and the like.

[0014]

[0015] Additionally, applicants recognize that some ML / DL / AI systems may be subject to poisoning attacks. For example, an attacker may perform a corruption poisoning by injecting a small or relatively small number of carefully selected malicious samples into a training data set, which may significantly impair or corrupt the operational accuracy of the ML / DL / AI system. In another example, an attacker may perform a backdoor poisoning by injecting a small or relatively small number of carefully selected malicious samples containing secret codes or indicia into a training data set, thereby causing the ML / DL / AI system to generally operate as intended when normal inputs are presented, and to classify certain types of inputs (containing similar secret codes or indicia) according to the attacker's intent when such inputs are provided to the ML / DL / AI system. Applicant recognizes that a poisoning attack may attempt to modify the behavior of an ML / DL / AI system by injecting or introducing one or more malicious training data items into a training dataset (e.g., injecting a relatively small number of malicious data items into the training dataset compared to the overall large size of the training dataset) so that the ML / DL / AI system behaves in a particular way desired by the attacker, either generally or in response to a particular subsequent query or type of query. In some circumstances, such poisoning attacks may be conducted in real-time or near real-time, as a one-off event or a periodic set of events, and may be directed directly at the AI ​​engine or AI system and / or indirectly at one or more units that collect data used by the AI ​​engine and that periodically update or append data, including (for example) a recommender system or other AI-based system.

[0015]

[0016] Applicants recognize that there is a need to protect ML / DL / AI systems from such attacks, as well as other types of attacks and attempts to compromise them.

[0016]

[0017] Referring to Figure 1, Figure 1 illustrates a schematic block diagram of a system 100 according to some example embodiments. The system 100 may include an ML / DL / AI engine 101, which is a unit (or subsystem, or system, or computer, or server, or set of computerized modules) that is protected against attacks, exploits, or malicious activity. The ML / DL / AI engine 101 may also be referred to herein as a "protected ML / DL / AI engine 101" or a "protected or evaluated ML / DL / AI engine 101."

[0017]

[0018] The ML / DL / AI engine 101 may include or use, for example, a training database 102, which may be a large dataset of samples or labelled samples or already classified samples, a model builder unit 103 for building and / or updating a model 104, a classifier unit 105 (or feature-izer or featurer or labeler unit), and additional units or components, such as, for example, an evaluator unit, a performance monitoring unit, a ground truth collector unit, an orchestrator unit for controlling and commanding the other units, a model server optionally associated with an application program interface (API) to enable efficient input reception and output generation, a front-end unit for providing metrics and generating text classification on numerical results, etc.

[0018]

[0019] According to some embodiments, the ML / DL / AI protection unit 120 can protect the ML / DL / AI engine 101 from or against attempts to compromise its operation or cyber-attacks, and from one or more specific attacks against such ML / DL / AI systems. For example, the ML / DL / AI protection unit 120, or one or more of its components or units or subsystems (e.g., the online protection unit 123 and / or other components in the ML / DL / AI protection unit 120) can be implemented with a modular architecture that can enable scalability and provide a cloud server for multiple users, can be installed locally as licensed software components, can be installed locally or on a network as a hybrid hardware and software dedicated unit, and can operate as a "firewall device for the ML / DL / AI engine", which , for example a dedicated local or remote or cloud based machine or unit operating as a locally installed "Firewall Device For ML / DL / AI Engine", or a co-located "Firewall Device For ML / DL / AI Engine", or a server based "Firewall Device For ML / DL / AI Engine", or a cloud computing based "Firewall Device For ML / DL / AI Engine", or as a software unit that may be purchased or licensed, or as a hardware / software hybrid unit, or with any other suitable arrangement or configuration scheme.Further, it should be noted that the implementation schemes and / or architectures described above may be used in some embodiments to implement the entire ML / DL / AI protection unit 120, or specific components of the ML / DL / AI protection unit 120, or specific subsets or groups of components of the ML / DL / AI protection unit 120.

[0019]

[0020] The ML / DL / AI protection unit 120 may include, for example, a penetration testing unit 121 configured to automatically or semi-automatically test or evaluate the robustness of the ML / DL / AI engine 101 against various threats, an offline protection unit 122 that operates to modify the structure and / or operation of the ML / DL / AI engine 101 to enhance the security and resilience of the ML / DL / AI engine 101, and an online protection unit 123 that can perform one or more defensive services to protect the ML / DL / AI engine 101 in real time or near real time.

[0020]

[0021] In some embodiments, the offline protection unit 122 can implement offline defense techniques to protect the ML / DL / AI engine 101. Applicant recognizes that some ML / DL / AI engines may assume an offline configuration, where the ML / DL / AI pipeline is hardened or secured during its development stage, and the resulting secured pipeline is then deployed and made available for receiving and processing incoming queries or inputs. In contrast to this approach, Applicant recognizes that online hardening techniques act to apply defenses to the ML / DL / AI engine 101 online, by external components or units that have an interface to the AI ​​pipeline in the deployment, or that are otherwise operatively associated with the ML / DL / AI engine 101, or that are configured to act as a gateway or filter or monitoring unit for incoming queries.

[0021]

[0022] Further, applicant recognizes that the online defense techniques according to some embodiments may provide various advantages, such as, for example, allowing dynamic and / or modular application of defense techniques, allowing easy and / or fast integration of defense techniques, allowing protection of "black box" pipelines, and / or allowing better separation between development of security measures and development of ML / DL / AI functions, and / or by providing an online or cloud-based or remote protection system or layer (e.g., a remote or cloud-based detector unit) that is non-intrusive and does not modify (and / or interfere with the normal operation of) the original ML / DL / AI models that a user (i.e., an owner or operator or administrator or developer of the ML / DL / AI system) spends efforts and resources to build and fine-tune.

[0022]

[0023] Applicant also recognizes that "offline protection," such as via offline protection unit 122, may provide additional and / or different benefits or advantages. According to some embodiments, "offline protection" of the MLDL / AI engine 101 may include, for example, receiving or analyzing or accessing the MLDL / AI engine 101 prior to deployment and performing one or more operations and / or modifications and / or additions to upgrade or secure or harden the ML / DL / AI engine 101 against attacks, and generating or returning an updated or upgraded or augmented version of the ML / DL / AI engine 101 that is more secure or enhanced or more resilient than the original version of the ML / DL / AI engine 101, which may then be used and deployed.

[0023]

[0024] In some embodiments, the offline protection unit 122 can perform (or can be used to perform) one or more of several protection and hardening techniques, including: (A) dataset hardening, dataset “hardening,” dataset augmentation, preprocessing operations, data sanitization operations, data pruning operations; (B) modifying the structure and / or architecture of neural networks (or other AI / ML / DL algorithms), modifying the structure and / or architecture of AI / ML / DL pipelines, replacing uncertain elements (e.g., deep neural networks (DNNs) or other components) with secure or more secure versions thereof; (C) modifying system parameters, e.g., based on retraining, adversarial training, threshold modification, modifying or introducing threshold values, or modifying or introducing operating conditions; (D) architectural augmentation, e.g., by creating an ensemble of models, adding or modifying pre-processing elements or components, or adding or modifying post-processing elements or components; (E) adding attack detectors to the AI / ML / DL pipeline that can generate a notification or raise a flag or throw an exception when an input (or a series or set of inputs) is identified as malicious or suspected or determined to be malicious; or (F) other appropriate techniques.In some embodiments, the protection system or platform may actively add attack detector(s) to the pipeline of which the protected engine is a part, and / or may build such a pipeline by adding attack detector(s) to a standalone protected engine, and / or by adding or invoking online and / or offline attack detectors, and / or by adding or invoking stateless or stateless attack detectors (e.g., they analyze or classify each input signal by itself and ignore previous and / or subsequent received input signals), and / or by adding or invoking stateful or state-maintaining attack detectors (e.g., they analyze or classify each input signal not by itself but in conjunction with or in combination with one or more previous and / or subsequent received input signals, thereby making predictions or analyses across a set or sequence of multiple inputs).

[0024]

[0025] In some embodiments, at least some of the offline protection actions are performed automatically or semi-automatically by the offline protection unit 122. For example, the offline protection unit 122 may automatically analyze the structure and / or operation of the ML / DL / AI engine 101, may automatically detect that the ML / DL / AI engine 101 is missing a particular type of pre-processing or post-processing component, and may automatically or semi-automatically (e.g., subject to approval of a human administrator) augment the ML / DL / AI engine 101 with the missing component(s). Similarly, the offline protection unit 122 may automatically analyze the structure and / or operation of the ML / DL / AI engine 101, may automatically detect that the ML / DL / AI engine 101 is missing a particular filter mechanism (e.g., a rate-limiting component or filter that limits the number or frequency of incoming queries from a particular sender), and may add or install such a component or filter. In some embodiments, the protective action(s) and / or mitigation action(s) may be automatically selected by the protection platform based on a set of rules for selecting such actions, and / or based on an estimated or quantified severity of an estimated problem or estimated attack, and / or based on resources and protection technologies relevant and available for protecting this particular ML / DL / AI engine of this particular system.

[0025]

[0026] For example, the protection platform or its offline protection unit 122 may issue 500 rapid queries within one second to the protected ML / DL / AI engine and observe that the ML / DL / AI engine strives to generate output in a timely manner (e.g., on average within N milliseconds of each input signal) and may then infer that the ML / DL / AI engine is likely to lack or not apply a rate-limiting pre-processing filter or gate or mechanism, or that such a rate-limiting module may be present but is improperly configured, and accordingly the protection platform may automatically recommend adding such a pre-processing unit or module, and in some implementations may automatically add such a pre-processing module or unit automatically or semi-automatically.

[0026]

[0027] Similarly, the protection platform may detect when output signals generated by an ML / DL / AI engine suffer from anomalies or inconsistencies that may be due to missing (or misconfigured) post-processing units or modules, such as pruning routines, rule filtering schemes, knowledge integration schemes, modules for reducing or pruning or truncating decision trees or decision rules or association rules or classification rules, and / or other post-processing filters that attempt to remove or filter out “noisy” or inaccurate outputs.

[0027]

[0028] Similarly, the offline protection unit 122 can automatically detect when the ML / DL / AI engine is using certain conditions or threshold values ​​that make it more susceptible or vulnerable to attacks, and can automatically increase or decrease or change such threshold values, and can automatically change or cancel or augment conditions when a certain set of parameters or thresholds or conditions are detected, based on a list of predefined rules for making such changes. Other suitable techniques can also be used.

[0028]

[0029] The ML / DL / AI protection unit 120 may optionally further include an attack mitigation unit 124, which may be a separate unit from the above-mentioned units, may be integrated with, may operate in association with, may be included in, may include one or more of the above-mentioned units. For example, the attack mitigation unit 124 may collect or receive inputs or insights from one or more of the other units 121-123 and / or may fuse such inputs to generate further insights, and may trigger or initiate or perform one or more mitigation actions to reduce or eliminate or stop or degrade or stop a particular damage or abuse or fraudulent or adversarial use of the ML / DL / AI engine 101. Such mitigation actions may include, for example, blocking, quarantining or discarding some (or most or all) of the incoming traffic or queries or requests determined by the ML / DL / AI protection unit 120 to be from a malicious or hostile entity; blocking, quarantining or discarding some (or most or all) of the outgoing traffic or results or insights or output that were intended to be output towards an entity determined by the ML / DL / AI protection unit 120 to be from a malicious or hostile entity; significantly slowing down the rate or speed of processing requests or queries of such hostile entities and / or the rate or speed of sending results or output back to such hostile entities; modifying and / or manipulating the outputs and / or query results before sending them to such hostile entities, or (in some implementations) even intentionally sending false or erroneous or reversed or modified or random or pseudo-random results to such hostile entities;pruning or diluting or purging or cleaning the training set or training data set or sample data set used or employed by the ML / DL / AI engine 101, such as by removing or deleting or excluding or discarding one or more samples or data points or data items determined to be malicious, and optionally replacing or correcting or re-labeling or reclassifying one or more training samples or training data points by using a human classifier and / or by using another computerized service or another ML / DL / AI engine, reclassifying or re-labeling only certain such data points, and updating or pruning or purging or cleaning the training set or training data set or sample data set used or employed by the ML / DL / AI engine 101, such as by removing or deleting or excluding or discarding one or more samples or data points or data items determined to be malicious, and optionally reprogramming the training session of the ML / DL / AI model and / or reconstructing the ML / DL / AI model based on a diluted or sparse data set; and generating notification, warning or alert messages indicating misuse of the ML / DL / AI engine 101 and sending or transmitting such messages to one or more predefined parties (e.g., an administrative entity of the ML / DL / AI engine 101, law enforcement agencies) and / or to one or more dynamically defined parties (e.g., contacts of an entity determined to be a malicious or hostile entity, an Internet Service Provider (ISP) of such hostile entity, a domain registrar of such hostile entity, a server hosting facilities of such hostile entity). In some embodiments, in addition to or instead of dataset manipulation and / or reprogramming, other protection and mitigation methods can be applied, such as modifying the architecture (e.g., replacing and / or adding and / or removing and / or isolating elements or subunits of the protected ML / DL / AI engine 101 or modifying one or more operating parameters thereof), replacing a current version of the protected ML / DL / AI engine 101 with a previous or earlier version (e.g.,and modifying, updating, or replacing a defense or protection policy that allows a user (e.g., an administrator or owner of the ML / DL / AI engine 101) to configure and balance system requirements or operating parameters or constraints, such as by replacing a previous version of the protected ML / DL / AI engine 101 with a new one (obtained from a versioning unit or repository that stores a previous version of the protected ML / DL / AI engine 101), or by automatically switching from operating according to a first predefined protection policy to operating according to a second, different predefined protection policy (e.g., from among a pool, bank, or array of predefined protection policies).

[0029]

[0030] In some embodiments, the ML / DL / AI protection unit 120 can employ semi-automated and / or partially supervised processes for attack detection, attack mitigation, and protection. The ML / DL / AI protection unit 120 operates to achieve accurate classification of inputs provided to the ML / DL / AI engine 101 as either (I) legitimate inputs (e.g., non-adversarial, authentic, non-attack related) and (II) illegitimate or fraudulent inputs (e.g., adversarial inputs, attack related inputs, compromising inputs, malicious inputs, fake inputs, attack inputs, or exploitative or abusive inputs).

[0030]

[0031] In a demonstrative example, the ML / DL / AI engine 101 is configured to perform image classification of vehicle license plate numbers (or number sequences) found in a photograph or image. When an input (an image containing a license plate) is received at or just prior to the entry point of the ML / DL / AI engine 101, the ML / DL / AI protection unit 120 intercepts the incoming image using the input interceptor / analyzer unit 125 and applies an automatic attack detection algorithm to it, which is configured to operate rigorously to reduce false positive errors, such that if the output of the attack detection algorithm indicates that the image is not malicious, the image is not actually associated with an attack. Because the algorithm is rigorous, it does not have some percentage of false positives (e.g., determining that a particular photo is malicious or associated with an attack when in fact it is not). To remedy this, if an input is detected as being related to an attack, the input is then automatically sent to a computing terminal for manual human inspection by a human labeler or human classifier. If the human labeler's identification of the photo content is the same as the original identification by the ML / DL / AI protection unit 120, the detection of the attack is considered to be an error and such errors are ignored or discarded (and / or stored in an ancillary repository for further learning and to enable the ML / DL / AI protection unit 120 to learn from such errors and improve the system). In contrast, if the human labeler's identification of the photo content differs from the original identification made by the ML / DL / AI engine 101, this input (e.g., this photo) is considered to be malicious or adversarial or part of an attack against the ML / DL / AI engine 101 and is marked or reported as such, and the true or ground truth label set by the human labeler is used as the correct output.

[0031]

[0032] In some embodiments, optionally, instead of or in addition to using a human labeler (or human classifier), the ML / DL / AI engine 101, which is protected from attacks (or from malicious or adversarial inputs), may use a secondary (e.g., proprietary or publicly available) similar ML / DL / AI engine, or multiple such secondary ML / DL / AI engines, solely for the purpose of detecting such malicious input(s). For example, Company A develops and / or operates its own ML / DL / AI engine 101, which is specifically adapted to correctly classify detections in photos that are related to animals. Company A receives or acquires a particular photo, "Tiger.jpg", which has been pre-labeled or classified as a tiger. Rather than automatically relying on such pre-labeled or pre-made classifications, the ML / DL / AI protection unit 120 (which protects the ML / DL / AI engine 101) may send that particular photo to a third-party and / or public classification service that is not necessarily specialized in this domain (animals), such as Bing or Google's (or both) "reverse image search" engines, and such "secondary" engines may thereby indicate that the photo "Tiger.jpg" is in fact classified as a "sofa" or "couch" (e.g., it actually depicts a sofa with a tiger skin-like patterned fabric). Based on the classification by such a "secondary" engine, or a majority of such secondary engines, or at least one of such secondary engines, the protected ML / DL / AI engine 101 may determine that the photo "Tiger.jpg" should not be trusted as a tiger animal and / or should be discarded or deleted, or should be classified as a "sofa," or should otherwise be addressed or treated as a malicious or adversarial input.Such interrogation of the "secondary" ML / DL / AI engine(s) may be performed by an escort or filtering unit of the system, or by an adversarial input detection unit 126, which may escort the ML / DL / AI engine 101 against malicious inputs by using human labelers, a secondary ML / DL / AI engine 131, a set of secondary ML / DL / AI engines, and / or other methods, and may prevent such malicious inputs from entering or even reaching the ML / DL / AI engine itself. Some elements may optionally employ mechanisms for applying "wisdom of the crowd" techniques, in which each of a number of secondary ML / DL / AI engines (and optionally one or more human labelers) is queried about its output or results for a particular incoming query (coming to the protected ML / DL / AI engine 101), and the dominant result or majority result or most frequent result (or result in the output of a weighted formula assigning different weights to the various secondary ML / DL / AI engines based on their reputation or credibility or size or speed or other parameters) is prioritized and used as the "ground truth".

[0032]

[0033] In some embodiments, the evasion / poisoning attack mitigation unit 127 can be configured to mitigate or remedy evasion or poisoning attacks directed at the ML / DL / AI engine 101. This unit reflects a "backup" action plan that is triggered upon the detection of such an attack(s) and, in particular, upon the detection of an ongoing attack that cannot necessarily be immediately thwarted.The evasion / poisoning attack mitigation unit 127 has several functions and features, including: (A) a manual classification backup service is set up in a dedicated center and is ready to operate within a short period of time (e.g., 15 or 30 or 60 minutes, based on the request of the owner of the protected ML / DL / AI engine 101); (B) since detecting an attacking input is easier or more efficient or accurate than detecting a misclassified input (in some implementations), it detects attacks currently being performed against the ML / DL / AI engine 101 that result in a misclassification by using a threshold value or condition based on the percentage of inputs determined or estimated to be adversarial (e.g., inputs belonging to or associated with an attack) by the ML / DL / AI engine 101 for all or N percent of the input queries (e.g., N is a pre-defined threshold value, such as 50 or 33 or 25 or other values); (C) an attack is identified or suspected by one or more protection units, or by one or more protection mechanisms, or by other system metrics, system user reports, or sources (e.g., detection of an unusual or excessive number of inputs received in a minute or hour, a large number of inputs coming from a particular source or entity, or other indicators of a possible attack); (D) the attack mitigation unit is initiated or triggered to switch over to a manual (e.g., human) classification service that is quickly set up by a large number of human labelers familiar with the particular ML / DL / AI engine 101 task at hand, and the incoming query is sent or rerouted to the manual classification service rather than to the ML / DL / AI engine 101; and (E) once the attack is thwarted or terminated, the manual mitigation service is stopped and the query is rerouted back to the automated ML / DL / AI engine 101.

[0033]

[0034] In some embodiments, the protection platform (e.g., in an offline unit(s) and / or in an online unit(s) and / or in a local or on-premise implementation and / or via a cloud computing or remote unit implementation) can detect and protect against two or more repeated evasion attacks or groups of related evasion attacks, such that even if each evasion attack is not necessarily detected in itself, the aggregation of multiple events is detected as a distributed or multiple attacker, or as a coordinated repeated evasion attack by multiple attackers or multiple malicious devices, or by a single attacker or a single device performing repeated or repeating evasion attacks from the same device or from multiple devices (and which need not be identical to each other).

[0034]

[0035] In some embodiments, the system or platform may optionally operate by correlating or matching input signals or inputs originating from multiple distributed instances of ML / DL / AI models, e.g., particularly in a vehicular setting, and such correlation or matching may be used to detect attacks on such ML / DL / AI models or engines.

[0035]

[0036] Some embodiments may include and use an AI-based penetration testing (pen-testing) unit 128 that evaluates the vulnerability of the ML / DL / AI engine 101 to various types of attacks in various settings. The AI-based penetration testing unit 128 estimates the robustness or resilience level of the ML / DL / AI engine 101 to various attacks. The AI-based penetration testing unit 128 collects or acquires data about historical or past attacks, such as penetration attacks and penetration tests (and their corresponding results) performed on this particular ML / DL / AI engine 101 in particular, or on multiple ML / DL / AI engines of the same type as the ML / DL / AI engine 101 (e.g., historical attacks on an image classification engine), or on multiple ML / DL / AI engines in general, and builds a dataset representing (I) model features of the ML / DL / AI engines that were penetration tested, such as their type, size, architectural features (e.g., size and type of layers), etc., and (II) results of the penetration tests performed on such ML / DL / AI engines, such as results of the penetration tests (e.g., success or failure, level of success, percentage of success), accuracy of the ML / DL / AI engine that was penetration tested, robustness score or resilience score or vulnerability score against the specific penetration attack tested, etc. The AI-based pen testing unit 128 uses AI techniques, such as deep neural networks, to generate predictions regarding predicted, estimated or inferred pen testing results for the particular ML / DL / AI engine 101 being evaluated or protected, without actually performing a time-consuming and effort-intensive pen testing process.

[0036]

[0037] The system 100 may further include a policy design tool and a smart hardening process for protecting the ML / DL / AI engine 101. Applicant recognizes that hardening the ML / DL / AI engine 101 may be a complex, multi-objective, constrained optimization problem. The optimization includes, for example, finding or detecting the optimal subject of suitable and available defenses, and selecting or setting or determining the optimal or most efficient defense parameter values. The optimization is multi-objective because the defenses will optimize robustness against various attacks in various scenarios along with the accuracy of the natural model. The optimization is constrained because there are limits on the expected execution time and computational requirements of the output model, along with the duration of the hardening process and the memory requirements for the hardening. Other requirements from the output model may be that the new model does not change some attributes of the input model, or that the hardening process uses only "black box" access to the original model.

[0037]

[0038] According to some embodiments, the defense policy design tool or defense policy builder unit 140 may include a pre-fortification unit 141 or pre-fortification interface, which allows the owner or operator of the ML / DL / AI engine 101 to configure and define its requirements from the output model (or its operational constraints) or from the fortification process, such as the expected execution time, the expected computational cost, the total memory allocated to the output model, the fortification process access type to the original model (e.g., "black box" or "white box" or other), the duration of the fortification process, the price / computational cost / resource cost of the fortification process, the type of changes to the original model that are allowed (e.g., allowing or not allowing retraining, only allowing the addition of preprocessing, not changing the original model but adding an attack detection layer), etc. Optionally, the smart fortification unit 143 executes an algorithm to find a Pareto optimal solution of the constrained optimization problem. In some embodiments, the evaluation of each model proposal may optionally include at least one attempt at penetration testing. The selection process can use one or more of several techniques or considerations, including using a multi-objective optimization algorithm, using a budget-constrained search algorithm for a time-limited fortification process, using AI-based penetration testing for fast penetration testing approximation, using a covering algorithm to find the optimal defense subset that covers all relevant attacks, using a grid search algorithm, and using an evolutionary / genetic algorithm. Optionally, the post-fortification unit 142 or post-fortification interface can allow browsing and inspecting the results of various fortifications in order to select the particular fortification solution that best meets the requirements and constraints, optionally including an efficient mechanism for searching for Pareto optimal solutions.

[0038]

[0039] Optionally, the AI-based model enrichment unit 144 can perform AI-based prediction of the most appropriate or efficient enrichment policy and can determine an efficient or appropriate subset of available and appropriate defenses and their corresponding efficient defense parameter values. For example, the AI-based model enrichment unit 144 can collect historical or past data about the model, customer requirements and constraints, and their corresponding enrichment results. The AI-based model enrichment unit 144 then builds a data set of (I) model features such as type, size, architectural features (e.g., layer size and type), (II) requirements or constraints set by the owner or operator of the particular ML / DL / AI engine 101 being evaluated or protected, and (III) the enrichment method results and their corresponding enrichment parameter values. Using suitable AI techniques, such as deep neural networks, random forests, etc., a subset of suitable or highly efficient enhancement techniques and their respective parameters can be predicted or determined to provide the optimal or most efficient enhancement for the particular ML / DL / AI engine 101 being evaluated or protected, while the specified constraints are maintained and the specified requirements are achieved, and without actually performing multiple iterations of the time-consuming and computationally resource-intensive enhancement process on the particular ML / DL / AI engine 101.

[0039]

[0040] Some embodiments may include a region of interest (RoI) based protection unit 150 to provide protection and / or hardening for the particular ML / DL / AI engine 101 being evaluated or protected. For example, some traditional (non-AI based) algorithms for image processing tasks may be robust against adversarial attacks but have low accuracy. For example, the task of detecting what the regions of interest (RoI) are in a given image is a task that has been studied and for which there are various non-AI algorithms to address. Some embodiments may include defense techniques that combine both traditional algorithms (e.g., RoI detection algorithms) and AI based protection units to achieve both robustness and accuracy in protecting the ML / DL / AI engine 101.

[0040]

[0041] Thus, a gradient / RoI adversarial input detector unit 151 can be used. When classifying images using an ML model, particularly using a deep neural network (DNN), the system can perform a derivation of the model's gradient at the input, e.g., the gradient is a matrix that has the same shape as the input image. The values ​​in the gradient are (or indicate or correspond to) the level of importance of pixels in the classification process of a particular input image. For example, if the DNN correctly classifies an image of a cat, the gradient indicates that the important pixels in the classification process were the cat's nose, ears, and eyes.

[0041]

[0042] Thus, an input image can be determined to be an adversarial image by (a) calculating the gradient of the input image according to an ML model, (b) generating a region of interest (RoI) of the image using a traditional or deterministic or non-ML algorithm, and (c) comparing the gradient and the RoI, and if they match or coincide with each other, accept the image as a legitimate or non-adversarial input image, and if they do not match or contradict each other, reject or flag the input image as an adversarial input image. The comparison or matching process can be performed, for example, using a trained ML algorithm that is dedicated for this specific comparison task and / or using a deterministic algorithm that demonstrates that a predefined threshold value of important pixels (e.g., at least 70 percent, or at least N percent) are indeed within the RoI.

[0042]

[0043] In some embodiments, the RoI preprocessing unit 152 may be used to perform preprocessing operations on a given input image for the above tasks, such as by (a) detecting or determining the RoI, and (b) replacing all pixels of that input image that are outside the RoI with one of the following: (i) a constant value (e.g., 0); (ii) a value corresponding to a smoothing or strong smoothing of the original image pixels; (iii) a random or pseudo-random value; or (iv) a combination of the above values.

[0043]

[0044] Some embodiments may include an adaptive hardening unit 155 or a hybrid online / offline protection unit 156 that may operate to further protect the ML / DL / AI engine 101 from attacks or adversarial or malicious inputs. Model hardening is a multi-objective and constrained optimization task. The task involves finding a model that is currently optimal in terms of accuracy and parameters, and in terms of its robustness against various threats, but subject to user-requested or other constraints on predicted execution time, predicted CPU usage, and memory. In addition to solving optimization problems in a static setting, it is beneficial to solve optimization problems in a dynamic setting. For example, CPU / processing costs, memory usage costs, database usage costs, and / or other costs of resource usage may change dynamically over time, as may queries loaded to the ML / DL / AI engine 101 over time, as well as the probabilities of various threats over time (e.g., on a particular day, ML / DL / AI engines in one region or around the world may be known to be under attack from a particular adversary or due to a particular vulnerability or malware currently being spread and operational). Thus, the system employs algorithms to dynamically adjust or modify or reconfigure the model itself, i.e., the ML / DL / AI engine 101 itself and its internal model, in a dynamic or online manner (e.g., once a day, twice a week, or even once an hour, or at other time intervals) according to changing demands and changing values ​​of the above-mentioned parameters. For example, the ML / DL / AI engine 101 enhancement process can be scheduled to run every H hours or every D days, and / or can use AI-based algorithms and online optimization algorithms to provide further response to a dynamic environment where threats change and / or parameter values ​​change.

[0044]

[0045] Some embodiments may employ an online defense and protection unit 157 capable of performing real-time or near real-time ML / DL / AI engine 101 protection operations. Such services may be web-based and may include locally installed software and / or hardware components, optionally supporting application programming interfaces (APIs) for improved configuration and use. Such operations may include, for example, (A) online input processing, in which the protected ML / DL / AI engine 101 first sends an input item (e.g., an input image) to the online defense and protection unit 157 and then receives from that unit a “sanitized” or “modified” input (e.g., by applying one or more modifications to the input that may cancel or mitigate or reduce the negative effects of an adversarial operation performed on it) or approved input (e.g., the input image is returned with an indication that it is legitimate or non-adversarial or non-malicious), or receives an indication that the input item is malicious or adversarial and should be discarded or not processed by the protected ML / DL / AI engine 101, and the input received from the online defense and protection unit 157 is then passed to the protected ML / DL / AI engine 101 for further processing (if approved). In a demonstrative example, the protected ML / DL / AI engine 101 operates to classify human images as male or female, and the online defense and protection unit 157 adds a pre-processing filter or analysis or check as to whether the input image is (fully) human or non-human (e.g., dog, chair), thereby protecting the protected ML / DL / AI engine from such input signals that may be associated with attacks and / or may cause the ML / DL / AI engine to malfunction or output erroneous results.(B) Online Adversarial Input Identification, in which the protected ML / DL / AI engine 101 sends an input item (e.g., an input image) to the online defense and protection unit 157 and receives a classification therefrom indicating whether the input item is adversarial or non-adversarial; and (C) Online Attack Pattern Detection, in which the protected ML / DL / AI engine 101 sends an input item (e.g., an input image) to the online defense and protection unit 157 and receives therefrom a classification indicating whether the input item is adversarial or non-adversarial. Therein, the protected ML / DL / AI engine 101 (or a monitoring component associated therewith) sends all or substantially all or most of some of the inputs received by the protected ML / DL / AI engine 101 to an online defense and protection unit 157, which then analyzes the sequence of inputs to identify an attack pattern (e.g., identifying a sequence of images of cats that have been pre-labeled as dogs), and upon detection of an attack pattern (e.g., by detecting an attack pattern that is a sequence of inputs (or images) that are very similar to each other but differ only in a few bits or bytes (or pixels), or a sequence of images that appear to be the result of a search / optimization process), the online defense and protection unit 157 can generate a trigger to notify an owner or operator of the protected ML / DL / AI engine 101 that such an attack has been detected and / or to block or quarantine or discard or avoid processing certain queries or inputs from particular sources or with common characteristics. (D) online monitoring of attacks and generating associated models(s) or graphs or charts representing the detected attacks and Key Performance Indicators (KIPs) associated with ML security;(E) Online Time Delay: in which the protected ML / DL / AI engine 101 first sends the input item along with an identifier of the source of the input to the online defense and protection unit 157, which injects or intentionally adds a time delay (e.g., T seconds or T minutes, where T is a predetermined number, or T is selected randomly or pseudo-randomly within a predetermined range of values) before returning the input to the protected ML / DL / AI engine 101 for processing, or T is selected based on penetration testing of the ML / DL / AI engine 101 or based on output generated by the smart hardening unit 143, the time delay may be a function of the input source and / or the input item itself, and the time delay may result in preventing or weakening or slowing down or eliminating certain attacks, such as theft attacks that require multiple queries to the protected ML / DL / AI engine 101. (F) Query quota or rate limiting: in which the protected ML / DL / AI engine 101 first sends an input item along with an identifier of the source of the input to the online defense and protection unit 157, which then decides whether to block the particular source according to a query quota associated with the particular source, which can be predefined or dynamically assigned and / or configured based on one or more parameters or conditions, such as considering the level of similarity among multiple inputs / multiple queries arriving from the same source within the past T minutes, and / or other parameters (e.g., outputs or values ​​generated by the smart reinforcement unit 143). (G) Blacklist process: in which the protected ML / DL / AI engine 101 first sends an input along with an identifier of the source of the input to the online defense and protection unit 157, which then decides whether to block the source according to a blacklist of sources and / or a blacklist of specific inputs or queries.

[0045]

[0046] In some embodiments, in this manner, the online defense and protection unit 157 may enable protection that does not require modification of the original ML model, may enable efficient integration and use of complex protection techniques with minimal or no modification of the protected ML / DL / AI engine 101 itself, may enable behavioral defense methods that detect attack patterns across (or from) a set of inputs or a set of queries, may enable the application and use of confidentiality defense methods provided by a remote server of the online defense and protection unit 157 as a “black box” solution that provides an output (e.g., “the input is legitimate” or “the input is adversarial”) to the protected ML / DL / AI engine 101 without exposing the method(s) used by such remote “black box” unit to reach such a decision, and may enable a separation between the ML / DL / AI engine 101 (and its original model) and the units that provide the protection layer to it. In some embodiments, the online defenses provided by the system may enable each of the two parties to keep their confidential information secret; the protected ML / DL / AI engine 101 itself and its internal methods of operation may be kept undisclosed while being protected from attack, and / or the protection service (which protects the ML / DL / AI engine 101) may be provided as a web-based or remote or locally installed or co-located or cloud-based "black box" protection service that does not disclose or expose the internal methods of operation.

[0046]

[0047] Some embodiments may further employ an online AI-based defense unit 158, which may employ one or more (remote or cloud-based) AI / ML / DL engines (e.g., using DNNs, random forests, evolutionary / genetic algorithms, or other techniques) to make selections as to which protection schemes or protection actions to apply (or not apply) with respect to a particular input (or set of inputs) coming into and / or to the protected ML / DL / AI engine 101, as well as which pre-processing or pre-classification actions to apply or perform on such input(s), which re-trained model(s) to apply, which detection algorithms to apply, and / or other actions that may be selectively determined to be applicable or appropriate, or which are selected from a pool of pre-defined attack mitigation actions based on such AI / ML / DL engine outputs of the online AI-based defense unit 158. In some embodiments, the functionality of the online AI-based defense unit 158 ​​can assist in achieving dynamic enhancements that enable the system to handle dynamic demands. For example, a database or dataset can be created with features of various types of ML / DL / AI engines, with features of their specific datasets and models, with features of known attacks detected against them, with mitigation / protection actions associated with results (success / failure, or a score or percentage value of success) for each type of engine and dataset and attack, and specific mitigation / protection actions with results against specific attacks against the specific engine or dataset or engine / dataset combination.Such data is subjected to ML / DL / AI analysis or classification in order to estimate or predict or deduce which attack vectors may be relevant to a particular (e.g., new) protected ML / DL / AI engine and / or which mitigation or protection techniques would be useful or effective against such attack(s) on such protected ML / DL / AI engine. In some embodiments, a data integrity analysis unit may operate as part of the system or platform to estimate and / or quantify the level of integrity of a dataset currently acted upon by an ML / DL / AI engine and / or by its models and / or used to train such engine and / or used to build such models, and the system may invoke one or more mitigation or retraining or adversarial training or dataset purifying or replacing or augmenting operations in response to a data integrity score below a predefined threshold value.

[0047]

[0048] The online AI-based defense unit 158 ​​may be responsible for selecting the appropriate (or optimal) online defense for each particular input that is coming (or is about to come) towards (or into) the protected ML / DL / AI engine 101. The online AI-based defense unit 158 ​​may be implemented (or operated) based on a history of dynamic reinforcement and / or based on the output of the smart reinforcement unit 143 and / or based on the output of the AI-based model reinforcement unit 144, which combines constantly improving model(s) based on experience or insights from the reinforcement history.

[0048]

[0049] The system may further include an online defense monitoring unit 159, which generates on-screen displays or notifications or dashboards that enable a user (e.g., the owner or operator of the protected ML / DL / AI engine 101) to monitor attacks that have been or are being attempted against the user's ML / DL / AI engine 101, as well as protection mechanisms that have been or have been applied and their results. The online defense monitoring unit 159 may enable a user to, for example, obtain graphs or charts or lists or other representations of the history of attacks detected and / or blocked and / or avoided, and the success rate of such attacks and / or mitigation actions applied; manually label certain usage statistics and / or configure certain parameters and constraints used by the system; obtain data on CPU / memory usage and execution time usage for each online defense mechanism; obtain defense effectiveness statistics and defense success data, such as the percentage of inputs (out of all inputs) identified as hostile and the percentage of adversarial inputs that were blocked in time or were subject to pre-processing actions or quarantine or discard; obtain information or generated insights about the penetration testing history of the protected ML / DL / AI engine 101 (e.g., especially when protection attributes or protection-related features are dynamically changed); and / or other related parameters.

[0049]

[0050] The system may further include an attack investigation console unit 160 that enables an attack investigator and / or an operator of the ML / DL / AI engine 101 to perform investigative actions regarding one or more attacks launched against the ML / DL / AI engine 101, thereby providing or generating further significant information or insight regarding any necessary or appropriate response or mitigation actions. For example, the attack investigation console unit 160 may generate and provide data about the distribution of adversarial labels for evasion attacks, in which an attacker attempts to trick a model into misclassifying some inputs, indicating the distribution of classifications the attacker was trying to achieve (e.g., indicating that the attacker attempted to attack the classification of images of animals but not the classification of images of vehicles), and / or the distribution of the type of attack (e.g., model theft, dataset information theft, poisoning exploitation, evasion, or other attack), and / or data indicating whether the attack was a “black box” attack or a “white box” attack, which, when detected, may cause the system to generate a warning that the model itself is presumed to have been stolen, and / or data indicating whether the attack is a known attack (e.g., an attack using one or more open source tools or previously seen attack tools) or an unknown attack (e.g., an attack using a previously unseen attack tool or attack vector or attack surface or attack technique), which may then also generate an indication as to the level of threat or risk associated with the attack, and / or other insights.The attack investigation console unit 160 may further be capable of efficiently routing or rerouting suspicious inputs for manual re-labeling or reclassification, and / or generating insight into the true or overall scope or extent of an attack, and / or initiating sandboxing or quarantine actions with respect to specific components and / or data points and / or datasets(one or more) and / or engine(s), and / or enabling visualization and / or initiating sandboxing actions with respect to one or more features or functions, and / or providing a visual or graphical representation of two or more protection or hardening technologies operating together or in parallel or in sequence. It may be possible to generate a visual or on-screen representation of the results of the penetration tests, and / or generate a dashboard that represents or shows currently deployed and / or currently available and / or currently unavailable protection and / or hardening techniques, and / or generate a visual or on-screen representation that represents the hardening or protection of a particular ML / DL / AI unit(s) within the composite ML / DL / AI system or engine, and / or take or trigger other mitigating actions (e.g., allowing an investigator to manually quarantine a set of inputs or inputs received from a particular source or that share particular characteristics).

[0050]

[0051] Applicant recognizes that in some situations, an owner or operator of an ML / DL / AI engine 101 may wish to determine the robustness of the ML / DL / AI engine 101 without sharing information about the ML / DL / AI engine 101 or its internal model(s) or component(s) or logic. However, Applicant recognizes that some types of attacks may be “white-box” type attacks, in which case certain or all knowledge of the ML / DL / AI engine 101 being attacked is already in the hands of the attacker. Applicant recognizes that in some situations, a “white-box” attack may require the attacker to know only the results of a particular calculation (e.g., the gradient of the model for a particular input) in order to perform the attack. Thus, the undisclosed model penetration testing unit 162 may perform “white-box” penetration testing of the ML / DL / AI engine 101 while at the same time maintaining the confidentiality of the models or components of the protected ML / DL / AI engine 101. For example, a server or computer hosting or operatively associated with the ML / DL / AI engine 101 may be configured by installing a test module thereon (e.g., optionally implemented as an open source module or as a module having source code that can be inspected by the owner of the ML / DL / AI engine 101 or a trusted auditor), where the installed test module performs only the calculations required for a "white-box" attack (e.g., calculating the gradient of the model with respect to specific inputs). The system performs penetration testing, which performs the "white-box" attack by sending a request to the test module to perform a calculation, and then sending the result(s) back to a non-disclosed model penetration testing unit 162 (e.g., implemented as a remote penetration testing server) for further evaluation or analysis.

[0051]

[0052] Furthermore, the system can provide protection against attacks even in the absence of information about the ML / DL / AI engine 101 or its models, by selectively activating only remote or cloud-based non-intrusive protection units(s), such as units that detect malicious or suspicious inputs or queries independent of the operating manner of the ML / DL / AI engine 101. Optionally, the system can perform a search for available protection options and, for each option, evaluate the robustness of the protected ML / DL / AI engine 101 using a closed model penetration testing unit 162, or using techniques that test for robustness against traditional "black box" attacks, or using other techniques.

[0052]

[0053] Some embodiments may use one or more penetration testing units or modules, such as one or more of units 121, 128, 162 described above, to evaluate the robustness of the ML / DL / AI engine 101. The evaluation may be performed using experimental or deterministic methods, such as by performing specific attacks on the ML / DL / AI engine 101 and analyzing the results of the attacks, and / or other methods, such as by using mathematical or computational analysis of the structure of the ML / DL / AI engine 101 if the structure is partially or completely known to the penetration testing unit. The penetration testing unit may be a modular tool that may be efficiently updated to allow new attacks and new theoretical or computational evaluation methods to be tested.

[0053]

[0054] Some embodiments may optionally use or provide intrusion defenses for the ML / DL / AI engine 101, which may be provided using, for example, secure pre-trained models, adversarial training, architectural modifications, ensemble training, model retraining, dataset pre-processing, dataset augmentation, dataset sanitation or sanitization, and / or other techniques.

[0054]

[0055] For example, the adversarial training unit 163 may operate to increase the robustness of the protected ML / DL / AI engine 101 by applying adversarial training defenses against one or more known attacks, and optionally by using components or modules installed locally on or at the server as components running or operatively associated with the ML / DL / AI engine 101. Optionally, pre-processing based adversarial training and / or post-processing based adversarial training may be used, including adversarial training on the ML / DL / AI engine 101 considered as a non-disclosed model. Optionally, some embodiments may use adversarial training of pre-processing elements added to the ML / DL / AI engine 101 and / or adversarial training of post-processing elements added to the ML / DL / AI engine 101, including adversarial training on the ML / DL / AI engine 101 considered as a non-disclosed model.

[0055]

[0056] In some embodiments, adversarial training may be applied in other manners, for example using Neural Architecture Search (NAS) techniques, where a neural architecture or model used by the protected ML / DL / AI engine, or one or more features or characteristics of the neural architecture or model, is part of a hyper-parameter search space, and the NAS unit predicts or estimates an optimal architecture or model for a given task, and optionally, the preferred or optimal model or architecture is then trained using an adversarial training protocol to achieve hardening or robustness against adversarial attacks.

[0056]

[0057] Optionally, one or more non-intrusive defense units (1 or more) 164 may be used in the system, for example to perform pre-processing and / or post-processing, classification of evasive inputs, evasive classification defenses, and / or other non-intrusive defenses that do not require partial or complete retraining of the original model. The incorporation of such defenses may be done in a modular manner, allowing new defenses to be efficiently added to the system as they are developed.

[0057]

[0058] In some embodiments, the poisoning detection and repair unit 165 can be operated to detect and repair models and / or datasets that have been subject to a poisoning attack. This unit can operate under one or more user-defined constraints, such as the constraint that retraining is now allowed, or undisclosed datasets, or undisclosed models. This unit can be built in a modular way, allowing new detection methods and / or new repair methods to be efficiently added or used.

[0058]

[0059] Some embodiments may provide real-time defenses as real-time services to the ML / DL / AI engine 101 running on healthy (e.g., “production”) servers and environments. Such defenses may include, for example, real-time pre-processing, real-time post-processing, real-time adversarial input classifiers, real-time pre-processing of ensembles, real-time backdoor poisoning mitigation, real-time backdoor poisoning exploitation detection, and / or other services that may be dynamically provided as cloud computing based services or as remote services.

[0059]

[0060] Some embodiments may employ a real-time model access restriction unit 166 to enable protection of the ML / DL / AI engine 101 running in “production” by selectively restricting or limiting access to the ML / DL / AI engine 101, for example by introducing or injecting or intentionally imposing response time delays and / or user query quotas and / or rate limits, which may be effective against some “black box” evasion attacks, “black box” reprogramming attacks, theft attacks, and / or other attacks. This unit also assists in thwarting various attacks by using or enforcing user-defined or user-configurable blacklists. Optionally, restriction parameters may be dynamically selected and enforced based on penetration testing of the model or the ML / DL / AI engine 101.

[0060]

[0061] The system may thus provide real-time detection, attack prevention, attack mitigation, and attack monitoring, for example by monitoring and inspecting inputs coming into the ML / DL / AI engine 101 and / or outputs generated by the ML / DL / AI engine 101 while the ML / DL / AI engine 101 is running and operating ("in production") and analyzing such inputs and outputs to detect and mitigate attacks. Once an attack is detected and identified, the system may trigger real-time or near real-time query blocking, real-time or near real-time user blocking, send relevant attack information to a monitoring system, generate notifications to be sent to administrators or one or more recipients, and / or take other mitigation actions (e.g., rate limiting, configuring and enforcing query quotas, injecting time delays, quarantining or discarding inputs or queries, sanitizing inputs or queries, intentionally modifying or randomizing outputs (e.g., to counter or confuse an attacker), taking one or more components of the ML / DL / AI engine 101 offline, or other mitigation actions).

[0061]

[0062] Some embodiments may use a protection policy configuration unit 167 or a similar design and configuration tool that allows a user to configure, modify, and optimize multiple dependent and independent parameters, such as model accuracy for natural inputs, model accuracy under various types of attacks, computational sources for model predictions, and / or other parameters or constraints, to increase the robustness of the ML / DL / AI engine 101 against attacks and threats. The protection policy configuration unit 167 allows a user to explicitly consider, select, approve, and / or disapprove tradeoffs of various parameters, and to select, define, or configure a protection scheme that best suits the user's requirements, goals, and constraints.

[0062]

[0063] It is clarified that the ML / DL / AI engine 101 covered by some embodiments need not necessarily be a single unit or a stand-alone unit or a single component unit; rather, some embodiments may similarly cover ML / DL / AI engines that include two or more (or many) ML / DL / AI engines or units, multiple component ML / DL / AI engines or subsystems, or multiple model ML / DL / AI engines or subsystems, or a combination or hybrid of multiple units or components, that may be chained or a set or a group or a batch or other collection of a "computing farm" and that may be operatively related to each other or may operate in parallel and / or serial and / or in cooperation and / or simultaneously and / or in sequence. It is emphasized that, according to some embodiments, the protection system may serve and protect multiple different clients or multiple different independent ML / DL / AI engines simultaneously, cooperatively or in parallel, where the multiple different clients or multiple different independent ML / DL / AI engines may be completely independent from each other, and / or may be separate from each other, and / or may be owned or operated by different entities, and / or may be of different types.

[0063]

[0064] For purposes of illustration, some portions of this description may relate to a protected ML / DL / AI engine that receives images or photographs or graphical data items as input, however, these are merely examples and not by way of limitation, and some embodiments may similarly protect other types of ML / DL / AI engines and / or protect ML / DL / AI engines that receive (or process or act on) other types of data, such as, for example, textual data, numerical data, alphanumeric data, tabular or tabulated data, data arranged or formatted as an array or matrix or table of data items or values, fields or records, non-graphical data, graphical data, video data, audio data, audio and video data, raw data, processed data, parts, or parts of data. input may be partially processed data, user-provided data, machine-generated data, data collected or received from a sensor or sensing unit (e.g. accelerometer data, gyroscope data, temperature data, spatial alignment data), biometric data (e.g. fingerprint scan, retina scan, eye scan, face scan or face image), financial data, finance-related data, data intended for OCR, data obtained from (or provided as) a text file or a spreadsheet file or an XML file or a CSV file or a database file, input that is a combination of two or more types of data, input provided as a combination of two or more data formats, data items accompanied by metadata (e.g. indicating how or where or when the data was collected or obtained), and / or other types of data.

[0064]

[0065] Reference is now made to Figure 2, which illustrates a schematic block diagram of a system 200 according to some exemplary embodiments. Optionally, system 200 may be an implementation of the exemplary architecture of system 100 of Figure 1. System 200 uses a modular architecture that allows for scalability and may be implemented as a cloud service or cloud computing service for a single user or multiple users, or may be implemented and installed locally (e.g., as licensed software or modules co-located with a protected ML / DL / AI engine). The architecture and components of system 200 may support several features of the protection scheme, including: (a) penetration testing through evaluation of the robustness of the protected ML / DL / AI engine against various threats or attacks; (b) offline protection by modifying or reconfiguring the protected ML / DL / AI engine to make it more secure or resilient against attacks; (c) online protection by providing defensive services for the protected ML / DL / AI engine in real-time or near real-time; (d) attack mitigation services; and (e) a protection sandbox environment that allows manual / custom experimentation by users to investigate or test or research specific vulnerabilities and protection schemes.

[0065]

[0066] In some embodiments, a hybrid approach may be used, such as by using hybrid adversarial training, where one or more tasks or operations of the protected service are performed on a cloud-based platform or a remote server that is located remotely (or at a distance) from the protected ML / DL / AI engine, and one or more other tasks or operations of the protected service are performed on the client side or at the protected ML / DL / AI engine, or where, for example, the cloud-based platform generates adversarial samples and provides them to the protected ML / DL / AI engine, and the protected ML / DL / AI engine trains on a dataset that now includes the adversarial samples. Other hybrid or distributed implementations may also be used.

[0066]

[0067] Some embodiments may optionally use a cascade of multiple detectors or protection units or a detector cascade subsystem, for example by implementing a set or series or a line or group of detectors and applying only one or some (but not all) of those detectors for any given input signal (e.g., directed to a protected ML / DL / AI engine) such that the output(s) generated by the already activated detector(s) dictates whether (i) a decision has been reached properly for the particular input signal, or (ii) one or more additional detectors should be invoked for that particular input signal and their outputs should be analyzed as well (e.g., whether a decision can be reached or whether additional detector(s) should be invoked). In some embodiments, the detector cascade may place faster or less resource demanding detectors before slower or more resource demanding detectors, allowing the system to first attempt to reach a decision based on a faster, less resource demanding detection or decision process before invoking a slower, more resource demanding detection or decision process.

[0067]

[0068] In some embodiments, the protection platform or system may include or provide a "sandbox" experimentation tool or feature that allows a user (e.g., an owner or administrator of a protected ML / DL / AI engine) to experiment and investigate and evaluate different protection methods or different protection schemes or different protection configurations, and / or different vulnerabilities or attacks in a controlled or "sandboxed" environment that does not actually damage the protected ML / DL / AI engine. For example, a temporary copy of the protected ML / DL / AI engine may be replicated so that the temporary copy can be tested or experimented on without affecting the "real" protected ML / DL / AI engine, which is maintained in a temporary manner and out of contact.

[0068]

[0069] In some embodiments, one or more features or functions of the platform may be accessed and / or programmed via an application programming interface (API) and / or a software development kit (SDK) or “dev-kit” that enables a user (e.g., an owner or administrator of a protected ML / DL / AI engine) to interact with the platform and / or to create scripts and / or code portions that automate one or more interactions with the platform and / or automatically provide commands to the protection platform and / or automatically apply one or more pre-programmed rules to outputs (or alerts) generated by the protection platform.

[0069]

[0070] In some embodiments, a “cloud-based” implementation of the protection platform or some (but not necessarily all) of its functionality may be implemented using a remote cloud-based computer and / or using an “on-prem” or “on-premise” deployment, in which some (or all) of the functionality is deployed locally, on the premises of a user (the owner or administrator of the protected ML / DL / AI engine) and / or on such user's private infrastructure, and / or via a private cloud-based deployment or other suitable type(s) of Software-as-a-Service (SaaS) deployment.

[0070]

[0071] In some embodiments, multiple protection policies (or defense policies) can be predefined in the system and / or can be configured or defined or modified from time to time by a user (the owner or administrator of the protected ML / DL / AI engine), and one or more rules can be further defined to enable the protection system to automatically switch from a first protection policy to a second, different defense policy when one or more predefined conditions are valid, and / or when a triggering event occurs or is detected, and / or when the value of one or more particular parameters exceeds a predefined threshold value, and / or based on other types of conditions or triggers. For example, the protection or defense policy may reflect several conditions or goals or constraints that may be configured by a user (e.g., an owner or administrator of the protected ML / DL / AI engine), such as (i) a target accuracy or validity or correctness of the output generated by the protected ML / DL / AI engine, (ii) a target level of protection desired or intended to be achieved, (iii) resource-related constraints (e.g., processing cycles, processing time, time-to-decision, memory resources, storage resources), (iv) constraints or goals that may be related to other business-related or technical-related features (e.g., the ability to verify or explain performance-related verdicts of operations), (v) cost or budget constraints. Thus, the user (e.g., an owner or administrator of the protected ML / DL / AI engine) may modularly define a defense or protection strategy that is so reflected in the protection or defense policy and that takes into account and reflects the above goals and / or other goals or constraints.The protection platform then advises the user as to which protection suite(s) or which particular features or functions of the protection platform may be appropriate or usable to achieve such goals and / or comply with such constraints, and in some embodiments the protection platform may even automatically or autonomously choose to switch from a first set of protection actions (e.g., a first protection suite) to a second, different set of protection actions (e.g., a second protection suite) and / or change the values ​​of one or more parameters or constraints.

[0071]

[0072] For example, a user may define goals such as (i) no more than 8 percent of inputs should be blocked or considered malicious, (ii) up to 6 gigabytes of RAM can be allocated to analyzing inputs and detecting attacks, (iii) up to 450 clock cycles of two processing cores can be allocated to analyzing inputs and detecting attacks, etc., and when the implementation of a particular protection suite reaches those constraints at a particular time, the protection suite (or optional parameters it employs) will be automatically adapted or modified by the protection platform in order to stay within the goals or constraints defined in the protection policy.

[0072]

[0073] Thus, some embodiments may employ dynamic or dynamically changing or dynamically changeable protection policies and dynamic or dynamically changing or dynamically changeable or dynamically switchable protection suites. The dynamic property is important because, for example, the monetary cost of computing resources may change dynamically, and / or because new vulnerabilities may cause a sudden increase in attacks of a particular type, and / or because a certain event (e.g., war between two countries, or business conflict between two companies) may cause a sudden increase in attacks on a particular ML / DL / AI engine. Thus, the system may provide a dynamically changing or dynamically adjusting set of protection capabilities that take into account (and comply with) the constraints and the goals, which may be implemented in an online or cloud-based platform, and / or in an on-premise deployed unit(s), in the attack mitigation unit 124, and / or in other suitable units of the system.

[0073]

[0074] In some embodiments, the system may dynamically or periodically optimize or change or adjust (or switch) the defense policy and / or defense techniques or protection features that are applied to enforce or achieve a particular defense or protection policy or its goals or constraints based on one or more rules or parameters or conditions or analyses, for example, by considering and analyzing the historical performance of the defense / protection optimization and / or changes, and by considering features (e.g., specific ML / DL / AI features being protected, system type, dataset type, dataset size, overall dataset features, dataset features generated using an autoencoder, model types, etc.) from a particular use case. This is done by extracting a set of effective defensive suites and / or protective features (e.g., a combination of effective defensive suites or sets of effective protection features) that would reach a goal (or a maximum number of goals) from the objectives defined by the protection policy, and may also be trained to predict or estimate an effective combination or set of effective defensive suites and / or protective features that would reach a goal (or a maximum number of goals) from the objectives defined by the protection policy, while still subject to system usage constraints (e.g., operational constraints, processing resource constraints, memory constraints, time constraints, false positive threshold constraints, false negative threshold constraints) or a maximum number of such constraints.In some embodiments, a first differential weight formula can be optionally used with respect to various objectives defined in the protection policy, since a particular objective (e.g., blocking 98 percent of attacks) can be assigned a greater weight than another particular objective (e.g., having at most 7 percent false positive errors), and a second differential weight formula can be optionally used with respect to various constraints defined in the protection policy or by a system administrator (e.g., the owner or administrator of the protected ML / DL / AI engine), since a particular constraint (e.g., having at most 5 gigabytes of RAM usage) can be assigned a greater weight or greater stringency than another particular constraint (e.g., having all relevant computations for any input signal completed within 6 seconds of receiving that input signal). The output of the weighted formula can be used as a consideration in dynamically optimizing or changing (or switching) the applied protection policy and / or protection features.

[0074]

[0075] Some embodiments provide a system including one or more processors configured to execute code, the one or more processors in operative association with one or more memory units configured to store the code. The system is configured to protect a protected engine, which is a unit selected from a group including a machine learning (ML) unit, a deep learning (DL) unit, an artificial intelligence (AI) unit, and a neural network (NN) unit. The system is configured to protect the protected engine from malicious inputs and malicious queries and malicious training data and malicious attacks. The system includes at least one of: (a) an offline protection unit for analyzing one or more characteristics of the protected engine and for offline hardening of the protected engine against attacks by changing one or more operating characteristics or parameters of the protected engine and reducing its vulnerability to attacks; and (b) an online protection unit for analyzing at least one of (i) one or more inputs intended to be input to the protected engine and (ii) one or more outputs generated by the protected engine and, based on the analysis, for dynamically hardening the protected engine online against attacks by dynamically changing one or more operating characteristics or parameters of the protected engine and reducing its vulnerability to attacks (and, in some implementations, both).

[0075]

[0076] In some embodiments, the protected engine uses a protected model selected from a group including an ML model, a DL model, and an AI model, and at least one of the offline protection unit and the online protection unit automatically modifies the protected model from an unmodified model version that is vulnerable to a particular attack to a modified model version that is not vulnerable or is less vulnerable to the particular attack.

[0076]

[0077] In some embodiments, the protected engine operates based on a protected dataset, and at least one of the offline protection unit and the online protection unit automatically modifies the protected dataset from a pre-modification dataset version that renders the protected engine vulnerable to a particular attack to a post-modification dataset version that renders the protected engine not vulnerable or less vulnerable to the particular attack.

[0077]

[0078] In some embodiments, the protected engine operates based on a protected dataset, and at least one of the offline protection unit and the online protection unit automatically modifies the protected dataset from a pre-modified dataset version that renders the protected engine vulnerable to a particular attack to a post-modified dataset version that renders the protected engine not vulnerable or less vulnerable to the particular attack by generating and appending adversarial data to the protected dataset and initiating re-training of the protected engine based on the post-modified dataset version.

[0078]

[0079] In some embodiments, the protected engine uses a protected architecture having a particular number of pipeline operations using particular values ​​of operational parameters, and at least one of the offline protection unit and the online protection unit automatically modifies the protected architecture from a pre-modified architecture version that renders the protected engine vulnerable to a particular attack to a post-modified architecture version that renders the protected engine not vulnerable or less vulnerable to the particular attack, by changing the number or order of the pipeline operations or by changing one or more of the particular values ​​of the operational parameters.

[0079]

[0080] In some embodiments, the protected engine uses a protected architecture having a particular number of pipeline operations using particular values ​​of operating parameters, and at least one of the offline protection unit and the online protection unit automatically modifies the protected architecture from a pre-modified architecture version that renders the protected engine vulnerable to a particular attack to a post-modified architecture version that renders the protected engine not vulnerable or less vulnerable to the particular attack by adding a pre-processing unit to the protected architecture that performs further operations on inputs received by the protected engine.

[0080]

[0081] In some embodiments, the protected engine uses a protected architecture having a particular number of pipeline operations using particular values ​​of operating parameters, and at least one of the offline protection unit and the online protection unit automatically modifies the protected architecture from a pre-modified architecture version that renders the protected engine vulnerable to a particular attack to a post-modified architecture version that renders the protected engine not vulnerable or less vulnerable to the particular attack by adding a post-processing unit to the protected architecture that performs further operations on output generated by the protected engine before the output is sent to an intended recipient.

[0081]

[0082] In some embodiments, the protected engine uses a protected architecture having a particular number of pipeline operations using particular values ​​of operating parameters, and at least one of the offline protection unit and the online protection unit automatically modifies the protected architecture from a pre-modification architecture version that renders the protected engine vulnerable to a particular attack to a post-modification architecture version that renders the protected engine not vulnerable or less vulnerable to the particular attack by adding an attack detection unit to the pipeline of the protected architecture that analyzes inputs and determines whether a particular input or a particular group of inputs is associated with an attack on the protected engine.

[0082]

[0083] In some embodiments, the attack detection unit is not necessarily an essential part of the protection array or offline protection unit or online protection unit, but rather may be an auxiliary unit that is capable of detecting an attack or inferring that a particular input (or set of inputs) may be part of an attack or may be malicious or may not be legitimate, and sending an "attack detected / inferred" signal to other parts of the system's units to invoke protective and / or mitigating actions.

[0083]

[0084] In some embodiments, one or more of the auxiliary units of the platform or system, such as an offline protection unit, may operate by actively modifying or adjusting the internal structure of the model (digital representation of the model) on which the ML / DL / AI engine runs or operates, and / or by adding pre-processing and / or post-processing operations, filtering operations, sanitization operations, re-training, adversarial training, and / or other techniques, and may operate to make the ML / DL / AI engine and / or its model more resilient or less vulnerable or more robust against attacks.

[0084]

[0085] In some embodiments, the protected engine uses a protected architecture having a particular number of pipeline operations with particular values ​​of operating parameters, and at least one of the offline protection unit and the online protection unit automatically modifies the protected architecture by doing at least one of the following: (i) rate limiting the number of queries that a particular recipient can submit to the protected engine; (ii) discarding at least some of the queries incoming to the protected engine; (iii) quarantining at least some of the queries incoming to the protected engine; (iv) intentionally delaying output generated by the protected engine or a particular output or a particular subset of outputs before it is sent to an associated or intended recipient(s); or (v) alternating between two or more of the above actions or performing actions different from the above actions from time to time.

[0085]

[0086] In some embodiments, the protected engine uses a protected architecture having a particular number of pipeline operations using particular values ​​of operating parameters, and at least one of the offline protection unit and the online protection unit automatically modifies the protected architecture by intentionally delaying responses output from the protected engine to particular recipients.

[0086]

[0087] In some embodiments, the protected engine uses a protected architecture having a particular number of pipeline operations using particular values ​​of operating parameters, and at least one of the offline protection unit and the online protection unit automatically modifies the protected architecture by intentionally providing erroneous output from the protected engine toward particular recipients determined to be the source of malicious queries.

[0087]

[0088] In some embodiments, at least one of the offline protection unit and the online protection unit performs a secondary classification of an input query received from a particular source entity using a secondary engine selected from the group including a secondary ML engine, a secondary DL engine, and a secondary AI engine, and determines whether the input query is part of a malicious attack on the protected engine based on the classification of the input query performed by the secondary engine.

[0088]

[0089] In some embodiments, the system further includes an AI-based penetration testing unit (or an ML- or DL-based penetration testing unit) configured to perform automated penetration testing on the protected engine in a controlled manner and to estimate or predict (e.g., via ML / DL / AI techniques) a quantitative level of resilience of said protected engine against various types of attacks. In some embodiments, every penetration test performed on a protected engine adds new data samples or data points that are then used by the AI-based (or ML- or DL-based) penetration testing unit to retrain and thereby improve the penetration testing prediction model(s). In some embodiments, the AI-based (or ML-based or DL-based) penetration testing unit may predict or estimate the most effective attack or attack type that can be used against the protected engine and / or may predict or estimate hyper-parameters of such attacks or the most effective manner in which such attacks can be executed against the protected engine, and the ML-predicted hyper-parameters of the ML-predicted attacks against the protected engine may be used by the protection platform to invoke or apply or select relevant or effective protective or mitigating actions (e.g., via ML-based / AI-based / DL-based prediction or selection, or by a deterministic selection process based on a set of predetermined selection rules).

[0089]

[0090] Similarly, the offline protection units and / or the online protection units may be AI-based or ML-based or DL-based, and thus they may be continuously (periodically) retrained based on the ongoing monitored performance of the protected engine and / or protection platform, and thus may learn and improve autonomously.

[0090]

[0091] In some embodiments, the system includes a defense policy builder unit configured to build a user-configurable defense policy defined by a plurality of user-configurable operational constraints for the protected engine, and configured to dynamically select, invoke and enforce a particular protection scheme from a pool of a plurality of available protection schemes, such that the particular protection scheme complies with the user-configurable operational constraints while protecting the protected engine from attacks.

[0091]

[0092] In some embodiments, the system includes an attack mitigation unit that is configured to select and activate one or more deterministic and non-AI based mitigation techniques in response to determining that the protected engine is under attack.

[0092]

[0093] In some embodiments, the system includes an attack mitigation unit that is configured to select and activate one or more AI-based non-deterministic mitigation techniques in response to determining that the protected engine is under attack.

[0093]

[0094] In some embodiments, the system includes a non-intrusive prevention unit configured to perform pre-processing and / or post-processing operations to thwart detected attacks against a protected engine without modifying the models used by the protected engine, without retraining the protected engine on a modified data set, and without access to the internal pipeline of operations performed by the protected engine.

[0094]

[0095] In some embodiments, the protected engine is protected from malicious queries and malicious input by the system, which is a remote cloud computing based system separate from and not co-located with the protected engine.

[0095]

[0096] In some embodiments, the system includes a defense policy construction unit that enables a user to input (i) defense goals for protecting a protected engine and (ii) operational constraints to be followed by the protected engine and / or system, and a defense optimization unit that operates based on the defense goals and operational constraints provided by the user and is configured to dynamically select and apply a set of specific defense techniques to prevent or mitigate attacks against the protected engine.

[0096]

[0097] In some embodiments, the defense optimization unit operates based on an AI-based estimation or prediction of effective defense techniques to be selected and applied to protect the protected engines, based on an AI-based analysis of historical data for the protection of a number of different AI engines and / or ML engines and / or DL ​​engines.

[0097]

[0098] In some embodiments, the protected engine is a protected engine in a combination or cascade or pipeline of two or more AI or ML or DL ​​engines, and the system performs protection operations to protect at least a portion of the combination or cascade or pipeline.

[0098]

[0099] In some embodiments, the online protection unit is configured to deploy and invoke a pre-processing module that performs defensive pre-processing of input signals directed to said protected engine.

[0099]

[0100] In some embodiments, the online protection unit is configured to implement and invoke a post-processing module that performs defensive post-processing of output signals generated and output by the protected engine before the output signals are sent to their intended recipients.

[0100]

[0101] In some embodiments, the online protection unit is configured to change the operating architecture of the protected engine from (I) a standalone or independent protected engine to (II) a protected engine that is part of a multi-engine pipeline or multi-engine cascade structure that is more resilient to attacks than a standalone or independent protected engine.

[0101]

[0102] Some embodiments may include a non-transitory storage medium or storage article that stores instructions or code that, when executed by a processor of a machine or hardware, cause the processor of the machine or hardware to perform the described method.

[0102]

[0103] Some embodiments may use an automated method or process, or a machine-implemented method or process, or a semi-automated or partially automated method or process, or a set of steps or operations that may be performed or carried out by a computer or machine or system or other device.

[0103]

[0104] Some embodiments may be implemented using suitable hardware and / or software components, a processor (e.g., a central processing unit (CPU), digital signal processor (DSP), integrated circuit (IC), logic unit, etc.) capable of executing code or programs or machine-readable instructions, storage units (e.g., hard disk drives, solid state devices) for storing received and / or processed data, memory units (e.g., random access memory (RAM) memory, flash memory) for short-term storage of data, transceivers capable of sending and receiving data using one or more wireless communication protocols (e.g., Wi-Fi communication, cellular communication, TCP / IP communication), and other suitable components (e.g., input units such as a keyboard, mouse, etc., output units such as a display unit, a power supply, etc.).

[0104]

[0105] Some embodiments may use code or program code or machine readable instructions or machine readable code that may be stored on a non-transitory storage medium or non-transitory storage item (e.g., CD-ROM, DVD-ROM, physical memory unit, physical storage unit) that, when executed by a processor or machine or computer, causes the processor or machine or computer to perform the methods or processes described herein. Such code or instructions may be or may include, for example, one or more of software, a software module, an application, a program, a subroutine, an instruction, a set of instructions, computing code, a word, a value, a symbol, a string, a variable, source code, compiled code, interpreted code, executable code, static code, dynamic code, including, but not limited to, code or instructions in a high level programming language, a low level programming language, an object oriented programming language, a visual programming language, a compiled programming language, an interpreted programming language, C, C++, C#, Java, JavaScript, SQL, Ruby on Rails, Go, Cobol, Fortran, ActionScript, AJAX, XML, JSON, Lisp, Eiffel, Verilog, Hardware Description Language (HDL), Register-Transfer Level (RTL), BASIC, Visual BASIC, Matlab, Pascal, HTML, HTML5, CSS, Perl, Python, PHP, machine language, machine code, assembly language, etc.

[0105]

[0106] The description herein uses terms such as, for example, “processing,” “calculating,” “estimating,” “generating,” “determining,” “establishing,” “analyzing,” “examining,” “detecting,” “measuring,” and the like, which may refer to operation(s) and / or process(es) of a processor, computer, computing platform, computing system, or other electronic or computing device that may automatically and / or autonomously manipulate and / or transform data represented as physical (e.g., electronic) quantities in registers and / or accumulators and / or memory units and / or storage units into other data or perform other suitable operations.

[0106]

[0107] Some embodiments may perform steps or actions such as, for example, "determining," "identifying," "comparing," "examining," "interrogating," "searching," "matching," and / or "analyzing," which may include, for example, using predefined threshold values ​​that may be compared to one or more parameter values; using a comparison between (i) the sensed or measured or calculated value(s) and (ii) predefined or dynamically generated threshold value(s) and / or value ranges and / or upper and / or lower limits and / or maximum and / or minimum values; using a look-up table or legend of possible values ​​or ranges between the sensed or measured or calculated data; This may be accomplished by using a comparison or matching between one or more values ​​stored in a look-up table, legend list or database; by using a comparison or matching or search process that searches for matches and / or identical and / or similar results among multiple values ​​or ranges stored in a database or look-up table; by using one or more equations, formulas, weighted formulas, and / or calculations to determine similarity or consistency between or among parameters or values; by using comparison units, look-up tables, threshold values, conditions, conditioning logic, Boolean operators (one or more), and / or other suitable components and / or operations.

[0107]

[0108] As used herein, the terms "plurality" and "a plurality" include, for example, "multiple" and "two or more." For example, "a plurality of items" includes two or more items.

[0108]

[0109] The terms "one embodiment," "embodiment," "exemplary embodiment," "various embodiments," "some embodiments," and / or similar terms may indicate that the embodiment(s) so described may optionally include a particular feature, structure, or characteristic, but not all embodiments necessarily include that particular feature, structure, or characteristic. Moreover, repeated use of the phrase "in one embodiment" may, but does not necessarily, refer to the same embodiment. Similarly, repeated use of the phrase "in some embodiments" may, but does not necessarily refer to the same set or group of embodiments.

[0109]

[0110] The use of ordinal adjectives such as "first," "second," "third," "fourth," etc., used herein to describe items or objects, unless otherwise specified, is intended merely to indicate that reference is being made to various instances of such items or objects, and is not intended to imply that the items or objects so described should be in a particular given order in time or space or in ranking or any other manner of ordering.

[0110]

[0111] Some implementations may be used in or in connection with a variety of devices and systems, including, for example, a personal computer (PC), a desktop computer, a mobile computer, a laptop computer, a notebook computer, a tablet computer, a server computer, a handheld computer, a handheld device, a personal digital assistant (PDA) device, a handheld PDA device, a tablet, an on-board device, an off-board device, a hybrid device, a vehicular device, a non-vehicular device, a mobile or portable device, a consumer electronics device, a home ... This includes but is not limited to, equipment, non-vehicular or non-portable devices, appliances, wireless communication stations, wireless communication devices, wireless access points (APs), wired or wireless routers or gateways or switches or hubs, wired or wireless modems, video devices, audio devices, audio / video (AV) devices, wired or wireless networks, wireless area networks, wireless video area networks (VWANs), local area networks (LANs), wireless LANs (WLANs), personal area networks (PANs), and wireless PANs (WPANs).

[0111]

[0112] Some implementations may be used in connection with one-way and / or two-way wireless communication systems, cellular radiotelephone communication systems, mobile phones, cellular telephones, wireless telephones, personal communication systems (PCS), PDAs or handheld devices incorporating wireless communication capabilities, mobile or portable Global Positioning System (GPS) devices, devices incorporating GPS receivers or transceivers or chips, devices incorporating RFID elements or chips, multiple-input multiple-output (MIMO) transceivers or devices, single-input multiple-output (SIMO) transceivers or devices, multiple-input single-output (MISO) transceivers or devices, devices with one or more internal and / or external antennas, digital video broadcast (DVB) devices and systems, multi-reference radio devices or systems, wired or wireless handheld devices such as smartphones, Wireless Application Protocol (WAP) devices, and the like.

[0112]

[0113] Some implementations may include or be implemented using an "app" or application, which may be downloadable or obtained, for free or for a fee, from an "app store" or "application store," which may be pre-installed on a computing or electronic device, or which may otherwise be transported and / or installed on such computing or electronic device.

[0113]

[0114] Any feature, operation, component, and / or feature described herein with reference to one or more implementations may be combined or used in combination with one or more other features, operations, components, and / or features described herein with reference to one or more other implementations. Some embodiments may include any possible or suitable combination, rearrangement, assembly, reassembly, or other use of some or all of the modules or functions or components or units described herein, even if the modules or functions or components or units described herein are described in different locations or different sections in the preceding description, or even if they are described in different figures or across multiple figures.

[0114]

[0115] While certain features of several illustrative embodiments have been illustrated and described herein, various modifications, substitutions, alterations, and equivalents will occur to those skilled in the art, and it is therefore intended that the claims cover all such modifications, substitutions, alterations, and equivalents.

Claims

1. 1. A system comprising: one or more processors configured to execute code; the one or more processors are operatively associated with one or more memory units configured to store code; the system is configured to protect a protected engine, the protected engine being a unit selected from the group consisting of a machine learning (ML) unit, a deep learning (DL) unit, an artificial intelligence (AI) unit, and a neural network (NN) unit; the system is configured to protect the protected engine against malicious inputs, malicious queries, malicious training data, and malicious attacks; The system is (a) an offline protection unit for analyzing one or more characteristics of the protected engine and for performing offline hardening of the protected engine against attacks by altering one or more operating characteristics or parameters of the protected engine to reduce the vulnerability of the protected engine to attacks; (b) an online protection unit for performing an analysis of at least one of (i) one or more inputs directed as inputs to the protected engine and (ii) one or more outputs generated by the protected engine, and for dynamically performing an online hardening of the protected engine against attacks by dynamically varying one or more operating characteristics or parameters of the protected engine based on the analysis to reduce vulnerability of the protected engine to attacks; A system comprising at least one of:

2. 2. The system of claim 1, the protected engine uses a protected model selected from the group consisting of an ML model, a DL model, and an AI model; at least one of the offline protection unit and the online protection unit automatically modifies the protected model from a pre-modification model version that is vulnerable to a particular attack to a post-modification model version that is not vulnerable or has a low vulnerability to the particular attack; system.

3. 2. The system of claim 1, the protected engine operates on a protected data set; At least one of the offline protection unit and the online protection unit automatically modifies the protected dataset from a pre-modification dataset version that renders the protected engine vulnerable to a particular attack to a post-modification dataset version that renders the protected engine not vulnerable or less vulnerable to the particular attack.

4. 2. The system of claim 1, the protected engine operates on a protected data set; At least one of the offline protection unit and the online protection unit: Automatically modifying the protected dataset from a pre-modified dataset version that renders the protected engine vulnerable to a particular attack to a post-modified dataset version that renders the protected engine not vulnerable or less vulnerable to the particular attack; by generating and appending adversarial data to the protected dataset and by initiating a retraining of the protected engine based on the modified dataset version; A system that performs the above.

5. 2. The system of claim 1, the protected engine uses a protected architecture having a particular number of pipeline operations using particular values ​​of operational parameters; At least one of the offline protection unit and the online protection unit: Automatically modifying the protected architecture from a pre-modification architecture version that renders the protected engine vulnerable to a particular attack to a post-modification architecture version that renders the protected engine not vulnerable or less vulnerable to the particular attack, by changing the number or order of said pipeline operations or by changing one or more of the particular values ​​of said operational parameters; A system that performs the above.

6. 2. The system of claim 1, the protected engine uses a protected architecture having a particular number of pipeline operations using particular values ​​of operational parameters; At least one of the offline protection unit and the online protection unit: Automatically modifying the protected architecture from a pre-modification architecture version that renders the protected engine vulnerable to a particular attack to a post-modification architecture version that renders the protected engine not vulnerable or less vulnerable to the particular attack, by adding a pre-processing unit to the protected architecture that performs further operations on inputs received by the protected engine; A system that performs the above.

7. 2. The system of claim 1, the protected engine uses a protected architecture having a particular number of pipeline operations using particular values ​​of operational parameters; At least one of the offline protection unit and the online protection unit: Automatically modifying the protected architecture from a pre-modification architecture version that renders the protected engine vulnerable to a particular attack to a post-modification architecture version that renders the protected engine not vulnerable or less vulnerable to the particular attack, by adding a post-processing unit to the protected architecture which performs further operations on the output generated by the protected engine before the output is sent to an intended recipient; A system that performs the above.

8. 2. The system of claim 1, the protected engine uses a protected architecture having a particular number of pipeline operations using particular values ​​of operational parameters; At least one of the offline protection unit and the online protection unit: Automatically modifying the protected architecture from a pre-modification architecture version that renders the protected engine vulnerable to a particular attack to a post-modification architecture version that renders the protected engine not vulnerable or less vulnerable to the particular attack, by adding an attack detection unit to the pipeline of the protected architecture that analyzes inputs to determine whether a particular input, or a particular group of inputs, is associated with an attack on the protected engine; A system that performs the above.

9. 2. The system of claim 1, the protected engine uses a protected architecture having a particular number of pipeline operations using particular values ​​of operational parameters; At least one of the offline protection unit and the online protection unit: automatically modifying the protected architecture, (i) rate limiting the number of queries that a particular recipient may submit to the protected engine; (ii) discarding at least some of the queries incoming to the protected engine; and (iii) isolating at least some of the queries incoming to the protected engine; and by performing at least one of the following: A system that performs the above.

10. 2. The system of claim 1, the protected engine uses a protected architecture having a particular number of pipeline operations using particular values ​​of operational parameters; At least one of the offline protection unit and the online protection unit: automatically modifying the protected architecture, by intentionally delaying responses output from said protected engines to specific recipients; A system that performs the above.

11. 2. The system of claim 1, the protected engine uses a protected architecture having a particular number of pipeline operations using particular values ​​of operational parameters; At least one of the offline protection unit and the online protection unit: automatically modifying the protected architecture, by intentionally providing erroneous output from said protected engine to particular recipients determined to be the source of malicious queries; A system that performs the above.

12. 2. The system of claim 1, At least one of the offline protection unit and the online protection unit: performing a secondary classification of the input query received from the particular source entity using a secondary engine selected from a group consisting of a secondary ML engine, a secondary DL engine, and a secondary AI engine; and determining whether the input query is part of a malicious attack on the protected engine based on the classification of the input query executed by the secondary engine; system.

13. 2. The system of claim 1, further comprising:

1. An AI-based infiltration test unit, comprising: configured to perform automated penetration testing on the protected engine in a controlled manner; configured to estimate a quantitative level of resilience of the protected engine against various types of attacks; A system including an infiltration testing unit.

14. 2. The system of claim 1, a defense policy builder unit, configured to construct a user-configurable defense policy defined by a plurality of user-configurable operational constraints for the protected engine; configured to dynamically select, invoke, and enforce a particular protection scheme from a pool of a plurality of available protection schemes, the particular protection scheme complying with the user-configurable operational constraints while protecting the protected engine against attacks; A system including a defense policy builder.

15. 2. The system of claim 1, a defense policy construction unit that enables a user to input (i) defense objectives for protecting the protected engine, and (ii) operational constraints that the protected engine and / or the system should follow; a defense optimization unit that operates based on the defense objectives and the operational constraints provided by the user, the defense optimization unit being configured to dynamically select and apply a set of specific defense techniques to prevent or mitigate attacks against the protected engine; Including, the system.

16. 2. The system of claim 1, The system, wherein the defense optimization unit operates based on an AI-based estimation or prediction of effective defense techniques to be selected and applied to protect the protected engines based on an AI-based analysis of historical data for the protection of a plurality of different AI engines, ML engines, and / or DL ​​engines.

17. 2. The system of claim 1, a system comprising an attack mitigation unit configured to select and activate one or more deterministic non-AI based mitigation techniques in response to determining that the protected engine is under attack;

18. 2. The system of claim 1, an attack mitigation unit, 11. The system, comprising: an attack mitigation unit configured to select and activate one or more AI-based non-deterministic mitigation techniques in response to determining that the protected engine is under attack.

19. 2. The system of claim 1, A system comprising: a non-intrusive prevention unit configured to perform pre-processing and / or post-processing operations to thwart detected attacks against the protected engine without modifying a model used by the protected engine, without retraining the protected engine on a modified data set, and without access to an internal pipeline of operations performed by the protected engine.

20. 2. The system of claim 1, A system that protects against malicious queries and malicious inputs, the system being a remote cloud computing based system where the protected engine is separate from and not co-located with the protected engine.

21. 2. The system of claim 1, the protected engine is one protected engine in a combination, cascade, or pipeline of two or more AI, ML, or DL ​​engines; the system performing a protection action to protect at least a portion of the combination, cascade, or pipeline. system.

22. 2. The system of claim 1, The system, wherein the online protection unit is configured to install and invoke a pre-processing module that performs defensive pre-processing of input signals directed to the protected engine.

23. 2. The system of claim 1, The system, wherein the online protection unit is configured to install and invoke a post-processing module that performs defensive post-processing of an output signal generated and output by the protected engine before the output signal is sent to an intended recipient.

24. 24. A system according to any one of claims 1 to 23, comprising: the online protection unit is configured to change the operating architecture of the protected engine from a standalone or independent protected engine to a protected engine that is part of a multi-engine pipeline or multi-engine cascade structure that is more resilient to attacks than a standalone or independent protected engine, system.

25. A computerized method implemented by using one or more processors configured to execute code, the one or more processors operatively associated with one or more memory units configured to store the code; The computerized method comprises: protecting a protected engine, the protected engine being a unit selected from the group consisting of a machine learning (ML) unit, a deep learning (DL) unit, an artificial intelligence (AI) unit, and a neural network (NN) unit; Protecting the protected engine against malicious inputs and queries, malicious training data, and malicious attacks, comprising: (a) operating an offline protection unit to perform offline hardening of the protected engine against attacks by analyzing one or more characteristics of the protected engine and altering one or more operating characteristics or parameters of the protected engine to reduce vulnerability of the protected engine to attacks; (b) operating an online protection unit to perform an analysis of at least one of (i) one or more inputs directed as inputs to the protected engine and (ii) one or more outputs generated by the protected engine, and based on the analysis, to dynamically perform an online hardening of the protected engine against attacks by dynamically varying one or more operating characteristics or parameters of the protected engine to reduce vulnerability of the protected engine to attacks; by performing at least one of the following: A method comprising:

26. A non-transitory storage medium storing instructions that, when executed by a machine, cause the machine to perform a computerized method, the method comprising: protecting a protected engine, the protected engine being a unit selected from the group consisting of a machine learning (ML) unit, a deep learning (DL) unit, an artificial intelligence (AI) unit, and a neural network (NN) unit; Protecting the protected engine against malicious inputs, malicious queries, malicious training data, and malicious attacks, comprising: (a) operating an offline protection unit to perform offline hardening of the protected engine against attacks by analyzing one or more characteristics of the protected engine and altering one or more operating characteristics or parameters of the protected engine to reduce vulnerability of the protected engine to attacks; (b) operating an online protection unit to perform an analysis of at least one of (i) one or more inputs directed as inputs to the protected engine and (ii) one or more outputs generated by the protected engine, and based on the analysis, to dynamically perform an online hardening of the protected engine against attacks by dynamically varying one or more operating characteristics or parameters of the protected engine to reduce vulnerability of the protected engine to attacks; by performing at least one of the following: (c) a storage medium.

Images