System for detecting the exploitation of an AI model and a method for doing so

Abstract

The present disclosure proposes a system (10) for detecting the exploitation of an AI model (11) and a method (200) for doing so. The system (10) comprises an AI model (11), a transformation module (12), a media-to-text AI model (14), and at least one processor (16). The AI ​​model (11) is designed to receive an input prompt (I) and generate a visual media output (A). The transformation module (12) is designed to transform the visual media output (A) into multiple secondary outputs (B). The media-to-text AI model (14) is designed to process the multiple secondary outputs (B) and generate multiple tertiary outputs (C). The processor (16) is designed to perform a comparative analysis of the tertiary outputs (C) and the received input prompts (I) to detect exploitation of the AI ​​model.

Description

Field of the invention

[0001] The present disclosure relates to the field of artificial intelligence security for generative AI models. In particular, the present disclosure proposes a method for detecting the exploitation of an AI model based on text-to-visual outputs in an AI system. Background of the invention

[0002] Since the advent of data science, data processing and decision-making systems have been implemented using artificial intelligence (AI) modules. AI modules employ various techniques such as machine learning, neural networks, deep learning, and others. Most AI-based systems receive large amounts of data and process this data to train AI models. Trained AI models generate outputs based on user-requested use cases. Typically, AI systems are used in computer vision, speech recognition, natural language processing, audio recognition, healthcare, autonomous driving, manufacturing, robotics, and others, where they process data to generate required outputs based on specific rules / relevant information acquired through appropriate training.

[0003] To process the inputs and deliver the desired output, AI systems use various models / algorithms that are trained using the training data. Once the AI ​​system has been trained using the training data, the AI ​​systems use the models to analyze the real-time data and generate an appropriate output. The models can be fine-tuned in real time based on the results. The models in AI systems form the core of the system. A great deal of effort, (tangible and intangible) resources, and knowledge are invested in developing these models.

[0004] Generative AI models are capable of generating text, images, or other media. Recent technological developments in the field of generative AI and so-called base models (such as ChatGPT) present new opportunities and challenges. These models are capable of generating content such as software code, text, images, and music. What makes them particularly powerful compared to previous AI models is the fact that they are trained on vast amounts of data. Recently, a number of generative AI systems have emerged that are known for accepting natural language prompts as input. These include LLM (Large Language Model) chatbots such as ChatGPT, Bing Chat, Bard, and LLaMA, and text-to-image AI art systems such as Stable Diffusion, Midjourney, and DALL-E.

[0005] Text-to-image generation is a unique area in the computer vision domain; while some attacks have been explored, few defenses have been developed. Attackers pose a significant threat to the integrity and reliability of such AI systems through various techniques such as fabricating adversarial data (poisoning) or finding adversarial examples. Conventional approaches that rely on a single model for prediction are vulnerable to targeted attacks, making it imperative to explore more robust security measures. In this invention, we propose a robust framework / system and method for detecting attacks on such AI models designed to receive a prompt and generate a visual media output. Brief description of the attached drawings

[0006] An embodiment of the invention will be described with reference to the accompanying drawings: Fig. 1 shows a system for detecting the exploitation of an AI model; Fig. 2 illustrates method steps (200) for detecting the exploitation of an AI model. Detailed description of the drawings

[0007] It is important to understand some aspects of artificial intelligence (AI)-based technologies and AI-based systems. Some key aspects of AI technology and AI systems can be explained as follows. Depending on the architecture, an AI system may include many components. One such component is an AI model. An AI module can be explained with reference to this disclosure as a component that executes a model. A model can be defined as a reference or inference set of data that uses different types of correlation matrices. Using these models and the data from these models, correlations can be established between different types of data to arrive at some logical understanding of the data.Those skilled in the art will be familiar with the different types of AI models, such as linear regression, naive Bayesian classifiers, support vector machines, neural networks, and the like. Those skilled in the art will also recognize that the AI ​​model can be implemented as a set of software instructions, as hardware (such as neural network chips), or as a simultaneous combination of software and hardware.

[0008] Some typical tasks performed by AI systems are classification, clustering, regression, etc. The majority of classification tasks depend on labeled datasets, meaning the datasets are manually labeled so that a neural network can learn the correlation between labels and data. This is known as supervised learning. Some of the typical applications of classification are: face recognition, object identification, gesture recognition, speech recognition, etc. Clustering, or grouping, is the detection of similarities in the inputs. Cluster learning techniques do not require labels to detect similarities. Learning without labels is called unsupervised learning. Unlabeled data forms the majority of data in the world. A law of machine learning states: the more data there is to train an algorithm on, the more accurate it will be.Therefore, unsupervised learning models / algorithms have the potential to produce accurate models as the size of the training dataset increases.

[0009] In model extraction attacks (MEA), the attacker obtains information about the model's internals by analyzing inputs, outputs, and other external information. Stealing such a model exposes the organization's critical intellectual property and allows the attacker to develop other adversarial attacks, such as evasion. The attacker typically generates random queries the size and shape of the input specifications and begins querying the model with these arbitrary queries. These queries generate input / output pairs for random queries and generate a secondary dataset derived from the pre-trained model. The attacker then takes these I / O pairs and trains the new model from scratch using the secondary dataset. This is a black-box model attack vector, requiring no prior knowledge of the original model.As the prior information regarding the model becomes available and increases, the attacker can conduct increasingly sophisticated attacks. The attacker selects a relevant dataset available to them to perform model extraction more efficiently. This is an attack vector based on a domain intelligence model. Using these approaches, it is possible to demonstrate a model theft attack across different models and datasets.

[0010] In general, adversarial AI threats can be broadly categorized into model extraction attacks, inference attacks, evasion attacks, and data poisoning attacks. In model extraction attacks (MEA), the attacker obtains information about the model's internals by analyzing inputs, outputs, and other external information. Stealing such a model exposes the organization's critical intellectual property and allows the attacker to develop other adversarial attacks, such as evasion attacks. The attacker typically generates random queries the size and shape of the input specifications and begins querying the model with these arbitrary queries. These queries generate input / output pairs for random queries and generate a secondary dataset derived from the pre-trained model.The attacker then takes these I / O pairs and trains the new model from scratch using the secondary dataset. This is a black-box model attack vector, which requires no prior knowledge of the original model. As prior information regarding the model becomes available and increases, the attacker can conduct increasingly intelligent attacks. The attacker selects a relevant dataset available to them to perform model extraction more efficiently. This is a domain intelligence model-based attack vector.

[0011] In poisoning attacks, the attacker injects carefully crafted data to contaminate the training data, ultimately compromising the functionality of the AI-based system. Because generative AI models rely on a comprehensive and diverse training dataset, they are highly vulnerable to poisoning attacks. Inference attacks attempt to derive the training data from the corresponding outputs or other information leaked by the target model. Studies have shown that it is possible to recover training data associated with arbitrary model outputs. In evasion attacks, the attacker manipulates the inputs of the AI ​​algorithm to find small perturbations that lead to large modifications of the associated outputs (i.e., decision errors), resulting in evasion of the AI ​​model.Because generative AI models are deployed in open environments that expose them to diverse and numerous inputs, they are vulnerable to evasion attacks.

[0012] Fig. 1 shows a system (10) for detecting the exploitation of an AI model (11). The system (10) comprises an AI model (11), a transformation module (12), a media-to-text AI model (14), and at least one processor (16). The output of the system (10) can be displayed on an output interface (20). It should be noted that each component of the system (10) can be implemented in different architectural frameworks depending on the application. In one embodiment of the architectural framework, all components of the system (10) are implemented in hardware, i.e., each building block can be hard-programmed into a microprocessor chip. This is particularly possible when the components are physically distributed across a network, with each component located on a single computer system within the network.In another embodiment of the system's architectural framework, the components are implemented as a combination of hardware and software, i.e., some building blocks are hard-coded into a microprocessor chip, while other building blocks are implemented in software that can be located either in a microprocessor chip or in the cloud.

[0013] The AI ​​model (11) is a generative AI model designed to receive a prompt (I) and generate a visual media output. Such AI models take a natural language description as input and generate an image or video that matches this description. Some well-known text-to-media AI generators are Jasper.ai Art, Dall-E, Midjourney, Stable Diffusion, Starry Al, etc.

[0014] The transformation module (12) is configured to transform the visual media output (A) into multiple secondary outputs. The transformation module (12) may perform transformations including, but not limited to, at least one or more of the spatial, pixel-based, random additive noise transformation. Spatial transformations include rotating, distorting, and scaling the visual media output. Pixel transformations transform the color, hue, and saturation of the visual media output (A). The random additive noise transformation adds calculated noise across all pixels in the visual media output.

[0015] The media-to-text AI models (14) are also known as reverse subtitling models. They attempt to generate subtitles / descriptions for the image / video inputs fed to them. The media-to-text AI model (14) is designed to receive the multiple secondary outputs (B) generated by the transformation module (12). They process the multiple secondary outputs to generate multiple tertiary outputs (C). The tertiary outputs (C) include a textual description of the visual media in the secondary output.

[0016] The processor (16) is a logic circuit or a software program that responds to and processes logical instructions to produce a meaningful result. A hardware processor (16) may be implemented in the system (10) as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, one or more microchips or integrated circuits interconnected using a motherboard, hard-wired logic, software stored by a memory device and executed by a microprocessor, firmware, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and / or any component that acts on signals based on operating instructions.In the present invention, the processor (16) includes a comparator. The comparator may be a conventional electronic comparator or a specialized electronic comparator that is either embedded in neural networks or executes another AI model to extend or enhance their functions.

[0017] The processor (16) is configured to perform a comparative analysis for the tertiary outputs C (i.e., textual description of the visual media in the secondary output) and the received prompts (I). The processor (16) detects exploitation of the AI ​​model based on the comparative analysis. The underlying concept is further explained using method steps (200). When performing the comparative analysis, a multi-label classifier is used that is trained to classify the multiple labels based on the important keywords in the textual description and the prompts (I). The processor is further configured to perform a loss calculation as well as a word error rate and similarity measurement for the textual description and the prompts (I).

[0018] As used in this patent application, the terms "component," "model," "module," and "interface" are intended to refer to a computer-related entity, or to an entity related to or part of an operating device with one or more specific functionalities, where such entities may be hardware, a combination of hardware and software, software, or executing software. As yet another example, interfaces may include input / output (I / O) components, as well as associated processor, application, or API (Application Programming Interface) components.

[0019] It should be understood at the outset that, although example embodiments are illustrated in the figures and described below, the present disclosure is in no way intended to be limited to the example implementations and techniques illustrated in the drawings and described below.

[0020] Fig. Figure 2 illustrates method steps for detecting the exploitation of an AI model (11). The AI ​​model (11) is located, as in accordance with Fig. 1, in the system (10). The system (10) comprises the AI ​​model (11), a transformation module (12), a media-to-text AI model (14), and at least one processor (16). It is again pointed out that the AI ​​model is configured to receive an input prompt (I) and generate a visual media output (A).

[0021] Method step 201 comprises transforming the visual media output (A) into a plurality of secondary outputs (B) using a transformation module (12). The transformations include, but are not limited to, at least one or more of the spatial, pixel-based, random additive noise transformation. Spatial transformations include rotating, distorting, and scaling the visual media output. In pixel transformations, the color, hue, and saturation of the visual media output (A) are transformed. In the random additive noise transformation, calculated noise is added across all pixels in the visual media output. Therefore, the secondary outputs are transformed visual media outputs.

[0022] Method step 202 comprises processing the plurality of secondary outputs (B) to generate a plurality of tertiary outputs (C) using a media-to-text AI model (14). The tertiary outputs include a textual description of the visual media in the secondary output. In other words, the tertiary outputs are a textual description of the transformed visual media outputs.

[0023] Method step 203 includes performing a comparative analysis of the tertiary outputs (C) and the received input requests (I) to detect exploitation of the AI ​​model by a processor (16). The processor (16) can perform further reactive measures, such as blocking the output at the output interface (20), if exploitation of the AI ​​model is detected.

[0024] When performing the comparative analysis, the processor (16) uses a multi-label classifier trained to classify the multiple labels based on the important keywords in the textual description of the secondary output (B) and the prompts (I). For example, a prompt (I) is "A bird is sitting in a tree." Now, for this prompt, the processor (16) has the following analysis: Class: Bird, Confidence: 0.998, Class: Tree, Confidence: 0.991, Class: Chair, Confidence: 0.453. Now, when analyzing the textual description of the secondary output (B), it looks for the classes Bird and Tree as the most important keywords. If the textual description of the secondary output (B) does not contain the keywords Bird or Tree, it concludes that the AI ​​model has been exploited.

[0025] When performing the comparative analysis, the processor is further configured to perform a loss calculation, word error rate, and similarity measurement for the textual description and prompts (I). The loss calculation and word error rate measure the qualitative and quantitative differences between the prompt and the tertiary output (C) (i.e., the textual description of the secondary output (B)). For example, consider the prompt "A bird sits on a tree" and the tertiary output "An eagle sits on a tree"; here, there are two word substitutions (a-a [in English, "a" bird vs. "an" eagle], bird-eagle), resulting in a word error rate of 2 / 6. The loss calculation measures the loss per word and as a whole; for example, the loss between bird and eagle would be lower because they belong to the same class.The similarity measurement calculates the loss in a 3D spatial vector space and is displayed graphically.

[0026] The core idea behind this invention is that for non-manipulative prompts or prompts that do not aim to exploit the AI ​​model (11), the textual description of the transformed visual media output (A) does not differ significantly from the prompt (I). The transformations for the visual media output (A) extend or enhance these hidden features that are characteristic of an attack vector / exploitation attempt with regard to an AI model. These features are then reflected in the textual description. Finally, the processor (16) compares the original prompt (I) with the textual description. If any anomalies or significant differences are found, the processor (16) concludes that an attempt was made to exploit the AI ​​model.In one embodiment of the present invention, the processor (16) also monitors attack patterns as a reference for future purposes.

[0027] There are several real-world, critical use cases for the proposed system and method. Content Generation Platforms: Content generation platforms that use AI models to generate articles, images, or videos based on user input can use this system to ensure that the generated content is not exploited or tampered with. It can help maintain the integrity of the content generation process by ensuring that the generated content is consistent with the user's original intent and free from tampering. Social Media Platforms: Social media platforms that use artificial intelligence (AI) to generate image or video descriptions can use this system to prevent malicious users from exploiting AI to generate inappropriate or misleading content.This technology can protect the quality of platform content by ensuring that user-generated media descriptions match the actual content and comply with platform policies.

[0028] Those skilled in the art will recognize that, although these method steps describe only a series of steps for achieving the objectives, these methodologies can be implemented with modifications to the system (10) described herein. It should be understood that the embodiments discussed in the foregoing detailed description are for illustrative purposes only and do not limit the scope of the present invention. Any variations and adaptations to the system (10) and methods for detecting exploitation of the AI ​​model are contemplated and constitute a part of this invention. The scope of this invention is limited only by the claims.

Claims

[1] A system (10) for detecting the exploitation of an AI model (11), wherein the AI model (11) is configured to receive an input request (I) and generate a visual media output (A), the system (10) comprising: a transformation module (12) configured to transform the visual media output (A) into a plurality of secondary outputs (B); a media-to-text AI model (14) designed to: Receiving the multiple secondary outputs; processing the multiple secondary outputs to generate multiple tertiary outputs (C); a processor (16) designed to: Conducting a comparative analysis on the tertiary outputs (C) and the received prompts (I); Detecting exploitation of the AI model (11) based on the comparative analysis. [2] The system (10) for detecting exploitation of an AI model (11) of claim 1, wherein the transformations include, but are not limited to, at least one or more of the spatial pixel-based random additive noise transformation. [3] A system (10) for detecting exploitation of an AI model (11) according to claim 1, wherein the tertiary outputs comprise a textual description of the visual media in the secondary output. [4] A system (10) for detecting exploitation of an AI model (11) according to claim 1, wherein the processor (16) uses a multi-label classifier when performing the comparative analysis that is trained to classify the multiple labels based on the important keywords in the textual description of the secondary output (B) and the input prompts (I). [5] A method (200) for detecting the exploitation of an AI model (11), wherein the AI model (11) is configured to receive an input prompt (I) and generate a visual media output, the method comprising: Transforming the visual media output (A) into multiple secondary outputs by means of a transformation module (12); Processing the plurality of secondary outputs (B) to generate a plurality of tertiary outputs (C) using a media-to-text AI model (14); Performing a comparative analysis for the tertiary outputs (C) and the received input requests (I) to detect exploitation of the AI model by means of a processor (16). [6] A method (200) for detecting exploitation of an AI model (11) according to claim 5, wherein the transformations include, but are not limited to, at least one or more of the spatial, pixel-based, random additive noise transformation. [7] A method (200) for detecting exploitation of an AI model (11) according to claim 5, wherein the tertiary outputs comprise a textual description of the visual media in the secondary output. [8] A method (200) for detecting the exploitation of an AI model (11) according to claim 5, wherein the processor (16) uses a multi-label classifier when performing the comparative analysis, which is trained to classify the multiple labels based on the important keywords in the textual description of the secondary output (B) and the input prompts (I). [9] A method (200) for detecting the exploitation of an AI model (11) according to claim 5, wherein the processor, when performing a comparative analysis, is further configured to perform a loss calculation and a word error rate and similarity measurement for the textual description and the input prompts (I).

Images