A method and framework of identifying a vulnerable ai model in a repository
Patent Information
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2024-07-26
- Publication Date
- 2026-06-24
AI Technical Summary
Existing technologies face challenges in identifying and scanning artificial intelligence (AI) models in repositories for vulnerabilities, as traditional file repository scanning tools are not specialized in checking AI models and internal contents, and there is a need for a method to detect changes within AI models to determine the necessity of a complete repository scan or individual model scans.
A method and framework that involves searching for AI model files in a repository, designating a previously scanned model as a reference, comparing checksums, executing the model, comparing internal architectures, and identifying vulnerabilities based on pre-determined thresholds and acceptable limits.
This solution enables the automatic identification of vulnerable AI models, streamlines vulnerability scanning, and provides comprehensive reporting, thereby enhancing the efficiency and effectiveness of AI model security assessments.
Smart Images
Figure EP2024071368_27022025_PF_FP_ABST
Abstract
Description
COMPLETE SPECIFICATIONTitle of the Invention:A method and framework of identifying a vulnerable Al Model in a repositoryComplete Specification:The following specification describes and ascertains the nature of this invention and the manner in which it is to be performed.Field of the invention
[0001] The present disclosure relates to the field of artificial intelligence (Al) and Al security along with cybersecurity. In particular, the present invention discloses method and framework of identifying a vulnerable Al Model in a repository.Background of the invention
[0002] With the advent of data science, data processing and decision-making systems are implemented using artificial intelligence modules. The artificial intelligence modules use different techniques like machine learning, neural networks, deep learning etc. Most of the Al based systems, receive large amounts of data and process the data to train Al models. Trained Al models generate output based on the use cases requested by the user. Typically, the Al systems are used in the fields of computer vision, speech recognition, natural language processing, audio recognition, healthcare, autonomous driving, manufacturing, robotics etc. where they process data to generate required output based on certain rules / intelligence acquired through training.
[0003] To process the inputs and give a desired output, the Al systems use various models / algorithms which are trained using the training data. Once the Al system is trained using the training data, the Al systems use the models to analyze the real time data and generate appropriate result. The models may be fine-tuned in real-time based on the results. The models in the Al systems form the core of the system. Lots of effort, resources (tangible and intangible), and knowledge goes into developing these models.
[0004] Companies working on Al development are having a tough time identifying models in the repositories and then configuring and triggering API calls for vulnerability analysis of Models and Artefacts. Repositories contain trained models, python scripts, package managers and other key artefacts necessary for model development and deployment. Traditional file repository scanning tools do not specialize in checking Al models and the internal contents. Additionally, identification of trivial and substantial changes within the Al Models is a key in deciding complete repo scan vs individual models’ scans. There is a need for a method to identify or discover Al models in the repository that require scanning afresh.Brief description of the accompanying drawings
[0005] An embodiment of the invention is described with reference to the following accompanying drawings:
[0006] Figure 1 depicts a framework (10) identifying a vulnerable Al Model;
[0007] Figure 2 illustrates method steps for identifying a vulnerable Al Model in a repository (16);
[0008] Figure 3 is a process flow diagram for the method step 200.Detailed description of the drawings
[0009] Figure 1 depicts a framework (10) identifying a vulnerable Al Model. The framework (10) comprises a repository (16) in communication with an inpul / output interface via a processor (14). The repository (16) refers to the centralized digital storage that developers used to make and manage changes to an Al model(s). The repository (16) comprises said Al model(s), associated artefacts, model files, configuration files and the like collection of objects. For example, the repository contains “n” number of Al models, each having different number of layers (say M, ni,n2and the like). These Al models may be different versions of one original Al model M that has been altered differently by different developers.
[0010] An Al Model is a program or algorithm that may be embedded in a specialized hardware, that utilizes a set of datasets to recognize certain patterns. Using these models and the data from these models, correlations can be established between different types of data to arrive at some logical understanding of the data. A person skilled in the art would be aware of the different types of Al models such as linear regression, naive bayes classifier, support vector machine, neural networks and the like. It must be understood that this disclosure is not specific to the type of model being executed in the Al module and can be applied to any Al module irrespective of the Al model being executed.
[0011] A person skilled in the art will also appreciate that the Al module may be implemented as a set of software instructions, combination of software and hardware or any combination of the same. For example, a neural network is embodied within an electronic chip. Such neural network chips are specialized silicon chips, which incorporate Al technology and are used for machine learning. Other Al models such as linear regression, naive bayes classifier, support vector machine can be a set of software instructions residing in a cloud.
[0012] The processor (14) is configured to exchange and manage the processing of information between the components of the framework (10) such as the repository (16) and input-output interface (12). The processor (14) may be implemented as any or acombination of one or more microchips or integrated circuits interconnected using a parent board, hardwired logic, software stored by a memory device and executed by a microprocessor, firmware, an application specific integrated circuit (ASIC), and / or a field programmable gate array (FPGA).
[0013] In particular the processor (14) is configured to : search for Al model files in the repository (16); designate a previously scanned Al model as the reference Al model; compare the checksum of at least one searched Al model with the checksum of the reference Al model; execute the searched Al model if the checksum of the searched Al model is different from the reference Al model; compare the internal architecture of the searched Al model with the reference Al model in dependance of the successful execution of the searched Al model; identify the searched Al model as vulnerable in dependence of the said comparison.
[0014] Further, the processor (14) compares the internal architecture by comparing the number of layers and dimension of each layer of the searched Al model with the reference Al model. The processor (14) performs a vulnerability analysis on the searched Al model if the number of layers and dimension of the searched Al model differs with the reference Al model beyond a pre-determined threshold. Furthermore, the processor (14) further compares the internal architecture by : comparing the weights and attributes of each layer of the searched Al model with reference Al model; calculating a distance between the weights using mean square error. The processor (14) performs a vulnerability analysis on the searched Al model if the calculated distance is beyond an acceptable limit.
[0015] The input-output interface (12) is a combination of software and hardware both that receives input from at least one user and displays the corresponding output. The input-output interface (12) displays the list of Al models identified as vulnerable. It further displays the results of vulnerability assessment performed on each of the Al model (s).
[0016] As used in this application, the terms "component," "framework (10)," "module," "interface," are intended to refer to a computer-related entity or an entity related to, or that is part of, an operational apparatus with one or more specific functionalities, wherein such entities can be either hardware, a combination of hardware and software, software, or software in execution. As further yet another example, interface(s) can include input / output (I / O) components as well as associated processor, application, or Application Programming Interface (API) components. The framework (10) could be a hardware combination of these modules or could be deployed remotely on a cloud or server.
[0017] It should be understood at the outset that, although exemplary embodiments are illustrated in the figures and described below, the present disclosure should in no way be limited to the exemplary implementations and techniques illustrated in the drawings and described below.
[0018] Figure 2 illustrates method steps for identifying a vulnerable Al Model in a repository (16). The framework (10) on the repository (16) that is used to implement these method steps has been explained in accordance with figure 1 . For the purposes of clarity, it is reiterated that the framework (10) comprises the repository (16) that is in communication with an inpul / output interface via a processor (14).
[0019] Method step 201 comprises searching for Al model files in the repository (16) by means of the processor (14). In this method step all files, artefacts and collection of like objects in the repository (16) are scanned based on their file extension (for example .pdf, .doc, etc.). Files that are actually Al models will be listed out separately.
[0020] Method step 202 comprises designating a previously scanned Al model as the reference Al model. The reference Al model refers to the Al model predecessor to the searched Al model against which the searched Al model will be compared. The reference model is the previous (snapshot in time) saved version of the model in the repository. Typically, a developer would work on this reference Al model for improvements and updates and save the next version to the repository. In absence of areference Al model, the searched model will be identified as vulnerable and processed appropriately.
[0021] Method step 203 comprises comparing the checksum of at least one searched Al model with the checksum of the reference Al model by means of the processor (14). A checksum is a unique sequence of numbers and letters assigned to every version of every file that is uploaded on the repository (16). By checking the checksum, we basically check if searched Al model is same as the original / reference Al model or not. Every time the Al model is altered a new checksum is created for each version.
[0022] Method step 204 comprises executing the searched Al model if the checksum of the searched Al model is different from the reference Al model. This is basically to ascertain the legitimacy of the searched Al model i.e. to validate that the searched Al model is not corrupt.
[0023] Method step 205 comprises comparing the internal architecture of the searched Al model with the reference Al model in dependance of the successful execution of the searched Al model by means of the processor (14). Comparing the internal architecture follows a top-down approach i.e. first we do comparison on the overall architecture then within each level of the Al model. Method step 206 comprises identifying the searched Al model as vulnerable in dependence of the said comparison by means of the processor (14).
[0024] The first level of comparison comprises comparing the number of layers and dimension of each layer of the searched Al model with the reference Al model. A vulnerability analysis is performed or recommended to the user via the input-output interface (12) on the searched Al model if the number of layers and dimension of the searched Al model differs with the reference Al model beyond a pre-determined threshold.
[0025] The second level of comparison of the internal architecture comprises comparing the weights and attributes of each layer of the searched Al model with reference Al model. A vulnerability analysis is performed or recommended to the uservia the input-output interface (12) on the searched Al model if the calculated distance is beyond an acceptable limit.
[0026] Figure 3 is a process flow diagram for the method step 200. The model input or Mi is searched Al model. If the checksum of Mi matches with that of Mref, we exit the process i.e. there has been no change in the reference Al model or the searched Al model is same as the reference Al model, hence no vulnerability assessment is warranted. If the checksum differs we proceed with sanity check that is method step 204. If the execution fails that means the searched Al model is corrupt, hence no vulnerability assessment is warranted. On a successful execution, we proceed to method step 205.
[0027] First the overall Al model architecture is compared as mentioned in para
[0024] . If no substantial change is detected, it mean the searched Al model has not altered significantly, Hence, no vulnerability assessment is warranted. If a substantial change is detected, the internals of the model architecture is compared as mentioned in para
[0025] . Finally, based on this comparison we know whether or not a vulnerability assessment is warranted.
[0028] A person skilled in the art will appreciate that while these method steps describes only a series of steps to accomplish the objectives, these methodologies may be implemented modifications to the method step and customization to the framework (10).
[0029] This idea to develop a method and framework (10) of identifying a vulnerable Al Model in a repository (16) can automatically identify models, their formats, and metadata within an organization’s complex collection of Al models in repository (16). It can further help to automatically trigger an API call when a new model is discovered or when an existing model undergoes a significant change. Additionally, it provides comprehensive reporting detailing the overview of all models present in the repository (16). This method will streamline and automate the vulnerability scanning process that will make further adversarial vulnerability assessment of the Al models smooth.
[0030] It must be understood that the embodiments explained in the above detailed description are only illustrative and do not limit the scope of this invention. Any modification to the method and framework (10) of identifying a vulnerable Al Model in repository (16) are envisaged and form a part of this invention. The scope of this invention is limited only by the claims.
Claims
We Claim:1 . A method (200) of identifying a vulnerable Al Model in a repository (16), said Al model stored in a repository (16) amongst other artefacts and collection of objects, the repository (16) in communication with a processor (14), the method comprising: searching (201) for Al model files in the repository (16) by means of the processor (14), characterized in that method:Designating (202) a previously scanned Al model as the reference Al model; comparing (203) the checksum of at least one searched Al model with the checksum of the reference Al model by means of the processor (14); executing (204) the searched Al model if the checksum of the searched Al model is different from the reference Al model; comparing (205) the internal architecture of the searched Al model with the reference Al model in dependance of the successful execution of the searched Al model by means of the processor (14); identifying (206) the searched Al model as vulnerable in dependence of the said comparison by means of the processor (14).
2. The method (200) of identifying a vulnerable Al Model in a repository (16) as claimed in claim 1 , wherein comparing (205) the internal architecture comprises comparing the number of layers and dimension of each layer of the searched Al model with the reference Al model.
3. The method (200) of identifying a vulnerable Al Model in a repository (16) as claimed in claim 1 , wherein a vulnerability analysis is performed on the searched Al model if the number of layers and dimension of the searched Al model differs with the reference Al model beyond a pre-determined threshold.
4. The method (200) of identifying a vulnerable Al Model in a repository (16) as claimed in claim 1 , wherein comparing (205) the internal architecture further comprises: comparing the weights and attributes of each layer of the searched Al model with reference Al model; calculating a distance between the weights using mean square error.
5. The method (200) of identifying a vulnerable Al Model in a repository (16) as claimed in claim 1 , wherein a vulnerability analysis is performed on the searched Al model if the calculated distance is beyond an acceptable limit.
6. A framework (10) for identifying a vulnerable Al Model in a repository (16), the framework (10) comprising a repository (16) in communication with at least a processor (14), the framework (10) characterized by the processor (14) configured to: search for Al model files in the repository (16): designate a previously scanned Al model as the reference Al model; compare the checksum of at least one searched Al model with the checksum of the reference Al model; execute the searched Al model if the checksum of the searched Al model is different from the reference Al model; compare the internal architecture of the searched Al model with the reference Al model in dependance of the successful execution of the searched Al model; identify the searched Al model as vulnerable in dependence of the said comparison.
7. The framework (10) for identifying a vulnerable Al Model in a repository (16) as claimed in claim 6, wherein the processor (14) compares the internal architecture by comparing the number of layers and dimension of each layer of the searched Al model with the reference Al model.
8. The framework (10) for identifying a vulnerable Al Model in a repository (16) as claimed in claim 6, wherein the processor (14) performs a vulnerability analysis on the searched Al model if the number of layers and dimension of the searched Al model differs with the reference Al model beyond a pre-determined threshold.
9. The framework (10) for identifying a vulnerable Al Model in a repository (16) as claimed in claim 6, wherein the processor (14) further compares the internal architecture by : comparing the weights and attributes of each layer of the searched Al model with reference Al model; calculating a distance between the weights using mean square error.
10. The framework (10) for identifying a vulnerable Al Model in a repository (16) as claimed in claim 6, wherein the processor (14) performs a vulnerability analysis on the searched Al model if the calculated distance is beyond an acceptable limit.