AI model security vulnerability detection system based on model output consistency analysis

Abstract

This invention discloses an AI model security vulnerability detection system based on model output consistency analysis. It includes an input perturbation module responsible for generating semantically equivalent diverse input variants to simulate potential attack scenarios and trigger inconsistent behavior in the AI ​​model; a multimodal output collector responsible for collecting multimodal output data of the AI ​​model after being subjected to input perturbations; a consistency analysis engine that detects abnormal inconsistent patterns in the model output through statistical analysis and machine learning methods; and a vulnerability localization and visualization module that locates security vulnerabilities in the AI ​​model based on the abnormal information detected by the consistency analysis engine, determines the location and related modules of the vulnerability, and visualizes the vulnerability information. This invention, employing the aforementioned AI model security vulnerability detection system based on model output consistency analysis, achieves multi-dimensional coverage, dynamically adapts to model scenario updates, improves the detection rate of hidden vulnerabilities, reduces detection latency, and enhances real-time performance and broad applicability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, and in particular to an AI model security vulnerability detection system based on model output consistency analysis. Background Technology

[0002] In recent years, artificial intelligence (AI) technology has been widely applied in fields such as computer vision, natural language processing, autonomous driving, and medical diagnosis. However, with the increasing complexity of AI models and the expansion of application scenarios, the security threats they face are becoming increasingly severe. Research shows that current mainstream deep learning models (such as CNN and Transformer) have multiple security vulnerabilities. Attackers can manipulate model behavior through adversarial examples, backdoor implantation, model theft, and data leakage, leading to serious security risks.

[0003] For example, adversarial attacks: Attackers construct adversarial examples through subtle perturbations (such as FGSM and PGD attacks) to cause the model to misclassify. For instance, in autonomous driving scenarios, slightly modifying road sign pixels can lead to traffic sign recognition errors. Backdoor attacks: Malicious triggers (such as specific pixel blocks or keywords) are implanted during the training phase, causing the model to output results preset by the attacker under specific inputs. These attacks are highly covert and difficult to detect in conventional testing. Model theft: Attackers reconstruct model parameters through API queries (such as black-box access), stealing the intellectual property of commercial models. Methods such as Jacobian-based architecture extraction have proven that the model structure can be reproduced with a limited number of queries. Data leakage: Training data is inferred from model output or gradient information (such as member inference attacks), leading to the exposure of private data. The 2023 MITRE assessment report shows that the average failure rate of commercial AI models in adversarial example testing is as high as 61%, highlighting the urgency of security protection.

[0004] Currently, AI security detection methods mainly include static analysis, dynamic testing, formal verification, and adversarial training defense. However, static analysis methods rely on code auditing and architecture verification (such as the CleverHans vulnerability detection library), which can only discover problems at the model structure level and cannot detect dynamic vulnerabilities related to training data (such as data poisoning). Dynamic testing methods use black-box fuzzing (such as DeepXplore's neuron coverage guidance) to generate random perturbation inputs to trigger anomalous behavior. However, the test cases generated by such methods differ significantly from real attack patterns, and experiments show that they can only detect 28% of backdoor behaviors. Formal verification proves the robustness of the model through mathematical constraints (such as the Marabou verification framework), but due to computational complexity, it is only suitable for small fully connected networks and is difficult to handle modern large-scale deep learning models. Adversarial training defense injects adversarial examples during the training phase (such as the Madry defense framework) to improve model robustness. However, the IEEE S&P 2022 research points out that this method can lead to a 5% to 15% decrease in the model's accuracy in normal scenarios, affecting practical usability.

[0005] While various AI security detection technologies exist, existing methods often focus on the input-output mapping relationship while neglecting the consistency analysis of internal model features (such as attention mechanisms and gradient distributions). This single detection dimension makes it difficult to discover hidden vulnerabilities. Furthermore, there is a lack of unified quantitative metrics to assess the security status of models, making it difficult to compare results from different detection methods. Traditional detection methods (such as formal verification) are far more time-consuming than model inference time, lacking real-time performance and failing to meet the demands of high real-time scenarios (such as autonomous driving).

[0006] In summary, current AI security detection technologies still face challenges such as low coverage, poor generalization, and insufficient real-time performance, urgently requiring an innovative solution that can comprehensively analyze the multi-dimensional output of models while balancing detection accuracy and efficiency. This invention proposes a vulnerability detection system based on model output consistency analysis, aiming to fill this technological gap. Summary of the Invention

[0007] The purpose of this invention is to provide an AI model security vulnerability detection system based on model output consistency analysis, which can achieve multi-dimensional coverage, dynamically adapt to model scenario updates, improve the detection rate of hidden vulnerabilities, reduce detection latency, and improve real-time performance and wide applicability.

[0008] To achieve the above objectives, this invention provides an AI model security vulnerability detection system based on model output consistency analysis, including an input perturbation module, a multimodal output collector, a consistency analysis engine, and a vulnerability location and visualization module; The input perturbation module is responsible for generating semantically equivalent diverse input variants to simulate potential attack scenarios and trigger inconsistent behavior of the AI ​​model; The multimodal output acquisition unit is responsible for collecting the multimodal output data of the AI ​​model after it has been subjected to input perturbations; The consistency analysis engine uses statistical analysis and machine learning methods to detect abnormal inconsistencies in the model output. The vulnerability localization and visualization module locates security vulnerabilities in the AI ​​model based on the anomaly information detected by the consistency analysis engine, determines the location of the vulnerability and the related modules, and visualizes the vulnerability information.

[0009] Preferably, the input perturbation module includes a data parsing and type identification unit, a perturbation strategy library and selection unit, a perturbation execution engine, a perturbation intensity control unit, and a perturbation data verification unit; The data parsing and type recognition unit first parses the format of the raw data input to the AI ​​model; then extracts the feature parameters of the data; and finally establishes a mapping relationship between data types and perturbation strategies. The perturbation strategy library and selection unit are divided into a basic perturbation library and an advanced attack perturbation library. Perturbation combination strategies are selected according to the AI ​​model type, application scenario and test target. The perturbation execution engine deploys dedicated perturbation executors, among which the image perturbation executor is based on OpenCV to implement pixel-level geometric transformations and noise injection; the text perturbation executor is based on an NLP toolkit to implement word-level / sentence-level feature adjustment; and the audio perturbation executor is based on librosa to implement spectral features and waveform perturbation. The disturbance intensity control unit performs quantification and classification of disturbance intensity, where the data disturbance difference is as follows: ; in, This is the original data; The data is after the disturbance; It is an L2 norm; To find the minimum value, prevent the denominator from being 0; The data perturbation difference is defined, ranging from [0,1].

[0010] Preferably, the multimodal output acquisition unit includes a modality recognition and adaptation unit, a distributed acquisition execution unit, a data preprocessing and standardization unit, and a data association and synchronization unit; The modality recognition and adaptation unit first automatically identifies the modality type of the model output by using the metadata of the input data, thus realizing modality type detection; then, it automatically selects the appropriate acquisition protocol for the output interface of the AI ​​model; finally, the configuration information is synchronized to the acquisition execution unit in real time to realize dynamic configuration management. The distributed data acquisition and execution unit first constructs a multi-threaded data collector, allocating independent data acquisition threads for each modality; then, for AI models deployed on edge devices, a lightweight data acquisition agent is deployed to achieve data acquisition at edge nodes; finally, a priority queue mechanism is used to allocate data acquisition resources. The data preprocessing and standardization unit first converts the raw output data of different modalities into a unified intermediate format; then, it extracts the metadata of each modal output; finally, it removes invalid output data and filters the data through validation rules, triggering an alarm when the proportion of invalid data is greater than 10%. The data association and synchronization unit first adds a high-precision timestamp to each acquired multimodal output data and calibrates the clocks of each acquisition node through the NTP protocol to ensure time consistency across modal data. Then, it generates a globally unique output data ID based on the input perturbation ID, model version number, and timestamp to achieve the association and traceability of multimodal output data. Finally, for asynchronously output multimodal data, a sliding time window is used for data alignment, and the multimodal data within the window is marked as an associated data group. The storage and indexing unit first adopts a hybrid storage architecture, with structured metadata stored in a MySQL database and unstructured data stored in a distributed file system. Then, indexes are built based on modality type, timestamp, output confidence, and perturbation ID dimensions to support real-time queries. Incremental backups are automatically performed at fixed times every day, and full backups are performed every Sunday. Backup data is stored in an off-site disaster recovery center and retained for 90 days.

[0011] Preferably, the consistency analysis engine includes a data acquisition and integration submodule, a statistical analysis submodule, a machine learning analysis submodule, a cross-modal consistency verification submodule, and a real-time monitoring and feedback submodule; The data acquisition and integration submodule includes a multi-source data access unit, a data standardization processing unit, and a data association indexing unit; The statistical analysis submodule includes a baseline model building unit, an anomaly calculation unit, and a threshold adaptation unit; The machine learning analysis submodule includes a supervised learning unit, an unsupervised learning unit, and a model evaluation unit; The cross-modal consistency verification submodule includes a modal feature extraction unit, a consistency measurement unit, and a conflict detection unit; The real-time monitoring and feedback submodule includes a streaming computing unit, an anomaly response unit, and a log auditing unit.

[0012] Preferably, in the data acquisition and integration submodule, the multi-source data access unit supports real-time access to data sources such as training data pipelines, inference service interfaces, and model parameter servers, and uses a distributed message queue to achieve throughput data buffering to meet terabyte-level data acquisition requirements. The data standardization processing unit uses different processing operations for structured and unstructured data. For structured data, field alignment is achieved through schema mapping, and missing values ​​are filled using the mean or mode method. For unstructured data, images are uniformly converted to RGB format and 256×256 resolution, text is segmented using BPE to unify sequence length, and audio is standardized to a 16kHz sampling rate and mono. The data association index unit establishes a three-dimensional index system of "input data - model output - parameter snapshot", and enables cross-source data association queries through timestamps and request IDs.

[0013] Preferably, in the statistical analysis submodule, the baseline model construction unit uses a sliding window algorithm to update the statistical baseline of the normal output in real time, specifically including: A. Classification task: mean vector μ and covariance matrix Σ of the category probability distribution; B. Regression task: Mean μ and standard deviation σ of the output values; C. Generation task: Generate the distance distribution between the generated samples and the real samples; The anomaly calculation unit designs dedicated anomaly indicators for different task types and outputs a comprehensive anomaly score by combining a multi-indicator fusion strategy. Threshold adaptive unit: The detection threshold is dynamically adjusted based on the ratio of "normal" to "abnormal" samples, and a dual verification mechanism of the 3σ rule and quantile method is adopted.

[0014] Preferably, in the modal feature extraction unit of the cross-modal consistency verification submodule, the text modality uses a BERT-based pre-trained model to extract the 768-dimensional semantic feature vector corresponding to its special symbol [CLS]; the image modality uses a ResNet50 deep convolutional network to obtain a 2048-dimensional visual feature vector after global average pooling; the audio modality first converts the audio signal into a Mel spectrogram, and then extracts a 512-dimensional acoustic feature vector through a convolutional neural network; finally, a lightweight fully connected network is deployed as a modal adapter to uniformly transform the above multimodal features of different dimensions into a 512-dimensional shared feature space; The consistency measurement unit measures directional consistency by calculating cosine similarity; measures numerical closeness by calculating Euclidean distance; and measures statistical correlation by calculating mutual information. The conflict detection unit sets a multimodal decision consistency threshold. When more than 30% of the modal pairs have a similarity below the threshold, it is determined to be a cross-modal conflict. The multimodal decision consistency is as follows: ; in, The number of modes; This is the consensus threshold for multimodal decision-making, with a default value of 0.7. This represents the intermodal consistency function, used to calculate the degree of consistency between the decisions of the i-th mode and the j-th mode; For indicator functions; when When this occurs, a cross-modal conflict alarm is triggered.

[0015] Preferably, in the real-time monitoring and feedback submodule, the streaming computing unit, based on the Flink framework, achieves real-time analysis with millisecond-level latency and supports parallel processing of 10,000+ samples per second. The anomaly response unit is divided into three levels based on the anomaly score: yellow (5-7 points), orange (7-9 points), and red (≥9 points), to achieve graded alarms. Moreover, when a red alarm triggers the model degradation strategy, i.e., switching to the backup model, an orange alarm automatically initiates parameter freezing. The log auditing unit uses blockchain technology to store abnormal event logs, ensuring immutability and traceability.

[0016] Preferably, the vulnerability location and visualization module includes a vulnerability location submodule and a visualization submodule; the vulnerability location submodule includes an anomaly feature extraction unit, a model structure mapping unit, and a vulnerability tracing and reasoning unit; the visualization submodule includes a data processing and conversion unit, a visualization rendering unit, and an interactive control unit.

[0017] Preferably, in the vulnerability location submodule, the anomaly feature extraction unit extracts key anomaly features from the anomaly information output by the consistency analysis engine. The model structure mapping unit establishes the mapping relationship between the internal structure of the AI ​​model and its inputs and outputs; The vulnerability tracing and reasoning unit is based on the mapping relationship between abnormal features and model structure, and uses a reverse reasoning algorithm to trace back from the abnormal output to the internal module or parameter area of ​​the model that caused the vulnerability. In the visualization submodule, the data processing and conversion unit processes and converts the vulnerability location, abnormal data, and other information output by the vulnerability location submodule into a format suitable for visualization; the visualization rendering unit selects an appropriate visualization method and renders the data according to the visualization requirements; and the interaction control unit provides interactive functions, allowing users to operate on the visualization results.

[0018] Therefore, the AI ​​model security vulnerability detection system based on model output consistency analysis described above, as used in this invention, has the following beneficial effects: (1) This invention analyzes logic output, internal features and gradient information at the same time to achieve multi-dimensional coverage and improve the detection rate of hidden vulnerabilities. (2) The present invention has dynamic adaptability, and the baseline is automatically updated according to the model usage scenario, avoiding misjudgment caused by fixed threshold; (3) The present invention uses a lightweight statistical method, which makes the detection delay significantly lower than the model inference time, and has high real-time efficiency; (4) This invention is applicable to a variety of scenarios, such as security auditing of autonomous driving models, adversarial robustness testing of financial risk control AI, and backdoor attack detection of medical diagnostic models, and has wide applicability.

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] Figure 1 This is an architecture diagram of the AI ​​model security vulnerability detection system based on model output consistency analysis, which is based on the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Example like Figure 1 As shown, the AI ​​model security vulnerability detection system based on model output consistency analysis includes an input perturbation module, a multimodal output collector, a consistency analysis engine, and a vulnerability location and visualization module.

[0023] I. Input Disturbance Module.

[0024] The input perturbation module includes a data parsing and type recognition unit, a perturbation policy library and selection unit, a perturbation execution engine, a perturbation intensity control unit, and a post-perturbation data verification unit. As the core front-end component of this system, the input perturbation module is responsible for generating semantically equivalent diverse input variants to simulate potential attack scenarios and trigger inconsistent behavior in the AI ​​model.

[0025] (1) Data parsing and type recognition unit first parses the original data input to the AI ​​model, supporting the recognition of multiple data types such as text, image, audio, and structured data; then extracts key feature parameters of the data, such as the resolution and number of channels of the image; the word vector dimension and sentence length of the text; the sampling rate and duration of the audio; and finally establishes the mapping relationship between data type and perturbation strategy, providing a basis for the selection of subsequent perturbation methods.

[0026] (2) The perturbation strategy library and selection unit are divided into a basic perturbation library and an advanced attack perturbation library.

[0027] The basic perturbation library includes: a subset of noise perturbations (Gaussian noise, salt-and-pepper noise, Poisson noise, etc.), a subset of geometric transformations (rotation, scaling, translation, shearing, flipping, etc.), and a subset of feature adjustments (eigenvalue scaling, offsetting, masking, permutation, etc.).

[0028] The advanced attack perturbation library includes: an adversarial perturbation subset (classic adversarial attack methods such as FGSM, PGD, and CW), a distribution offset perturbation subset (distribution migration perturbation in domain-adaptive scenarios), and a temporal perturbation subset (timestamp offset, sequence truncation / insertion, etc. for time-series data).

[0029] Among them, the adversarial perturbation (FGSM) is as follows: ; ; in, To counteract disturbances, For the disturbance intensity, The loss function with respect to the input The gradient; This is the original data; The data is after the disturbance; It is a symbolic function.

[0030] Based on the type of AI model (such as CNN, RNN, Transformer), application scenario (such as classification, detection, generation) and testing objective, the perturbation combination strategy can be automatically recommended or manually selected.

[0031] (3) The perturbation execution engine deploys dedicated perturbation executors for different data types, supports batch perturbation processing and parallel computing, and improves the efficiency of large-scale data testing. Among them, the image perturbation executor implements pixel-level geometric transformation and noise injection based on OpenCV; the text perturbation executor implements word-level / sentence-level feature adjustment based on NLP toolkit; and the audio perturbation executor implements spectral features and waveform perturbation based on librosa.

[0032] (4) The disturbance intensity control unit realizes the quantitative classification of disturbance intensity (level 1-10), supports linear / nonlinear intensity adjustment, establishes an intensity-impact assessment model, and predicts the possible impact range of different intensity disturbances on the model output.

[0033] The data perturbation difference is shown below: ; in, It is an L2 norm; This is the minimum value (to prevent the denominator from being 0); The data perturbation difference is defined as [0,1]. A larger value indicates a stronger disturbance.

[0034] (5) The data verification unit after perturbation verifies the validity of the data after perturbation to ensure that it meets the input format requirements of the AI ​​model. It calculates the similarity between the data before and after perturbation to avoid invalid perturbation (such as excessive perturbation causing the data to lose semantic information).

[0035] II. Multimodal output acquisition unit.

[0036] The multimodal output acquisition unit is responsible for collecting the multimodal output data of the AI ​​model after it has been subjected to input perturbations. For different types of AI models, such as models involving images, text, and audio, the corresponding output results are collected separately to ensure that comprehensive model output information can be obtained.

[0037] The multimodal output acquisition unit includes a modality recognition and adaptation unit, a distributed acquisition execution unit, a data preprocessing and standardization unit, and a data association and synchronization unit.

[0038] (1) Modality recognition and adaptation unit: First, it automatically recognizes the modality type of the model output by the metadata of the input data (such as file format, data header identifier) ​​to realize modality type detection. It supports mainstream modalities such as images (.jpg, .png, etc.), text (.txt, .json, etc.), audio (.wav, .mp3, etc.), video (.mp4, .avi, etc.) and structured data (table, vector).

[0039] Then, it has built-in interfaces for multiple communication protocols such as HTTP / HTTPS, gRPC, and MQTT, and automatically selects the appropriate acquisition protocol for the output interface of different AI models (such as REST API and Socket service) to ensure the compatibility and stability of data transmission.

[0040] Finally, a visual configuration interface is provided, allowing users to customize the acquisition parameters of each modality (such as sampling frequency, data format, and storage path). The configuration information is synchronized to the acquisition execution unit in real time, realizing dynamic configuration management.

[0041] (2) Distributed acquisition execution unit: First, a multi-threaded acquisition unit is built, and an independent acquisition thread is allocated for each modality. The number of threads can be dynamically adjusted according to the amount of modal data (default number of image threads is 4, text threads are 8, and audio threads are 2) to avoid single-modal data congestion affecting the overall acquisition efficiency.

[0042] Then, for AI models deployed on edge devices, a lightweight data acquisition agent (resource usage ≤50MB) is deployed to achieve edge node data acquisition, support offline caching (maximum cache capacity 10GB), and automatically synchronize data after network recovery.

[0043] Finally, a priority queue mechanism is adopted to allocate collection resources to high-priority tasks (such as output data in real-time detection scenarios). The task priority can be dynamically adjusted according to business needs (levels 1-5, with level 5 being the highest).

[0044] (3) Data preprocessing and standardization unit: First, the original output data of different modalities are converted into a unified intermediate format. Image data is converted into PNG format (compression quality 90%), text data is converted into JSON format with UTF-8 encoding, and audio data is converted into WAV format with a sampling rate of 16kHz.

[0045] Then, key metadata for each modality output is extracted: resolution, color space, confidence level, etc. are extracted from image data; word vector dimension, sentiment value, etc. are extracted from text data; and duration, spectral features, etc. are extracted from audio data.

[0046] Finally, invalid output data (such as empty images and text with a length of 0) is removed, and valid data is filtered by validation rules (such as image pixel value range [0,255], text length ≥ 10 characters). An alarm is triggered when the proportion of invalid data exceeds 10%.

[0047] (4) Data association and synchronization unit: First, add a high-precision timestamp (millisecond level) to each acquired multimodal output data, and calibrate the clock of each acquisition node through the NTP protocol to ensure the time consistency of cross-modal data (time deviation ≤50ms).

[0048] The time synchronization of output data from different modes under the same input disturbance is evaluated using the time deviation, as shown below: ; in, Time deviation (unit: ms); for The timestamp of the modal output data; when If this occurs, it is marked as a synchronization error.

[0049] Then, a globally unique output ID is generated based on the input perturbation ID, model version number, and timestamp to achieve the correlation and traceability of multimodal output data.

[0050] Finally, for asynchronously output multimodal data (such as text output lagging behind image output), a sliding time window (window size 1s) is used for data alignment, and the multimodal data within the window is marked as associated data groups.

[0051] The correctness of multimodal data association is measured by calculating the association accuracy rate, as shown below: ; in, For correlation accuracy; The number of correctly associated multimodal data sets; This represents the total number of associated data groups; when At the same time, optimize the parameters of the correlation algorithm.

[0052] (5) Storage and indexing unit: First, a hybrid storage architecture is adopted. Structured metadata is stored in a MySQL database, and unstructured data (images, audio) is stored in a distributed file system (such as MinIO). Data lifecycle management is supported (default storage is 30 days, which can be configured).

[0053] Then, indexes are built based on dimensions such as modality type, timestamp, output confidence level, and perturbation ID to support fast queries (query response time ≤ 100ms), such as querying "output data of image modality with confidence level < 0.5 between 08:00 and 10:00 on August 19, 2025". Incremental backups are automatically performed every day at 3:00 AM, and full backups are performed every Sunday. Backup data is stored in an off-site disaster recovery center and retained for 90 days.

[0054] III. Consistency Analysis Engine.

[0055] The consistency analysis engine comprises a data acquisition and integration submodule, a statistical analysis submodule, a machine learning analysis submodule, a cross-modal consistency verification submodule, and a real-time monitoring and feedback submodule. The consistency analysis engine uses statistical analysis and machine learning methods to detect abnormal inconsistencies in the model output.

[0056] (1) The data acquisition and integration submodule includes a multi-source data access unit, a data standardization processing unit, and a data association indexing unit.

[0057] The multi-source data access unit supports real-time access to data sources such as training data pipelines, inference service interfaces, and model parameter servers. It uses distributed message queues (such as Kafka) to achieve high-throughput data buffering and adapt to terabyte-level data acquisition requirements.

[0058] The data standardization processing unit uses different processing operations for structured and unstructured data. For structured data, field alignment is achieved through schema mapping, and missing values ​​are handled using mean imputation (for numerical data) or mode imputation (for categorical data). For unstructured data, images are uniformly converted to RGB format and 256×256 resolution, text is standardized in sequence length through BPE segmentation (padding with zeros if insufficient, truncating if too long), and audio is standardized to a 16kHz sampling rate and mono.

[0059] The data association index unit establishes a three-dimensional index system of "input data - model output - parameter snapshot", and enables cross-source data association queries through timestamps and request IDs.

[0060] (2) The statistical analysis submodule includes a baseline model building unit, an anomaly calculation unit, and a threshold adaptation unit.

[0061] The baseline model building unit uses a sliding window algorithm (window size is configurable, default is 10,000 samples) to update the statistical baseline of the normal output in real time, specifically including: A. Classification task: mean vector μ and covariance matrix Σ of the category probability distribution; B. Regression task: Mean μ and standard deviation σ of the output values; C. Generation task: Generate the distance distribution between the generated samples and the real samples (such as the FID value distribution).

[0062] The anomaly calculation unit designs dedicated anomaly indicators for different task types and outputs a comprehensive anomaly score by combining a multi-indicator fusion strategy (weighted voting method).

[0063] Threshold adaptive unit: The detection threshold is dynamically adjusted based on the ratio of "normal" to "abnormal" samples, and a dual verification mechanism is adopted using the 3σ rule (applicable to normal distribution) and the quantile method (applicable to non-normal distribution).

[0064] (3) The machine learning analysis submodule includes a supervised learning unit, an unsupervised learning unit, and a model evaluation unit.

[0065] In the supervised learning unit, the classifier set includes models such as SVM, XGBoost, and lightweight CNN, and ensemble learning (Stacking strategy) is used to improve detection accuracy. Then, statistical features (mean, entropy), temporal features (first-order difference), and high-frequency component features (wavelet transform) of the output vector are extracted; incremental learning mode is supported, and model fine-tuning is triggered every 1000 accumulated labeled data.

[0066] In the unsupervised learning unit, the autoencoder adopts a bottleneck layer structure (1 / 3 of the input dimension), and the reconstruction error threshold is determined by the 95th percentile method; the isolated forest is set with 100 base trees, and the outlier score threshold is set to 0.6 (which can be dynamically adjusted); density clustering uses the DBSCAN algorithm, which automatically optimizes the neighborhood radius ε through the silhouette coefficient.

[0067] The model evaluation unit calculates precision, recall, and F1 score in real time, and automatically triggers model reconstruction when F1 < 0.7.

[0068] (4) The cross-modal consistency verification submodule includes a modal feature extraction unit, a consistency measurement unit, and a conflict detection unit.

[0069] In the modality feature extraction unit, the text modality uses a BERT-based pre-trained model to extract a 768-dimensional semantic feature vector corresponding to its special symbol [CLS]. The image modality uses a ResNet50 deep convolutional network to obtain a 2048-dimensional visual feature vector after global average pooling. The audio modality first converts the audio signal into a Mel spectrogram, and then uses a convolutional neural network (CNN) to extract a 512-dimensional acoustic feature vector. Finally, a lightweight fully connected network is deployed as a modality adapter to uniformly transform the above multimodal features of different dimensions into a 512-dimensional shared feature space.

[0070] The consistency measurement unit measures directional consistency by calculating cosine similarity; measures numerical closeness by calculating Euclidean distance; and measures statistical correlation by calculating mutual information.

[0071] The conflict detection unit sets a multimodal decision consistency threshold (default 0.7). When more than 30% of the modal pairs have a similarity below the threshold, it is judged as a cross-modal conflict.

[0072] Multimodal decision consensus is shown below: ; in, The number of modes; This is the consensus threshold for multimodal decision-making, with a default value of 0.7. This represents the intermodal consistency function, used to calculate the degree of consistency between the decisions of the i-th mode and the j-th mode; For indicator functions (1 if true, 0 otherwise); when When this occurs, a cross-modal conflict alarm is triggered.

[0073] (5) The real-time monitoring and feedback submodule includes a streaming computing unit, an anomaly response unit, and a log auditing unit.

[0074] The streaming computing unit, based on the Flink framework, enables real-time analysis with millisecond-level latency and supports parallel processing of 10,000+ samples per second.

[0075] The anomaly response unit categorizes anomalies into three levels (yellow 5-7 points, orange 7-9 points, and red ≥9 points) based on anomaly scores (0-10 points), enabling tiered alarms. Furthermore, when a red alarm triggers a model degradation strategy, i.e., switching to a backup model, an orange alarm automatically initiates parameter freezing.

[0076] The log auditing unit uses blockchain technology to store abnormal event logs (including input data hashes, abnormal characteristics, and processing results) to ensure immutability and traceability.

[0077] IV. Vulnerability Location and Visualization Module.

[0078] The vulnerability localization and visualization module includes a vulnerability localization submodule and a visualization submodule. Based on anomaly information detected by the consistency analysis engine, the module locates security vulnerabilities in the AI ​​model, determining the approximate location of the vulnerability and related modules. It then presents the vulnerability information in an intuitive visualization format, such as generating vulnerability distribution heatmaps and outputting anomaly data comparison charts, facilitating user viewing and analysis.

[0079] (1) The vulnerability location submodule includes an anomaly feature extraction unit, a model structure mapping unit, and a vulnerability tracing and reasoning unit.

[0080] The anomaly feature extraction unit extracts key anomaly features from the anomaly information output by the consistency analysis engine, such as the type of input perturbation corresponding to the anomaly output and the inconsistency feature vector of the multimodal output.

[0081] The model structure mapping unit establishes the mapping relationship between the internal structure of the AI ​​model (including each network layer, module component, etc.) and its inputs and outputs. For example, for a neural network model, it records information such as the input and output dimensions, parameter range, and the contribution weight of each layer to the final output.

[0082] The vulnerability tracing and inference unit, based on the mapping relationship between anomaly characteristics and model structure, employs a reverse inference algorithm to trace back from the anomaly output to the internal modules or parameter regions of the model that may contain vulnerabilities. For example, using the idea of ​​gradient backpropagation, it calculates the gradient contribution of the anomaly at each layer of the model to determine the approximate location of the vulnerability.

[0083] The reliability of vulnerability location results is measured by calculating the confidence level of the vulnerability location, as shown below: ; in, Confidence level for vulnerability location; This is the number of relevant features used to calculate the confidence level; For the first The weights of each feature (the weights are determined based on the importance of the feature to vulnerability location; the higher the importance, the greater the weight). For the first The matching score between each feature and the vulnerability location (the score ranges from 0 to 1, with higher scores for better matching).

[0084] A linear mapping method is used to map the severity of vulnerabilities or related numerical values ​​to colors on a heatmap, as shown below: ; in, The final mapped color value (which can be represented as a value in a color space such as RGB or HSV); This represents the minimum value in the range of color values. This represents the maximum value within the range of color values. The values ​​are related to the current vulnerability; This represents the minimum value related to the vulnerability. This represents the maximum value related to the vulnerability.

[0085] The degree of anomalous difference between different data is quantified by calculating the difference between anomalous data, as shown below: ; in, For the degree of difference; The number of feature dimensions of the data; For the first set of data, the first One eigenvalue; For the second set of data, the first One eigenvalue; It is a minimum value (to prevent the denominator from being 0).

[0086] (2) The visualization submodule includes a data processing and conversion unit, a visualization rendering unit, and an interactive control unit.

[0087] The data processing and conversion unit processes and converts the vulnerability location, abnormal data and other information output by the vulnerability location submodule into a format suitable for visualization, such as converting the model layer structure into drawable graphic elements and converting abnormal values ​​into color-coded values.

[0088] The visualization rendering unit selects and renders appropriate visualization methods based on different visualization needs. For example, it uses heatmap rendering technology to generate vulnerability distribution heatmaps and utilizes chart drawing libraries to generate anomaly data comparison charts.

[0089] The interactive control unit provides interactive functions, enabling users to manipulate the visualization results, such as zooming and panning the heatmap, and selecting to view anomaly details of specific modules.

[0090] Therefore, this invention employs the aforementioned AI model security vulnerability detection system based on model output consistency analysis, simultaneously analyzing logical output, internal features, and gradient information to achieve multi-dimensional coverage and improve the detection rate of hidden vulnerabilities. This invention features dynamic adaptability, with the baseline automatically updated according to the model's usage scenario, avoiding false positives caused by fixed thresholds. It utilizes lightweight statistical methods, resulting in detection latency significantly lower than model inference time, demonstrating high real-time efficiency. This invention is applicable to various scenarios such as security auditing of autonomous driving models, adversarial robustness testing of financial risk control AI, and backdoor attack detection of medical diagnostic models, exhibiting broad applicability.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An AI model security vulnerability detection system based on model output consistency analysis, characterized in that: It includes an input perturbation module, a multimodal output acquisition unit, a consistency analysis engine, and a vulnerability location and visualization module; The input perturbation module is responsible for generating semantically equivalent diverse input variants to simulate potential attack scenarios and trigger inconsistent behavior of the AI ​​model; The multimodal output acquisition unit is responsible for collecting the multimodal output data of the AI ​​model after it has been subjected to input perturbations; The consistency analysis engine uses statistical analysis and machine learning methods to detect abnormal inconsistencies in the model output. The vulnerability localization and visualization module locates security vulnerabilities in the AI ​​model based on the anomaly information detected by the consistency analysis engine, determines the location of the vulnerability and the related modules, and visualizes the vulnerability information.

2. The AI ​​model security vulnerability detection system based on model output consistency analysis according to claim 1, characterized in that: The input disturbance module includes a data parsing and type identification unit, a disturbance strategy library and selection unit, a disturbance execution engine, a disturbance intensity control unit, and a post-disturbance data verification unit. The data parsing and type recognition unit first parses the format of the raw data input to the AI ​​model; then extracts the feature parameters of the data; and finally establishes a mapping relationship between data types and perturbation strategies. The perturbation strategy library and selection unit are divided into a basic perturbation library and an advanced attack perturbation library. Perturbation combination strategies are selected according to the AI ​​model type, application scenario and test target. The perturbation execution engine deploys dedicated perturbation executors, where the image perturbation executor uses OpenCV to implement pixel-level geometric transformations and noise injection; and the text perturbation executor uses an NLP toolkit to implement word-level / sentence-level feature adjustment. The audio perturbation actuator implements spectral characteristics and waveform perturbations based on librosa; The disturbance intensity control unit performs quantification and classification of disturbance intensity, where the data disturbance difference is as follows: ； in, This is the original data; The data is after the disturbance; It is an L2 norm; To find the minimum value, prevent the denominator from being 0; The data perturbation difference is defined, ranging from [0,1].

3. The AI ​​model security vulnerability detection system based on model output consistency analysis according to claim 1, characterized in that: The multimodal output acquisition unit includes a modality recognition and adaptation unit, a distributed acquisition execution unit, a data preprocessing and standardization unit, and a data association and synchronization unit; The modality recognition and adaptation unit first automatically identifies the modality type of the model output by using the metadata of the input data, thus realizing modality type detection; then, it automatically selects the appropriate acquisition protocol for the output interface of the AI ​​model; finally, the configuration information is synchronized to the acquisition execution unit in real time to realize dynamic configuration management. The distributed data acquisition and execution unit first constructs a multi-threaded data collector, allocating independent data acquisition threads for each modality; then, for AI models deployed on edge devices, a lightweight data acquisition agent is deployed to achieve data acquisition at edge nodes; finally, a priority queue mechanism is used to allocate data acquisition resources. The data preprocessing and standardization unit first converts the raw output data of different modalities into a unified intermediate format; then, it extracts the metadata of each modal output; finally, it removes invalid output data and filters the data through validation rules, triggering an alarm when the proportion of invalid data is greater than 10%. The data association and synchronization unit first adds a high-precision timestamp to each acquired multimodal output data and calibrates the clocks of each acquisition node through the NTP protocol to ensure time consistency across modal data. Then, it generates a globally unique output data ID based on the input perturbation ID, model version number, and timestamp to achieve the association and traceability of multimodal output data. Finally, for asynchronously output multimodal data, a sliding time window is used for data alignment, and the multimodal data within the window is marked as an associated data group. The storage and indexing unit first adopts a hybrid storage architecture, with structured metadata stored in a MySQL database and unstructured data stored in a distributed file system. Then, indexes are built based on modality type, timestamp, output confidence, and perturbation ID dimensions to support real-time queries. Incremental backups are automatically performed at fixed times every day, and full backups are performed every Sunday. Backup data is stored in an off-site disaster recovery center and retained for 90 days.

4. The AI ​​model security vulnerability detection system based on model output consistency analysis according to claim 1, characterized in that: The consistency analysis engine includes a data acquisition and integration submodule, a statistical analysis submodule, a machine learning analysis submodule, a cross-modal consistency verification submodule, and a real-time monitoring and feedback submodule. The data acquisition and integration submodule includes a multi-source data access unit, a data standardization processing unit, and a data association indexing unit; The statistical analysis submodule includes a baseline model building unit, an anomaly calculation unit, and a threshold adaptation unit; The machine learning analysis submodule includes a supervised learning unit, an unsupervised learning unit, and a model evaluation unit; The cross-modal consistency verification submodule includes a modal feature extraction unit, a consistency measurement unit, and a conflict detection unit; The real-time monitoring and feedback submodule includes a streaming computing unit, an anomaly response unit, and a log auditing unit.

5. The AI ​​model security vulnerability detection system based on model output consistency analysis according to claim 4, characterized in that: In the data acquisition and integration submodule, the multi-source data access unit supports real-time access to data sources such as training data pipelines, inference service interfaces, and model parameter servers. It uses a distributed message queue to achieve throughput data buffering and adapt to terabyte-level data acquisition requirements. The data standardization processing unit uses different processing operations for structured and unstructured data; For structured data, field alignment is achieved through schema mapping, and missing values ​​are filled using the mean or mode method; for unstructured data, images are uniformly converted to RGB format and 256×256 resolution, text is segmented using BPE to unify sequence length, and audio is standardized to a 16kHz sampling rate and mono. The data association index unit establishes a three-dimensional index system of input data, model output, and parameter snapshots, enabling cross-source data association queries through timestamps and request IDs.

6. The AI ​​model security vulnerability detection system based on model output consistency analysis according to claim 4, characterized in that: In the statistical analysis submodule, the baseline model construction unit uses a sliding window algorithm to update the statistical baseline of the normal output in real time, specifically including: A. Classification task: mean vector μ and covariance matrix Σ of the category probability distribution; B. Regression task: Mean μ and standard deviation σ of the output values; C. Generation task: Generate the distance distribution between the generated samples and the real samples; The anomaly calculation unit designs dedicated anomaly indicators for different task types and outputs a comprehensive anomaly score by combining a multi-indicator fusion strategy. Threshold adaptive unit: The detection threshold is dynamically adjusted based on the ratio of normal to abnormal samples, and a dual verification mechanism of the 3σ rule and quantile method is adopted.

7. The AI ​​model security vulnerability detection system based on model output consistency analysis according to claim 4, characterized in that: In the modal feature extraction unit of the cross-modal consistency verification submodule, the text modality uses a BERT-based pre-trained model to extract a 768-dimensional semantic feature vector corresponding to its special symbol [CLS]; the image modality uses a ResNet50 deep convolutional network to obtain a 2048-dimensional visual feature vector after global average pooling; the audio modality first converts the audio signal into a Mel spectrogram, and then extracts a 512-dimensional acoustic feature vector through a convolutional neural network; finally, a lightweight fully connected network is deployed as a modal adapter to uniformly transform the above multimodal features of different dimensions into a 512-dimensional shared feature space; The consistency measurement unit measures directional consistency by calculating cosine similarity and numerical closeness by calculating Euclidean distance. Statistical correlation is measured by calculating mutual information; The conflict detection unit sets a multimodal decision consistency threshold. When more than 30% of the modal pairs have a similarity below the threshold, it is determined to be a cross-modal conflict. The multimodal decision consistency is as follows: ； in, The number of modes; This is the consensus threshold for multimodal decision-making, with a default value of 0.

7. This represents the intermodal consistency function, used to calculate the degree of consistency between the decisions of the i-th mode and the j-th mode; For indicator functions; when When this occurs, a cross-modal conflict alarm is triggered.

8. The AI ​​model security vulnerability detection system based on model output consistency analysis according to claim 4, characterized in that: In the real-time monitoring and feedback submodule, the streaming computing unit, based on the Flink framework, achieves real-time analysis with millisecond-level latency and supports parallel processing of 10,000+ samples per second. The anomaly response unit is divided into three levels based on the anomaly score: yellow (5-7 points), orange (7-9 points), and red (≥9 points), to achieve graded alarms. Moreover, when a red alarm triggers the model degradation strategy, i.e., switching to the backup model, an orange alarm automatically initiates parameter freezing. The log auditing unit uses blockchain technology to store abnormal event logs, ensuring immutability and traceability.

9. The AI ​​model security vulnerability detection system based on model output consistency analysis according to claim 1, characterized in that: The vulnerability localization and visualization module includes a vulnerability localization submodule and a visualization submodule; the vulnerability localization submodule includes an anomaly feature extraction unit, a model structure mapping unit, and a vulnerability tracing and reasoning unit; the visualization submodule includes a data processing and conversion unit, a visualization rendering unit, and an interactive control unit.

10. The AI ​​model security vulnerability detection system based on model output consistency analysis according to claim 9, characterized in that: In the vulnerability localization submodule, the anomaly feature extraction unit extracts key anomaly features from the anomaly information output by the consistency analysis engine. The model structure mapping unit establishes the mapping relationship between the internal structure of the AI ​​model and its inputs and outputs; The vulnerability tracing and reasoning unit is based on the mapping relationship between abnormal features and model structure, and uses a reverse reasoning algorithm to trace back from the abnormal output to the internal module or parameter area of ​​the model that caused the vulnerability. In the visualization submodule, the data processing and conversion unit processes and converts the vulnerability location, abnormal data and other information output by the vulnerability location submodule into a format suitable for visualization. The visualization rendering unit selects an appropriate visualization method and renders the data based on the visualization requirements; the interactive control unit provides interactive functions, allowing users to interact with the visualization results.

Images