Knowledge constraint-based image description confrontation sample detection method and system

Abstract

The invention relates to the technical field of image analysis and recognition, and discloses an image description confrontation sample detection method and system based on knowledge constraints. The method comprises the following steps: constructing a category entity lexicon containing common objects and scenes, then when an image description model outputs and generates a word, searching the generated word in the entity lexicon to determine whether the generated word exists in the entity lexicon, if so, inputting the word and an attention thermodynamic diagram corresponding to the word into a large language model, and if not, inputting the word and an attention thermodynamic diagram corresponding to the word into the large language model; according to the method, the words and key features displayed by the thermodynamic diagram are analyzed and judged, whether the features displayed by the thermodynamic diagram can be matched with word semantics or not is analyzed, and in addition, final description generated by an image description model is also used as input; and inputting the description to the large model to check whether the phenomenon described by the description violates common phenomena in the real world, and outputting confidence of an adversarial sample by the large model to realize detection of the adversarial sample.

Description

Technical Field

[0001] This invention relates to the field of image analysis and recognition technology, specifically to a knowledge-constrained image description adversarial sample detection method and system. Background Technology

[0002] In recent years, with the rapid development of natural language processing and computer vision, security issues have gradually emerged, such as adversarial attacks. Adversarial attacks involve adding noise that is imperceptible to the human eye to the input data to mislead the model into generating incorrect output. Image description tasks, as an intersection of these two fields, also suffer from the aforementioned security problems. For example, describing a "ship" in the original image as an "airplane," or ultimately outputting "a monkey driving a car on the road"—descriptions that seem extremely illogical to us humans—has significant drawbacks in existing adversarial example detection techniques. Traditional pixel-level detection methods (such as statistical analysis based on image perturbation amplitude) can only identify "pixel anomalies" and cannot correlate "semantic consistency between text description and visual features," rendering them completely ineffective against adversarial examples where "pixel perturbations are small but semantic deviations are severe." Existing semantic-level detection methods mostly rely on "grammar rule verification" or "simple vocabulary matching," failing to handle descriptions that are "grammatically correct but contradict common sense" (such as "an elephant standing on a person's shoulders"), and lacking the ability to verify the "correspondence between attention heatmaps and word semantics." Some detection schemes do not incorporate external knowledge (such as real-world common sense or object feature databases), relying solely on the image description model's own output for judgment. When the model is misled by adversarial attacks, the detection results are prone to deviating from the model's erroneous output, resulting in insufficient robustness.

[0003] The aforementioned deficiencies make it difficult for existing technologies to effectively cope with adversarial attacks in the field of image description, leading to serious practical impacts: in intelligent transportation scenarios, misrepresenting "pedestrians" as "streetlights" may cause autonomous driving systems to misjudge road conditions; in medical auxiliary diagnosis scenarios, misrepresenting "tumor areas" as "normal tissue" may delay diagnosis; and in content security review scenarios, misrepresenting "illegal images" as "normal content" through adversarial perturbations may lead to the spread of illegal information. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a knowledge-constrained image description adversarial example detection method and system, which has the advantages of accurately identifying adversarial examples targeting image descriptions and solves the aforementioned technical problems.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a knowledge-constrained image description adversarial example detection method, comprising the following steps:

[0006] S1: Preprocess the input image data;

[0007] S2: Input the preprocessed image data from S1 into the image description model to generate an attention heatmap and natural language description;

[0008] S3: Input the generated attention heatmap and natural language description obtained in S2 into the knowledge-constrained image description model within the adversarial example detection framework for detection.

[0009] As a preferred technical solution of the present invention, the preprocessing process in S1 specifically includes: generating perturbation samples through FGSM to construct an adversarial sample dataset, selecting images from the MSCOCO dataset to construct a training set, and combining the images selected from the MSCOCO dataset and the adversarial sample dataset constructed through FGSM to form a validation set and a test set.

[0010] As a preferred embodiment of the present invention, the image description model in S2 adopts a long short-term memory model, and its update equation at time t is as follows:

[0011]

[0012]

[0013]

[0014]

[0015]

[0016] in, , and These represent the input gate, forget gate, and output gate at time t, respectively. This represents element-wise multiplication. express The hidden state at all times express The hidden state at all times This represents the sigmoid function. Represents the hyperbolic tangent function. It refers to the state of a memory unit. , , Let represent the input weight matrix of the input gate, the hidden state of the output gate, and the context vector weight matrix of the input gate, respectively. Used to input at the current time Mapped to the input gate dimension, Used to hide the state at time t-1 Mapped to the input gate dimension, Used to transfer the context vector at time t Mapped to the input gate dimension, , , Let represent the input weight matrix of the forget gate, the hidden state weight matrix of the forget gate, and the context vector weight matrix of the forget gate, respectively. Used to input at the current time Mapped to the forget gate dimension, Used to hide the state at time t-1 Mapped to the forget gate dimension, Used to transfer the context vector at time t Mapped to the forget gate dimension, , , These represent the input weight matrix of the output gate, the hidden state weight matrix of the output gate, and the context vector weight matrix of the output gate, respectively. Used to input at the current time Mapped to the output gate dimension, Used to hide the state at time t-1 Mapped to the output gate dimension, Used to transfer the context vector at time t Mapped to the output gate dimension, , , These represent the input weight matrix, the hidden state weight matrix, and the context vector weight matrix of the memory unit, respectively. Used to input at the current time Mapped to the dimension of memory units, Used to hide the state at time t-1 Mapped to the dimension of memory units, Used to transfer the context vector at time t Mapped to the dimension of memory units, Indicates time The context vector is expressed as follows:

[0017]

[0018] in, The index representing image features ranges from 1 to L; This represents the total number of image features; This represents the local feature vector of the i-th image. The attention distribution is obtained through softmax, and the specific expression is as follows:

[0019]

[0020] in, Represents an exponential mapping. This represents a temporary index used for attention weight normalization, ranging from 1 to L, and is used to iterate through the alignment scores of all image features. , express Hide state at all times Image features The alignment score is calculated for vectors, and the specific expression is as follows:

[0021]

[0022] in, This represents the attention alignment score function, used to compute the hidden state at time t-1. With the local feature vector of the i-th image The degree of correlation between them.

[0023] As a preferred embodiment of the present invention, the image description model is trained using a training set, and its training loss function is as follows:

[0024]

[0025] in, Sentence Length, In the image description model G, given the i-th image... The descriptive sentence corresponding to the image When considering all words preceding the k-th word, the model generates the k-th word. The conditional probability, , , , These represent the descriptive sentences corresponding to the i-th image. The sentence corresponding to the k-th word and the i-th image. The prefix sequence from the first word to the (k-1)th word, the i-th image in the training set, and the total number of image-description pairs in the training set. This represents a logarithmic mapping.

[0026] As a preferred technical solution of the present invention, the knowledge-constrained image description model adversarial example detection framework in S3 includes the following steps:

[0027] S3.1: Construct an entity lexicon, obtain the vocabulary mapping file generated during the training of the image description model, and obtain all entries;

[0028] S3.2: For large language models, pre-set dedicated prompt word templates and interaction flows corresponding to local visual attention consistency judgment and global semantic rationality judgment, and implement the judgment mechanism according to the following rules:

[0029] Default scenario: A consistency determination mechanism based on local visual attention and a common-sense reasonableness determination mechanism based on overall semantic logic operate together. The confidence score of either mechanism outputting a local value higher than the corresponding threshold is considered valid. Global This is considered an adversarial example.

[0030] Contains only entity words, local close-up images, and is resource-constrained: Only a consistency determination mechanism based on local visual attention is implemented, using local confidence. As the sole basis for judgment;

[0031] For samples with no entity words, no explicit entity images, and global semantic tampering: only a common-sense reasonableness judgment mechanism based on overall semantic logic is executed, using global confidence. As the sole basis for judgment;

[0032] The final judgment is based on a combination of the results of the two mechanisms: the confidence level after the two mechanisms are executed in sequence is compared with the threshold. If either mechanism exceeds the threshold, it is judged as adversarial; if both mechanisms are below the threshold, it is judged as normal.

[0033] As a preferred embodiment of the present invention, the consistency determination mechanism based on local visual attention specifically includes the following steps:

[0034] S3.2.a1: Image description model generates word sequences word by word Output the attention map corresponding to each word. ;

[0035] in, This indicates the length of the natural language description sentence generated by the image description model;

[0036] S3.2.a2: Determine the word Does it exist in the entity dictionary? If it does, proceed to the discrimination process.

[0037] S3.2.a3: Attention Map Analyze and extract pixels with values ​​greater than 100%. area The specific expression is as follows:

[0038]

[0039] in, , This represents the saliency threshold for pixel values ​​in the attention map, used to filter the attention map output from the image description model. Extract the current word The visual area that contributes the most;

[0040] S3.2.a4: Transfer the region from the original image... Corresponding image content and words Input the data into a large language model and ask it to determine whether the region contains a word. The characteristics it possesses;

[0041] S3.2.a5: Obtain the confidence score of the large model output. If the final result is greater than the threshold If it is, then it is identified as an adversarial example.

[0042] As a preferred technical solution of the present invention, the common sense rationality determination mechanism based on overall semantic logic specifically includes the following steps:

[0043] S3.2.b1: Extracting word sequences generated by the image description model ;

[0044] S3.2.b2: will Input a large language model, and the large language model outputs a confidence score. ;

[0045] S3.2.b3: If the confidence level is higher than the preset threshold If the statement contains a semantic conflict or logical contradiction, it is considered an adversarial sample.

[0046] The present invention also provides a knowledge-constrained image description adversarial example detection system for performing the above-described knowledge-constrained image description adversarial example detection method.

[0047] Compared with existing technologies, this invention provides a knowledge-constrained image description adversarial example detection method and system, which has the following beneficial effects:

[0048] 1. This invention proposes an image description model that generates each category of words and its corresponding attention distribution region. By combining this model with a large language model, it determines whether the features displayed in the region match the features of the corresponding category of words. This allows for the accurate identification of adversarial sample mechanisms targeting image descriptions. This mechanism effectively enhances the robustness of the image description model while maintaining its original performance.

[0049] 2. This invention examines adversarial examples by judging whether the phenomenon described by the complete description output by the image description model conforms to normal phenomena in the real world. Compared with traditional examination methods that rely only on the pixel level or grammatical rules, this mechanism can identify absurd information at the semantic level, thus improving the coverage and accuracy of adversarial example detection.

[0050] 3. This invention filters the words in the original Cocoa word mapping file, only filtering entity-type words and scene-type words, thereby reducing invalid detection and computational resource consumption, and improving the model's response speed and accuracy. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the data processing of the present invention;

[0052] Figure 2 This is a diagram illustrating the mismatch between word semantics and attention region features resulting from the attack.

[0053] Figure 3 This is a diagram illustrating the effect of a local feature matching but overall logical contradiction caused by an attack.

[0054] Figure 4 Flowchart for adversarial sample detection model. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Please see Figure 1 - Figure 4 A knowledge-constrained image description adversarial example detection method includes the following steps:

[0057] S1: Preprocess the input image data;

[0058] This invention uses the general dataset MSCOCO, which contains more than 120,000 everyday scene images, meeting the requirements for training and final verification of image description models;

[0059] Data input and standardization:

[0060] Description: Standardize the 5 manually annotated figure descriptions into a structured format:

[0061] {

[0062] "image_id": "COCO_val2014_000000391895.jpg",

[0063] "captions": [

[0064] "A man with a red helmet on a small moped on a dirt road.",

[0065] "Man riding a motor bike on a dirt road on the countryside." ,

[0066] "A dirt path with a young person on a motor bike rests to theforeground of a verdant area with a bridge and a background of cloud-wreathedmountains.",

[0067] "A man riding on the back of a motorcycle.",

[0068] "A man in a red shirt and a red hat is on a motorcycle on a hillside.", ]

[0070] };

[0071] Adversarial example generation: 3,000 high-quality perturbation samples were generated using FGSM to construct an adversarial example dataset. The attack perturbation was set to 0.03 to ensure that the generated adversarial examples were visually indistinguishable from the original images, but could successfully induce the model to generate incorrect descriptions.

[0072] Training set: 82,783 images and their corresponding descriptions from the MSCOCO training set are used to train the image description model.

[0073] Validation set: Composed of 5000 images from the MSCOCO validation set and adversarial examples generated using the FGSM attack, used for hyperparameter tuning and threshold setting.

[0074] Test set: Composed of 5000 images from the MSCOCO test set and adversarial examples generated using the FGSM attack, used for the final model performance evaluation and comparative experiments;

[0075] FGSM, proposed by Goodfellow et al., is a classic adversarial example generation algorithm. Its core idea is to utilize the model's gradient information to significantly alter the model's prediction results through small perturbations. This attack method significantly exposes the vulnerabilities of deep learning models, driving research into their robustness. It should be noted that since the traditional FGSM method is designed for image classification tasks, while this invention targets image description tasks, the following adaptive adjustments have been made to its original loss function, gradient calculation, and attack effectiveness evaluation:

[0076] Loss function adjustment: In the traditional FGSM algorithm, the loss function is usually the cross-entropy loss function, which is used to measure the difference between the predicted class label and the true class.

[0077]

[0078] in Indicates the number of categories. This is an indicator function, representing the condition when the true category is... Take 1 at time. The model indicates that the input image belongs to a certain category. The predicted probability is a scalar category; while the output of the image description model is a word sequence. Since it is not a single-category label, its loss function needs to cover the entire sequence prediction process:

[0079] It is the cross-entropy loss over time steps. It is the actual t-th word, and It is the t-th word predicted by the model. It is a regularization term used to constrain the distribution of attention. As weight.

[0080] Gradient calculation method adjustment: In classification tasks, the gradient is calculated based on the output of a single classification, and is directly applied to the input image.

[0081]

[0082] Compared to classification tasks, gradients for image description tasks are influenced not only by image features but also by the attention mechanism and time dependence during sequence decoding.

[0083]

[0084] Successful Attack Criteria: For image classification tasks, a successful FGSM attack is defined as a change in the model's final predicted class. The criterion for a successful FGSM attack on image description tasks is that the sequence of words in the final generated description exhibits semantic deviation. ;

[0085] S2: Input the preprocessed image data from S1 into the image description model to generate an attention heatmap and natural language description;

[0086] Framework for image description models with attention mechanisms:

[0087] In this framework, by taking an image I as input, a natural language sentence s is ultimately generated:

[0088]

[0089]

[0090] The core idea of ​​training an image captioning model is to maximize the probability of a description given an input image, thus generating words one after another until the end. Its loss function is expressed as:

[0091]

[0092] Where N is the total number of samples in the training. It is a sentence Among them, the first One word, T is a sentence. Length, Sentence Length, In the image description model G, given the i-th image... The descriptive sentence corresponding to the image When considering all words preceding the k-th word, the model generates the k-th word. The conditional probability, , , , These represent the descriptive sentences corresponding to the i-th image. The sentence corresponding to the k-th word and the i-th image. The prefix sequence from the first word to the (k-1)th word, the i-th image in the training set, and the total number of image-description pairs in the training set. This represents a logarithmic mapping. Show, Attend, and Tell extract spatial features at the encoding end using a convolutional neural network, and a recurrent neural network is used as the decoder to generate subtitles. In this invention, Long Short-Term Memory (LSTM) is used as our language model. It takes the output of the previous time step, the input of the current time step, and the context extracted from image I as its input. The update equation at time t is:

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] , and Let represent the input gate, forget gate, and output gate at time t, respectively. It refers to the state of a memory unit. It is in a hidden state. It is a sequence element with time step t. , , Let represent the input weight matrix of the input gate, the hidden state of the output gate, and the context vector weight matrix of the input gate, respectively; where Used to input at the current time Mapped to the input gate dimension, Used to hide the state at time t-1 Mapped to the input gate dimension, Used to transfer the context vector at time t Mapped to the input gate dimension, all three components participate in the activation calculation of the input gate. , , Let represent the input weight matrix of the forget gate, the hidden state weight matrix of the forget gate, and the context vector weight matrix of the forget gate, respectively; where, Used to input at the current time Mapped to the forget gate dimension, Used to hide the state at time t-1 Mapped to the forget gate dimension, Used to transfer the context vector at time t Mapped to the forget gate dimension, all three factors participate in the activation calculation of the forget gate. , , These represent the input weight matrix of the output gate, the hidden state weight matrix of the output gate, and the context vector weight matrix of the output gate, respectively. Used to input at the current time Mapped to the output gate dimension, Used to hide the state at time t-1 Mapped to the output gate dimension, Used to transfer the context vector at time t Mapped to the output gate dimension, all three factors participate in the activation calculation of the output gate. , , These represent the input weight matrix, the hidden state weight matrix, and the context vector weight matrix of the memory unit, respectively. Used to input at the current time Mapped to the dimension of memory units, Used to hide the state at time t-1 Mapped to the dimension of memory units, Used to transfer the context vector at time t Mapped to the memory cell dimension, all three participate in the calculation of the candidate state of the memory cell (the part within ϕ in the formula). This represents element-wise multiplication. () represents the sigmoid function, and ( ) represents the hyperbolic tangent function, and W.. represents the model parameters. Unlike the traditional encoder-decoder framework, the context vector in Show, Attend, and Tell is not dynamically calculated at each time step through an attention mechanism. Specifically, when the model generates the t-th word, it calculates the context vector based on the previous hidden state. Image features Vector alignment score calculation:

[0099]

[0100] The index representing the image feature ranges from 1 to L and is used to traverse all local features of the image to calculate the contribution weight of each feature to the current text generation. This represents the total number of image features; This represents the local feature vector of the i-th image;

[0101] The attention distribution is then obtained using softmax:

[0102]

[0103] Represents an exponential mapping. This represents a temporary index used for attention weight normalization, ranging from 1 to L, and is used to iterate through the alignment scores of all image features. This ensures that the denominator of the softmax function is the exponential sum of the feature alignment scores, thus making... It satisfies the probability distribution property that "the sum of all weights is 1";

[0104] And by summing these values ​​using weighted methods, we obtain the context vector:

[0105]

[0106] This mechanism ensures that at each time step, the decoder can selectively focus on the most relevant regions of the image based on the historically generated words and hidden states.

[0107] It should be noted that the role of the image description model in this invention is to generate attention heatmaps and natural language descriptions. Its output is used in the final semantic consistency verification and adversarial example detection process. Therefore, the technical innovation of this invention does not lie in the image description model itself, but in the image description adversarial example detection mechanism constructed by using its output and the massive knowledge and reasoning ability of the large language model.

[0108] S3: Input the generated attention heatmap and natural language description obtained in S2 into the knowledge-constrained image description model within the adversarial example detection framework for detection;

[0109] This invention proposes a knowledge-constrained image description model adversarial example detection architecture. This architecture obtains the final description of the image description model and the scene and object category words within the description, along with their corresponding attention heatmaps. Combined with the massive knowledge base and powerful reasoning capabilities of a large language model, it detects anomalous outputs in the features displayed in the heatmaps and the real-world object features of their corresponding words, identifying anomalous outputs that reveal facts or semantic logic within the description. In this invention, the entire detection process is divided into two stages: a word-level attention verification stage and a description-level common sense verification stage. The specific steps are as follows:

[0110] S3.1: Entity Lexicon Construction

[0111] To improve detection efficiency and semantic targeting, this invention first constructs an "entity lexicon," which contains a large number of object category words (such as "wing," "car," etc.) and scene category words (such as "forest," "sky," etc.) to limit the set of words participating in consistency judgment in subsequent detection.

[0112] The specific steps are as follows: First, obtain all the vocabulary entries by acquiring the vocabulary mapping file generated during the training of the image description model using the COCO dataset; then, filter out all object-related words and scene-related words through manual screening, which will serve as the basis for determining whether the image description model needs to be fed into the large model for judgment when outputting description words.

[0113] The introduction of this mechanism can effectively limit the set of words involved in the judgment during the subsequent detection process, avoid invalid judgments caused by abstract semantic terms, and improve the system's operating efficiency and detection accuracy.

[0114] S3.2: This invention designs dedicated prompt word templates and interaction processes for two types of tasks: local visual attention consistency determination and global semantic rationality determination.

[0115] Local visual attention consistency cue word design:

[0116] You are a visual AI assistant specializing in fine-grained visualevidence analysis. Your task is to determine if the highlighted area in the image does indeed contain the object '[objectiw_ORD]'.

[0117] Instructions:

[0118] Focus only on pixels within the highlighted display area. Ignore all other contexts outside of this area.

[0119] Analyze the visual attributes of the highlighted area. Systematicallycheck for the existence of key features that define a typical '[OBECT-WORD]'.consider:

[0120] Texture and surface patterns: for example, animal fur, clothingfabrics, vehicle metal.

[0121] Key components: For example, for a "horse": look for a head withears, mane, long neck, torso, four legs, and tail. For "cars": search forwheels, windows, headlights.

[0122] Make a decision based solely on the visual evidence present in thehighlighted area. Do not speculate or assume based on the surroundingenvironment. If the area only displays common textures or blurry parts thatmay belong to multiple objects, the confidence level must be low.

[0123] Output format: You must respond using only a JSON object thatcontains the following two fields:

[0124] Reasoning ":" Summarize the most decisive visual evidence for oragainst in a concise sentence. mention specific features (such as' the areadisplays the structural features of the car's wheels and headlights')

[0125] Confidence ": A floating-point number between 0 (unmatched) and 1(perfectly matched). Calibrate this score based on the clarity and completeness of the identified defined features. The higher the confidencelevel, the greater the likelihood that the example becomes an adversarialsample.

[0126] Design Description: The model is defined as an expert focused on "fine-grained visual feature analysis." By clearly defining the feature dimensions to be analyzed, such as shape, texture, and key components, a reasoning direction is provided for the model's visual inference, effectively guiding the model to conduct interpretable, feature-based analysis. Furthermore, the design emphasizes two requirements: the model should only examine "highlighted areas" and be "purely based on visual features," preventing the model from being misled by biases across the entire image. In the final output stage, the model is required to provide specific features and explanations, which greatly enhances the interpretability of the detection results. The final confidence score evaluation criterion is also clearly defined: higher confidence indicates a greater likelihood of adversarial examples, making the scores more consistent and comparable.

[0127] Global semantic rationality prompt word design:

[0128] You are an artificial intelligence assistant designed to assess the common sense validity of sentences describing images. Please strictly follow the following steps:

[0129] Task: Evaluate whether the following description is reasonable and does not violate common sense or the laws of physics: "[FULL [ACTION]".

[0130] Attention: Consider typical object properties, object behavior, spatial relationships, and physical possibilities (such as gravity, time). If a description describes an impossible or highly unlikely situation, then the description is unreliable.

[0131] Output format: You must respond only in the following JSON format: {"Inference": "One sentence explanation", "Confidence": Floating point number between 0 and 1}. The confidence score should reflect the certainty of your description being reasonable. High confidence implies that the description may be adversarial.”

[0132] Design Description: The model's role is clearly defined as an "assistant in judging the rationality of image description sentences." Its task is to evaluate the rationality of descriptions. By listing factors such as "physical attributes, behavior, and spatial relationships," the model is effectively guided to utilize relevant knowledge for deep reasoning, resulting in more accurate judgments. In the model's output phase, the confidence score represents the probability of being an adversarial example. Therefore, a high confidence score implies that the description may be problematic, and the higher the probability that it is an adversarial example.

[0133] This prompt word template achieves its design solidification through 'standardized analysis dimensions + fixed output format', and the template will not be changed during subsequent testing to ensure the comparability of test results for different samples. Human intervention in the template design stage is only used to define the 'core detection dimensions' and 'output specifications', and does not interfere with the model inference process or the final confidence output. The entire detection process is automated.

[0134] For large language models, dedicated prompt word templates and interaction flows are pre-defined for local visual attention consistency judgment and global semantic rationality judgment, and the judgment mechanism is based on the following rules:

[0135] Default scenario: A consistency determination mechanism based on local visual attention and a common-sense reasonableness determination mechanism based on holistic semantic logic operate together. The confidence score of either mechanism outputting a value higher than the corresponding threshold (local) is considered valid. Global This is considered an adversarial example.

[0136] Contains only entity words / close-up images / resource-constrained: Only performs a consistency determination mechanism based on local visual attention, using local confidence. As the sole basis for judgment;

[0137] For samples with no entity words / no explicit entity images / global semantic tampering: only a common-sense reasonableness judgment mechanism based on overall semantic logic is executed, using global confidence. The sole criterion for judgment is the combined result of the two mechanisms. This judgment process relies on the aforementioned "local / global confidence threshold" and "serial termination rule": by comparing the confidence level after the serial execution of the two mechanisms with the threshold, the logic is "if either mechanism exceeds the threshold, it is judged as adversarial; if both mechanisms are below the threshold, it is judged as normal".

[0138] Consistency determination mechanism based on local visual attention:

[0139] This step starts with the category words and their attention heatmaps output by the image description model. The category words and their corresponding attention heatmaps filtered through the entity lexicon are then input into the large model, which determines whether the features shown in the heatmap can match the category words.

[0140] The specific steps are as follows: (1) The model generates word sequences character by character. Output the attention map corresponding to each word. (2) Determine the word Does it exist in the entity vocabulary? If it does, proceed to the discrimination process; (3) For the attention map Analyze and extract pixels with values ​​greater than 100%. (In this invention, take) The region with a value of 0.8) (4) Combine the regions in the original image Corresponding image content and words Input the same data into a large language model with multimodal understanding capabilities (such as GPT-4V), and ask it to determine whether the region is a word. The features it possesses; (5) Obtain the confidence score of the large model output. (0-1 interval), if the final result is greater than the threshold (In this invention, take) =0.8 (the value judgment is introduced later), then it is judged as an adversarial example: .

[0141] This mechanism effectively solves the attention shift problem caused by input image perturbation. For example... Figure 2 As shown, the original description "a people riding on the back of a horse" is compared using the entity lexicon method described above. The words "people" and "horse," along with their corresponding attention views, are obtained and fed into the large model. The model then assigns a confidence score based on the words and the features displayed in their heatmaps. Since the above are normal samples, and the features displayed by the words and their corresponding heatmaps are highly plausible, the model outputs a low confidence score. However, for... Figure 2 The adversarial example shown, after an adversarial attack, the original description becomes "a monkeyriding on the top of a bed". After extracting the entity lexicon, the words "monkey" and "bed" and their corresponding attention views are obtained and input into the large model. Although the word "monkey" has similar features to "people", since monkeys do not wear hats or clothing, the large model infers based on common sense and gives a high confidence score (0.70). However, the other word "bed" has completely different features from the correct word "horse", such as horse tail, horse head, and horse mane. Therefore, the large model gives a very high confidence score (0.90). This score is significantly higher than the set maximum threshold score (0.80), so it is judged as an adversarial example.

[0142] Common sense rationality judgment mechanism based on holistic semantic logic

[0143] This step starts with the entire descriptive statement and uses a large language model to perform common-sense reasoning analysis on the phenomena described in the image description, judging from the overall description whether the phenomena described conform to the common sense and laws of the real world.

[0144] The operation process is as follows: (1) Extract the most central description generated by the image description model. (2) will Input the large language model and ask whether the described phenomenon conforms to common sense; (3) The large model outputs a confidence score. (0-1 interval), indicating the probability that the sample corresponding to the description is an adversarial sample; (4) if the confidence level is higher than the preset threshold (In this invention, take) If the value is 0.85, it indicates that the sentence contains a semantic conflict or logical contradiction, and is therefore classified as an adversarial sample. .

[0145] This module can be used to capture structural absurdities in descriptions, compensating for the shortcomings of local attention mechanisms in handling complex semantic combinations, for example... Figure 3 In the example above, the adversarial example cannot be detected by the above mechanism. This is because the adversarial example can also extract the words "man" and "elephant" and their corresponding attention views, which are highly consistent with the words and attention views corresponding to the normal example. This will cause the above method to completely fail. However, the final description of the adversarial example, "an elephant is on the man," is almost impossible to happen in the physical world because it is usually the man on top of the elephant, not the other way around. Therefore, when this description is sent to a large model for evaluation, the large model will output a high confidence score (0.92). This confidence score is significantly greater than the set threshold, so it will be judged as an adversarial example.

[0146] The detection system proposed in this invention consists of a local visual attention consistency module and a phenomenon description rationality module, and has the following characteristics: (1) The two modules can work independently or in sequence, and the final judgment result output by the system is based on the detection result of either module; (2) , , The threshold parameters are determined using their respective methods to ensure an optimal balance between precision and recall. The specific implementation schemes are as follows: The parameter settings are based on those given in existing papers on Show, Attend, and Tell; the other two parameters... and The grid search method is used to determine the following: First, a validation set is constructed (containing clean samples and their corresponding adversarial samples and their respective label values, i.e. whether they are adversarial samples. Here, this invention defines whether they are adversarial samples as 0 and 1). Then, discrete candidate values ​​are set for the above two parameters, with a step size of 0.05 in the range [0.5, 0.95]. Then, each set of parameters is run through the validation set in the detection system to obtain its own F1 score. The calculation formula is F1 = 2 * (Precision * Recall) / (Precision + Recall). Finally, the set of parameters with the highest F1 score is selected as the final threshold setting. This mechanism can effectively ensure that the model can adapt to different models and automatically obtain the optimal detection performance, thereby enhancing the universality and robustness of the method; (3) The judgment reason can be output for each detection process, which is beneficial for manual review and system audit.

[0147] This invention utilizes the massive knowledge of large models to constrain malicious samples where the output words and features of the image description model do not match and the final described phenomena do not conform to common real-world phenomena. It constructs a robust, generalizable, and highly interpretable image description adversarial sample detection framework, which improves the system's anti-interference capability without affecting the quality of the model description.

[0148] The present invention also provides a knowledge-constrained image description adversarial example detection system for performing the above-described knowledge-constrained image description adversarial example detection method.

[0149] This invention has the following application scenarios:

[0150] 1. Enhance the adversarial robustness of image description systems:

[0151] Due to the rapid development of AI technology, image description technology has been widely applied in intelligent vision systems, such as smart transportation, robot navigation, and smart homes. However, image description models are susceptible to adversarial examples, which can lead to serious output deviations in these intelligent vision systems, affecting their reliability and security. The knowledge-constrained detection mechanism proposed in this invention can effectively identify and filter interfering input samples without affecting the original description quality, significantly improving the resilience of the original model. Furthermore, its detection results provide clear explanatory evidence, enhancing the system's reliability and security.

[0152] 2. Facilitating the Reliable Evaluation and Supervision of Multimodal Large Models:

[0153] With the rapid development and widespread application of multimodal large models in recent years, the authenticity and accuracy of their outputs have gradually attracted attention. The detection mechanism of this invention can be applied not only to image description models, but also to multimodal large models, improving the credibility of large models in high-risk and high-requirement fields such as medical auxiliary diagnosis and legal consultation.

[0154] 3. Security support applied to intelligent review and content security:

[0155] Image description technology is not only applied to intelligent vision systems but also to large-scale content generation environments such as short video platforms and news portals. However, these environments are also highly susceptible to adversarial attacks, which can lead to the output of misleading information and the risk of false information dissemination. This invention, when facing adversarial attacks, can not only identify adversarial samples but also output clear and easily understood detection reasons. Combined with manual review, this reduces the spread of false information and enhances the security of automatically generated content on the platform.

[0156] 5. Standardization Development of Highly Reliable Artificial Intelligence Systems:

[0157] For tasks requiring high robustness and interpretability, such as disaster monitoring, aerial remote sensing, and military image analysis, this invention proposes a feasible solution for the development of highly reliable artificial intelligence systems through two detection mechanisms: "local semantic attention feature consistency" and "global semantic logic rationality".

[0158] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A knowledge constraint-based image description adversarial sample detection method, characterized in that: The method comprises the following steps: S1: preprocessing input image data; S2: inputting the image data preprocessed in S1 into an image description model to generate an attention heat map and a natural language description; S3: inputting the generated attention heat map and the natural language description obtained in S2 into a knowledge constraint-based image description model to detect an adversarial sample bounding box.

2. The knowledge constraint-based image description adversarial sample detection method according to claim 1, wherein: The preprocessing process in S1 specifically comprises: generating a perturbed sample through FGSM to construct an adversarial sample dataset, selecting pictures from the dataset MSCOCO to construct a training set, selecting pictures from the dataset MSCOCO and generating a perturbed sample through FGSM to construct an adversarial sample dataset, and combining the two to form a validation set and a test set.

3. The knowledge constraint-based image description adversarial sample detection method according to claim 2, characterized in that: The image description model in S2 adopts a long short-term memory model, and its update equation at time t is as follows: wherein denotes the current input, , and denote the input gate, the forget gate and the output gate at time t, respectively, denotes element-wise multiplication, denotes the hidden state at time t, denotes the hidden state at time t, denotes the sigmoid function, denotes the hyperbolic tangent function, is the state of the memory cell, , , denote the input weight matrix of the input gate, the hidden state of the output gate, the context vector weight matrix of the input gate, respectively, is used to map the input at the current time to the input gate dimension, is used to map the hidden state at time t-1 to the input gate dimension, is used to map the context vector at time t to the input gate dimension, , , denote the input weight matrix of the forget gate, the hidden state weight matrix of the forget gate, the context vector weight matrix of the forget gate, respectively, is used to map the input at the current time to the forget gate dimension, is used to map the hidden state at time t-1 to the forget gate dimension, is used to map the context vector at time t to the forget gate dimension, , , denote the input weight matrix of the output gate, the hidden state weight matrix of the output gate, the context vector weight matrix of the output gate, respectively; is used to map the input at the current time to the output gate dimension, is used to map the hidden state at time t-1 to the output gate dimension, is used to map the context vector at time t to the output gate dimension, , , denote the input weight matrix of the memory cell, the hidden state weight matrix of the memory cell, the context vector weight matrix of the memory cell, respectively; for mapping an input at the current time to the dimension of the memory cell, for mapping a hidden state at time t-1 to the dimension of the memory cell, for mapping a context vector at time t to the dimension of the memory cell, denotes a context vector at time , which is expressed as follows: wherein, the index value range of the image feature is 1 to L; the total number of image features is represented by L; the local feature vector of the i-th image is represented by the attention distribution obtained by softmax is represented by where, denotes the exponential mapping, denotes the summation, denotes the temporary index for attention weight normalization, ranging from 1 to L, used to traverse all image feature alignment scores , denotes the hidden state at time the image feature vector to compute the alignment score, which is expressed as follows: wherein, denotes an attention alignment score function for computing the hidden state at time t-1 the relevance between the i-th image local feature vector and the i-th text local feature vector.

4. The knowledge constraint-based image description adversarial sample detection method according to claim 3, characterized in that: The image description model is trained by using the training set, and its loss function is as follows: wherein, denotes the length of the sentence, denotes the conditional probability that the model generates the k-th word in the sentence given all the words preceding the k-th word in the sentence corresponding to the i-th image in the image description model G, denote the k-th word in the sentence corresponding to the i-th image, the prefix sequence from the 1st word to the k-1th word in the sentence corresponding to the i-th image, the total number of image-sentence pairs in the training set, denotes the summation, denotes the logarithmic mapping.​​​​​ 5. The knowledge constraint-based image description adversarial sample detection method according to claim 1, wherein: The knowledge constraint-based image description model adversarial sample detection framework in S3 comprises the following steps: S3.1: constructing an entity library, obtaining a vocabulary mapping file generated by the image description model during the training process, and obtaining all word entries; S3.2: determining the corresponding special prompt word templates and interaction processes of the local visual attention consistency determination and the global semantic rationality determination of the large language model according to the following rules: Default scenario: the consistency determination mechanism based on local visual attention and the common sense rationality determination mechanism based on overall semantic logic run together, and the output confidence of any mechanism is higher than the corresponding threshold local , global , that is, determined as an adversarial sample; Entity words only, partial close-up images, resource constrained: only perform consistency decision mechanism based on partial visual attention, with partial confidence as the only decision basis; No entity words, no explicit entity images, global semantic tampering samples: only perform common sense rationality judgment mechanism based on overall semantic logic, with global confidence as the only basis for determination; Finally, the results of the two mechanisms are comprehensively determined: the confidence after the serial execution of the two mechanisms is compared with a threshold value, and if either mechanism exceeds the threshold value, it is determined to be adversarial, and if both mechanisms are below the threshold value, it is determined to be normal.

6. The knowledge constraint-based image description adversarial sample detection method according to claim 5, characterized in that: The consistency determination mechanism based on local visual attention specifically comprises the following steps: S3.2.a1: image description model generates word sequence verbatim and outputs an attention map for each word ; wherein, denotes the length of the natural language description sentence generated by the image description model; S3.2.a2: judge the word whether it exists in the entity library, if it exists, enter the judgment process; S3.2.a3: Attention Map Analyze and extract pixels with values ​​greater than 100%. area The specific expression is as follows: wherein, represents a visual region extraction operation, represents a saliency filtering threshold for attention pixel values from the attention map output by the image description model wherein, the visual region with the highest contribution to the current word is extracted. S3.2.a4: The region in the original image corresponding to the region the image content and the word together input a large language model and ask it to judge whether the region is a word the features possessed; S3.2.a5: Obtain the confidence score of the large model output If the final result is greater than the threshold Then it is determined as an adversarial sample.

7. The knowledge constraint-based image description adversarial sample detection method according to claim 5, characterized in that: The common sense rationality determination mechanism based on overall semantic logic specifically comprises the following steps: S3.2.b1 : Extracting an image description model to generate a word sequence ; S3.2.b2: outputting, by the large language model, a confidence score inputting the large language model, outputting, by the large language model, a confidence score ; S3.2.b3: If the confidence is higher than a preset threshold then it is determined that the sentence has semantic conflict or logical contradiction, and is determined as a confrontation sample.

8. A knowledge constraint based image description adversarial sample detection system, characterized in that: A knowledge constraint-based image description adversarial sample detection method for executing any one of claims 1-7.

Images