Evaluation method, system, device and medium for TI-ReID system

Abstract

The application discloses a kind of evaluation methods, systems, equipment and medium for TI-ReID system, they are corresponding scheme, in scheme, through gradual multimodal fusion mechanism and residual-based perturbation information generation mode, TI-ReID system can be well evaluated in target and no target attack The security performance;Moreover, based on the perturbation information generated by soft restraint function, the gradient reverse propagation beyond the constraint threshold is allowed, so as to optimize the attack effect, and further more effectively evaluate the security performance of TI-ReID system.The present application achieves consistent excellent performance on different data sets and models, can well evaluate the security performance, is also the first system to study TI-ReID security method, provides a new security evaluation scheme for the field.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security, and in particular to an evaluation method, system, device and medium for the TI-ReID system. Background Technology

[0002] In recent years, Person Re-Identification (ReID) technology has become a core component of intelligent surveillance systems, playing a crucial role in security tasks such as crime investigation and tracking people in public areas. Traditional ReID primarily relies on RGB (red, green, blue) image matching across cameras to retrieve target individuals, but in real-world scenarios, it often faces challenges such as blurry images, occlusions, or missing data. To overcome this limitation, Text-to-Image Person Re-Identification (TI-ReID) technology has emerged. It significantly expands the flexibility of application scenarios by fusing text descriptions provided by witnesses (such as "a man wearing a red coat and carrying a black backpack") with visual data for cross-modal matching. According to industry reports, the global intelligent security market will expand at a CAGR of 16.8% between 2022 and 2030, with TI-ReID becoming a deployment hotspot due to its superior performance in low-quality image scenarios.

[0003] However, the TI-ReID system relies heavily on visual language models and cross-modal alignment mechanisms, making it susceptible to inherent security vulnerabilities in deep neural networks. Attackers can manipulate model decisions by injecting subtle perturbations: on the one hand, causing non-targeted failures (such as completely excluding the target person from the search results), and on the other hand, enabling dangerous targeted identity forgery (such as associating a suspect with an innocent individual, resulting in a malicious incident where "a teacher wearing a blue shirt is mistakenly identified as a wanted criminal").

[0004] Current research on adversarial attacks suffers from three major setbacks: First, traditional ReID attack methods are designed only for single-modal images and cannot handle the modal gap between text and images. Their perturbation generation mechanisms lack cross-modal fusion capabilities, leading to transfer failures on TI-ReID systems. Second, there are bottlenecks in fundamental technologies: traditional hard-clamp functions with invisibility constraints block gradient backpropagation outside the threshold, causing perturbation optimization to get stuck in local optima; while alternatives such as the tanh function (hyperbolic tangent function) severely distort the output when dynamically adjusting the perturbation amplitude. Third, the attack dimensions are limited; existing solutions can only achieve non-directional interference, failing to meet the attack and defense verification needs of the security field, which requires simulating complex attack scenarios (such as specific identity forgery).

[0005] More concerning is that current research almost entirely ignores the security analysis of TI-ReID and has failed to establish a targeted security assessment framework. This research gap leads to two major risks: industry's lack of awareness of system vulnerabilities leads to blind deployment, while academia struggles to develop effective defense mechanisms due to the absence of attack benchmarks. Given that TI-ReID is already being used in high-risk scenarios such as airport security and smart cities, there is an urgent need to establish a security performance assessment scheme to promote the design and deployment of robust systems.

[0006] In view of this, the present invention is hereby proposed. Summary of the Invention

[0007] The purpose of this invention is to provide an evaluation method, system, device, and medium for TI-ReID systems. Based on the reconstruction residual, the security performance of the system is evaluated by generating adversarial perturbations, which can provide technical support for improving the robustness of text-image person re-identification systems.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] An evaluation method for TI-ReID systems includes:

[0010] Step 1: Obtain the original image and adversarial cue text;

[0011] Step 2: Encode the original image using an image encoder to obtain image features; encode the adversarial cue text using a text encoder to obtain text features; fuse the image features and text features using a progressive multimodal fusion mechanism to obtain adversarial image features; and generate a reconstructed image using the adversarial image features.

[0012] Step 3: Calculate the residual between the reconstructed image and the original image, and generate perturbation information based on the soft clamping function;

[0013] Step 4: Apply the perturbation information to the original image to obtain an image for security assessment;

[0014] Step 5: Input the image used for security assessment into the TI-ReID system to be evaluated, and evaluate the security performance based on the output results of the TI-ReID system to be evaluated; wherein, the TI-ReID system is a text-image pedestrian re-identification system.

[0015] An evaluation system for TI-ReID systems, used to implement the aforementioned method, includes:

[0016] The information acquisition module is used to acquire the original image and adversarial prompt text;

[0017] The reconstruction module is used to encode the original image using an image encoder to obtain image features, and to encode the adversarial prompt text using a text encoder to obtain text features; it then uses a progressive multimodal fusion mechanism to fuse the image features and text features to obtain adversarial image features; and finally uses the adversarial image features to generate the reconstructed image.

[0018] A residual perturbation generator is used to calculate the residual between the reconstructed image and the original image, and to generate perturbation information based on a soft clamping function;

[0019] A perturbation application module is used to apply the perturbation information to the original image to obtain an image for security assessment;

[0020] The security performance evaluation module is used to input the image used for security evaluation into the TI-ReID system to be evaluated, and to evaluate the security performance based on the output results of the TI-ReID system to be evaluated; wherein, the TI-ReID system is a text-image pedestrian re-identification system.

[0021] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0022] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0023] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0024] As can be seen from the technical solution provided by the present invention, the security performance of the TI-ReID system under targeted and untargeted attacks can be well evaluated by using the progressive multimodal fusion mechanism and the residual-based perturbation information generation method. Moreover, the generation of perturbation information based on the soft clamping function allows gradient backpropagation beyond the constraint threshold, thereby optimizing the security assessment effect and thus more effectively evaluating the security performance of the TI-ReID system. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of an evaluation method for a TI-ReID system provided in an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of an evaluation method for a TI-ReID system provided in an embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram of the overall workflow of the reconfiguration module provided in an embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram illustrating the overall principle of the progressive multimodal fusion mechanism provided in an embodiment of the present invention.

[0030] Figure 5 This is a schematic diagram of the residual disturbance generator provided in an embodiment of the present invention.

[0031] Figure 6 Push-pull loss provided in the embodiments of the present invention

[0032] Figure 7 This is a schematic diagram of the matching loss provided in an embodiment of the present invention.

[0033] Figure 8 This is a schematic diagram of an evaluation system for the TI-ReID system provided in an embodiment of the present invention.

[0034] Figure 9 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0036] First, the following explanations are provided for the terms that may be used in this article:

[0037] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0038] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0039] The following provides a detailed description of an evaluation method, system, device, and medium for the TI-ReID system provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, conventional conditions or conditions recommended by the manufacturer in the art should be followed. Reagents or instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0040] Example 1

[0041] This invention provides an evaluation method for TI-ReID systems, such as... Figure 1 As shown, it mainly includes the following steps:

[0042] Step 1: Obtain the original image and the target text (i.e., the adversarial prompt text).

[0043] The original image is the pedestrian image to be tested (real identity). The adversarial cue text is pre-set and describes a "designated identity" (target identity) that is completely different from the identity in the original image. The purpose of this step is to prepare test data so that the TI-ReID system, when processing the original image, will no longer recognize its real identity, but instead incorrectly point its search results to the "designated identity" described in the text.

[0044] Step 2: Reconstruct the image.

[0045] In this embodiment of the invention, an image encoder is used to encode the original image to obtain image features, and a text encoder is used to encode the adversarial prompt text to obtain text features; a progressive multimodal fusion mechanism is used to fuse the image features and text features to obtain adversarial image features; and the reconstructed image is generated using the adversarial image features.

[0046] In this embodiment of the invention, the multimodal fusion mechanism includes multi-layer multi-head cross-attention calculation, adopts layer adaptive attention scaling, and performs image feature and text feature fusion by adjusting the weights of text and image features layer by layer.

[0047] In this embodiment of the invention, adjusting the weights of text and image features layer by layer includes: setting a scaling factor. The scaling factor, used to control the weights of text features in multi-head cross-attention calculation, is updated as follows:

[0048] ;

[0049] in, Here, L and L+1 represent the L-th and L+1-th layers, respectively. This is the scaling factor.

[0050] In this embodiment of the invention, the step of fusing image features and text features by adjusting the weights of text and image features layer by layer includes: for the Lth layer, using a scaling factor. The weights of text features in multi-head cross-attention computation are controlled. The input features of layer L contain the output features of layer L-1 (the input of layer 0 is the image features obtained by encoding the original image using an image encoder), and multi-head cross-attention computation is performed on them. To preserve the original feature information, the features obtained from the multi-head cross-attention computation are added to the input features (i.e., residual connection), and the result is normalized to obtain intermediate features. Subsequently, multi-head cross-attention computation is performed using the intermediate features and text features. During this process, to control the degree of influence of text information on image features, a preset weight is used. The text features are weighted and multiplied by the text features extracted by the text encoder to obtain weighted text features; finally, the result of the multi-head cross-attention calculation is multiplied... Multiply by the intermediate feature The results are added together and then normalized. The obtained normalized features are then fed forward and added to the normalized features again for normalization, thus completing the fusion of text features and image features to obtain the final output of the Lth layer.

[0051] Finally, transposed convolution is used to generate the reconstructed image.

[0052] In this embodiment of the invention, both the image encoder and the text encoder are pre-trained. The structural similarity index loss between the image used for security assessment and the original image is used as the pre-training loss, and the pre-training loss is then used to pre-train both the image encoder and the text encoder simultaneously.

[0053] Step 3: Generate disturbance information.

[0054] In this embodiment of the invention, the residual between the reconstructed image and the original image is calculated, and perturbation information is generated based on a soft clamping function.

[0055] In this embodiment of the invention, a soft clamping function is introduced to control the disturbance amplitude. The soft clamping function is expressed as follows:

[0056] ;

[0057] in, For disturbance values, The threshold value is defined by the infinite norm constraint, and n is the smoothing temperature. The output value of the soft clamping function represents the constrained disturbance information.

[0058] Step 4: Generate images for security assessment.

[0059] In this embodiment of the invention, the perturbation information is applied to the original image to obtain an image for security assessment.

[0060] Step 5: Security performance assessment of the TI-ReID system.

[0061] In this embodiment of the invention, the image used for security assessment is input to the TI-ReID system (text-image person re-identification system) to be evaluated, and the security performance is evaluated based on the output results of the TI-ReID system to be evaluated.

[0062] In this embodiment of the invention, step 2 is implemented through a reconstruction module. To balance training efficiency and attack robustness, this invention employs a pairwise batch sampling strategy, specifically including:

[0063] (1) Data sampling: Two batches of “descriptive text-image pairs” are sampled as training data, denoted as batch A training set. and batch B training set Where I represents an image, T represents text, and the superscript represents a batch, with each batch containing... Regarding the data, the descriptive text-image pairs here are corresponding; that is, the descriptive text describes the identity of the pedestrian in the image.

[0064] (2) Cross-interchange to generate adversarial examples: Descriptive text from two batches is interchanged as adversarial cue text to generate image perturbations. For images in batch A of the training set... Using the description text of batch B training set As adversarial cue text designed to mislead specific targets, perturbation information is generated based on steps 2 and 3 and applied to the image. The above image of batch A used for security assessment was obtained. This constitutes the adversarial batch A training set. Similarly, for the images in batch B of the training set... Using the description text of batch A training set As adversarial cue text designed to mislead specific targets, perturbation information is generated based on steps 2 and 3 and applied to the image. Above, images of batch B used for safety assessment were obtained. This constitutes the adversarial batch B training set. .

[0065] Next, the adversarial batch A training set and the adversarial batch B training set are input into the TI-ReID system to be evaluated. The training loss is constructed by combining the output of the TI-ReID system to be evaluated, and the reconstruction module is trained.

[0066] In this embodiment of the invention, the TI-ReID system to be evaluated is used to... and By performing feature extraction separately, we can obtain feature sets for different batches, i.e. and Then, a training loss is constructed using the push-pull loss and the matching loss, and the reconstruction module is trained using the training loss.

[0067] Push-pull loss Represented as:

[0068] ;

[0069] ;

[0070] in, Indicates comparative loss, This represents the InfoNCE loss (contrast loss based on information noise contrast estimation), with the superscript -1 indicating the reciprocal.

[0071] Matching loss Represented as:

[0072] ;

[0073] in, This represents the set of image features in the training set of the adversarial batch j. This represents the set of text features in the training set of the adversarial batch k. This represents the set of text features in the training set of the adversarial batch j. Let i represent the image features in the i-th descriptive text-image pair of the adversarial batch j training set. This represents the text feature in the i-th descriptive text-image pair of the adversarial batch k training set; when At that time, that is or At that time, the label , indicates a mismatch, when At that time, that is or At that time, the label , indicating matching, in other words, sampling training samples with the same index i simultaneously from both the adversarial batch A training set and the adversarial batch B training set within the entire batch, for the calculated feature sets of different batches, i.e. and ,use As and The tag, using As and Tags; express and The fused probability output, more specifically, is obtained by using the fusion encoder of the TI-ReID system to be evaluated. and The features are fused to obtain cross features, and then multilayer perceptions (MLPs) combined with the Softmax (normalized exponential function) activation function are used to classify the cross features to obtain the evaluation results of the TI-ReID system. The probability of determining whether a pair is matched. This represents the cross-entropy loss.

[0074] The above-mentioned solution provided by the embodiments of the present invention can effectively evaluate the security performance of the TI-ReID system under targeted and untargeted attacks by using a progressive multimodal fusion mechanism and a residual-based perturbation information generation method. Moreover, by generating perturbation information based on a soft clamping function, gradient backpropagation beyond the constraint threshold is allowed, thereby optimizing the attack effect and thus more effectively evaluating the security performance of the TI-ReID system.

[0075] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0076] I. Overall Overview of the Plan

[0077] like Figure 2The diagram illustrates the above-described scheme provided by an embodiment of the present invention. The input information includes adversarial prompt text and the original image. First, the pre-trained encoders (text encoder and image encoder) in the reconstruction module encode the input information respectively, then perform progressive multimodal fusion to obtain adversarial image features, and then decode the reconstructed image through transposed convolution. Next, a residual perturbation generator uses the residual between the reconstructed image and the original image to generate perturbation information. This perturbation information is applied to the original image to obtain the attacked image (i.e., the image used for security assessment), which is then input into the TI-ReID system to be evaluated. The security performance is then evaluated based on the output of the TI-ReID system.

[0078] The above scheme employs an attack method based on reconstructed residuals, which can effectively test the security of the TI-ReID system.

[0079] II. Detailed introduction of the plan.

[0080] 1. Refactor the module.

[0081] like Figure 3 The diagram illustrates the overall workflow of the refactoring module. Figure 3 The flame symbol in the diagram indicates that the parameters of the relevant module need to be trained, and the snowflake symbol indicates that the parameters are frozen. This indicates that there are no training parameters.

[0082] The following is a detailed introduction to each step.

[0083] In this embodiment of the invention, adversarial perturbations are generated using reconstruction residual techniques. Specifically, a pre-trained text encoder encodes the input adversarial cue text description into a latent state to obtain text features. Subsequently, an adversarial perturbation generator generates adversarial perturbations in the latent space. These perturbations are designed to mislead the TI-ReID system, causing it to incorrectly match images with incorrect text descriptions.

[0084] The reconstruction module is the core component of this invention. It employs an autoencoder structure to generate adversarial images, injecting adversarial semantic depth into the image feature space while maintaining visual similarity. This module operates through a dual-branch encoder-decoder architecture. Specifically, the input raw image (typically a pedestrian image) is processed by a pre-trained image encoder to generate image features. The image encoder segments the raw image into a sequence of blocks of a specified size (e.g., 16*16) and extracts global features using a multi-head self-attention mechanism. Adversarial text prompts (e.g., "man in a blue jacket") are transformed into text features by a pre-trained text encoder. Subsequently, a progressive multimodal fusion mechanism is used to fuse the two types of features to achieve the attack operation.

[0085] For example, a text encoder could employ a BERT (Bidirectional Encoder Representation based on Transformer) encoder, which utilizes a bidirectional Transformer (Transformer Neural Network) to capture semantic context and ensure the integrity of identity features in adversarial cues. An image encoder could employ a Vision Transformer (ViT, Visual Transformer Neural Network) encoder.

[0086] Both encoders mentioned above are pre-trained encoders. They can be combined with the structural similarity between the image used for security assessment and the original image to calculate the pre-training loss and then pre-trained together. This will be introduced in detail later.

[0087] To overcome the differences between text and image modalities, this invention designs a Transformer-based progressive multimodal fusion module, which deeply integrates adversarial text prompts and image features using a progressive multimodal fusion mechanism. The core of this mechanism lies in layer-adaptive attention scaling, which achieves efficient fusion by adjusting the weights of text and image features layer by layer. Figure 4 The diagram illustrates the overall principle of the progressive multimodal fusion mechanism. Taking layer L as an example, the input features include the output features of layer L-1 (when L=0, the input features of layer L are the image features obtained by encoding the original image). Multi-head cross-attention is performed on these features, and the resulting features are added to the input features. The result is then normalized to obtain intermediate features. The scaling factor is then... Multiply the intermediate text features to obtain weighted text features. Then, perform multi-head cross-attention calculation on the weighted text features and intermediate features. Finally, multiply the result with a scaling factor. Multiply, and multiply with the intermediate feature The results are added together, and the added result is normalized to obtain normalized features. The normalized features are then fed forward and added to the normalized features, and then normalized again to obtain the output features of the Lth layer.

[0088] In this embodiment of the invention, a scaling factor is defined. The weights of text features in the multi-head cross-attention calculation are controlled by the following update formula:

[0089] ;

[0090] in The scaling factor ensures that the weights gradually decrease. L is the layer index in the Transformer model, starting from 0; for example, it can be set to... , .

[0091] In the initial layer, Larger text features dominate multi-head cross-attention computation, injecting adversarial identity cues into visual features. At a deeper level, The focus shifts to image self-attention to refine pixel-level perturbations, while residual learning preserves the original visual semantics. The fused features are decoded through transposed convolution and mapped back to the image space, ensuring that the attacked image is visually similar to the original image.

[0092] 2. Residual disturbance generator.

[0093] In this embodiment of the invention, the residual perturbation generator uses the residual between the reconstructed image and the original image output by the reconstruction module to generate perturbation information (adversarial perturbation). Specifically, the residual between the reconstructed image and the original image is calculated as the initial adversarial perturbation, and a soft clamping function is used to control the perturbation amplitude to achieve an effective and covert perturbation, thus obtaining the final adversarial perturbation.

[0094] To ensure that the perturbation is both effective and concealed, this invention introduces a soft-clamping function to control the perturbation amplitude. The soft-clamping function overcomes the limitations of traditional methods. The limitations of norm (infinite norm) constraints allow for backpropagation of gradients exceeding the constraint threshold, thereby optimizing attack performance. The soft clamping function is defined as:

[0095] ;

[0096] in, The residual (i.e., the initial adversarial perturbation) represents the amount of change in image pixels or features. for The threshold for the norm constraint, for example, can be set to 8 / 255. n is the smoothing temperature, for example, can be set to 10, controlling the smoothness of the function.

[0097] The derivative of the soft clamping function is:

[0098] ;

[0099] in, For soft clamping function pairs The derivative of is used for gradient backpropagation. The output value of the soft clamping function represents the constrained perturbation.

[0100] like Figure 5 The diagram shown illustrates the schematic of a residual perturbation generator, illustrating the input-output relationship of the soft-clamp function. The soft-clamp function dynamically adjusts... This ensures the disturbance is visually imperceptible while maintaining the attack's effectiveness. Furthermore, privacy considerations are taken into account. Figure 2 , Figure 3 as well as Figure 5The facial areas in the relevant images have been pixelated.

[0101] 3. Loss function.

[0102] In this embodiment of the invention, the loss function mainly includes the pre-training loss of the two encoders and the training loss of the entire reconstruction module.

[0103] (1) Pre-training loss.

[0104] To further ensure the visual concealment of the perturbation, this invention introduces the Structural Similarity Index (SSIM) loss as a pre-training loss, which further guarantees the visual concealment of the perturbation, such as... Figure 5 As shown. The structural similarity index is a metric used to measure the structural similarity between two images. The formula for calculating the structural similarity index loss is:

[0105] ;

[0106] in, and The images represent the original image and the image used for security assessment, respectively. and They are respectively and The average intensity reflects the brightness information of the image; and They are respectively and The variance measures the magnitude of change in the contrast or brightness of an image. for and The covariance is used to capture the correlation of structural information between two images; and It is a small constant used to stabilize the denominator and prevent it from approaching zero.

[0107] In this embodiment of the invention, by minimizing this loss, the perturbed image is ensured. Compared with the original image The high visual similarity further ensures the visual concealment of the perturbation. Combined with dynamic adjustment of the soft clamping function, this invention significantly improves the invisibility of the perturbation while maintaining attack efficiency, making it suitable for security assessments of TI-ReID systems.

[0108] (2) Training loss of the reconstruction module.

[0109] In this embodiment of the invention, the reconstruction module is optimized using push-pull loss and matching loss, respectively targeting the alignment and fusion stages of the TI-ReID model.

[0110] As mentioned earlier, during training, two batches of descriptive text-image pairs are sampled as training data, denoted as batch A training set. and B batch training set Adversarial samples are generated through cross-interchange to obtain adversarial batches. and And thereby obtain the feature set. and .

[0111] (2.1) Push-pull loss.

[0112] The push-pull loss is designed for the alignment stage based on contrastive learning, aiming to bring the attacking image closer to the adversarial cue while pushing away the similarity of the original image-text pairs. It is defined as follows:

[0113] ;

[0114] in, This represents the set of image features in the training set of the adversarial batch j. This represents the set of text features in the training set of the adversarial batch k. This represents the set of text features in the training set of the adversarial batch j. This indicates a comparative loss.

[0115] The contrast loss can be the InfoNCE-based contrast loss (contrast loss based on information noise contrast estimation), which can be expressed as:

[0116] ;

[0117] ;

[0118] in, Represents a single image feature. Represents a single text feature; The CLS token represents image features and indicates global image features. and Both represent CLS tags for text features, i and Index the samples (images and text) in a batch. express and The cosine similarity measures the similarity between features. N represents the batch size.

[0119] Push-pull loss impairs TI-ReID's alignment capabilities by increasing the similarity of incorrect pairings while decreasing the similarity of correct pairings. For example... Figure 6As shown, a schematic diagram of push-pull loss is presented, where the left side shows the calculation principle of push-pull loss, and the right side shows the loss optimization direction of push-pull loss in the potential space. Figure 6 The visual encoder and language encoder shown are both from the TI-ReID system to be evaluated.

[0120] (2.2) Matching loss.

[0121] Matching loss, specifically for the fusion stage based on supervised learning, increases the probability of incorrect pairings and decreases the probability of correct pairings by misleading the probability output of the reordering mechanism. Figure 7 The diagram shown illustrates the matching loss.

[0122] The matching loss is expressed as:

[0123] ;

[0124] ;

[0125] in, Let P represent the cross-entropy loss, used to optimize the probability distribution, where P represents the predicted probability and Y represents the label. At that time, that is or At that time, the label , indicates a mismatch, when At that time, that is or At that time, the label , indicates a match; express and The fused probability output is defined as:

[0126] ;

[0127] in, Softmax is a normalized exponential function, and Fusion represents the fusion layer.

[0128] Figure 7 In this study, both the fusion encoder and MLPs (multilayer perceptrons) are derived from the TI-ReID system to be evaluated. The MLPs are responsible for predicting the aforementioned probability outputs, and the snowflake symbol represents the frozen parameters.

[0129] Combining the above two losses, we obtain the training loss. By jointly optimizing the push-pull loss and the matching loss, this invention can simultaneously attack the alignment and fusion phases of TI-ReID, ensuring maximum attack effectiveness.

[0130] The above-described scheme provided in this invention, through the reconstruction residual and progressive multimodal fusion mechanism, performs excellently in both targeted and untargeted attacks, with a significantly higher attack success rate than existing methods. Furthermore, by introducing a soft clamping function and structural similarity index loss, it ensures that the perturbation is visually imperceptible while maintaining attack efficiency. Based on this, this invention achieves consistently excellent performance on different datasets and models, effectively evaluating security performance. It is also the first method to systematically study the security of TI-ReID, providing a new security assessment scheme for the field.

[0131] III. Example Explanation.

[0132] Based on the above explanation, the following section, using public datasets and standard models, provides examples of the implementation process for both targeted and untargeted attacks.

[0133] The CUHK-PEDES dataset was used, containing 15,013 pedestrian images and their corresponding text descriptions (e.g., "man wearing a red coat and carrying a black backpack"). The dataset was split into 80% training (12,010 images) and 20% test (3,003 images). Images were resized to 224*224 pixels, and adversarial cue text was segmented before being input into the BERT encoder. A pre-trained Vision Transformer (ViT-B / 16) was used as the image encoder, where B indicates the selection of the base model, and 16 represents the patch size (image patch size), i.e., the image is segmented into a 16*16 block sequence, and global features are extracted through a multi-head self-attention mechanism. The text encoder used a pre-trained BERT-base-uncased model to transform the adversarial text cue into text features (768 dimensions), where base indicates the base model and uncased indicates that it is not case-sensitive.

[0134] The decoder employs progressive multimodal fusion and transposed convolution to fuse features from different modalities (i.e., image and text features) and decode them back into image space. An adversarial perturbation generator generates adversarial perturbations in the latent space to mislead the TI-ReID system into incorrectly matching images and text. A pre-trained autoencoder uses structural similarity index loss to ensure visual similarity between the reconstructed image and the original image. Random adversarial text prompts, such as "woman in a blue jacket," are generated to disrupt the alignment capabilities of the TI-ReID system. In the progressive multimodal fusion mechanism, a scaling factor η = 0.9 is set to control the scaling factor. Scaling rate during update. The fused features are decoded back to image space via transposed convolution. The residual between the reconstructed image and the original image is calculated to generate an adversarial perturbation. A soft clamping function is used to control the perturbation amplitude, where the smoothing temperature n=10.

[0135] For the alignment stage, a push-pull loss is used; for the fusion stage, a matching loss is used. The total loss is the sum of the push-pull and matching losses. The Adam optimizer (adaptive moment estimator) was used with a learning rate of 0.001, a batch size of 32, and training for 10 epochs on an NVIDIA RTX 3090 GPU, taking approximately 12 hours. On the CUHK-PEDES test set, the original TI-ReID model achieved an accuracy of 76.5%, which dropped to 1.5% after the attack, while the identity forgery success rate reached 85.7%.

[0136] By combining the accuracy of the original TI-ReID model with the accuracy after the attack, users can determine the security and robustness of the TI-ReID model. Specifically, the smaller the difference in accuracy before and after the attack, the stronger the security and robustness of the TI-ReID model; the larger the difference in accuracy before and after the attack, the weaker the security and robustness of the TI-ReID model. In addition, the higher the success rate of identity forgery, the lower the security of the model, and the lower the success rate of identity forgery, the higher the security of the model.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0138] Example 2

[0139] This invention also provides an evaluation system for TI-ReID systems, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 8 As shown, the system mainly includes:

[0140] The information acquisition module is used to acquire the original image and adversarial prompt text;

[0141] The reconstruction module is used to encode the original image using an image encoder to obtain image features, and to encode the adversarial prompt text using a text encoder to obtain text features; it then uses a progressive multimodal fusion mechanism to fuse the image features and text features to obtain adversarial image features; and finally uses the adversarial image features to generate the reconstructed image.

[0142] A residual perturbation generator is used to calculate the residual between the reconstructed image and the original image, and to generate perturbation information based on a soft clamping function;

[0143] A perturbation application module is used to apply the perturbation information to the original image to obtain an image for security assessment;

[0144] The security performance evaluation module is used to input the image used for security evaluation into the TI-ReID system to be evaluated, and to evaluate the security performance based on the output results of the TI-ReID system to be evaluated; wherein, the TI-ReID system is a text-image pedestrian re-identification system.

[0145] Since the main technical details of the above system have been described in detail in the previous embodiments, they will not be repeated here.

[0146] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0147] Example 3

[0148] The present invention also provides a processing device, such as Figure 9 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0149] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0150] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0151] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0152] The output device can be a display terminal;

[0153] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0154] Example 4

[0155] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0156] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0157] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. An evaluation method for TI-ReID systems, characterized in that, include: Step 1: Obtain the original image and adversarial cue text; Step 2: Encode the original image using an image encoder to obtain image features, and encode the adversarial prompt text using a text encoder to obtain text features; A progressive multimodal fusion mechanism is used to fuse the image features and text features to obtain adversarial image features; the reconstructed image is then generated using the adversarial image features. The progressive multimodal fusion mechanism for fusing image and text features includes: the multimodal fusion mechanism comprises multiple layers of multi-head cross-attention computation, employs layer-adaptive attention scaling, and fuses image and text features by adjusting the weights of text and image features layer by layer; wherein, adjusting the weights of text and image features layer by layer includes: setting a scaling factor. The scaling factor, used to control the weights of text features in multi-head cross-attention calculation, is updated as follows: ; Here, L and L+1 represent the updated scaling factor, respectively, the L-th layer and the L+1-th layer. This refers to the scaling factor; Step 3: Calculate the residual between the reconstructed image and the original image, and generate perturbation information based on the soft clamping function; Step 4: Apply the perturbation information to the original image to obtain an image for security assessment; Step 5: Input the image used for security assessment into the TI-ReID system to be evaluated, and evaluate the security performance based on the output results of the TI-ReID system to be evaluated; wherein, the TI-ReID system is a text-image pedestrian re-identification system.

2. The evaluation method for a TI-ReID system according to claim 1, characterized in that, The process of fusing image and text features by adjusting the weights of text and image features layer by layer includes: For the Lth layer, use the scaling factor Controlling the weights of text features in multi-head cross-attention computation; The input features of layer L contain the output features of layer L-1. Multi-head cross-attention is performed on these features, and the resulting features are added to the input features. The sum is then normalized to obtain intermediate features. The scaling factor is then applied. Multiply the intermediate text features to obtain weighted text features. Then, perform multi-head cross-attention calculation on the weighted text features and intermediate features. Finally, multiply the result with a scaling factor. Multiply, and multiply with the intermediate feature The results are added together, and the added result is normalized to obtain normalized features. The normalized features are then fed forward and added to the normalized features, and then normalized again to obtain the output features of the Lth layer. When L=0, the input is the image features obtained by encoding the original image using an image encoder.

3. The evaluation method for a TI-ReID system according to claim 1, characterized in that, Both the image encoder and the text encoder are pre-trained. The structural similarity index loss between the image used for security assessment and the original image is calculated as the pre-training loss, and the pre-training loss is then used to pre-train both the image encoder and the text encoder simultaneously.

4. The evaluation method for a TI-ReID system according to claim 1, characterized in that, The generation of perturbation information based on the soft clamping function includes: A soft-clamp function is introduced to control the disturbance amplitude. The soft-clamp function is expressed as follows: ； in, For disturbance values, The threshold value is the infinite norm constraint, and n is the smoothing temperature. The output value of the soft clamping function represents the constrained disturbance information.

5. The evaluation method for a TI-ReID system according to claim 1, characterized in that, Step 2 is implemented through a reconstruction module, which obtains adversarial batch A and adversarial batch B training sets by generating adversarial examples through data sampling and cross-interchange. These training sets are then input into the TI-ReID system to be evaluated, and a training loss is constructed based on the output of the TI-ReID system. The reconstruction module is then trained accordingly. Data sampling includes sampling two batches of descriptive text-image pairs as training data, denoted as batch A and training set respectively. and batch B training set Where I represents an image, T represents text, and the superscript represents a batch, with each batch containing N pairs of data; Cross-interchange generative adversarial examples include: images from the training set of batch A. Using the description text of batch B training set As adversarial cue text, perturbation information is generated based on steps 2 and 3 and applied to the image. The above image of batch A used for security assessment was obtained. This constitutes the adversarial batch A training set. Images for batch B training set Using the description text of batch A training set As adversarial cue text, perturbation information is generated based on steps 2 and 3 and applied to the image. Above, images of batch B used for safety assessment were obtained. This constitutes the adversarial batch B training set. .

6. The evaluation method for a TI-ReID system according to claim 5, characterized in that, Training losses include: push-pull loss and matching loss; in: Push-pull loss Represented as: ； Matching loss Represented as: ； in, This represents the set of image features in the training set of the adversarial batch j. This represents the set of text features in the training set of the adversarial batch k. This represents the set of text features in the training set of the adversarial batch j. Let i represent the image features in the i-th descriptive text-image pair of the adversarial batch j training set. The text features in the i-th descriptive text-image pair of the adversarial batch k training set are represented by the text features extracted by the TI-ReID system to be evaluated. Indicates comparative loss; when At that time, that is or At that time, the label , indicates a mismatch, when At that time, that is or At that time, the label , indicates a match; express and The probability output after fusion This represents the cross-entropy loss.

7. An evaluation system for TI-ReID systems, characterized in that, To implement the method according to any one of claims 1 to 6, comprising: The information acquisition module is used to acquire the original image and adversarial prompt text; The reconstruction module is used to encode the original image using an image encoder to obtain image features, and to encode the adversarial prompt text using a text encoder to obtain text features; it then uses a progressive multimodal fusion mechanism to fuse the image features and text features to obtain adversarial image features; and finally uses the adversarial image features to generate the reconstructed image. A residual perturbation generator is used to calculate the residual between the reconstructed image and the original image, and to generate perturbation information based on a soft clamping function; A perturbation application module is used to apply the perturbation information to the original image to obtain an image for security assessment; The security performance evaluation module is used to input the image used for security evaluation into the TI-ReID system to be evaluated, and to evaluate the security performance based on the output results of the TI-ReID system to be evaluated; wherein, the TI-ReID system is a text-image pedestrian re-identification system.

8. A processing apparatus, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images