High-precision segmentation boundary guided open vocabulary detection large model confrontation sample generation method

Abstract

The invention belongs to the technical field of high-precision segmentation, and particularly relates to an open vocabulary detection large model adversarial sample generation method guided by a high-precision segmentation boundary. The method specifically comprises the following steps: re-parameterizing an original YOLO-World model f into a model f 'by using a text part L as a basis; adding the original sample X and the current disturbance epsilon n bit by bit to generate an adversarial sample Xn; inputting the adversarial sample Xn and the text part L into a fine tuning model f 'for cross fusion, and calculating current bounding box loss Lbbox, confidence loss Lconf and fusion loss LVL through a vision-text fusion loss module in the cross fusion process; the disturbance epsilon n is optimized through the calculated loss; and obtaining the effective disturbance epsilon n meeting an iteration ending condition, obtaining a mask M of the original sample, and carrying out film covering on the currently obtained effective disturbance epsilon n through the mask M to generate a final adversarial sample epsilon '.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of high-precision segmentation, and particularly relates to an open vocabulary detection large model adversarial sample generation method guided by high-precision segmentation boundaries. BACKGROUND

[0002] High-precision segmentation is one of the core technologies in the field of computer vision, aiming to achieve pixel-level accurate division of target objects in images. With the rapid development of artificial intelligence technology, this technology has evolved from traditional methods such as threshold segmentation and edge detection to complex model-driven paradigms based on deep learning. In recent years, the introduction of the Transformer architecture has broken through the receptive field limitations of traditional CNNs, enabling long-range dependency modeling through self-attention mechanisms, such as Swin-UNet and SegFormer. At the same time, three-dimensional segmentation technology has greatly improved segmentation accuracy through adaptive preprocessing strategies, such as nnUNet, which has improved the Dice coefficient to 0.91 in the BraTS brain tumor segmentation task, driving the clinical landing of medical image analysis. Current technology frontiers focus on multi-modal fusion, weakly supervised learning, and real-time lightweight deployment, and these advances are driving intelligent changes in fields such as autonomous driving, industrial quality inspection, and remote sensing interpretation.

[0003] At the same time, the rapid development of large models in recent years has provided strong support for open vocabulary detection (OVD). Large models often have large parameter quantities and complex network structures, and are fully pre-trained on large-scale datasets, enabling them to learn more rich and abstract feature representations. For the OVD task, the multi-modal information processing capabilities of large models can effectively combine information from different sources such as vision and text to assist in identifying new classes of objects. Some OVD models take advantage of these strengths of large models and have achieved excellent detection accuracy. Minderer et al. used the powerful representation capabilities of the existing large model CLIP to change the picture-level classification to patch-level classification, classifying and regressing each patch to achieve the task of open vocabulary detection; Li et al. combined image information and text information to propose GLIP, which can achieve open vocabulary detection accuracy exceeding all current SOTA models without using additional annotations.

[0004] However, as the capabilities of large models improve, their security and reliability are also facing serious challenges. In the process of continuously improving detection accuracy, the ability to resist malicious interference has become an important indicator of model capability. Adversarial sample attacks, as an effective means of testing model robustness, are gradually gaining attention. Adversarial samples are special inputs designed for deep learning networks that, while remaining almost unchanged to human observers, can cause advanced deep learning models to deviate.

[0005] The concept of adversarial samples was first proposed by Szegedy et al. in 2013, and an L-BFGS algorithm was used as an optimization tool to attack image classification networks. Since the target detection task not only needs to classify the target, but also needs to detect the position of the object, most adversarial attack methods on image classification are ineffective when used in target detection, and even the attack completely fails. Therefore, some adversarial sample generation methods specifically for target detection have appeared to meet the multi-target requirements in the task. According to the generation logic of adversarial samples, the existing target detection adversarial sample generation methods can be divided into three categories: optimization-based, generation-based, and gradient-based.

[0006] Optimization-based methods often find the optimal perturbation position and perturbation amount through an iterative optimization process, such as FGSM, BIM, PGD, DeepFool, etc. Although this method can achieve good attack results, the computational resources and time consumption caused by iteration cannot be ignored; generation-based methods generally adopt the way of training a generative adversarial network (GAN) to directly generate adversarial samples, such as PhysGAN, UEA, etc. This method has a great advantage in generation speed, but also has problems such as difficulty in converging during training, large perturbation amount, and easy detection; gradient-based methods are between the two, which can not only make full use of the gradient information of the victim model, but also generate adversarial samples with good attack effect at a faster speed.

[0007] As the current mainstream research direction, gradient-based algorithms have had many related research results in recent years. Xie et al. designed the DAG method for Faster R-CNN detector, which uses the loss function containing gradient information to guide the generation process of perturbation, forming dense perturbation distributed on the entire image; Li et al. proposed the RAP method for multiple two-stage target detectors, combining the position information and classification information of the detection frame, creating a robust adversarial perturbation that can still be effective under different transformations (rotation, scaling, etc.); Zhang et al. proposed a non-target attack method CAP, which adds a context loss based on the classification loss and position loss, which can effectively capture the strong features of objects in the image and add noise to these features; Chow et al. combined different types of gradient information in multiple ways as the basis for generating perturbation, designed the TOG method, which can achieve target disappearance, fake label, classification error, and other adversarial results according to actual needs.

[0008] While the above methods have achieved impressive results against adversarial attacks, they are all based on conventional object detection models. Due to the complex structure and strong robustness of large models, it is difficult to extend adversarial attacks from such conventional models to OVD models, or even large OVD models. Among the many large OVD models, YOLO-World has quickly become a research hotspot due to its superior performance and innovative design. As an extension of the YOLO series of models, YOLO-World significantly improves detection accuracy in open vocabulary scenarios by incorporating multimodal pre-training techniques while retaining its original real-time and accuracy advantages. Specifically, YOLO-World learns rich semantic representations through large-scale visual-text joint training, enabling the model to accurately identify unknown objects in images after receiving textual cues without additional training steps. For large models like YOLO-World, which integrate a wealth of knowledge, the existence of adversarial examples is not only a technical challenge but also poses profound social security risks. Summary of the Invention

[0009] In view of this, the present invention proposes a high-precision segmentation boundary-guided adversarial sample generation method for large models of open vocabulary detection. The adversarial samples generated by this method can prevent the model from experiencing problems such as confidence drop, detection box drift or loss when detecting adversarial samples, providing a basis for evaluating the robustness and generalization of large models.

[0010] To achieve the above object, the present invention is achieved through the following technical solutions:

[0011] A high-precision segmentation boundary-guided adversarial sample generation method for large open-word detection models. This method targets the large open-word detection model YOLO-World. The specific process is as follows:

[0012] Step 1: Extract and normalize the text information T, and use the obtained text part L as the basis to re-parameterize the original YOLO-World model f into model f′;

[0013] Step 2: Combine the original sample X with the current perturbation ε n Generate adversarial sample X by bitwise addition n ; The adversarial sample X n And the text part L is input into the fine-tuning model f′;

[0014] Adversarial example X n After the feature extraction of the backbone network and feature pyramid, the visual part V of the input information is formed, and the visual part L is input into the visual-text fusion module in YOLO-World for cross fusion to generate the final detection result Y; in the cross fusion process, the current bounding box loss L is calculated through the visual-text fusion loss modulebbox , a confidence loss l conf , and a fusion loss L VL ; the calculated loss is used to optimize the disturbance ε n ;

[0015] Step three, set a maximum disturbance threshold e, so that the total change of the disturbance ε is controlled between [0, e), and judge whether the iteration end condition is met, if not, return to step two, otherwise go to step four;

[0016] Step four, obtain the mask M of the original sample, and generate the final adversarial sample ε' by covering the current effective disturbance ε n with the mask M.

[0017] Optionally, the adversarial sample X n after feature extraction by the backbone network and the feature pyramid forms the visual part V of the input information.

[0018] The adversarial sample X n after feature extraction by the backbone network and the feature pyramid obtains feature maps of small, medium and large sizes, and the three sizes of feature maps are adaptively pooled to be consistent in size, then flattened and spliced to form the visual part V.

[0019] Optionally, the visual-text similarity s k between any sub-region v j in the visual part V and a specific text w k,j is calculated.

[0020] The fusion loss L VL is designed as:

[0021] L VL = E ε [max(s k,j )]

[0022] Wherein, E ε is a minimum optimization process on the disturbance vector ε, which continuously adjusts the vector ε by iteration to find the optimal disturbance that minimizes the index in the brackets, that is, E ε represents the target that the content finally reaches the minimum.

[0023] Optionally, the visual-text similarity s k,j is:

[0024]

[0025] Wherein, indicates the element-wise multiplication, L2Norm(·) is the L2 norm normalization, which is used for matrix normalization, so that its L2 norm is equal to 1; alpha is a scaling factor, and beta is an offset factor.

[0026] Optionally, the application sets a maximum perturbation threshold e, so that the total change of the perturbation epsilon is controlled between [0, e). The specific process is: using L ∞ norm and L2 norm constraints, the specific process is:

[0027] First, L ∞ norm constraint

[0028]

[0029] epsilon <- clip (|| epsilon || 2, -e1, e1) ∞

[0030] Wherein, epsilon i represents the i-th element in epsilon, and e1 represents the set pixel maximum perturbation threshold;

[0031] Second, L2 norm constraint

[0032]

[0033] Wherein, e2 represents the set vector maximum perturbation threshold.

[0034] Optionally, the application adopts a stochastic gradient descent (SGD) algorithm to optimize the perturbation epsilon n .

[0035] Optionally, the application extracts and normalizes the text information T, and the specific process is:

[0036] First, the n-gram algorithm is used to segment the natural language text to obtain the text information T, which contains n nouns, modifiers and a placeholder in the original natural language text, that is, the total length is n+1.

[0037] Second, the corresponding text embedding C is the length of the text T, and D is the embedding dimension;

[0038] Finally, the text part L is obtained by dynamically adjusting the text embedding W using the YOLO-World network. Optionally, the feature map of the small, medium and large sizes in the application has a dimension of ​Adaptive pooling is performed on the feature maps of three sizes to unify their size to (3, 3, 512), and then the feature maps are further flattened and spliced to form an image vector (1, 27, 512), i.e., the visual part V.

[0039] Optionally, the iteration end condition of the present application is that if the attack effect of the current obtained adversarial sample reaches the expected effect or reaches the set maximum iteration round, the iteration is ended.

[0040] Optionally, the present application obtains the mask M of the original sample through nnUNet.

[0041] Beneficial effects

[0042] Firstly, for the RepVL-PAN module unique to YOLO-World, the present application proposes a visual-text fusion loss L VL , which can effectively utilize the intermediate information generated in the forward propagation process, reduce the relevance between text and image, and effectively destroy the function of RepVL-PAN.

[0043] Secondly, the present application sets a threshold for the L ∞ norm and L2 norm to control the size of the total perturbation, so that the perturbation is distributed as much as possible in the positions that have a greater impact on the model decision, and a sparse and effective adversarial sample is generated.

[0044] Thirdly, the present application proposes a boundary guiding module based on high-precision segmentation, introduces the nnUNet framework to realize pixel-level accurate segmentation, extracts accurate boundary information of the target in the original sample, i.e., the mask M, and uses the boundary information to guide the generation of the adversarial sample, effectively avoids unnecessary modification of the background area, and to some extent reduces the perturbation of the perturbation vector.

[0045] Fourthly, the adversarial sample generation method designed for YOLO-World guided by high-precision segmentation boundary of the present application has double value for current large model research. Firstly, this helps to test the robustness of the OVD large model when facing malicious attacks, and verifies whether the performance of the OVD large model under extreme conditions can meet the high safety standard. Secondly, the practice of adversarial attack can reveal the potential weaknesses of YOLO-World or other similar large models in the OVD large model, and provide important feedback information for the design of more robust open vocabulary detection systems in the future. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0047] Figure 1 A whole flow schematic diagram of an open vocabulary detection large model adversarial sample generation method based on text-image fusion loss provided by the present application is provided.

[0048] Figure 2 A construction and calculation mode schematic diagram of a fusion feature loss provided by the present application is provided.

[0049] Figure 3 An adversarial perturbation vector perturbation amount optimization method schematic diagram provided by the present application is provided. DETAILED DESCRIPTION

[0050] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0051] It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict; and all other embodiments obtained by those skilled in the art based on the embodiments in the present disclosure without creative labor are within the scope of protection of the present disclosure.

[0052] It should be noted that the various aspects of the embodiments described below are within the scope of the appended claims. It should be apparent that the aspects described herein can be embodied in a wide variety of forms and that any specific structure and / or function described herein is merely illustrative. Based on the teachings provided herein one skilled in the art should appreciate that an aspect described herein can be implemented independently of any other aspects and that an aspect can be implemented both as any number of devices and / or as any number of methods. For example, an aspect can be implemented as a device that comprises any number of the features described herein. Additionally, an aspect can be implemented as any number of methods comprising any number of the features described herein. Additionally, one skilled in the art will appreciate that an aspect can be implemented as any number of software, firmware, hardware, and / or combinations thereof features described herein, and that the method aspects can be implemented any number of ways. For example, the methods can be implemented using software, hardware and / or any combination of these methods. One skilled in the art will appreciate that a software implementation of the described aspects can be implemented as a program.

[0053] The present application provides an open vocabulary detection large model adversarial sample generation method mainly based on visual-text loss. The method is aimed at the current most popular open vocabulary detection large model YOLO-World, and forms an adversarial sample with adversarial attack on the basis of original samples mainly composed of natural images. Specifically, the method takes the RepVL-PAN module involved in the cross fusion of images and texts in the YOLO-World model as the main adversarial attack target, designs a processing mode with a double-path structure for visual information and text information, and inputs the images and texts into the fusion module after being processed by the respective branches; in the fusion module, the image and text fusion capability of the model is invalidated according to the intermediate gradient loss calculation method, which further affects the detection accuracy of the whole model, as shown in Figure 1 .

[0054] Step 1: Extract and normalize the text information T, and use the obtained text part L as the basis to re-parameterize the original YOLO-World model f into model f′;

[0055] Step 2: Compare the original sample X with the current perturbation ε n Generate adversarial sample X by bitwise addition n ; The adversarial sample X n And the text part L is input into the fine-tuning model f′;

[0056] Adversarial example X n After the feature extraction of the backbone network and feature pyramid, the visual part V of the input information is formed, and the visual part L is input into the visual-text fusion module in YOLO-World for cross fusion to generate the final detection result Y; in the cross fusion process, the current bounding box loss L is calculated through the visual-text fusion loss module bbox , confidence loss l conf and fusion loss L VL ; Use the calculated loss to perturb ε n Optimize

[0057] Step 3: Set the maximum disturbance threshold e so that the total change of the disturbance ε is controlled between [0, e) and determine whether the iteration end condition is met. If not, return to step 2, otherwise go to step 4.

[0058] Step 4: Obtain the mask M of the original sample through nnUNet, and use M to obtain the effective perturbation ε n Lamination is performed to generate the final adversarial sample ε′.

[0059] In this embodiment, for the original sample X encoded in RGB format, the YOLO-World model f needs to provide text information T of the content that the user expects to detect in the image. Therefore, in step 1, before processing the original sample X, the model f is fine-tuned according to the text information T to form a fine-tuned model f′.

[0060] This embodiment resists sample X n After the feature extraction of the backbone network and feature pyramid, the visual part V of the input information is formed. YOLO-World has designed a visual-text fusion module to fuse multimodal information. The visual part V and the text part L of the input information are cross-fused, and the bounding box information bbox and target confidence information conf are generated by mutual guidance. They are then matched with the text information T to form the final detection result Y. At the same time, the gradient information during the propagation process is recorded, and the current bounding box loss L is calculated through the visual-text fusion loss module. bbox , confidence loss L conf and fusion loss L VLand the random gradient descent (SGD) algorithm is used to optimize the perturbation ε n This step aims to optimize the perturbation ε n of the current iteration according to the gradient information fed back in the most critical visual-text module in the victim model, supplemented by the regular bounding box information and confidence information in the target detection model, so that the pixels with larger perturbation values tend to be in positions that have a greater impact on the model gradient.

[0061] In step three, the perturbation constraint is set, that is, the maximum perturbation threshold e is set, so that the total change of the perturbation ε is controlled between [0, e). This step aims to limit the perturbation added to the original sample, on the one hand, to limit the change of the adversarial sample compared to the original sample to avoid being recognized by the naked eye, and on the other hand, to make the high-perturbation pixel points as much as possible to be distributed in positions with more sensitive gradients, further improving the perturbation efficiency. The iteration end condition can be that the attack effect reaches the expected effect or reaches the set maximum iteration number.

[0062] In step four, the mask M of the original sample is obtained by nnUNet, and the perturbation generated in the previous step is guided. After all the iteration rounds are completed, the original sample X' in the converted format is segmented by nnUNet with high precision, and the object mask M of the original sample is formed. The effective perturbation ε n is generated by covering the mask M, and the final adversarial sample ε' is generated. This step aims to limit the perturbation added to the original sample to the key area rather than the entire sample, which can effectively reduce the calculation amount of invalid perturbation; at the same time, the perturbation mainly generated in the foreground can make the adversarial sample more similar to the original image, avoiding obvious noise in the background, thereby enhancing the concealment.

[0063] The adversarial sample generated by the embodiment can cause the model to have problems such as confidence drop, detection box drift or missing when detecting the adversarial sample, provide a basis for robustness and generalization evaluation of large models, and reveal some basic limitations of the current model, providing an improvement direction for subsequent research.

[0064] The above process will be described in detail as follows:

[0065] Step one: extract and normalize the text information T, and parameterize the original YOLO-World model f into f' according to the text part L obtained.

[0066] For the original sample X encoded in RGB format, the YOLO-World model f needs to provide the text information T expected by the user to detect the content in the image, and fine-tune the model f according to the text information T to form a fine-tuned model f' before processing the original sample X.

[0067] For the text information T meeting the natural language specification, information collection needs to be performed by a text encoding module before inputting the model f', extracting the nouns and modifiers, and normalizing them into a uniform length vector to form the text part L of the input information. The obtained text part L is used as the basis for fine-tuning the model and is also input into the fine-tuned model f'.

[0068] The specific process of text information extraction and normalization processing is as follows: in the method, the natural language text from the user or the data set is first segmented by a simple n-gram algorithm to obtain the text T, which contains n important words (usually nouns) in the original natural text and a placeholder, i.e., the total length is n+1. Then, the corresponding text embedding W is extracted by the CLIP pre-trained Transformer text encoder where C is the length of the text T, and D is the embedding dimension, generally selected as 512. The text embedding W is dynamically adjusted by the YOLO-World network to obtain the text part L, which is also directly involved in the visual-text fusion loss calculation together with the feature map generated by the visual detection part.

[0069] Step two: input the original sample X (i.e., image data) and the current perturbation ε n into the model f' to generate the adversarial sample X n , and input the text part L into the fine-tuned model f'.

[0070] Step three: input the adversarial sample X n and the text part L into the fine-tuned model f', record the gradient information in the propagation process, and calculate the current boundary box loss L bbox , confidence loss L conf , and fusion loss L VL using the visual-text fusion loss module, and optimize the current perturbation ε n using the loss functions.

[0071] When the first execution of step three is performed, according to the boundary box information bbox, the target confidence information conf, and the decision-making information VL generated in the visual-text fusion module, different loss functions are designed during the propagation process of inputting the adversarial sample X1 and the text part L into the model f' to generate the detection result Y, and the losses L bbox , L conf , and V LV are calculated.

[0072] YOLO-World designs a visual-text fusion module to fuse multimodal information. It cross-fuses the visual part L and the text part L of the input information, and guides each other to generate bounding box information bbox, target confidence information conf, and category information described by symbolic text information T to form the final detection result Y.

[0073] like Figure 2 As shown, the fusion loss L VL The construction process is:

[0074] This method mainly relies on the designed combined loss function to optimize the perturbation vector ε. The combined loss function can be divided into four parts: the fusion feature loss L VL As the key part, the construction method is shown in 2. After the image (original sample) of size (H, W) is input into the model, the features are extracted through the backbone network and the path aggregation network to form feature maps of small, medium and large sizes. The dimensions are generally In order to fuse the features of the two different formats of image and text, adaptive pooling is performed on the feature maps of three sizes to unify their sizes to (3,3,512). At the same time, they are further flattened and spliced ​​to form an image vector V of (1,27,512). For any subregion v in V k , which is related to a specific text w j Visual-text similarity s k,j Determined by the following formula:

[0075]

[0076] Where ⊙ represents element-by-element multiplication, L2Norm(·) is L2 norm normalization, which is used to standardize the matrix so that its L2 norm is equal to 1; α is the scaling factor, and β is the offset factor.

[0077] Based on this, the fusion feature loss designed by the present invention is:

[0078] L VL =E ε [max(s k,j )] (2)

[0079] Among them, E ε This is the minimization optimization process on the perturbation vector ε. By iteratively adjusting the vector ε, we seek the optimal perturbation that minimizes the index in the brackets. Same below; max(·) represents the maximum value of visual-text similarity. E ε It is a symbolic representation of a minimization operator, which can be understood as E ε [f(·)]=min ε f(·).

[0080] By the fusion feature loss designed in this way, the image-text mutual perception ability of the RepVL-PAN module in the YOLO-World model can be effectively destroyed, and thus the performance of the entire model is reduced.

[0081] Step three: set a maximum perturbation threshold e, so that the total change of the perturbation ε is controlled between [0, e), and use the stochastic gradient descent (SGD) algorithm to optimize the perturbation ε n .

[0082] Through model back propagation, the gradient of the loss function to the input adversarial sample X n is calculated, reflecting the area of the input sample that has the greatest impact on the model output, which is used as the basis for perturbation optimization. Based on the recorded gradient information, the stochastic gradient descent (SGD) algorithm is used to iteratively optimize the perturbation.

[0083] Optimizing the perturbation vector ε through the above process can improve the adversarial attack of the formed adversarial sample, but this will increase the perturbation amount of ε to some extent, and thus the concealment of the formed sample will decrease.

[0084] The perturbation amount of the current perturbation vector is often measured by the L p norm. Different calculation methods of the norm reflect different emphases of the perturbation amount of the perturbation vector, such as the L2 norm, which represents the square root of the sum of the squares of all elements in the perturbation vector, reflecting the total perturbation amount of the vector; the L ∞ norm represents the element with the largest absolute value in the perturbation vector, reflecting the maximum perturbation value of a single pixel, etc.

[0085] In this embodiment, the combination of L2 and L ∞ norm constraint methods is used to limit the perturbation vector, as shown in Figure 3 , and the specific implementation is as follows:

[0086] In this method, the value of the initial perturbation vector ε is set to zero, and the maximum pixel perturbation threshold e1 and the maximum vector perturbation threshold e2 are set. In any optimization iteration,

[0087] First, L ∞ norm constraint is performed to limit the maximum perturbation value of a single pixel and enhance the local perturbation ability of the sample. The calculation method of the L ∞ norm is as follows:

[0088]

[0089] Where ε i represents the i-th element (pixel) in ε.

[0090] Thus, the L ∞ norm of the perturbation vector can be constrained in the following way:

[0091] ε←clip(‖ε‖ ∞ ,-e1,e1)

[0092] Secondly, L2 norm constraint is performed to limit the total energy of the perturbation, so that the sample is smoother. Similar to the L ∞ norm, the calculation method of the L2 norm is as follows:

[0093]

[0094] where ε i represents the i-th element (pixel) in ε.

[0095] Thus, the L2 norm of the perturbation vector can be constrained in the following way:

[0096]

[0097] In the above way, the maximum perturbation of the pixels of the perturbation vector and the maximum perturbation of the vector can be effectively constrained within e1 and e2, respectively, to ensure the concealment of the formed adversarial sample.

[0098] The above optimization process represents the optimization process in one iteration process. After multiple iterations until the attack effect of the current adversarial sample on the model reaches the expected effect, or after reaching the maximum iteration round, the iteration is ended to form the final effective adversarial sample.

[0099] When the optimized perturbation is obtained, it is returned as the current perturbation to step two.

[0100] The optimized ε perturbation is added to the original sample X again, forming the iterated adversarial sample X2, and X2 is input into the model f', and the perturbation ε' is further optimized according to the above description until an effective adversarial attack is achieved.

[0101] When the attack effect reaches the expected effect or reaches the set maximum iteration round, the current perturbation is the optimal perturbation.

[0102] This step designs a perturbation amount loss L L2 , which sets a threshold for the L2 norm of the perturbation vector to control the size of the total perturbation amount, so that the perturbation is distributed as much as possible in positions that have a greater impact on the model decision, resulting in a sparse and effective adversarial sample.

[0103] Step four: obtain the mask M of the original sample through nnUNet, and guide the perturbation generated in the previous step.

[0104] After all the iteration rounds are completed, the nnUNet is used to perform high-precision segmentation on the converted original sample X' to form an object mask M in the original sample, and the effective perturbation ε obtained through the previous optimization is adjusted through M n The film is generated to form a final adversarial sample ε'.

[0105] The application proposes an adversarial sample generation method for YOLO-World guided by high-precision segmentation boundaries, which has double value for current large model research. First, this helps to test the robustness of the OVD large model when facing malicious attacks, and verifies whether the performance of the OVD large model under extreme conditions can meet the high safety standard. Secondly, the practice of adversarial attack can reveal the potential weaknesses of YOLO-World or other similar large models in the OVD large model, and provide important feedback information for the subsequent design of more robust open vocabulary detection systems.

[0106] To sum up, the above is only a preferred embodiment of the application, and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A high-precision segmentation boundary-guided adversarial sample generation method for the open vocabulary detection large model, the method is targeted at the open vocabulary detection large model YOLO-World, characterized by: The specific process is: Step 1: Extract and normalize the text information T, and use the obtained text part L as the basis to re-parameterize the original YOLO-World model f into model f′; Step 2: Combine the original sample X with the current perturbation ε n Generate adversarial sample X by bitwise addition n ; The adversarial sample X n And the text part L is input into the fine-tuning model f′; Adversarial example X n After the feature extraction of the backbone network and feature pyramid, the visual part V of the input information is formed, and the visual part L is input into the visual-text fusion module in YOLO-World for cross fusion to generate the final detection result Y; in the cross fusion process, the current bounding box loss L is calculated through the visual-text fusion loss module bbox , confidence loss L conf and fusion loss L VL ; Use the calculated loss to perturb ε n Optimize; Step 3: Set the maximum disturbance threshold e so that the total change of the disturbance ε is controlled between [0, e) and determine whether the iteration end condition is met. If not, return to step 2, otherwise go to step 4. Step 4: Get the mask M of the original sample and use the mask M to get the effective perturbation ε n Lamination is performed to generate the final adversarial sample ε′.

2. The method for generating adversarial samples for open vocabulary detection using a high-precision segmentation boundary-guided large model according to claim 1 is characterized in that: Compute any sub-region v in the visual part V k With a specific text w j Visual-text similarity s k,j ; Design the fusion loss L VL for: L VL =E ε [max(s k,j )] Among them, E ε It is a minimization optimization process on the perturbation vector ε. By iteratively adjusting the vector ε, we seek the optimal perturbation that minimizes the index in the brackets.

3. The method for generating adversarial samples for open vocabulary detection using a high-precision segmentation boundary-guided large model according to claim 2 is characterized in that: The visual-textual similarity s k,j for: s k,j =α⊙L2Norm(v k )⊙L2Norm(w j ) T +β Where ⊙ represents element-by-element multiplication, L2Norm(·) is L2 norm normalization, which is used to standardize the matrix so that its L2 norm is equal to 1; α is the scaling factor, and β is the offset factor.

4. The method for generating adversarial samples for open vocabulary detection using high-precision segmentation boundary guidance according to claim 2 is characterized in that: The specific process of setting the maximum disturbance threshold e so that the total change of disturbance ε is controlled between [0, e) is as follows: using L ∞ Norm and L2 norm are constrained, the specific process is: First, perform L-norm constraint ε←clip(‖ε‖ ∞ ,-e1,e1) Among them, ε i represents the i-th element in ε, and e1 represents the set maximum pixel disturbance threshold; Secondly, perform L2 norm constraint Wherein, e2 represents the set maximum vector disturbance threshold.

5. The method for generating adversarial samples for open vocabulary detection using high-precision segmentation boundary guidance according to claim 2 is characterized in that: The stochastic gradient descent algorithm is used to n Optimize.

6. The method for generating adversarial samples for open vocabulary detection using a high-precision segmentation boundary-guided large model according to claim 2, characterized in that: The text information T is extracted and normalized, and the specific process is as follows: First, the n-gram algorithm is used to segment the natural language text to obtain text information T. T contains n nouns, modifiers, and a placeholder in the original natural language text, that is, the total length is n+1; Secondly, the corresponding text embedding is extracted through the CLIP pre-trained Transformer text encoder C is the length of the text T, and D is the embedding dimension; Finally, the YOLO-World network is used to dynamically adjust the text embedding W to obtain the text part L.

7. The method for generating adversarial samples for open vocabulary detection using a high-precision segmentation boundary-guided large model according to claim 2, characterized in that: The adversarial sample X n The specific process of forming the visual part V of the input information after feature extraction through the backbone network and feature pyramid is as follows: Adversarial example X n After feature extraction through the backbone network and feature pyramid, feature maps of three sizes (small, medium, and large) are obtained. The feature maps of the three sizes are adaptively pooled to make them of the same size, and then flattened and spliced ​​to form the visual part V.

8. The method for generating adversarial samples for open vocabulary detection using a high-precision segmentation boundary-guided large model according to claim 7, characterized in that: The feature maps of the three sizes of small, medium and large have dimensions of The feature maps of the three sizes are adaptively pooled to make their sizes uniform (3, 3, 512), and are further flattened and concatenated to form an image vector of (1, 27, 512), namely the visual part V.

9. The method for generating adversarial samples for open vocabulary detection using a large model guided by high-precision segmentation boundaries according to claim 2, characterized in that: The iteration end condition is: if the attack effect of the current adversarial sample reaches the expected effect or reaches the set maximum number of iteration rounds.

10. The method for generating adversarial samples for open vocabulary detection using a large model guided by high-precision segmentation boundaries according to claim 2, characterized in that: Obtain the mask M of the original sample through nnUNet.

Images