Target detection model robustness test method based on cross-task learning

Abstract

The invention provides a target detection model robustness test method based on cross-task learning, features extracted by a semantic segmentation model are teacher features, features extracted by a target detection model are student features, the similarity between the teacher features and the student features is evaluated by using a query-key attention mechanism, and the robustness of the target detection model is tested. The optimization process of stabilizing the gradient by adopting a historical gradient stabilization strategy is adopted, the conflict gradient direction is adjusted through conflict gradient projection, better gradient compatibility is obtained in combination with alternate joint task element optimization, and the test performance is improved. According to the method, diversified gradient information with strong correlation required by the target detection task is distilled from the semantic segmentation task, so that the target detection model can learn more universal and robust characteristics, the gradient direction is further stabilized and optimized, and the performance of the model robustness test is improved.

Description

Technical Field

[0001] The present invention relates to the field of robustness testing of multi-task learning models, and specifically proposes a method and system for generating highly transferable test samples by distilling target detection task-related information from semantic segmentation auxiliary tasks, learning more general and robust features. The entire system uses a feature linking method based on an attention mechanism to perform knowledge distillation, construct a target detection proxy model set, and improve the effective test rate of the generated test samples for the black box model. Background Art

[0002] Model robustness testing refers to the tester generating test samples by adding subtle perturbations that are difficult for humans to perceive to the original input data, making the model output wrong. This is of great significance to the research and improvement of the security of the model. Researching the robustness test of the target detection model can provide a deeper understanding of the limitations and vulnerabilities of the target detection model and provide guidance for the improvement and enhancement of the model. Key fields such as autonomous driving, security monitoring, and medical imaging diagnosis are highly dependent on the accuracy of the target detection model. Therefore, the robustness test problem of the target detection model has aroused extensive research and attention in the industry. Existing robustness testing techniques for target detection models can only achieve good testing results under white-box settings. White-box testing allows attackers to understand the network structure, parameters, and training data of the target detection model. In real scenarios, testers often face black-box models with unknown information, which is more challenging than white-box testing.

[0003] At present, the black-box robustness test for target detection models mainly adopts a transfer-based method, which means that the tester uses a known white-box model to generate test samples, hoping that the generated test samples can also be migrated to the black-box model with unknown information to produce the same test effect. However, the test effect based on migration is limited by the similarity between the source white-box model and the target black-box model. If there are large differences between the models, the test effect may be affected and it is difficult to apply to different types of target detection models and scenarios. This patent aims to solve the problem of insufficient transferability in the robustness test of black-box models based on migration, and proposes a new method to generate test samples to ensure that it has a good effective test rate in different types of target detection models. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a robustness testing method for target detection models based on cross-task learning, which can utilize the relevant and diverse gradient information provided by semantic segmentation auxiliary tasks, supplement the inherent missing knowledge of target detection tasks, and generate highly transferable test samples.

[0005] The main modules of the technical solution of the present invention include the following: the system includes two parts, the first part is the process of training multiple target detection proxy models using a knowledge distillation method based on an attention mechanism, and the second part is a multi-task optimization process based on historical alternating testing and gradient projection. In the first part, the features extracted by the semantic segmentation model are teacher features, and the features extracted by the target detection model are student features. The query-key attention mechanism is used to evaluate the similarity between the teacher features and the student features, which is used to control the distillation strength of all possible feature pairs, and the channel average pooling method is used to process the features, extract the main features and reduce the dimensions, and combine the feature pair similarity for knowledge distillation. In the second part, the historical gradient stabilization strategy is used to stabilize the gradient optimization process, and the conflicting gradient direction is adjusted through conflicting gradient projection, and combined with alternating joint task meta-optimization to obtain better gradient compatibility and improve test performance.

[0006] The technical solution adopted by the present invention to solve the technical problem comprises the following main steps:

[0007] Step 1: A target image is passed through the semantic segmentation model to obtain a set of teacher feature sets Then input the target image into the backbone network of the target detection model to obtain the student feature set , and Represents the number of features from the teacher and the student respectively, each with its own feature map size and channel dimension;

[0008] Step 2: Use the query-key attention mechanism to determine the correlation value between any set of teacher features and student features; each teacher feature generates a query , each student feature is identified as a key , by using the query and key, the association value between teacher and student features is calculated;

[0009] Step 3: Perform channel average pooling operation on teacher features and student features;

[0010] Step 4: Information Enhancement Proxy Model Training Loss Function The final expression is as follows:

[0011] (1)

[0012] in, represents the classification and regression losses of the target detection model, and the hyperparameters is the weight parameter of the impact of distillation loss on the total loss, Denotes the distillation term loss; During the training process, the target detection model and the network based on the query-key attention mechanism are guided by the total loss at the same time to jointly improve the performance of the model; Two target detection white-box proxy models are obtained through training, which are represented as models and Model , the proxy model set is ;

[0013] Step 5: Establish the corresponding loss function of the corresponding proxy model in the model robustness test phase; the specific steps are:

[0014] Define the input sample as , the detection output of the target detection test model is ,in Include detection frames, each detection frame is express, ; , Indicates the horizontal coordinate of the upper left corner of the detection box. Indicates the vertical coordinate of the upper left corner of the detection box. Indicates the width of the detection box. Indicates the height of the detection box. Indicates the category of the detection box, Indicates the confidence of the detection box; Randomly select the test box , ; Select the test detection box category as ; All categories that have appeared in the training data set are a set , in the collection Randomly select the target category , modify the test detection box Category For the target category , and obtain a new set of test results ,Will As input image The true value of the test is used to measure the actual test results. and truth value The differences:

[0015] (2)

[0016] in, , U represents the set of proxy models, represents the proxy models contained in the set U, Indicates the detection results of each detection box and truth value The sum of the bounding box regression losses, Indicates the detection results of each detection box and truth value The sum of the categorical cross entropy losses, Represents the corresponding loss function of the corresponding proxy model in the model robustness testing phase;

[0017] Step 6: To generate test samples, modify the input samples by adding changes Iteratively modify the pixels in;

[0018] Step 7: Define an internal iteration count , and then calculate according to formula (3) Make changes and record the corresponding gradient of the changes;

[0019] Step 8: The calculated changes are added to the input sample to generate test samples and pass the calculation results to another model Calculate its corresponding gradient in

[0020] Step 9: Gradient and The conflict gradient projection technique is applied gradient by gradient, namely:

[0021] (4)

[0022] in, , thus obtaining the model Gradient group after applying conflicting gradient projection technique ;

[0023] Step 10: Aggregate all gradient information and update the model Changes on , the calculation formula is as follows:

[0024] (5)

[0025] in, , is to update the hyperparameters, is the learning rate, is the output of each iteration change, and the Sign(·) function returns the sign of the input data. Indicates that the first branch selected in the iteration is When the internal iteration reaches After the changes calculated times reach the maximum number of external iterations, the final change result is obtained and added to the original image to obtain the required test sample.

[0026] In step 2, query and key The formula is as follows:

[0027] (6)

[0028] (7)

[0029] in, and The indices representing the features of teachers and students respectively, , , represents the global average pooling operation, and are the activation functions for query and key, respectively, and the parameters and They are Teacher characteristics and The linear transformation parameters of the student characteristics, is the set of real numbers, is the maximum number of channels, the value is 128, and Respectively represent Teacher characteristics and The number of channels parameter of the student characteristics, and Mapping raw features to query and key spaces allows for better comparison of similarities between teacher and student features.

[0030] The calculation formula of the association value is:

[0031]

[0032] (8)

[0033] in, represents the activation function; represents the transpose function; and introduces an additional bilinear weight parameter, the weight parameter is , ; represents the bilinear weights used to generalize the relevance values ​​from different sources, and the position encoding is and , Represents the positional encoding of the first student feature; the query and key are extracted from features of different dimensions and are weighted by bilinear Generalizing relevance values ​​from different sources; positional encoding It is used to share common information between different features; Indicates The query corresponding to the first student feature under the teacher feature, Indicates The first The query corresponding to the student characteristics, the correlation value is an attention vector that captures the The relationship between teacher characteristics and all student characteristics is obtained by using Selectively transfer knowledge of teacher features to student features.

[0034] In step 4, the final distillation loss is:

[0035] (9)

[0036] in, Represents the channel average pooling layer and The normalized combination function is used to process the teacher and student characteristics; in addition, is from The feature maps of the student features are upsampled or downsampled to match the size of the teacher features in order to better compare and analyze the differences between the teacher and student features, thereby helping the process of knowledge distillation.

[0037] In step 6, the maximum number of external iterations is defined , in each iteration, take turns from the collection Select a branch from the list. The order of branch selection does not affect the final performance. Let the first selected branch in the iteration be , updated changes Calculated by formula (3):

[0038] (3)

[0039] in, represents a random initialization change, is the learning rate, represents the symbolic function, is to update the hyperparameters, Indicates the selected branch of loss, represents the gradient corresponding to the loss, express The projection function for the constraint.

[0040] The calculation in step 7 The formula for making changes and recording the corresponding gradient of the changes is:

[0041] (10)

[0042] (11)

[0043] in, Indicates that the first branch selected in the iteration is When the internal iteration reaches The calculated changes are express The modified group obtained by this calculation; Indicates The gradient value corresponding to the change obtained by the calculation is express The gradient group obtained by this calculation.

[0044] In step 8, the calculation formula of the corresponding gradient is as follows:

[0045] (12)

[0046] in, , , Indicates that the Calculated changes Added to the attack image and passed to the branch The gradient value calculated when , thus getting , express Changes are delivered to the branch The gradient value group calculated later.

[0047] An electronic device comprises: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method as described above.

[0048] A computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method as described above.

[0049] The beneficial effect of the present invention is that through cross-task learning, it is possible to distill the diverse and highly correlated gradient information required for the target detection task from the semantic segmentation task, so that the target detection model can learn more general and robust features. At the same time, the feature linking method based on the attention mechanism effectively captures the degree of correlation between the teacher features and the student features in the knowledge distillation process, making the distillation process more accurate. In addition, the multi-task optimization strategy of historical alternation test and gradient projection further stabilizes and optimizes the direction of the gradient, and improves the performance of the model robustness test. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram of the structure of the robustness testing method of the target detection model based on cross-task learning in the present invention.

[0051] Figure 2 Schematic diagram of the knowledge distillation network structure based on the attention mechanism of the present invention.

[0052] Figure 3 It is a schematic diagram of the structure of the multi-task optimization algorithm based on historical alternating test and gradient projection of the present invention.

[0053] Figure 4a is the detection result of the original clean sample of image 1 in the Libra detector; Figure 4b Image 1 test sample detection results in the Libra detector, Figure 4c is the detection result of the original clean sample of image 2 in the Libra detector; Figure 4d It is the detection result of the image 2 test sample in the Libra detector. DETAILED DESCRIPTION

[0054] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0055] The present invention provides a method for testing the robustness of a target detection model based on cross-task learning. The overall structure of the method is as follows: Figure 1 The specific implementation process is as follows:

[0056] Step 1. Proxy model preparation

[0057] The PspNet and DeepLab semantic segmentation models pre-trained on the COCO dataset in the MMSegmentation library are selected as teacher models, and the Faster RCNN and RetinaNet target detection models in the MMDetection library are selected as student models.

[0058] Step 2: Target image preprocessing

[0059] Input the target image, adjust the image size, and use the bilinear interpolation method to adjust the image dimension to , adapt to the image encoder's requirements for images; perform image enhancement processing to The image is horizontally flipped with a probability of , and the bounding box annotations are adjusted accordingly.

[0060] Step 3: Extract candidate feature sets

[0061] For the Faster RCNN and PspNet student-teacher model group (the same method is used for the RetinaNet and DeepLab student-teacher model group), the batch size is set to 4. First, the Faster RCNN backbone network is extracted. The features of the residual layer are obtained Dimensions respectively, , , , The student features of the PspNet backbone network are extracted The features of the residual layer are obtained The dimensions are , , , The same method is used for the RetinaNet and DeepLab student-teacher model groups to obtain The dimensions are , , , characteristics of students, and The dimensions are , , , Teacher characteristics.

[0062] Step 4: Calculate feature correlation

[0063] like Figure 2 As shown in , for each teacher feature and each student feature, global average pooling, linear transformation, batch normalization, and ReLU activation operations are performed in sequence, and the teacher features and student features after the operations are stacked on a new dimension, and the shape of the stacked student features is reshaped, and the final shape is The query vector and the shape is Key Vector . Initialize a shape The learnable parameter matrix , learn the location information of the feature. and key vector Perform bilinear transformation on each feature map in , perform matrix multiplication on any transformed teacher-student feature pair, and add the calculated result to the learnable parameter matrix Add and proceed Operation, finally get a shape of , represents the correlation value matrix between the student feature map and the teacher feature map .

[0064] Step 5: Calculation of distillation loss

[0065] For each teacher-student feature pair, an adaptive average pooling layer is created to adjust the height and width of the input feature map to the height and width of the teacher feature map, and the spatial size of the student feature map is adjusted using adaptive average pooling. The teacher feature map and the adjusted student feature map are average pooled to obtain the sizes of , , , The student features of the same size and the student features of the same size are matrix-differed for each teacher-student feature pair. Normalization operation is performed, and the result is multiplied by the correlation value of the teacher-student feature pair. The results of all teacher-student feature pairs are accumulated to obtain the distillation loss. The distillation loss is multiplied by the weight parameter 200, and the respective losses of the Faster RCNN and RetinaNet models in the original MMDetection library are combined as their respective total loss functions.

[0066] Step 6: Model training

[0067] The entire training process is end-to-end training. The model optimizer uses a momentum factor of And the weight decay is The batch size of the input data is set to Total training epochs, the overall learning rate is set to The learning rate scheduler uses a linear scheduler that linearly increases the learning rate within the first 500 iters and a multi-step scheduler that decays the learning rate in the 8th and 11th cycles.

[0068] Step 7: Test implementation

[0069] The test implementation process is as follows Figure 3 As shown, set the number of external iterations to , the number of internal iterations is , , , The values ​​are , , For a shape The target image is initialized with a shape-matched modification. First, the Faster RCNN proxy model branch is selected and the calculation Iteration gradient and changes , switching to another RetinaNet proxy model branch, calculated based on the five changes in the Faster RCNN proxy model branch Iteration gradient ,Will Element-wise projection onto Thus updating , add the first four updated gradients and multiply by the weight Then add it to the fifth gradient to get the total gradient of this external iteration, and calculate the change size based on it. Repeat the above process until the maximum number of external iterations is reached, and get the total change. After adding the total change to the original image, get the test sample generated by this test.

[0070] Step 8: Method application

[0071] After the above process, we get two knowledge-enhanced object detection proxy models. We select two COCO dataset test set images and use the Libra detector as a black box model for testing. At this time, the image data does not need to be enhanced. We only need to adjust the image to The size can be used as the input of the model, and the parameters of the entire network model are fixed. In the actual experiment, the test effect of the first picture is as follows Figure 4a and Figure 4b As shown, the selected test category is surfboard and the target category is bus, where Figure 4a The image represents the detection result of the original clean sample in the Libra detector; Figure 4b The image shows the detection result of the test sample in the Libra detector. It can be seen that the surfboard is successfully tested as a bus category. The test effect of the second picture is as follows Figure 4c and Figure 4d As shown, the selected test category is person and the target category is tie. Figure 4c The image represents the detection result of the original clean sample in the Libra detector; Figure 4d The image shows the detection result of the test sample in the Libra detector. It can be seen that the person is successfully tested as a tie category.

Claims

1. A robustness testing method for target detection models based on cross-task learning, characterized in that The steps include: Step 1: A target image is passed through the semantic segmentation model to obtain a set of teacher feature sets Then input the target image into the backbone network of the target detection model to obtain the student feature set , and denote the number of features from the teacher and the student, respectively, each with its own feature map size and channel dimension; Step 2: Use the query-key attention mechanism to determine the correlation value between any set of teacher features and student features; each teacher feature generates a query , each student feature is identified as a key , by using the query and key, the association value between teacher and student features is calculated; Step 3: Perform channel average pooling operation on teacher features and student features; Step 4: Information Enhancement Proxy Model Training Loss Function The final expression is as follows: ; in, represents the classification and regression losses of the target detection model, and the hyperparameters is the weight parameter of the impact of distillation loss on the total loss, Denotes the distillation term loss; During the training process, the target detection model and the network based on the query-key attention mechanism are guided by the total loss at the same time to jointly improve the performance of the model; Two target detection white-box proxy models are obtained through training, which are represented as models and Model , the proxy model set is ; Step 5: Establish the corresponding loss function of the corresponding proxy model in the model robustness testing phase; Step 6: To generate test samples, modify the input samples by adding changes Iteratively modify the pixels in; Step 7: Define an internal iteration count , then calculate Make changes and record the corresponding gradient of the changes; Step 8: The calculated changes are added to the input sample to generate test samples and pass the calculation results to another model Calculate its corresponding gradient in Step 9: Gradient and The conflict gradient projection technique is applied gradient by gradient, namely: ; in, , thus obtaining the model Gradient group after applying conflicting gradient projection technique ; Step 10: Aggregate all gradient information and update the model Changes on , the calculation formula is as follows: ; in, , is to update the hyperparameters, is the learning rate, is the output of each iteration change, and the Sign(·) function returns the sign of the input data. Indicates that the first branch selected in the iteration is When the internal iteration reaches After the changes calculated times reach the maximum number of external iterations, the final change result is obtained and added to the original image to obtain the required test sample.

2. The method for testing robustness of target detection models based on cross-task learning according to claim 1, characterized in that: In step 2, query and key The formula is as follows: （6）; （7）; in, and The indices representing the features of teachers and students respectively, , , represents the global average pooling operation, and are the activation functions for query and key, respectively, and the parameters and They are Teacher characteristics and The linear transformation parameters of the student characteristics, is the set of real numbers, is the maximum number of channels, the value is 128, and Respectively represent Teacher characteristics and The number of channels parameter of the student characteristics, and Mapping raw features to query and key spaces allows for better comparison of similarities between teacher and student features.

3. The method for testing robustness of target detection models based on cross-task learning according to claim 1, characterized in that: The calculation formula of the association value in step 2 is: ; （8）; in, represents the activation function; represents the transpose function; and introduces an additional bilinear weight parameter, the weight parameter is , ; represents the bilinear weights used to generalize the relevance values ​​from different sources, and the position encoding is and , Represents the positional encoding of the first student feature; the query and key are extracted from features of different dimensions and are weighted by bilinear Generalizing relevance values ​​from different sources; positional encoding It is used to share common information between different features; Indicates The query corresponding to the first student feature under the teacher feature, Indicates The first The query corresponding to the student characteristics, the correlation value is an attention vector that captures the The relationship between teacher characteristics and all student characteristics is obtained by using Selectively transfer knowledge of teacher features to student features.

4. The method for robustness testing of target detection models based on cross-task learning according to claim 1, characterized in that: In step 4, the final distillation loss is: （9）; in, Represents the channel average pooling layer and The normalized combination function is used to process the teacher and student characteristics; in addition, is from The feature maps of the student features are upsampled or downsampled to match the size of the teacher features in order to better compare and analyze the differences between the teacher and student features, thus helping the process of knowledge distillation.

5. The method for testing robustness of target detection models based on cross-task learning according to claim 1, characterized in that: In step 5, a corresponding loss function of the corresponding proxy model in the model robustness test phase is established; the specific steps are: Define the input sample as , the detection output of the target detection test model is ,in Include detection frames, each detection frame is express, ; , Indicates the horizontal coordinate of the upper left corner of the detection box. Indicates the vertical coordinate of the upper left corner of the detection box. Indicates the width of the detection box. Indicates the height of the detection box. Indicates the category of the detection box, Indicates the confidence of the detection box; Randomly select the test box , ; Select the test detection box category as ; All categories that have appeared in the training data set are a set , in the collection Randomly select the target category , modify the test detection box Category For the target category , and obtain a new set of test results ,Will As input image The true value of the test is used to measure the actual test results. and truth value The differences: （2）; in, , U represents the set of proxy models, represents the proxy models contained in the set U, Indicates the detection results of each detection box and truth value The sum of the bounding box regression losses, Indicates the detection results of each detection box and truth value The sum of the categorical cross entropy losses, Represents the corresponding loss function of the corresponding proxy model in the model robustness testing phase.

6. The method for robustness testing of target detection models based on cross-task learning according to claim 1, characterized in that: In step 6, the maximum number of external iterations is defined , in each iteration, take turns from the collection Select a branch from the list. The order of branch selection does not affect the final performance. Let the first selected branch in the iteration be , updated changes Calculated by formula (3): （3）; in, represents a random initialization change, is the learning rate, represents the symbolic function, is to update the hyperparameters, Indicates the selected branch of loss, represents the gradient corresponding to the loss, express The projection function for the constraint.

7. The method for robustness testing of target detection models based on cross-task learning according to claim 1, characterized in that: In step 7, calculate The formula for making changes and recording the corresponding gradient of the changes is: （10）; （11）; in, Indicates that the first branch selected in the iteration is When the internal iteration reaches The calculated changes are express The modified group obtained by this calculation; Indicates The gradient value corresponding to the change obtained by the calculation is express The gradient group obtained by this calculation.

8. The method for robustness testing of target detection models based on cross-task learning according to claim 1, characterized in that: In step 8, the calculation formula of the corresponding gradient is as follows: （12）; in, , , Indicates that the Calculated changes Added to the attack image and passed to the branch The gradient value calculated when , thus getting , express Changes are delivered to the branch The gradient value group calculated later.

9. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 8.

Images