Target detection model robustness test method based on uncertainty

Abstract

The invention relates to a target detection model robustness test method based on uncertainty, and belongs to the technical field of image processing. The method comprises the steps of seed variation, model prediction, uncertainty analysis, test case amplification and robustness test. Diversified data variation strategies such as affine transformation and target-level transformation of the image are introduced, and target information in the image is changed, so that diversity and aggressiveness of variation samples are enhanced. In addition, the probability of correct prediction of the target is measured through an uncertainty index measurement model. The test cases generated under the guidance of uncertainty have higher value for revealing model vulnerabilities or repairing model errors, potential defects in the aspect of robustness of the target detection model can be more efficiently mined, a large number of test cases can be automatically generated, and the model vulnerabilities can be efficiently found; and the robustness of the target detection system can be improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a robustness testing method for target detection models based on uncertainty. Background Technology

[0002] Object detection technology has been successfully applied in fields such as autonomous driving and security monitoring. However, in real-world applications, the system may still be affected by complex environmental factors, leading to instability or even erroneous behavior, posing potential risks to users. Therefore, robustness testing of object detection systems is necessary. Inspired by traditional software testing techniques, researchers have improved these techniques to address the characteristics of deep neural networks (DNNs), proposing various testing methods for DNNs. These methods incorporate mutation testing theory to ensure the model's robustness under different scenarios. Currently, the mainstream intelligent model testing methods fall into two categories: metamorphic testing, which defines the metamorphic relationship between input and output and determines whether the test passes by detecting whether the model's behavior meets expectations; and fuzz testing, which automatically generates a large number of error samples to detect defects in the model.

[0003] The testing and optimization of object detection systems rely heavily on large amounts of labeled data. However, traditional test data typically depends on limited scene acquisition, and images require manual labeling of all target categories and bounding boxes, resulting in high data labeling costs and insufficient test cases for model robustness testing. To improve the testing quality of object detection systems, several techniques based on mutation testing theory have been proposed, mainly including metamorphic testing and fuzzy testing. In metamorphic testing, the design of effective metamorphic relationships heavily relies on domain knowledge. For complex scenarios such as object detection, NLP generation, and reinforcement learning, it is often difficult to formulate suitable metamorphic rules. As for fuzzy testing, most existing fuzzy testing methods use coverage-guided test case generation, focusing on improving test coverage, with less consideration for domain knowledge guidance and improving the model's robustness to natural perturbations. Moreover, they are mostly applied to object classification system testing, and their testing efficiency needs improvement for object detection models with complex network structures.

[0004] The purpose of this invention is to propose an uncertainty-based robustness testing method for target detection models, focusing on the important attribute of robustness in target detection systems. Current robustness testing methods for target detection models have the following shortcomings: 1) the number of test cases available through manual annotation is limited; 2) the testing efficiency for target detection models needs improvement. Summary of the Invention

[0005] (a) Technical problems to be solved The technical problem this invention aims to solve is how to provide a robustness testing method for target detection models based on uncertainty, in order to address the issue that the number of test cases available through manual annotation is limited, and the testing efficiency for target detection models needs to be improved.

[0006] (II) Technical Solution To address the aforementioned technical problems, this invention proposes a robustness testing method for target detection models based on uncertainty, which includes the following steps: Step 1: Seed Mutation For each seed Different mutation methods are used to generate mutation data. Each seed Contains an image and corresponding tags ,Right now For each image ,use Indicates the coordinates of the center point of the target. This represents the width and height of the target's true bounding box. Indicate the target category, then the label Let be the set of all target labels in the image, represented as: ; Where n is the target quantity; Step 2: Model Prediction The object detection model makes predictions for a given mutated image; the detection results of the object detection model for a given image are considered as a set of predictions for each individual object in the image. The prediction result is ;in yes The prediction category These are the coordinates of the top-left vertex of the target bounding box. and These are the width and height of the bounding box, respectively. Indicates system to The prediction confidence level; Step 3: Uncertainty Analysis An uncertainty index is constructed to measure the probability that an object in an image may be correctly detected; Step 4: Expanding Test Cases First, for the given original test case set Each test case Contains an image and corresponding tags First, perform the seed mutation operation as described in step one; if the mutation method is applicable, generate the transformed image and corresponding labels. The generated test cases are represented as follows. ;Calculate the uncertainty using the uncertainty calculation method proposed in step three. Uncertainty fraction and compare it with a predefined threshold. The results are compared to determine whether to add them to the seed queue. Items exceeding a threshold are added to the seed queue for further mutation, while those below the threshold are added to the test case set. In the middle; repeat this process until the loop reaches the set maximum number of iterations, using the final retained... The use cases in the model will be used for the next step of model robustness testing; Step 5 Robustness Testing Using the expanded test case set Robustness testing was performed on the target detection system.

[0007] (III) Beneficial Effects This invention proposes an uncertainty-based robustness testing method for target detection models. This method can automatically generate a large number of test cases and efficiently discover model vulnerabilities, thus improving the robustness of the target detection system. Traditional mutation methods, such as brightness and blurring changes, often produce samples with high similarity to the original image after perturbation, failing to effectively alter the target's intrinsic information. Therefore, this invention introduces diverse data mutation strategies, such as affine transformations and target-level transformations, to change the target information in the image, enhancing the diversity and aggressiveness of the mutated samples. Furthermore, only a few test cases generated through mutation amplification can detect model vulnerabilities. To improve testing efficiency, this invention further proposes an uncertainty index, calculated by the confidence level of the model's target prediction. This index has lower computational complexity than traditional coverage indices and measures the probability of the model correctly predicting the target. Test cases generated based on uncertainty are more valuable for revealing model vulnerabilities or fixing model errors, enabling more efficient discovery of potential defects in the robustness of target detection models. Attached Figure Description

[0008] Figure 1 This is a diagram illustrating the robustness testing framework of the uncertainty-based target detection model of this invention. Figure 2 Here are examples of affine transformations; (a) is a clipping transformation, and (b) is a masking transformation. Figure 3 Examples of pixel-level transformations are shown: (a) contrast transformation, (b) brightness transformation, and (c) blur transformation. Figure 4Example image for target extraction from subgraph; Figure 5 Here are some examples of target-level transformations: (a) target insertion, (b) target deletion, and (c) target replacement. Figure 6 The following are examples of incorrect target detection behaviors: (a) shows a missed detection, and (b) shows a false detection. Detailed Implementation

[0009] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0010] To address this issue, this invention proposes an uncertainty-based robustness testing method for target detection models. This method can automatically generate a large number of test cases and efficiently discover model vulnerabilities, thus improving the robustness of the target detection system. Traditional mutation methods, such as brightness and blurring changes, generate samples with high similarity to the original image after perturbation, failing to effectively alter the information representation of the target itself in the image. Therefore, this invention introduces diverse data mutation strategies such as affine transformation and target-level transformation of the image to change the target information in the image, thereby enhancing the diversity and aggressiveness of the mutated samples. Furthermore, only a few test cases amplified through mutation can detect model vulnerabilities. To improve testing efficiency, this invention further proposes an uncertainty index, which is calculated by the confidence level of the model's target prediction. This index has lower computational complexity than traditional coverage indices and can measure the probability of the model correctly predicting the target. Test cases generated based on uncertainty are of higher value in revealing model vulnerabilities or fixing model errors, enabling more efficient discovery of potential defects in the robustness of target detection models.

[0011] The robustness testing framework for the target detection model based on uncertainty proposed in this invention is as follows: Figure 1 As shown. The main steps include: seed mutation, model prediction, uncertainty analysis, test case expansion, and robustness testing.

[0012] First, the original test dataset is used as the initial seed pool and added to the first round of data generation. Then, three image mutation rules are designed: affine transformation, pixel-level transformation, and target-level transformation, to mutate the data in the seed pool. During data generation, if all the new data obtained from the mutation of the original seed pool is added to the seed pool, it will contain many similar and redundant images. To improve detection efficiency, this invention proposes using an uncertainty index to filter the generated test dataset. Finally, the expanded test dataset is output for robustness testing of the target detection model.

[0013] This invention provides a robustness testing method for a target detection model based on uncertainty, the method comprising the following steps: Step 1: Seed Mutation For each seed Different mutation methods are used to generate mutation data. Each seed Contains an image and corresponding tags ,Right now For each image ,use Indicates the coordinates of the center point of the target. This represents the width and height of the target's true bounding box. Indicate the target category, then the label Let be the set of all target labels in the image, represented as:

[0014] in, n For the target quantity; Step 2: Model Prediction The object detection model makes predictions for a given mutated image; the detection results of the object detection model for a given image are considered as a set of predictions for each individual object in the image. The prediction result is ;in yes The prediction category These are the coordinates of the top-left vertex of the target bounding box. and These are the width and height of the bounding box, respectively. Indicates system to The prediction confidence level; Step 3: Uncertainty Analysis An uncertainty index is constructed to measure the probability that an object in an image may be correctly detected; Step 4: Expanding Test Cases First, for the given original test case set Each test case Contains an image and corresponding tags First, perform the seed mutation operation as described in step one; if the mutation method is applicable, generate the transformed image and corresponding labels. The generated test cases are represented as follows. ;Calculate the uncertainty using the uncertainty calculation method proposed in step three. Uncertainty fraction and compare it with a predefined threshold. The results are compared to determine whether to add them to the seed queue. Items exceeding a threshold are added to the seed queue for further mutation, while those below the threshold are added to the test case set. In the middle; repeat this process until the loop reaches the set maximum number of iterations, using the final retained... The use cases in the model will be used for the next step of model robustness testing; Step 5 Robustness Testing Using the expanded test case set Robustness testing was performed on the target detection system. Example

[0015] To generate large-scale test data with label information, it is necessary to process each seed. Different mutation methods are used to generate mutation data. Each seed Contains an image and corresponding tags ,Right now For each image ,use Indicates the coordinates of the center point of the target. This represents the width and height of the target's true bounding box. Indicate the target category, then the label Let be the set of all target labels in the image, represented as:

[0016] in, n The target quantity.

[0017] There are three common mutation methods: affine transformation, pixel-level transformation, and target-level transformation. Pixel-level transformation leaves the label information unchanged, while affine transformation and target-level transformation involve changes to information such as the number, size, and category of targets in the image, requiring the reconstruction of the corresponding labels. Information, generating mutant seeds The specific mutation methods are as follows: (1) Affine transformation.

[0018] Cropping: Removing a portion of the image simulates a reduction in the observation range and evaluates the model's robustness when some observation information is missing. In this case, the labels corresponding to the missing targets should be deleted from the original label file. The image transformation effect is as follows: Figure 2 As shown in (a).

[0019] Masking: Select a target and add a black mask, then set the pixels within the selected target's bounding box to zero. This simulates the loss of target information due to occlusion, testing the model's ability to recognize other targets around the occluded object. At this point, the labels corresponding to the occluded targets should be deleted from the original label file. The image transformation effect is as follows: Figure 2 As shown in (b).

[0020] (2) Pixel-level transformation Contrast: By adjusting the pixel distribution of an image, the difference between bright and dark areas is enhanced or reduced, simulating image changes under different lighting and visual effects in the same scene. The transformation effect is as follows: Figure 3 As shown in (a).

[0021] Brightness: By adding or subtracting a constant to each pixel of the image, the overall brightness is changed, simulating the changes in the image under different lighting conditions. The transformation effect is as follows: Figure 3 As shown in (b).

[0022] Blur: By using a Gaussian function to smooth the image, image details become less sharp, simulating the blurring effect caused by factors such as camera shake or focus inaccuracy. The transformation effect is as follows: Figure 3 As shown in (c).

[0023] (3) Target-level transformation Target insertion: Select a few single-target sub-images for each category from the test dataset, erase the background of the sub-images and keep only the target, so as to facilitate insertion into the image to be mutated, such as... Figure 4 As shown. Then, randomly select the target's scaling ratio and insertion position, and insert the target into the original image to simulate the situation where the target enters the observation range in the scene. At this time, the label of the inserted target needs to be added to the original label file. The image transformation effect is as follows. Figure 5 As shown in (a).

[0024] Target Removal: Based on the LAMA (Large Mask Inpainting) algorithm framework, a random target removal method is implemented to simulate the situation where a target object leaves a real-world scene. This method utilizes LAMA's inpainting mechanism to semantically fill in the target region, maintaining the semantic coherence of the image. At this point, the label corresponding to the target should be deleted from the original label file. The image transformation effect is as follows: Figure 5 As shown in (b).

[0025] Target replacement: Based on target deletion and insertion algorithms, a random target in the image is selected and deleted. Then, a new target of the same category as the deleted target is inserted at the corresponding position. This simulates changes in the appearance of targets within the observation range, and the transformation effect is as follows: Figure 5 As shown in (c).

[0026] Object detection models are used to predict the mutated features of a given image. Current object detection models include single-stage models such as SSD and YOLO, as well as two-stage models such as Faster R-CNN and SPP-Net.

[0027] Single-stage models treat object detection as a regression problem. Taking the YOLO series models as an example, firstly, for an input image, the entire image is divided into a grid of cells, with each cell responsible for detecting objects whose centers fall within that cell. Then, a convolutional neural network is used as the backbone to extract image features, transforming the image into a feature map. This feature map summarizes important features at different spatial locations in the image, such as edges and textures. Finally, for each cell, one or more bounding boxes are predicted, along with the confidence score and class probability of the object contained within each bounding box. Bounding boxes with low confidence scores are filtered out. Since an object may be predicted by multiple overlapping boxes, non-maximum suppression is used to filter the predicted boxes, and the final retained bounding boxes are the prediction results.

[0028] The two stages of the two-stage model are generating region proposals and refining and classifying the proposed regions. Taking Faster R-CNN as an example, after inputting the image, the first step is to process the image by using a convolutional neural network to extract image features. Then, a region proposal generation network is used to extract a large number of candidate regions containing the target, ultimately generating a large number of candidate boxes. Next, the input feature maps and candidate boxes are collected, and the region of interest pooling method is used to extract the feature maps of the regions of interest. Finally, fully connected layers are used to further classify and regress the feature maps, calculate the category of the detection boxes, and use boundary regression to obtain the final precise location of the detection boxes.

[0029] The detection results of an object detection model for a given image are considered as a set of predictions for each individual object in the image. The prediction result is .in yes The prediction category It is the coordinate of the top-left vertex of the object. and These are the width and height of the bounding box, respectively. Indicates system to The prediction confidence level.

[0030] To guide the process of generating test cases and further improve the ability of generated test cases to detect errors, this invention constructs an uncertainty index to measure the probability that a target in an image may be correctly detected. The calculation method is as follows: First, from each target in the image Prediction results In the process of obtaining prediction confidence, Then, this invention defines the uncertainty of a test case as the average prediction confidence of all targets contained in the image of a given test case, calculated as follows:

[0031] in, The total number of targets detected in the image This indicates that the object detection model is effective for images. The uncertainty. The larger the value, the stronger the model's ability to detect an instance.

[0032] The test case expansion method proposed in this invention is an iterative data generation process. First, given a set of original test cases... Each test case Contains an image and corresponding tags First, perform a series of seed mutation operations as per step one. If the mutation method is applicable, generate the transformed image and corresponding labels. The generated test cases are represented as follows: Next, based on the uncertainty calculation method proposed in step three, calculate... Uncertainty fraction and compare it with a predefined threshold. The results are compared to determine whether to add them to the seed queue. Items exceeding a threshold are added to the seed queue for further mutation, while those below the threshold are added to the test case set. In the middle. Repeat this process until the loop reaches the set maximum number of iterations, using the final retained... The use cases in the model will be used for the next step of model robustness testing.

[0033] Using the expanded test case set The robustness of the target detection system was tested. Figure 6 shows some examples of erroneous behaviors of the target detection system triggered by the generated test cases.

[0034] This invention uses the Intersection over Union (IOU) ratio to measure the similarity between the predicted and actual results. That is, for each target present in the image, it assumes... and Representing the same goal The ground truth bounding box and the predicted bounding box regions, function Represents the size of a fixed region, then It is the ratio of the intersection to the union of the areas of two regions, expressed by the formula:

[0035] The accuracy of the model's target location prediction can be evaluated. In The accuracy of the model's predictions for the target category can be evaluated. If the model correctly predicts the target category for each test case, and If all values ​​are greater than a certain threshold, the test is considered passed.

[0036] This invention uses the percentage of test cases that pass the test to measure the robustness of the target detection model. Robustness index The calculation method is as follows: A larger value indicates a stronger robustness of the model.

[0037]

[0038] In this invention, due to the expansion of the test case set If the test cases generated can effectively trigger erroneous behaviors in the object detection model, then combining some of the generated test cases into the training dataset to retrain the model can improve the robustness of the object detection model. This is of great significance for the practical application and transformation of intelligent software systems that include object detection models. This invention proposes an uncertainty-based robustness testing method for target detection models. This method can automatically generate a large number of test cases and efficiently discover model vulnerabilities, thus improving the robustness of the target detection system. Traditional mutation methods, such as brightness and blurring changes, generate samples with high similarity to the original image after perturbation, failing to effectively alter the information representation of the target itself in the image. Therefore, this invention introduces diverse data mutation strategies such as affine transformation and target-level transformation of the image to change the target information in the image, thereby enhancing the diversity and aggressiveness of the mutated samples. Furthermore, only a few test cases amplified through mutation can detect model vulnerabilities. To improve testing efficiency, this invention further proposes an uncertainty index, which is calculated by the confidence level of the model's target prediction. This index has lower computational complexity than traditional coverage indices and can measure the probability of the model correctly predicting the target. Test cases generated based on uncertainty are of higher value in revealing model vulnerabilities or fixing model errors, enabling more efficient discovery of potential defects in the robustness of target detection models. The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A robustness testing method for a target detection model based on uncertainty, characterized in that, The method includes the following steps: Step 1: Seed Mutation For each seed Different mutation methods are used to generate mutation data. Each seed Contains an image and corresponding tags ,Right now For each image ,use Indicates the coordinates of the center point of the target. This represents the width and height of the target's true bounding box. Indicate the target category, then the label Let be the set of all target labels in the image, represented as: ； Where n is the target quantity; Step 2: Model Prediction The object detection model makes predictions for a given mutated image; the detection results of the object detection model for a given image are considered as a set of predictions for each individual object in the image. The prediction result is ;in yes Prediction categories, These are the coordinates of the top-left vertex of the target bounding box. and These are the width and height of the bounding box, respectively. Indicates the system's... The prediction confidence level; Step 3: Uncertainty Analysis An uncertainty index is constructed to measure the probability that an object in an image may be correctly detected; Step 4: Expanding Test Cases First, for the given original test case set Each test case Contains an image and corresponding tags First, perform the seed mutation operation as described in step one; if the mutation method is applicable, generate the transformed image and corresponding labels. The generated test cases are represented as follows. ;Calculate the uncertainty using the uncertainty calculation method proposed in step three. Uncertainty fraction and compare it with a predefined threshold. The results are compared to determine whether to add them to the seed queue. Items exceeding a threshold are added to the seed queue for further mutation, while those below the threshold are added to the test case set. In the middle; repeat this process until the loop reaches the set maximum number of iterations, using the final retained... The use cases in the model will be used for the next step of model robustness testing; Step 5 Robustness Testing Using the expanded test case set Robustness testing was performed on the target detection system.

2. The robustness testing method for the target detection model based on uncertainty as described in claim 1, characterized in that, In step one, the mutation methods include: affine transformation, pixel-level transformation, and target-level transformation. Pixel-level transformation leaves the label information unchanged, while affine transformation and target-level transformation involve changes to the number, size, and category information of targets in the image, requiring reconstruction of the corresponding labels. information.

3. The robustness testing method for the target detection model based on uncertainty as described in claim 2, characterized in that, Affine transformations include: Cropping: Remove a portion of the image to simulate a reduction in the observation range and evaluate the robustness of the model when some observation information is missing. In this case, the labels corresponding to the missing targets should be deleted from the original label file accordingly. Masking: Select a target and add a black mask, then set the pixels in the selected target box to zero to simulate the loss of target information caused by occlusion. This tests the model's ability to recognize other targets around the occluded object. At this time, the labels corresponding to the occluded targets should be deleted from the original label file.

4. The robustness testing method for the target detection model based on uncertainty as described in claim 2, characterized in that, Pixel-level transformations include: Contrast: By adjusting the pixel distribution of an image, the difference between bright and dark areas is enhanced or reduced, simulating image changes under different lighting and visual effects in the same scene; Brightness: By adding or subtracting a constant to each pixel of the image, the overall brightness is changed to simulate the changes in the image under different lighting conditions; Blur: By using a Gaussian function to smooth the image, the image details become unclear, simulating the blurring effect caused by imaging jitter or inaccurate focus.

5. The robustness testing method for the target detection model based on uncertainty as described in claim 2, characterized in that, Target-level transformations include: Target insertion: Select a few single-target sub-images for each category from the test dataset, erase the background of the sub-images and keep only the target to facilitate insertion into the image to be mutated; then, randomly select the scaling ratio and insertion position of the target and insert the target into the original image to simulate the situation where the target enters the observation range in the scene. At this time, the label of the inserted target needs to be added to the original label file. Target Removal: Based on the LAMA algorithm framework, a random target removal method is implemented to simulate the situation where a target object leaves a real scene. This method uses the LAMA inpainting mechanism to semantically fill the target region and maintain the semantic coherence of the image. At this time, the label corresponding to the target should be deleted from the original label file. Target replacement: Based on target deletion and target insertion algorithms, randomly select any target in the image and delete it, then insert a new target of the same category as the deleted target at the corresponding position to simulate the change in the appearance of the target within the observation range.

6. The robustness testing method for the target detection model based on uncertainty as described in claim 1, characterized in that, The object detection model in step two is a single-stage model. The single-stage model treats the object detection task as a regression problem. First, for an input image, the entire image is divided into a grid of cells, and each cell is responsible for detecting objects whose centers fall within the cell. Then, a convolutional neural network is used as the backbone network to extract image features, transforming the image into a feature map. The feature map summarizes the important features of different spatial locations in the image. Finally, for each cell, one or more bounding boxes are predicted, along with the confidence score and class probability of the object contained in the corresponding bounding box. Bounding boxes with low confidence are filtered out. Since an object may be predicted by multiple overlapping boxes, a non-maximum suppression method is used to filter the predicted boxes, and the final retained bounding boxes are the prediction results.

7. The robustness testing method for the target detection model based on uncertainty as described in claim 1, characterized in that, The target detection model in step two is a two-stage model. The two stages of the two-stage model are: generating region proposals and refining and classifying the proposed regions. After inputting the image, the input image is first processed by using a convolutional neural network to extract image features. Then, a candidate region generation network is used to extract a large number of candidate regions containing the target, ultimately generating a large number of candidate boxes. Next, the input feature maps and candidate boxes are collected, and the region of interest pooling method is used to extract the feature maps of the regions of interest. Finally, a fully connected layer is used to further classify and regress the feature maps, calculate the category of the detection boxes, and use boundary regression to obtain the final precise location of the detection boxes.

8. The robustness testing method for a target detection model based on uncertainty as described in any one of claims 1-7, characterized in that, Step three includes: First, from each target in the image Prediction results In the process of obtaining prediction confidence, Then, the uncertainty of a test case is defined as the average prediction confidence of all targets contained in the image of a given test case, calculated as follows: in, The total number of targets detected in the image This indicates that the object detection model is effective for images. Uncertainty; The higher the value, the stronger the model's ability to detect an instance.

9. The robustness testing method for the target detection model based on uncertainty as described in claim 8, characterized in that, In step five, the Intersection over Union (IOU) ratio is used to measure the similarity between the predicted and actual results. That is, for each target present in the image, it is assumed that... and Representing the same goal The ground truth bounding box and the predicted bounding box regions, function Represents the size of a fixed region, then It is the ratio of the intersection to the union of the areas of two regions, expressed by the formula: The accuracy of the model's prediction of the target location can be evaluated. In The accuracy of the model's predictions for the target category can be evaluated. If the model correctly predicts the target category for each test case, and If all values ​​are greater than a certain threshold, the test is considered passed.

10. The robustness testing method for the target detection model based on uncertainty as described in claim 8, characterized in that, In step five, the robustness of the target detection model is measured by the percentage of test cases that pass the test. The robustness metric is... The calculation method is as follows: A larger value indicates a stronger robustness of the model; 。

Images