Target detection network metamorphic test method based on composite metamorphic relation

Abstract

The invention provides a target detection network metamorphic test method based on a composite metamorphic relation, and relates to the field of intelligent software testing, and the method comprises the steps: constructing an original test case; segmenting the picture into a foreground target and a background, and constructing a composite metamorphic relation to generate a derivative test case; calculating the similarity of the pictures by using a mean Hash algorithm, and rejecting derivative test cases with low similarity; inputting the derivative test case into a tested target detection network to obtain an output result; according to a target category in the result and the position of the detection frame, judging whether the result of the derivative test case and the original data conform to a composite metamorphic relationship; and adding the derivative test cases which do not conform to the composite metamorphic relationship into the training set to retrain the target detection model. Compared with other metamorphic test methods of the target detection network, the derivative test cases generated by the method are more diversified and have better error detection capability, and the precision of the target detection model can be further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent software testing, and particularly relates to a metamorphic testing method for object detection networks based on composite metamorphic relations. Background Art

[0002] Object detection is one of the important directions of deep learning. The great progress of deep learning technology has enabled object detection systems to achieve great development. With the wide application of object detection in daily work and life, the reliability problem of object detection systems has gradually attracted people's attention. Once the object detection systems applied in safety-critical fields such as autonomous driving, video surveillance, and industrial inspection fail, it will cause heavy losses of life and property. Therefore, improving the reliability of object detection networks has become one of the important goals in the field of object detection.

[0003] In the field of software testing, verifying the correctness of program outputs usually requires comparison with known expected outputs. However, for systems such as machine learning algorithms or complex mathematical models, it is often very difficult to construct expected outputs, which is called the "Oracle problem". To solve this problem, a metamorphic testing technology has been proposed. It verifies the correctness of the system by analyzing the specific relationship between inputs and outputs without the need to pre-construct the expected output corresponding to each input. Since intelligent software testing technology often faces problems such as difficulty in automatically constructing expected results and high manual annotation costs during testing, the use of metamorphic testing technology can effectively alleviate this problem. Although certain achievements have been made in the application of metamorphic testing technology in object detection network testing, there are still some limitations. For example, the application scenarios of existing methods are relatively single, the diversity of generated test cases is insufficient, and the cost of constructing test data is relatively high. To address the above problems, the present invention proposes a metamorphic testing method for object detection networks based on composite metamorphic relations. By constructing a composite metamorphic relationship between foreground and background, a variety of high-quality derivative test cases can be quickly generated, thereby effectively improving the test coverage and reliability of object detection networks. Summary of the Invention

[0004] Object of the Invention: The technical problem to be solved by the present invention is to provide a metamorphic testing method for object detection networks based on composite metamorphic relations in view of the problems existing in the metamorphic testing technology of existing object detection networks, such as relatively single application scenarios, low degree of test case diversity, and relatively high test case generation cost.

[0005] Technical Solution: A metamorphic testing method for object detection networks based on composite metamorphic relations, the method includes the following steps:

[0006] Step 1: Construct an original data set, including images and real label data of the images, and use the original data set to train a pre-trained model;

[0007] Step 2: Segment the image into foreground objects and background according to the segmentation mask of the image;

[0008] Step 3: Define a set of metamorphic relations, combine two metamorphic relations into a group to construct a composite metamorphic relation, perform different transformation methods on the foreground and background of the image, and generate derivative test cases;

[0009] Step 4: Use the mean hash algorithm to calculate the similarity between each pair of derivative test cases and the original test case, and eliminate the derivative test cases with a similarity lower than 0.9;

[0010] Step 5: Input the derivative test cases into the pre-trained model to obtain its output results. According to the true categories and detection box positions of the derivative test cases and the corresponding original test cases, determine whether the output results conform to the composite metamorphic relation. If not, add the derivative test case to the adversarial sample set;

[0011] Step 6: Add the adversarial sample set to the training data set of the pre-trained model and retrain the model.

[0012] Furthermore, in Step 1, standard data sets such as COCO, PASCAL VOC, and BDD can be used for the original data set. The true label data includes the object categories, bounding boxes, and segmentation masks in the image. The original data is divided into a training set, a validation set, and a test set according to common ratios, and a pre-trained model is obtained through training.

[0013] Furthermore, in Step 2, the segmentation mask is usually stored in the form of polygon coordinates. A binary mask is generated using the polygon coordinates. After reading the image through OpenCV, this mask is used for bitwise operations to extract the foreground; by inverting the mask and performing bitwise operations, the background is extracted.

[0014] Furthermore, in Step 3, a metamorphic relation (MR) is a certain expected rule or property between the input and the output. The metamorphic relations of the object detection network include brightness, noise, contrast, and blur. Define a set of metamorphic relations MR s ={MR 1 , MR 2 ...MR n}, MR s is a set of single metamorphic relations. Then, select metamorphic relations MR s and MR i from MR j . Use metamorphic relation MR i for the foreground and metamorphic relation MR j for the background, so as to construct a composite metamorphic relation MR ij . The formula is as follows:

[0015]

[0016] Among them, MR i represents the metamorphosis relationship applied in the foreground target, and MR j represents the metamorphosis relationship applied in the background, represents the composite metamorphosis combination operation.

[0017] Furthermore, in step 4, the mean hash algorithm is used to calculate the feature sequence x of the derived test case image and the feature sequence y of the original test case image. Then, the Hamming distance is calculated to evaluate the data quality. On this basis, the picture similarity is defined as the ratio of the number of bits with the same feature sequence to the length of the feature sequence. Therefore, the formula is as follows:

[0018]

[0019] On this basis, the picture similarity is defined as the ratio of the number of bits with the same feature sequence to the length of the feature sequence. The formula is as follows:

[0020]

[0021] In the above formula, the range of S(x, y) is 0 - 1. The closer it is to 1, the higher the quality of the generated picture. Derived test cases with a similarity lower than 0.9 are excluded.

[0022] Furthermore, in step 5, the object detection network can select the Yolov series, SSD network, etc. The output results include the picture category, the position coordinates of the detection box, and the confidence level. The criteria for judging that the derived test case conforms to the composite metamorphosis relationship are that the object category is correct and the IoU is greater than 0.5. The definition of IoU is as follows:

[0023]

[0024] Among them, S p is the area of the predicted bounding box, and S gt is the area of the ground truth bounding box. In practice, if the value of IoU is greater than 0.5 and the category of the inference result is correct, the prediction result is considered correct; otherwise, it is an invalid prediction result.

[0025] Furthermore, in step 6, the mAP is used to judge whether the model accuracy has been improved. The definitions of precision P and recall rate R are shown in the following formulas:

[0026]

[0027] In the above formula, TP is the number of instances that the model correctly predicts as the positive class, FP is the number of instances that the model incorrectly predicts as the positive class, and FN is the number of instances that the model incorrectly predicts as the negative class. Next, the calculation methods of AP and mAP are defined as follows:

[0028]

[0029] As shown in the above formula, each prediction box corresponds to a precision (P) and a recall (R). These two values can form a point (P, R), and all these points constitute the precision-recall curve. The area under the curve is called the average precision (AP). Since the AP for each class is relatively independent, the average AP for each class is called mAP (mean average precision), where C is the number of target classes.

[0030] The beneficial effects of the present invention are:

[0031] Aiming at the problem of ensuring the reliability of the target detection network, the present invention proposes a metamorphic testing method for the target detection network based on composite metamorphic relations. This method divides the input image of the target detection network into foreground objects and background, and applies different metamorphic relations to the image regions of the foreground objects and the background respectively, so as to construct a composite metamorphic relation; transforms the image according to the composite metamorphic relation to batch generate derivative test cases; after using a similarity algorithm to eliminate the low-naturalness images with too low similarity to the original image among the derivative test cases, high-quality test cases can be obtained. The generated test cases can be used to detect potential defects in the target detection network, and can also be used to retrain the target detection network to improve the model accuracy. This metamorphic testing method for the target detection network based on composite metamorphic relations can construct more diverse metamorphic relations, the generated test cases have high naturalness, and the test case generation rate is fast, bringing new methods and ideas to the field of metamorphic testing of the target detection network, and having high practical value and broad application prospects. Description of the Drawings

[0032] Figure 1 It is the overall framework of the metamorphic testing method for the target detection network based on composite metamorphic relations in the present invention.

[0033] Figure 2 It is the data generation process of the metamorphic testing method for the target detection network based on composite metamorphic relations in the present invention. Detailed Embodiments

[0034] The following further specifically describes the present invention in conjunction with the drawings and specific embodiments, and the above and other advantages of the present invention will become clearer.

[0035] This embodiment discloses the implementation of an object detection metamorphic testing method based on compound metamorphic relationships. The method includes:

[0036] Step 1: This paper proposes an object detection metamorphic testing method based on compound metamorphic relationships, and its overall framework is as Figure 1 shown. When selecting an object detection model, common multi-stage models can be used, such as the YOLO series and SSD networks. For the original dataset, standard datasets commonly used in object detection networks such as COCO, PASCAL VOC, and BDD are used. The ground truth data in these datasets should include the correct annotation of the object categories in the image, detection bounding boxes, and segmentation masks. In the division of the dataset, the original dataset is divided into a training set, a validation set, and a test set according to the ratio of 8:1:1. Through this division method, the effectiveness of model training and the comprehensiveness of evaluation can be ensured. During the model training process, the model needs to be trained until the loss value no longer decreases significantly to obtain a pre-trained model with stable performance.

[0037] Step 2: In order to segment the image into foreground objects and background, first we use the OpenCV library to process the image. The image file is read through the cv2.imread() function. The segmentation mask is usually stored in the form of polygon coordinates, so we need to create a zero matrix with the same size as the input image as the initial mask, which can be achieved by using the np.zeros() function of the numpy library. Next, a binary mask is generated using these polygon coordinates. The cv2.fillPoly() function of OpenCV is used to draw polygons on the mask. This function fills the specified area according to the provided polygon coordinates and sets the pixel values of these areas to 1. Subsequently, the cv2.bitwise_and() function of OpenCV is used to perform a bitwise AND operation between the binary mask and the input image to extract the foreground part of the image. To extract the background part, the cv2.bitwise_not() function of OpenCV is used to invert the binary mask. Finally, the inverted mask is used to perform a bitwise AND operation with the input image to extract the background part of the image.

[0038] Step 3: The metamorphic relationship is a certain expected rule or property between the input and the output. The metamorphic relationships of the object detection network include brightness, noise, contrast, and blur. Define the set of metamorphic relationships MR s ={MR 1 、MR 2 ...MR n}, and then select the metamorphic relationships MR s and MR i from MR j . The foreground uses the metamorphic relationship MR i, the metamorphosis relationship MR is used in the background j , thus constructing the composite metamorphosis relationship MR ij , the formula is as follows:

[0039]

[0040] Among them, MR i represents the metamorphosis relationship applied in the foreground, and MR j represents the metamorphosis relationship applied in the background, represents the composite metamorphosis combination operation.

[0041] In this way, n(n - 1) different composite metamorphosis relationships can be constructed. The data generation process of the object detection metamorphosis test method based on the composite metamorphosis relationship is as Figure 2 shown.

[0042] Step 4: Use the mean hash algorithm to calculate the feature sequence x of the derived test case image and the feature sequence y of the original test case image, and then calculate the Hamming distance to evaluate the data quality. On this basis, the picture similarity is defined as the ratio of the number of bits with the same feature sequence to the length of the feature sequence. Therefore, the formula is as follows:

[0043]

[0044] On this basis, the picture similarity is defined as the ratio of the number of bits with the same feature sequence to the length of the feature sequence. The formula is as follows:

[0045]

[0046] In the above formula, the range of the picture similarity S(x, y) is 0 - 1. The closer it is to 1, the higher the quality of the generated picture. Derived test cases with a similarity lower than 0.9 are removed.

[0047] Step 5: Input the derived test case into the object detection network for inference. The output results include the picture category, the position coordinates of the detection box, and the confidence. The criterion for determining whether the derived test case meets the composite metamorphosis relationship is that the object category is correct and the iou is greater than 0.5. Test cases that do not meet the composite metamorphosis relationship are added to the adversarial sample set.

[0048] Step 6: Add the adversarial samples to the training set to retrain the pre-trained model. The same model configuration as before is used during retraining. According to the mAP, it is judged whether the model accuracy has improved. First, the definitions of precision P and recall rate R are shown in the following formulas:

[0049]

[0050] In the above formula, TP is the number of instances that the model correctly predicts as the positive class, FP is the number of instances that the model incorrectly predicts as the positive class, and FN is the number of instances that the model incorrectly predicts as the negative class. Then, the calculation methods of AP and mAP are defined as follows:

[0051]

[0052] In the above formula, AP is the average precision, C is the number of target categories, and mAP is used as the standard to measure the accuracy of the object detection network.

[0053] In summary, the overall flowchart of the object detection metamorphic testing method based on the composite metamorphic relationship is as Figure 1 shown. First, construct the original dataset. Then, based on the characteristic that an image can be segmented into foreground objects and background, perform different metamorphic relationship transformations on the foreground objects and the background respectively to construct a composite metamorphic relationship and generate derivative test cases. Next, use the original test cases to train the pre-trained model, use the mean hash algorithm to screen the derivative test cases according to the image similarity, input the screened derivative test cases into the pre-trained model to obtain the output results. Finally, screen out adversarial samples by judging whether they conform to the composite metamorphic relationship, and retrain the pre-trained model with the derivative test cases that do not satisfy the composite metamorphic relationship and the original test cases together to improve the accuracy of the model.

[0054] The above is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modification or change made by those of ordinary skill in the art according to the content disclosed by the present invention shall be included in the protection scope recorded in the claims.

Claims

1. A target detection network degradation test method based on a composite degradation relationship, characterized in that: The following steps are involved: Step 1: Build an original dataset, which includes images and real label data of images. Use the original dataset to train the pre-trained model. Step 2: Segment the image into foreground object and background according to the segmentation mask of the image; Step 3: Define a set of transformation relationships, combine two transformation relationships into a group, construct a composite transformation relationship, perform different transformations on the foreground target and background of the image, and generate derivative test cases; Step 4: Use the mean hashing algorithm to calculate the similarity between each pair of derived test cases and the original test cases, and remove the derived test cases with a similarity lower than 0.9; Step 5: Input the derived test case into the pre-trained model to obtain its output result. According to the real category of the original test case corresponding to the derived test case and the position of the detection box, determine whether the output result conforms to the compound metamorphosis relationship constructed in step 2. If it does not conform to the compound metamorphosis relationship, the derived test case is included in the adversarial sample set for subsequent model training. Step 6: Add the adversarial sample set to the training dataset of the pre-trained model and retrain the model.

2. According to claim 1, a target detection network degradation test method based on a composite degradation relationship is characterized in that: In step 1, the original data set uses the standard data set of the target detection network. The real label data includes the object category, bounding box, and segmentation mask in the correct picture. The pre-trained model is trained using the original test case.

3. According to claim 1, a target detection network degradation test method based on a composite degradation relationship is characterized in that: In step 2, a binary mask is generated according to the polygon coordinates of the segmentation mask. After reading the image through OpenCV, the mask is used to perform bitwise operations to extract the foreground; the background is extracted by inverting the mask and performing bitwise operations.

4. According to claim 1, a target detection network degradation test method based on a composite degradation relationship is characterized in that: In step 3, the metamorphic relation (MR) is a certain expected law or property between input and output. The metamorphic relation of the target detection network includes brightness, noise, contrast and blur. The metamorphic relation set MR is defined as s = {MR1, MR2...MR n }, MR s is a single transformation relationship set, and then from MR s Transformation Relationship MR i With MR j , the foreground target uses the metamorphosis relation MR i , background uses metamorphosis relationship MR j , thereby constructing a composite transformation relationship MR ij , the formula is as follows: Among them, MR i represents the transformation relation applied in the foreground object, MR j represents the transformation relationship applied in the background, Represents a composite transformation combination operation.

5. According to claim 1, a target detection network degradation test method based on a composite degradation relationship is characterized in that: In step 4, the mean hash algorithm is used to calculate the feature sequence of the derived test case image and the feature sequence of the original test case image, and then the Hamming distance is calculated to evaluate the data quality. On this basis, the image similarity is defined as the ratio of the number of bits with the same feature sequence to the length of the feature sequence. The mean hash algorithm is used to calculate the feature sequence of the derived data x and the original data y, and then the Hamming distance is calculated. The formula is as follows: On this basis, the image similarity is defined as the ratio of the number of identical digits in the feature sequence to the length of the feature sequence. The formula is as follows: In the above formula, the range of S(x, y) is 0-1, and the closer it is to 1, the higher the quality of the generated image.

6. The target detection network degradation test method based on composite degradation relationship according to claim 1 is characterized in that: In step 5, the output results of the target detection network include the image category, the detection box position coordinates, and the confidence level. The standard for judging whether the derived test case meets the composite metamorphosis relationship is that the object category is correct and the intersection over union (IoU) is greater than 0.

5.

7. The target detection network degradation test method based on composite degradation relationship according to claim 1 is characterized in that: In step 6, the adversarial sample set is added to the training dataset of the pre-trained model, and retraining is performed according to the same model configuration as before. The improvement in model accuracy is evaluated by calculating the mean average precision (mAP). mAP is a key indicator for measuring the performance of the object detection model, and its calculation formula is as follows: In the above formula, C is the total number of target categories, AP i It is the average precision of each category. The higher the mAP value, the better the detection performance of the model on all target categories.

Images