A method and system for multi-level robustness evaluation of a machine learning object detection system

Abstract

The application discloses a kind of multilevel robustness evaluation method and system of machine learning target detection system, its method is: 1) for the definition of multilevel critical transition robustness of target detection system and corresponding critical transition robustness score;2) the specific test procedure of critical transition robustness;3) the calculation method of critical transition robustness score and the statistics of test result.Different levels of critical transition robustness can test and measure the robustness of target detection system under different strict level requirements.The application uses image conversion technology to simulate the input image accepted by target detection system under different real environments, under the condition of data enhancement on original test set samples, combined with multilevel critical transition robustness index evaluation system, the robustness of responding to different real environment condition changes;Finally, based on the guidance of critical transition robustness, existing target detection system can be repaired and enhanced.

Description

Technical Field

[0001] This invention relates to a multi-level robustness evaluation method and system for machine learning object detection systems, and relates to the fields of software engineering and artificial intelligence technology. Background Technology

[0002] In recent years, machine learning-based object detection technology has developed rapidly and has been widely applied in various social scenarios, such as autonomous driving systems, bringing great convenience to people's lives and work. A machine learning-based object detection model first needs to be trained on a manually labeled training dataset to obtain the decision logic for predicting the location and category of objects in an image. Then, the model is deployed to a real-world environment, receiving actual input images and performing object detection. However, compared to traditional software systems, the decision logic of machine learning-based object detection systems is uninterpretable. Therefore, traditional software testing techniques, such as logic coverage and static analysis, cannot be used to test the robustness of the object detection system in maintaining correct predictions under various deployment environment changes.

[0003] When deploying an object detection system in a real-world environment, the real-time input images received by the system may differ significantly from the original training set due to various environmental factors. For example, varying light intensity at different times of day can cause significant differences in the brightness of the input image, rainy weather can blur the input image, and camera shake and rotation can change the angle of the input image. Therefore, the robustness of the object detection system in dealing with different environmental changes still needs to be tested. Summary of the Invention

[0004] The purpose of this invention is to propose a multi-level robustness evaluation method and system for target detection systems to cope with different environmental changes. Based on the evaluation results, existing target detection systems can be repaired and enhanced, enabling their deployment in various real-world application environments. This invention uses image transformation technology to simulate the input images received by the target detection system under different real-world environments. After data augmentation of the original test set samples, it combines a multi-level critical transformation robustness index evaluation system to assess the robustness of the system to changes in different real-world environmental conditions. Finally, based on the guidance of critical transformation robustness, existing target detection systems can be repaired and enhanced.

[0005] To achieve the objectives of this invention, the technical solution adopted is as follows:

[0006] A multi-level robustness evaluation method for a machine learning object detection system, comprising the following steps:

[0007] 1) For a target detection system M to be evaluated, input the sample x in the training dataset D into the target detection system to obtain the prediction result; the target detection system M is a multi-target detection system;

[0008] 2) For the predicted result M(x), the sample x is iteratively transformed using transformation T. The increment of each iteration is δ, and the transformation parameter of transformation T is updated to θ in the (i+1)th transformation. i+1 =θ i +δ;The sample x after the i-th iteration transformation is T(x;θ i ), inputting it into the target detection system yields the prediction result M(T(x; θ) i Using multi-level critical transition robustness metrics to determine whether the prediction results meet the consistency conditions of the prediction results at the corresponding level, the critical transition robustness judgment M(T(x;θ)) is used. i When ))≠M(x), the transformation parameter θ of transformation T during the (i-1)th transformation is changed. i-1 As a sample x, the critical transformation robustness CTR(x;T,δ) of the target detection system M at the corresponding level under transformation T and increment δ;

[0009] 3) Calculate the mean critical transformation robustness of the target detection system M at the corresponding level under transformation T and increment δ.

[0010]

[0011] 4) Based on the robustness score of critical transition Determine the critical transformation robustness score of the target detection system M at the corresponding level under transformation T and increment δ; where θ max To find the maximum value of the critical transition robustness CTR(x; T,δ) for all samples in the training dataset D at the corresponding level, The average value of the critical transition robustness CTR(x; T,δ) for all samples in the training dataset D at the corresponding level;

[0012] 5) Determine the critical transition robustness of the target detection system M based on the critical transition robustness scores at each level.

[0013] Furthermore, the sample x is an image sample, and the multi-level critical transformation robustness metrics include image-level metrics, class-level metrics, and target-level metrics; the corresponding level of critical transformation robustness CTR(x; T,δ) is image-level critical transformation robustness, class-level critical transformation robustness, and target-level critical transformation robustness.

[0014] Furthermore, for image-level critical transformation robustness, if the number of target boxes in the i-th prediction output changes, then M(T(x; θ)) is determined. i ))≠M(x); For class-level critical transformation robustness, if the number of bounding boxes of any class in the i-th prediction output changes, then M(T(x; θ) is determined. i ))≠M(x); For target-level critical transformation robustness, if any target box in the i-th prediction output has a prediction class error or a position offset greater than a set value, then M(T(x; θ)) is determined to be M(T(x); θ) i ))≠M(x).

[0015] Furthermore, the steps for solving the Boolean variable satisfiability problem transformed by the output consistency judgment of critical transformation robustness at the target level include:

[0016] 1) The prediction result M(x) of the original image before conversion in the object detection system M is denoted as the set of triplets Triplets = {(x1,y1,d1),(x2,y2,d2),…,(x...} n ,y n ,d n )}, where n>0, x i ,y i Let d be the coordinates of the center point of the i-th bounding box. i The maximum allowable offset distance of the center point for the i-th target box;

[0017] 2) The prediction result M(T(x; θ) of the input transformed image in the target detection system M i Let this be denoted as the set of point pairs Pairs = {(x1,y1),(x2,y2),…,(x...} m ,y m )}, where m>0, x i ,y i Let the coordinates be the center point coordinates of the i-th bounding box;

[0018] 3) The set of Boolean variables to be solved, assigned_pairs, is initialized to empty;

[0019] 4) For each element *triplet* in the pre-transformation prediction triplet, match it with an element *pair* in the post-transformation prediction pair, where the matching condition is *Distance(triplet[x,y],pair[x,y])<=triplet[d]*, and *Distance* is the distance calculation function; assign_triplet_{i}_pair_{j}, a Boolean variable representing the matching result.

[0020] adding it to the set assigned_pairs, wherein i represents the i-th element in Triplets, j represents the j-th element in Pairs, and assign_triplet_{i}_pair_{j} represents matching the j-th element in Pairs to the i-th element in Triplets;

[0021] 5) using a constraint solver to determine the satisfiability of the Boolean variable set assigned_pairs. If the values of the Boolean variable set can be satisfied (SAT), the output consistency at the target level is established; otherwise, there is a conflict in the values of Boolean variables that cannot be satisfied (UNSAT), and the output consistency at the target level is not established.

[0022] The specific details of the above step 4) are as follows:

[0023] 4-1) cyclically traversing each element in Triplets;

[0024] 4-2) for the i-th element triplet in Triplets, traversing each element in Pairs. If the j-th element pair in Pairs can be matched to triplet, the Boolean variable assign_triplet_{i}_pair_{j} is true, and it is added to the set assigned_pairs. The Boolean variable assignment implied by this Boolean variable being true includes that all triplets before the i-th element in Triplets have not matched the j-th point pair in Pairs, that is, the list

[0025] ['assign_triplet_{index}_pair_{j}' for each index<i], all Boolean variables therein are false, and they are added to the set assigned_pairs. If the j-th element pair in Pairs cannot be matched to triplet, the Boolean variable assign_triplet_{i}_pair_{j} is false, and it is added to the set assigned_pairs;

[0026] 4-3) among the Boolean variables constructed for each element in Triplets, combining them using a logical OR operator, and adding the combination to the solving constraints of the constraint solver.

[0027] A multi-level robustness evaluation system for machine learning object detection, characterized by comprising an image-level critical transformation robustness evaluation module, a class-level critical transformation robustness evaluation module, an object-level critical transformation robustness evaluation module, and a comprehensive evaluation module;

[0028] Image-level critical transformation robustness evaluation module, used for image-level transformation T 1 The sample x in the training dataset D is iteratively transformed, with an increment of δ for each iteration. 1 During the (i+1)th transformation, the transformation parameters of transformation T are updated to θ. i+1 1 =θ i 1 +δ 1 The sample x after the i-th iteration transformation is T. 1 (x;θ i 1 ) 1 The result M(T) is input into the target detection system to obtain the prediction result. 1 (x;θ i 1 )) 1 When the converted output does not meet the requirements for output consistency at the image level, i.e., M(T) 1 (x;θ i 1 )) 1 ≠M(x) 1 When the transformation parameter θ of transformation T is used for the (i-1)th transformation, i-1 1 As a sample x, for the target detection system M in the transformation T 1 and increment δ 1 Image-level critical transformation robustness CTR(x;T) 1 ,δ 1 Then, the target detection system M is calculated during the conversion T. 1 and increment δ 1 Mean of image-level critical transformation robustness Based on critical transition robustness score Determine the target detection system M during the conversion T 1 and increment δ 1 The image-level critical transformation robustness score; where θ max To maximize the critical transformation robustness of all samples in the training dataset D, M(x) represents the average image-level critical transformation robustness of all samples in the training dataset D. 1 Input the prediction result obtained by the target detection system into sample x; the target detection system M is a multi-target detection system;

[0029] A class-level critical transition robustness evaluation module is used to evaluate the robustness of critical transitions using class-level transitions T. 2 The sample x in the training dataset D is iteratively transformed, with an increment of δ for each iteration.2 During the (i+1)th transformation, the transformation parameters of transformation T are updated to θ. i+1 2 =θ i 2 +δ 2 The sample x after the i-th iteration transformation is T. 2 (x;θ i 2 ) 2 The result M(T) is input into the target detection system to obtain the prediction result. 2 (x;θ i 2 )) 2 When the converted output does not meet the requirements for output consistency at the class level, i.e., M(T) 2 (x;θ i 2 )) 2 ≠M(x) 2 When the transformation parameter θ of transformation T is used for the (i-1)th transformation, i-1 2 As a sample x, for the target detection system M in the transformation T 2 and increment δ 2 Critical transition robustness at the lower class level (CTR(x;T)) 2 ,δ 2 Then, the target detection system M is calculated during the conversion T. 2 and increment δ 2 Mean of robustness to critical transition at the lower class level Based on critical transition robustness score Determine the target detection system M during the conversion T 2 and increment δ 2 The critical transition robustness score at the lower class level; among which, M(x) represents the average class-level critical transformation robustness of all samples in the training dataset D. 2 Input the prediction result obtained by the target detection system into sample x;

[0030] The target-level critical transition robustness evaluation module is used to evaluate the robustness of the transition T at the target level. 3 The sample x in the training dataset D is iteratively transformed, with an increment of δ for each iteration. 3 During the (i+1)th transformation, the transformation parameters of transformation T are updated to θ. i+1 3 =θ i 3 +δ 3 The sample x after the i-th iteration transformation is T. 3 (x;θ i 3) 3 The result M(T) is input into the target detection system to obtain the prediction result. 3 (x;θ i 3 )) 3 When the converted output does not meet the output consistency requirements at the target level, i.e., M(T) 3 (x;θ i 3 )) 3 ≠M(x) 3 When the transformation parameter θ of transformation T is used for the (i-1)th transformation, i-1 3 As a sample x, for the target detection system M in the transformation T 3 and increment δ 3 Critical transition robustness at the next target level (CTR(x;T)) 3 ,δ 3 Then calculate the target detection system M during the conversion T. 3 and increment δ 3 Mean of robustness to critical transition at the next target level Based on critical transition robustness score Determine the target detection system M during the conversion T 3 and increment δ 3 Robustness score for critical transitions at the next target level; among which... M(x) represents the mean of the target-level critical transformation robustness for all samples in the training dataset D. 3 Input the prediction result obtained by the target detection system into sample x;

[0031] Furthermore, regarding image-level critical transformation robustness, if the number of target boxes in the i-th prediction output changes, then M(T) is determined to be invalid. 1 (x;θ i 1 )) 1 ≠M(x) 1 For class-level critical transition robustness, if the number of bounding boxes for any class in the i-th prediction output changes, then M(T) is determined. 2 (x;θ i 2 )) 2 ≠M(x) 2 For target-level critical transformation robustness, if any target box in the i-th prediction output has a prediction category error or a position offset greater than a set value, then M(T) is determined to be... 3 (x;θ i 3 )) 3 ≠M(x) 3.

[0032] Furthermore, the steps for solving the Boolean variable satisfiability problem transformed by the output consistency judgment of critical transformation robustness at the target level include:

[0033] 1) Input the prediction result M(x) of the original image before transformation in the object detection system M. 3 Let the set of triplets be Triplets = {(x1,y1,d1),(x2,y2,d2),…,(x...} n ,y n ,d n )}, where n>0, x i ,y i Let d be the coordinates of the center point of the i-th bounding box. i The maximum allowable offset distance of the center point for the i-th target box;

[0034] 2) The prediction result M(T) of the input transformed image in the object detection system M 3 (x;θ i 3 )) 3 Let it be denoted as the set of point pairs Pairs = {(x1,y1),(x2,y2),…,(x...} m ,y m )}, where m>0, x i ,y i Let the coordinates be the center point coordinates of the i-th bounding box;

[0035] 3) The set of Boolean variables to be solved, assigned_pairs, is initialized to empty;

[0036] 4) For each element *triplet* in the pre-transformation prediction triplet, match it with an element *pair* in the post-transformation prediction pair, where the matching condition is *Distance(triplet[x,y],pair[x,y])<=triplet[d]*, and *Distance* is the distance calculation function; assign_triplet_{i}_pair_{j}, a Boolean variable representing the matching result.

[0037] Add it to the set assigned_pairs, where i represents the i-th element in Triplets, j represents the j-th element in Pairs, and assign_triplet_{i}_pair_{j} means matching the j-th element in Pairs to the i-th element in Triplets;

[0038] 5) Use a constraint solver to determine the satisfiability of the Boolean variable set assigned_pairs. If the value assignment of the Boolean variable set is satisfiable (SAT), the output consistency at the target level holds, that is, M(T 3 (x; θ i 3 )) 3 =

[0039] M(x) 3 ; otherwise, there is a conflict in the value assignment of Boolean variables and it is unsatisfiable (UNSAT), then the output consistency at the target level does not hold, that is, M(T 3 (x; θ i 3 )) 3 ≠M(x) 3 .

[0040] Specific details of the above step 4) are as follows:

[0041] 4-1) Circularly traverse each element in Triplets;

[0042] 4-2) For the i-th element triplet in Triplets, traverse each element in Pairs. If the j-th element pair in Pairs can be matched to triplet, the Boolean variable assign_triplet_{i}_pair_{j} is true, and it is added to the set assigned_pairs. The Boolean variable assignments implied by this Boolean variable being true include that none of all triplets before the i-th element in Triplets match the j-th point pair in Pairs, that is, all Boolean variables in the list

[0043] ['assign_triplet_{index}_pair_{j}' for each index<i] are all false, and they are added to the set assigned_pairs. If the j-th element pair in Pairs cannot be matched to triplet, the Boolean variable assign_triplet_{i}_pair_{j} is false, and it is added to the set assigned_pairs;

[0044] 4-3) Among the Boolean variables constructed for each element in Triplets, combine them using a logical OR operator, and add the combination to the solving constraints of the constraint solver.

[0045] In object detection tasks, the system should make the same prediction result for images with the same semantic information. Based on this criterion, this invention proposes a multi-level robustness evaluation method for object detection systems, which evaluates the system's robustness to complex environmental changes through various input transformation techniques. Object detection systems have multiple independent predicted bounding boxes and their respective categories for a single input image, thus belonging to multi-object prediction systems. First, for single-object prediction systems, the Critical Transformation Robustness (CTR) metric is defined as follows.

[0046] Critical transformation robustness of a single-objective prediction system is defined as follows: For a neural network M for single-objective prediction, sample x uses transformation T, and the transformation parameter θ is iteratively increased. i+1 =θ i +δ, θ0=0 is the initial value, δ>0, then when θ increases by a change of δ, assuming M(T(x;θ) is satisfied. i The minimum parameter value when ))≠M(x) is θ i Let θ i-1 Let CTR(x;T,δ) represent the critical transformation robustness of sample x for model M under transformation T and increment δ.

[0047] For a dataset D, the critical transformation robustness under transformation T and increment δ is defined as the mean of the critical transformation robustness of all samples in the dataset, that is:

[0048]

[0049] The mean represents the overall critical transformation robustness of samples in dataset D, used to measure the overall sensitivity of the dataset to input transformations. Based on the definition of critical transformation robustness for single-object prediction systems, the critical transformation robustness index for multi-level object detection systems is defined as follows.

[0050] The definition of multi-level critical transition robustness of object detection systems: Object detection systems belong to multi-object prediction systems, which simultaneously and independently predict various different objects in an image. Based on this characteristic, and combining the critical transition robustness of single-object prediction systems, the critical transition robustness of multi-level object detection systems is defined as image-level critical transition robustness, class-level critical transition robustness, and object-level critical transition robustness.

[0051] (1) Image-level critical transition robustness

[0052] At this level, it is determined that the prediction result of the target detection system has changed (i.e., M(T(x; θ)). iThe condition for M(x) ≠ M(x) is that the number of bounding boxes in the prediction result of the object detection system for a certain image has changed. The critical transformation robustness of this level meets the requirement that the number of predicted bounding boxes for the transformed image is the same as that for the original image. It is simple to implement, requires less computational resources and execution time, and the time complexity of processing the result returned by the single-sample object detection system is O(1).

[0053] (2) Class-level critical transition robustness

[0054] At this level, it is determined that the prediction result of the target detection system has changed (i.e., M(T(x; θ)). i The condition for M(x) ≠ M(x) is that the number of bounding boxes for at least one class of objects in the prediction results of the object detection system for the image changes. The critical transformation robustness requirement at this level is that the number of predicted bounding boxes for multiple object classes is the same in the prediction results of the transformed image and the image before transformation. Therefore, it is necessary to traverse the system's prediction results and count the number of predicted bounding boxes for different object classes. The implementation difficulty is moderate, the required computational resources and execution time are moderate, and the time complexity for processing the results returned by the single-sample object detection system is O(n).

[0055] (3) Target-level critical transition robustness

[0056] At this level, it is determined that the prediction result of the target detection system has changed (i.e., M(T(x; θ)). i The condition for M(x) ≠ M(x) is that at least one predicted bounding box in the image prediction results of the object detection system has changed. This change includes a change in the predicted category or a significant shift in the position of the predicted bounding box. The critical transformation robustness requirement at this level is that for each bounding box in the system's prediction results for the transformed and untransformed images, the predictions before and after the transformation should remain consistent, except for the necessary corresponding changes in the size and position of the bounding box due to the transformation. Under this critical transformation robustness requirement, it is necessary to traverse the system's prediction results with each increment of the transformation parameter and judge the consistency of the predictions before and after the transformation for each detection box. This is difficult to implement, requires significant computational resources and execution time, and the time complexity for processing the results returned by the single-sample object detection system is O(n^2). 2 This invention addresses the challenges of high difficulty and computational complexity in target box matching during the implementation of critical transition robustness at this level. It proposes a method that models the target box matching problem as a Boolean variable satisfiability problem, significantly improving computational speed. The algorithm for solving the Boolean variable satisfiability problem modeled by the target box matching problem is as follows:

[0057] Solving the Boolean Satisfactionability of Algorithm 1 for the Target Box Matching Problem

[0058]

[0059]

[0060] The three levels of critical transformation robustness are progressively stricter in determining whether the predicted results of the transformed image are consistent with those of the original image. Specifically, image-level critical transformation robustness is the most lenient, followed by class-level, and then target-level. If, after transformation, the target detection system's prediction does not meet the consistency requirements under image-level critical transformation robustness, it will also certainly not meet the consistency requirements under class-level critical transformation robustness. However, failing to meet class-level consistency requirements does not necessarily mean failing image-level consistency requirements. The relationship between target-level and class-level critical transformation robustness is similar. These three levels of critical transformation robustness characterize different levels of robustness requirements for target detection systems, and corresponding critical transformation robustness evaluation methods can measure and differentiate the robustness differences between different target detection systems.

[0061] After measuring the critical transformation robustness of an object detection system under a certain input transformation, in order to further accurately describe the proportion of the critical transformation robustness of the object detection system to the variable range of parameters of the input transformation on the entire dataset D, with input transformation T and parameter increment δ, this invention proposes the Critical Transformation Robustness Score (CTRS), defined as follows:

[0062] Critical Transformation Robustness Score Definition: The critical transformation robustness score is the ratio of the critical transformation robustness of the input transformation T to the variable range of the parameters of this transformation on the dataset D, i.e.:

[0063]

[0064] The critical transition robustness score can further intuitively describe the maximum change in samples that the model can withstand under an input transition, namely CTR(D;T,δ) and the maximum change that the transition can bring to the samples. The ratio between them. The critical transition robustness score index further makes the evaluation system more complete. Critical transition robustness can be used to evaluate the robustness of different target detection systems under the same environmental change conditions, while the critical transition robustness score can be used to evaluate the robustness of the same system under multiple different environmental change conditions.

[0065] In practical implementation, the robustness of multi-level critical transformations is crucial because the prediction results of object detection systems simultaneously predict the bounding box and the object's category. Certain input transformation techniques (such as rotation and translation) can cause the relative positions of the bounding boxes in the image to shift, even leading to some bounding boxes moving out of the image boundary. This impacts the prediction results of the object detection system to varying degrees. Therefore, when determining whether the prediction results of the object detection system change under a certain level of critical transformation robustness in the face of input transformations that cause bounding box movement, this invention proposes a conservative prediction principle, defined as follows.

[0066] The conservative prediction principle is defined as follows: If, after input transformation, some predicted bounding boxes in the original image are partially moved outside the image boundary, then for the transformed image, it is permissible for the object detection system to either successfully predict the bounding box or miss it. However, other unaffected bounding boxes should be correctly detected and successfully classified.

[0067] To ensure that the system's robustness in the face of different environmental changes is accurately tested, the test dataset D provided to the target detection system needs to meet the following consistency and diversity conditions.

[0068] Data consistency: For each sample, the number of detection boxes and the target category predicted by the target detection system are completely consistent with the label data.

[0069] Data diversity: The images in the dataset should be as diverse as possible, i.e., single-target, multi-target, multi-class, and different scene samples.

[0070] The specific evaluation process for critical transition robustness and critical transition robustness score is as follows: Figure 1 As shown. First, for the target detection model M to be tested and the test dataset D, determine the input transformation T and its parameter domain [θ]. min ,θ max [And parameter increment δ. Initialize the critical transformation robustness list CTR_List to be empty. For all untested samples x∈D, the prediction result of model M for x is M(x), and initialize the transformation parameter θ = θ min The sample x, after being transformed by input T and parameter θ, generates a sample T(x; θ), and the model's prediction result for the transformed sample is M(T(x; θ)). The transformation parameter θ is iteratively increased by increment δ until, under the critical transformation robustness requirement of a certain level of target detection system, M(x) and M(T(x; θ)) are inconsistent. The critical transformation robustness for sample x under transformation T is denoted as θ-δ. If the parameter θ is continuously increased to θ maxIf, under a certain level of critical transformation robustness requirement for target detection system, there is still no inconsistency between M(x) and M(T(x; θ)), then the critical transformation robustness of sample x under transformation T is denoted as θ. max Add the critical transition robustness of sample x to the critical transition robustness list CTR_List.

[0071] Finally, based on the target detection system's robustness score to a certain input transformation, the evaluation result of the target detection system's robustness in response to certain environmental changes can be obtained.

[0072] Compared with the prior art, the positive effects of the present invention are as follows:

[0073] First, this invention defines multi-level critical transition robustness indices and scores to measure the robustness of object detection systems to various environmental changes. For different robustness requirements, this invention designs multi-level critical transition robustness indices and scores with varying computational complexity, which are accurate and effective in measuring the robustness of object detection systems to environmental changes, solving the problem of robustness evaluation for object detection systems in complex and variable environments. Simultaneously, to address the high computational complexity of object-level critical transition robustness, a novel method is proposed, significantly improving computational speed. Finally, combined with evaluation criteria, the robustness of object detection systems to various real-world environmental changes can be accurately determined, guiding the upgrading and repair of object detection systems. Furthermore, this method can be well extended to other machine learning model testing frameworks, exhibiting good scalability. Attached Figure Description

[0074] Figure 1 This is a flowchart for testing the critical transition robustness score of a target detection system under input transformation. Detailed Implementation

[0075] The present invention will be further described in detail below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art fall within the scope defined by the appended claims.

[0076] This example uses a mask detection system based on the YoLoV5 architecture to demonstrate the evaluation process for the robustness of multi-level critical transitions. This system predicts the bounding boxes for faces and masks present in an image.

[0077] 1. Input Conversion Selection and Parameter Settings

[0078] This example uses brightness variation and rotation as input conversion techniques to further illustrate the testing process in detail. The test parameter settings for input conversion are shown in Table 1. The input conversion techniques used in this invention to simulate various changes in the real environment include, but are not limited to:

[0079] (1) Simulation of brightness change: Due to changes in the intensity of light sources in the environment (e.g., the intensity of sunlight at different times of the day, changes in the power of lighting lamps, etc.), the brightness of the image data input to the target detection system changes.

[0080] (2) Image blurring and image quality compression can simulate the situation where the image data quality input to the target detection system is reduced due to changes in the weather in the environment (such as rain, fog, etc.) or input device reasons, resulting in the target object being blurry.

[0081] (3) Changing the contrast can simulate the situation where the image contrast input to the target detection system is different due to the different configurations of the devices when using different input devices.

[0082] (4) Rotation can simulate situations where the angle of the image input to the target detection system changes significantly due to certain reasons (such as the input device being tilted).

[0083] (5) Translation can simulate the situation where the target object in the image appears in different relative positions in the image.

[0084] (6) Horizontal shearing can simulate the situation where the different positions of the input devices result in different viewing angles of the images input into the target detection system.

[0085] (7) Image scaling can simulate situations where the distance between the target and the input device varies, resulting in different sizes of the target in the entire image.

[0086] Table 1 Input Conversion and Parameter Settings

[0087]

[0088] The maximum and minimum values ​​of the input transformation parameters need to be set according to the specific input transformation technology, the variable range of the parameters, and the actual needs of the tester. For the target-level critical transformation robustness standard, this invention uses the criterion that a large shift in the target box position is determined by the distance between the center coordinates of the predicted bounding box in the transformed image and the mapped coordinates of the corresponding predicted bounding box in the original image after input transformation, which must be greater than half the diagonal distance of the corresponding target box in the original image. This criterion can be adapted to different requirements of the actual system.

[0089] 2. Calculation of Critical Transition Robustness Score and Statistical Analysis of Test Results

[0090] The critical transformation robustness of the object detection model M on the test dataset D with respect to the input transformation T is the mean of the critical transformation robustness of all samples in D. The initial average parameter of the test dataset D for the input transformation T is θ avg The critical transformation robustness score of model M on the test dataset D with respect to input transformation T is denoted as .

[0091] Tables 2, 3, and 4 show the results of the critical conversion robustness test of the mask target detection system under three different levels for two input conversions: brightness change and rotation.

[0092] Table 2 Image-level critical transition robustness

[0093]

[0094] Table 3 Critical transition robustness at different class levels

[0095]

[0096] Table 4. Critical transition robustness at the target level

[0097]

[0098] Experimental results show that the three levels of critical transformation robustness are progressively stricter in determining whether the predicted results of the transformed image are consistent with those of the original image. Specifically, image-level critical transformation robustness is the most lenient, followed by class-level, with image-level being the most stringent. When a sample, after input transformation, fails to meet the consistency requirement of the prediction under image-level critical transformation robustness, it will also certainly fail to meet the consistency requirement of the prediction under class-level critical transformation robustness. However, failing to meet the consistency requirement of the prediction under class-level critical transformation robustness does not necessarily mean failing to meet the consistency requirement of the prediction under image-level critical transformation robustness. As the experimental results show, the class-level critical transition robustness is equal to or less than the image-level critical transition robustness. Specifically, the class-level critical transition robustness for right rotation is significantly less than that for image-level critical transition robustness. This indicates that the image-level robustness only considers whether the number of bounding boxes in the prediction result has changed, without considering whether the predicted categories of the bounding boxes are consistent. In contrast, the class-level critical transition robustness considers both changes in the number of predicted bounding boxes and changes in the predicted categories of the detected boxes. Therefore, when the number of predicted bounding boxes in the image remains unchanged, but the predicted categories have changed due to the input transformation technique, the class-level critical transition robustness can effectively measure the system's robustness in this situation. However, the class-level critical transition robustness also has higher requirements for testing time than the image-level critical transition robustness. The relationship between the object-level and class-level critical transition robustness is similar.

[0099] In summary, the robustness testing method for a machine learning object detection system based on input transformation technology proposed in this invention can accurately and effectively test the robustness of the object detection system in response to different environmental changes, and is applicable to various machine learning models.

[0100] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0101] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0102] Other embodiments of the present invention:

[0103] This invention does not limit the input conversion technology used, its corresponding parameter range and parameter increment, or the dataset required for testing.

[0104] This invention does not limit the specific implementation method of the three different levels of target detection critical transition robustness determination or the specific critical transition robustness level classification.

[0105] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A multi-level robustness evaluation method for a machine learning object detection system, comprising the following steps: 1) For a target detection system to be evaluated , training dataset Samples in The target detection system obtains the prediction result by inputting the target data; the target detection system For multi-target detection systems; 2) Regarding the prediction results Use conversion This sample Perform iterative transformation, with each iteration increasing by an amount of change. , No. i +1 conversion The conversion parameters are updated to ;No. i Samples after the second iteration for The prediction result is obtained by inputting it into the target detection system. ; Multi-level critical transition robustness metrics are used to determine whether the prediction results meet the consistency conditions of the prediction results at the corresponding level. When the critical transition robustness at the corresponding level is judged... At that time, the first i -1 conversion Conversion parameters As a sample For the target detection system In conversion and increment Critical transition robustness at the next corresponding level ; wherein, the sample For image samples, the multi-level critical transformation robustness metrics include image-level metrics, class-level metrics, and target-level metrics; corresponding to the level of critical transformation robustness... For image-level critical transformation robustness, class-level critical transformation robustness, and target-level critical transformation robustness; for image-level critical transformation robustness, if the... i If the number of target boxes in the next prediction output changes, then it is determined that... For robustness to critical transitions at the class level, if the first... i If the number of bounding boxes for any category in the next prediction output changes, then a decision is made. For target-level critical transition robustness, if the first i If any bounding box in the next prediction output has a predicted category error or a position offset greater than a set value, then it is determined that... ; 3) Calculate the target detection system In conversion and increment Mean of robustness to critical transition at the next corresponding level ; 4) Based on the robustness score of critical transition Determine the target detection system In conversion and increment The robustness score of the critical transition at the corresponding level; among which... For training dataset Robustness of critical transitions at all corresponding levels for all samples The maximum value, For training dataset Robustness of critical transitions at all corresponding levels for all samples The average value; 5) Determine the target detection system based on the critical transition robustness score at each level. Critical transition robustness.

2. The method according to claim 1, characterized in that, The conversion Including samples Perform brightness and rotation angle changes.

3. The method according to claim 2, characterized in that, For target-level critical transition robustness, the determination The method for checking if they are consistent is as follows: 31) Input the original image before conversion in the target detection system Prediction results Let it be denoted as the set of triples Triplets ={( x 1, y 1, d 1), ( x 2, y 2, d 2), … , ( x n , y n , d n )},in, , x i , y i For the first i The center point coordinates of the target bounding box d i For the first i The maximum allowable offset distance of the center point of each target bounding box. For the set of triples Triplets The total number of bounding boxes in the target area; 32) Input the converted image into the target detection system Prediction results denoted as a set of point pairs Pairs ={( x 1, y 1), ( x 2, y 2), … , ( x m , y m )},in, , x i , y i For the first i The center point coordinates of the target bounding box For a set of point pairs Pairs The total number of bounding boxes in the target area; 33) The set of Boolean variables to be solved assigned_pairs Initialize to empty; 34) is the triplet of the prediction result before transformation. Triplets Each element in triplet Match a transformed prediction result Pairs one of the elements pair The matching condition is Distance ( triplet [ x , y ], pair [ x , y ]) <= triplet [ d ], in Distance This is the distance calculation function. triplet [ x , y ], pair [ x , y These are the coordinates of the center point of the target bounding box. triplet [ d [To retrieve the target box element] triplet The maximum allowed offset distance of the center point; a Boolean variable representing the matching result. assign_triplet_ { i } _pair_ { j Add to collection assigned_pairs Among them i express Triplets The Middle i One element, j express Pairs The Middle j One element, assign_triplet_ { i } _pair_ { j } indicates that Pairs The Middle j The element matches the Triplets The Middle i One element; 35) Use the constraint solver on the set of Boolean variables assigned_pairs The satisfiability is determined; if the values ​​of the Boolean variable set can satisfy the requirements, then the output consistency at the target level is achieved, i.e. Otherwise, if the values ​​of the Boolean variables conflict and cannot be satisfied, then the output consistency at the target level will not hold, i.e. .

4. A multi-level robustness evaluation system for a machine learning object detection system, characterized in that, It includes an image-level critical transition robustness evaluation module, a class-level critical transition robustness evaluation module, an object-level critical transition robustness evaluation module, and a comprehensive evaluation module; Image-level critical transformation robustness evaluation module, used for image-level transformations For the training dataset Samples in Perform iterative transformation, with each iteration increasing by an amount of change. , No. i +1 conversion The conversion parameters are updated to ;No. i Samples after the second iteration for The result is obtained by inputting it into the target detection system. ; When the converted output does not meet the requirements for output consistency at the image level, i.e. At that time, the first i -1 conversion Conversion parameters As a sample For the target detection system In conversion and increment Image-level critical transformation robustness Then calculate the target detection system In conversion and increment Mean of robustness to critical transformation at the next image level Based on the critical transition robustness score Determine the target detection system In conversion and increment The image-level critical transformation robustness score; where... For training dataset The maximum value of the critical transition robustness for all samples. For training dataset The average of the critical transformation robustness at the image level for all samples in the dataset; For the sample Input the prediction result obtained by the target detection system; the target detection system For multi-target detection systems; A class-level critical transition robustness evaluation module is used for class-level transitions. For the training dataset Samples in Perform iterative transformation, with each iteration increasing by an amount of change. , No. i +1 conversion The conversion parameters are updated to ;No. i Samples after the second iteration for The result is obtained by inputting it into the target detection system. When the converted output does not meet the requirements for output consistency at the class level, i.e. At that time, the first i -1 conversion Conversion parameters As a sample For the target detection system In conversion and increment Lower class level critical transition robustness Then calculate the target detection system In conversion and increment Mean of robustness to critical transition at the lower class level Based on the critical transition robustness score Determine the target detection system In conversion and increment Robustness score of critical transition at the lower class level; among which, For training dataset The average of the critical transition robustness of all sample classes at the class level; For the sample Input the prediction results obtained by the target detection system; The target-level critical transition robustness evaluation module is used for target-level transitions. For the training dataset Samples in Perform iterative transformation, with each iteration increasing by an amount of change. The transformation occurs during the (i+1)th transformation. The conversion parameters are updated to ;No. i Samples after the second iteration for The result is obtained by inputting it into the target detection system. When the converted output does not meet the output consistency requirements at the target level, i.e. At that time, the first i -1 conversion Conversion parameters As a sample For the target detection system In conversion and increment Critical transition robustness at the next target level Then calculate the target detection system. In conversion and increment Mean of robustness to critical transition at the next target level Based on the critical transition robustness score Determine the target detection system In conversion and increment Robustness score for critical transitions at the next target level; among which... For training dataset The average of the target-level critical transition robustness of all samples in the dataset; For the sample Input the prediction results obtained by the target detection system; The comprehensive evaluation module is used to assess the robustness scores of critical transitions at each level. , , Determine the target detection system Critical transition robustness.

5. The system according to claim 4, characterized in that, For image-level critical transformation robustness, if the number of target boxes in the i-th prediction output changes, then a decision is made. For robustness to critical transitions at the class level, if the first... i If the number of bounding boxes for any category in the next prediction output changes, then a decision is made. For target-level critical transition robustness, if the first i If any bounding box in the next prediction output has a predicted category error or a position offset greater than a set value, then it is determined that... .

6. The system according to claim 5, characterized in that, For target-level critical transition robustness, the determination The method for checking if they are consistent is as follows: 1) Input the original image before conversion in the target detection system Prediction results Let it be denoted as the set of triples Triplets ={( x 1, y 1, d 1), ( x 2, y 2, d 2), … , ( x n , y n , d n )},in, , x i , y i For the first i The center point coordinates of the target bounding box d i For the first i The maximum allowable offset distance of the center point of each target bounding box. For the set of triples Triplets The total number of bounding boxes in the target area; 2) Input the converted image into the target detection system Prediction results denoted as a set of point pairs Pairs ={( x 1, y 1), ( x 2, y 2), … , ( x m , y m )},in, , x i , y i For the first i The center point coordinates of the target bounding box For a set of point pairs Pairs The total number of bounding boxes in the target area; 3) The set of Boolean variables to be solved assigned_pairs Initialize to empty; 4) Triplets of the prediction results before transformation Triplets Each element in triplet Match a transformed prediction result Pairs one of the elements pair The matching condition is Distance ( triplet [ x , y ], pair [ x , y ]) <= triplet [ d ], in Distance This is the distance calculation function. triplet [ x , y ], pair [ x , y These are the coordinates of the center point of the target bounding box. triplet [ d [To retrieve the target box element] triplet The maximum allowed offset distance of the center point; a Boolean variable representing the matching result. assign_triplet_ { i } _pair_ { j Add to collection assigned_pairs Among them i express Triplets The Middle i One element, j express Pairs The Middle j One element, assign_triplet_ { i } _pair_ { j } indicates that Pairs The Middle j The element matches the Triplets The Middle i One element; 5) Use the constraint solver on the set of Boolean variables assigned_pairs The satisfiability is determined; if the values ​​of the Boolean variable set can satisfy the requirements, then the output consistency at the target level is achieved, i.e. Otherwise, if the values ​​of the Boolean variables conflict and cannot be satisfied, then the output consistency at the target level will not hold, i.e. ; The specific details of step 4) above are as follows: 4-1) Looping through Triplets Each element in; 4-2) For Triplets The first in i element triplet traversal Pairs Each element in; if Pairs The first in j element pair Can be matched to triplet Then Boolean variable assign_triplet_ { i } _pair_ { j If} is true, add it to the set. assigned_pairs In this context, the Boolean variable assignment implied by the Boolean variable being true includes... Triplets The Middle i All triples preceding the element were not matched. Pairs The first in j A list of pairs of dots, i.e., [' assign_triplet_ { index }_ pair _{ j }' foreach index < i All Boolean variables in the set are false; add them to the set. assigned_pairs In the middle; if the first in Pairs j element pair Unable to match triplet Then Boolean variable assign_triplet_ { i } _pair_ { j If the value is false, add it to the set. assigned_pairs middle; 4-3) For Triplets The Boolean variables constructed from each element in the algorithm are combined using a logical OR operator and added to the constraint solver.

7. A server, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing each step of the method of any one of claims 1 to 3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images