A Multidimensional Automated Evaluation and Test Adequacy Determination Method for Target Detection Models
By encapsulating standardized interfaces and using multi-dimensional evaluation metrics, combined with controlled disturbance generation and automated supplementary testing, the problems of inconsistent evaluation and insufficient testing of target detection models are solved, achieving more reliable multi-dimensional automated evaluation and closed-loop control.
Patent Information
- Application Number
- CN202611151039.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-31
- Publication Date
- 2026-11-17
AI Technical Summary
Existing object detection model evaluation methods lack a unified interface encapsulation, making it difficult to conduct comprehensive evaluations under multi-dimensional conditions. They also lack automated evaluation under scene changes, modal changes, and disturbance conditions, and lack a quantitative judgment mechanism for test sufficiency, resulting in unreliable evaluation results.
By encapsulating the model through standardized input and output interfaces, and combining parameterized controlled disturbance generation, multi-dimensional evaluation index calculation, and automated supplementary test closed-loop control, the multi-dimensional automated evaluation and test sufficiency determination of the target detection model are realized, including internal state extraction and multi-dimensional index construction.
It improves the consistency of evaluations among different models, comprehensively reveals the performance fluctuations of models in complex environments, ensures the comprehensiveness and reliability of testing, and realizes automated supplementary testing closed-loop control.
Smart Images

Figure CN123114945A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence testing and evaluation technology, specifically to an automated evaluation method for target detection tasks, applicable to the testing, comparative evaluation, version acceptance, and engineering selection of target detection neural network models under visible light images, infrared images, and multiple scene conditions. Background Technology
[0002] Object detection models are used to locate and classify objects in input images and are a core component of computer vision systems. With the development of convolutional neural networks, attention mechanisms, and multi-scale feature modeling techniques, object detection models have been widely applied in scenarios such as intelligent monitoring, security early warning, unmanned platform perception, industrial inspection, traffic analysis, and complex environment recognition. The effectiveness of object detection models in practical deployment depends not only on their single-shot accuracy on static test sets but also on their overall adaptability to scene changes, modal changes, input degradation, boundary samples, and internal state coverage.
[0003] Current methods for evaluating target detection models typically rely on average accuracy, detection results at a fixed intersection-over-union (IoU) threshold, or one-time statistical results on a single test set as the primary criteria. While these methods can characterize the model's performance on the current data distribution to some extent, they fail to comprehensively reflect the model's overall performance in complex application environments. On the one hand, a single accuracy metric cannot simultaneously reveal target localization bias, category recognition error, false positive control capability, false negative control capability, and performance differences between different categories. On the other hand, evaluation results obtained only under a single data distribution are insufficient to demonstrate whether the model can maintain stable operation under conditions of environmental changes, imaging variations, and input interference.
[0004] Furthermore, existing testing processes for different target detection models often lack unified computer execution constraints. Different models may employ different input preprocessing methods, output formats, inference parameters, post-processing thresholds, and evaluation scripts, making it difficult to directly compare the results of multiple models under the same technical caliber. Even if a model appears to achieve high performance, it is difficult to confirm whether the result truly stems from the model's inherent capabilities or from biases caused by differences in input / output interfaces, post-processing strategies, or statistical methods. Therefore, it is necessary to encapsulate the model under test into a unified, standardized input / output interface and conduct automated evaluation under unified data loading, unified inference recording, and unified result parsing conditions.
[0005] Besides insufficient accuracy evaluation, existing technologies also lack adequate analysis of the generalization ability and robustness of target detection models. Target detection models often exhibit significant performance fluctuations under different weather conditions, lighting conditions, occlusion levels, background complexity, target scales, and modalities. Particularly in combined visible and infrared scenarios, the same target may possess significantly different texture features, contrast distributions, and background response characteristics under different modalities. Without systematic analysis across scenes, modalities, and sample variations, the model's performance can only be described on the current test distribution, not whether it possesses stable generalization ability under changing distribution conditions.
[0006] Meanwhile, existing technologies typically lack automated robustness evaluation mechanisms under controlled perturbation conditions. Input degradation factors such as noise, blurring, compression, occlusion, brightness fluctuations, and contrast changes directly affect target bounding box regression, class determination, and confidence distribution. Without a unified method for generating perturbations and comparative analysis of performance before and after perturbation, it is difficult to quantify the model's ability to withstand interference in real-world complex environments. Although existing technologies include individual tests for perturbation conditions, they are usually not combined with unified data input, internal model state recording, and subsequent closed-loop testing, thus failing to form a complete automated testing system.
[0007] More importantly, existing technologies generally lack a quantitative judgment mechanism for "whether the test is sufficient." Even if a target detection model achieves high accuracy results in a round of testing, it does not mean that the model has been fully tested. If the test samples do not cover enough categories, scenarios, and modalities; if the number of high-uncertainty boundary samples, low-confidence samples, or high-failure-rate samples is insufficient; if the key evaluation results have not converged in multiple rounds of supplementary testing; or if the key layer neurons and feature channels within the model have not been fully activated, then the test results may still lack sufficient support. To solve this problem, it is necessary to establish a quantitative mechanism for test sufficiency at three levels: the sample layer, the model layer, and the conclusion layer, and to feed back low-level dimensions to the automated supplementary testing process to form a closed-loop control from testing, judgment, supplementary testing to retesting.
[0008] Existing technologies have not yet effectively solved the following problems:
[0009] Relying on a single performance metric is insufficient to fully characterize the comprehensive capabilities of an object detection model;
[0010] The lack of a unified interface encapsulation and a unified calculation process makes it difficult to guarantee the consistency of evaluations between different models;
[0011] There is a lack of automated, multi-dimensional evaluation mechanisms for scenario changes, modal changes, sample changes, and controlled disturbances;
[0012] There is a lack of a quantitative mechanism for determining test sufficiency based on the activation state of internal neurons and sample coverage;
[0013] There is a lack of a closed-loop supplementary testing process that can automatically retrieve and / or synthesize supplementary samples based on low-level sub-dimensions and inject them into the test set.
[0014] Therefore, there is an urgent need for a technical solution that can perform automated multidimensional evaluation, internal state extraction, sufficiency quantification judgment, and closed-loop supplementary test control for target detection neural network models, so as to improve the technical reliability, automation level, and engineering application value of test results. Summary of the Invention
[0015] To address the problems in existing technologies, such as single evaluation dimensions, discrete testing processes, unobservable internal states, insufficient sample coverage making quantification difficult, and lack of automated closed-loop control in the supplementary testing process, this invention provides a multi-dimensional automated evaluation and adequacy determination method for target detection models. This method encapsulates the target detection neural network model under test into a standardized input / output interface, performs unified inference and internal state extraction on the test image, and combines parameterized controlled perturbation generation, multi-dimensional evaluation index calculation, quantitative determination of test adequacy, and automated supplementary testing closed-loop control to form a complete technical process that reflects both the model's performance level and the adequacy of the testing.
[0016] To address the aforementioned technical problems, this invention provides a multi-dimensional automated evaluation and sufficiency determination method for target detection models, executed by a computer device, comprising the following steps.
[0017] S1. Model encapsulation and test data loading
[0018] A target detection neural network model is acquired and encapsulated using a standardized input / output interface. An initial test image dataset is loaded, comprising test images with ground truth bounding box labels. The standardized input / output interface unifies the input image format, output target box format, category label format, and confidence output format of the model under test, enabling target detection models with different structures to perform inference, recording, and analysis within the same automated process. Preferably, the initial test image dataset includes at least one of visible light images and infrared images, and can be organized according to category, scene, modality, perturbation, and boundary sample dimensions.
[0019] S2, Forward Reasoning and Internal State Extraction
[0020] The initial test image dataset is input into the encapsulated target detection neural network model, which outputs the target prediction box, target category, and corresponding confidence score. Simultaneously, the internal neuron activation state feature map of the target detection neural network model during the inference process is extracted. This internal neuron activation state feature map can come from at least one type of network layer, including the backbone layer, feature fusion layer, detection head layer, and attention layer, and is used to characterize whether the test sample truly reaches the key response region within the model. By synchronously recording the inference results and internal states, a unified computational foundation can be provided for subsequent correctness analysis, structural coverage analysis, failure disclosure analysis, and confidence reliability analysis.
[0021] S3. Controlled Disturbance Generation and Disturbance Condition Inference
[0022] A parameterized controlled perturbation is applied to the initial test image dataset to generate a perturbed test image sequence. This perturbed test image sequence is then input into the target detection neural network model to obtain target prediction results under perturbed conditions. The parameterized controlled perturbation can be applied in the pixel domain and / or frequency domain, and preferably includes at least one of noise perturbation, blur perturbation, compression perturbation, occlusion perturbation, brightness variation perturbation, and contrast variation perturbation. By establishing a one-to-one correspondence between the original samples and the perturbed samples, the stability and degradation resistance of the model under different perturbation conditions can be further compared.
[0023] S4. Construct a multi-dimensional evaluation index matrix
[0024] Based on the original inference results obtained in step S2 and the perturbation condition inference results obtained in step S3, a multidimensional evaluation index matrix is constructed. The multidimensional evaluation index matrix includes at least correctness evaluation indicators, generalization evaluation indicators, and robustness evaluation indicators.
[0025] Among them, the accuracy evaluation index is determined based on the spatial overlap matching relationship between the predicted target bounding box and the ground truth bounding box, and is used to characterize the performance of target localization and target classification; the generalization evaluation index is determined based on the difference in prediction results under different scene conditions, different modal conditions, and different sample variation conditions, and is used to characterize the model's ability to adapt to distribution changes; the robustness evaluation index is determined based on the prediction result deviation between the original test samples and the perturbed test samples, and is used to characterize the model's ability to maintain performance under controlled perturbation conditions. By constructing the above three types of indices simultaneously in a unified process, a complete performance characterization of the target detection model under multidimensional conditions can be obtained.
[0026] In one implementation, the basic quantity used in the correctness evaluation to determine whether the predicted bounding box matches the true bounding box can be the intersection-union ratio (IU), which is defined as:
[0027] in, Indicates the prediction box. This represents the true bounding box. Based on the intersection-union (IU) matching results, true positives, false positives, and false negatives can be further counted, and at least one of precision, recall, F1 score, mean precision, and mean mean precision can be calculated. Here, these metrics are not used as isolated mathematical formulas, but rather as calculation results in the automated evaluation module to characterize localization matching, category recognition, and erroneous behavior, for subsequent sufficiency quantification and loop closure supplementary testing.
[0028] In one embodiment, the model performance under undisturbed conditions is denoted as... The model performance under perturbation conditions is denoted as The disturbance hold rate can then be defined as:
[0029]
[0030] The performance degradation rate can be defined accordingly:
[0031]
[0032] in, This represents a minimal constant. The perturbation retention rate and performance degradation rate, as part of the robustness evaluation metrics, are used to characterize the performance stability of the model under different perturbation conditions.
[0033] S5. Quantitative judgment of test sufficiency
[0034] Based on the internal neuron activation state feature map extracted in step S2, the multidimensional distribution statistical features of the test samples, and the multidimensional evaluation index matrix obtained in step S4, a quantitative score for test sufficiency is calculated. The quantitative determination of test sufficiency includes determination of sample sufficiency, algorithm state coverage sufficiency, and test result convergence sufficiency.
[0035] S51. Determination of Sample Sufficiency
[0036] The sample sufficiency assessment determines whether the current test samples provide sufficient support in terms of category, scenario, modality, quantity, and boundary samples. Specifically, category sufficiency assesses whether the target category is fully covered and whether the number of samples in each category meets the minimum requirements; scenario sufficiency assesses whether different weather, time periods, and background conditions are basically covered; modality sufficiency assesses whether different modalities, such as visible light and infrared, are covered; quantity sufficiency assesses whether the current sample size has stabilized the key evaluation results; and boundary sample sufficiency assesses whether the current test set contains a sufficient number of high-uncertainty samples, low-confidence samples, and high-failure-rate samples to support the analysis of false positives, false negatives, and robustness risks.
[0037] In one implementation, high-uncertainty samples can be identified by statistically analyzing the variance of scores after performing multiple random forward inferences on the same object; low-confidence samples can be identified by identifying output objects with confidence levels close to the decision threshold; and high-failure-rate samples can be identified by the frequency of erroneous behavior triggered in repeated testing, perturbation testing, or mutation testing. By automatically statistically analyzing the above boundary samples, key constraints in determining sample sufficiency can be formed.
[0038] S52, Algorithm State Coverage Sufficiency Determination
[0039] The algorithm state coverage sufficiency determination is used to judge whether the current test has explored the internal state of the neural network model for the target detection under test in sufficient depth. This determination includes at least neuron activation sufficiency, structural coverage sufficiency, failure disclosure sufficiency, and confidence reliability sufficiency.
[0040] Among them, neuron activation sufficiency is used to determine whether the test sample truly drives the key layers and key channels inside the model to generate effective activation; structural coverage sufficiency is used to determine whether the test has covered a sufficient number of state regions inside the model; failure disclosure sufficiency is used to determine whether the current test truly has the ability to detect erroneous behaviors such as false detection, false negative detection, misclassification, severe localization bias or disturbance instability; confidence reliability sufficiency is used to determine whether the confidence level of the model output is consistent with the true correctness, in order to analyze whether there are high confidence errors or insufficient calibration problems.
[0041] In one embodiment, neuronal activation sufficiency can be quantified by statistically analyzing the proportion of effective neurons or effective channels that reach the activation threshold in the monitored layer; structural coverage sufficiency can be quantified by at least one of basic coverage rate, multi-interval coverage rate, or strong activation coverage rate; failure disclosure sufficiency can be analyzed by combining the kill rate and failure disclosure rate of the variant model; and confidence reliability sufficiency can be analyzed by combining calibration error and probability consistency evaluation results.
[0042] S53. Determination of the sufficiency of convergence of test results
[0043] The convergence sufficiency criterion for test results is used to determine whether the correctness evaluation results, generalization evaluation results, and robustness evaluation results have been adequately supported. If the class coverage is insufficient, boundary samples are missing, key results have not yet converged, or the model's internal coverage is insufficient, then the current conclusions cannot be considered sufficient. By analyzing the changes in results before and after multiple rounds of retesting, the average changes in the most recent rounds, and the stability of key conclusions, it can be determined whether the test results have reached a convergence state.
[0044] In one implementation, the Test Adequacy Quantification (ATA) score satisfies the following relationship:
[0045]
[0046] in, Indicates the sample sufficiency score. This represents the score for the sufficiency of state coverage in the algorithm. This indicates the convergence sufficiency score of the test results. , and Let represent the corresponding weight, and satisfy:
[0047]
[0048] This quantitative relationship is used to uniformly map the sufficiency judgment results of the three levels—sample layer, model layer, and conclusion layer—into a calculable comprehensive score, which serves as the triggering basis for closed-loop supplementary testing control.
[0049] S6, Automated Compensation Test Closed-Loop Control
[0050] When the test adequacy quantification score is lower than a preset threshold, an automated supplementary testing process is triggered. Based on the low-scoring dimension that caused the score to fail, supplementary image samples for that low-scoring dimension are automatically retrieved from the candidate sample library and / or synthesized and injected into the current test set. Then, the process returns to step S2 until the test adequacy quantification score meets the stopping condition.
[0051] The low-scoring dimensions may include at least one of the following: insufficient category coverage, insufficient scene coverage, insufficient modality coverage, insufficient boundary samples, insufficient neuron activation coverage, insufficient structural coverage, insufficient failure disclosure, or insufficient confidence reliability. The supplementary image samples may include at least one of the following: high-uncertainty boundary samples, low-confidence samples, high-failure-rate samples, specific scene samples, specific modality samples, and specific perturbation samples. Through this automated supplementary testing closed-loop control mechanism, the originally static, one-off testing process can be transformed into a sustainable, adaptive, iterative testing process, thereby significantly improving the credibility and engineering usability of the test conclusions.
[0052] Compared with the prior art, the present invention has at least the following beneficial effects:
[0053] 1. Based on the standardized interface encapsulation, unified image input, unified inference recording, and unified result parsing of target detection neural network models, this invention can improve the consistency and comparability of evaluation results among different target detection models.
[0054] 2. In addition to single-time performance statistics, this invention introduces automated multi-dimensional evaluation under conditions of scene change, modal change, sample change and controlled perturbation, which can more comprehensively reveal the performance fluctuation law of the target detection model in complex application environments.
[0055] 3. By synchronously extracting the feature maps of the activation state of internal neurons, and performing analyses of neuron activation sufficiency, structural coverage sufficiency, failure disclosure sufficiency, and confidence reliability sufficiency based on these features, this invention can determine whether the test has truly reached the key response region from the perspective of the model's internal state.
[0056] 4. This invention introduces a Test Adequacy Quantitative Score (ATA) consisting of sample adequacy, algorithm state coverage adequacy, and test result convergence adequacy, which transforms the question of "whether the current test is sufficient" into a calculable, comparable, and iteratively optimizeable technical quantity.
[0057] 5. This invention achieves automated supplementary testing closed-loop control driven by low-dimensional sub-items by automatically retrieving and / or synthesizing supplementary image samples and injecting them into the current test set, which can significantly improve the automation level of the testing process and the credibility of the final conclusion.
[0058] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.
[0059] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0060] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0061] Figure 1 This is a schematic diagram of the overall process for multidimensional automated evaluation and sufficiency determination of the target detection model designed in this invention.
[0062] Figure 2 This is a schematic diagram illustrating the sample sufficiency designed for this invention.
[0063] Figure 3 This is a schematic diagram illustrating the sufficiency of the algorithm designed in this invention.
[0064] Figure 4 This is a schematic diagram illustrating the sufficiency of the properties designed for this invention. Detailed Implementation
[0065] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0066] Example 1: Implementation of Automated Evaluation of a General Target Detection Model
[0067] In this embodiment, the target detection neural network model is first obtained and then encapsulated using a standardized input / output interface. This standardized interface maps the input image tensor format, output bounding box representation, category label format, and confidence representation format of different detection frameworks to a predetermined structure, enabling subsequent inference, recording, and analysis to be performed by the same automated process. Subsequently, an initial test image dataset with real bounding box labels is loaded and indexed and organized according to dimensions such as category, scene, modality, perturbation, and boundary samples. Here, the target detection neural network model can be a single-stage detection network, a two-stage detection network, or an attention-based detection network.
[0068] After model encapsulation and test data loading are completed, forward inference is performed on the initial test image dataset. During inference, the target prediction box, target category, and corresponding confidence score are output, and the internal neuron activation state feature map of the model during inference is extracted simultaneously. Internal state extraction preferably records at least one level from the deep feature map of the backbone network, the output of the feature fusion layer, the classification branch of the detection head, and the regression branch, so that subsequent neuron activation coverage analysis and structural coverage analysis can be performed based on actual observable data.
[0069] After obtaining the original inference results, controlled perturbations are applied to the initial test image dataset to generate a perturbed test image sequence. Preferably, at least one of the following perturbations is applied to the images according to the test target: noise perturbation, blur perturbation, compression perturbation, occlusion perturbation, brightness change perturbation, and contrast change perturbation. For each type of perturbation, multiple intensity levels can be further set to examine the behavior changes of the target detection neural network model under different degradation levels. Then, the perturbed test image sequence is input again into the model under test to obtain the target prediction results under perturbation conditions.
[0070] Based on the original inference results and the inference results under perturbation conditions, a multi-dimensional evaluation index matrix is constructed. In this matrix, the correctness evaluation index is automatically calculated based on the matching relationship between the predicted bounding box and the ground truth bounding box; the generalization evaluation index is automatically calculated based on the differences in prediction results under different scene conditions, different modal conditions, and different sample variation conditions; and the robustness evaluation index is automatically calculated based on the performance changes between the original samples and perturbation samples. This constructed multi-dimensional evaluation index matrix not only reflects the current static detection performance of the model under test, but also reflects its stability under changing conditions and input degradation.
[0071] After constructing the multi-dimensional evaluation index matrix, the test adequacy quantification stage begins. First, a sample adequacy score is calculated based on the coverage results of the test samples in terms of category, scenario, modality, quantity, and boundary samples. Second, an algorithm state coverage adequacy score is calculated based on the proportion of effective activation channels, structural coverage, failure disclosure, and confidence reliability statistics of the internal neuron activation state feature map. Third, a test result convergence adequacy score is calculated based on the changes and convergence of the correctness evaluation, generalization evaluation, and robustness evaluation results in multiple rounds of testing or retesting. Finally, the three scores are combined according to preset weights to obtain the test adequacy quantification score (ATA).
[0072] When the ATA falls below a preset threshold, an automated supplementary testing process is triggered. The system analyzes the low-level dimensions that cause the ATA to fall below the threshold and automatically performs supplementary sample retrieval and / or sample generation based on these low-level dimensions. For example, when boundary sample sufficiency is insufficient, high-uncertainty boundary samples and low-confidence samples can be prioritized for supplementation; when modality sufficiency is insufficient, specific modality test samples can be prioritized for supplementation; when neuron activation coverage is insufficient, high-difficulty samples that can trigger deep feature responses can be prioritized for supplementation; when robustness evaluation support is insufficient, specific perturbation condition samples can be prioritized for supplementation. After injecting the supplementary image samples into the current test set, forward inference, perturbation inference, multi-dimensional evaluation index construction, and sufficiency quantification are performed again until the ATA meets the stopping condition.
[0073] Example 2: Preferred implementation method using the YOLO series models as an example
[0074] In a preferred embodiment, the target detection neural network model adopts a YOLO series model, such as the YOLOv8 model based on anchor boxes or without anchor boxes. In this embodiment, the standardized input / output interface uniformly maps the input image to the tensor format required by the model, and uniformly parses the output results into target box coordinates, class labels, and confidence scores. When extracting the activation states of internal neurons, it is preferable to record deep feature maps for the key feature extraction module, feature pyramid fusion module, and classification and regression branches in the detection head in order to analyze the response states of small targets, large targets, and targets with complex backgrounds at different levels.
[0075] In this preferred embodiment, when the system detects a high false negative rate for small target categories and insufficient deep channel activation coverage, target samples that are small in size, heavily occluded, have low contrast, or are located in complex backgrounds can be prioritized for retesting. When the system detects significant degradation in robustness under specific perturbation conditions, higher-intensity or finer-grained perturbation samples can be generated for the perturbation type to further stress test the model. In this way, the monitoring of neuron states within the model, the analysis of external input perturbations, and the sample retesting strategy can be organically combined to form a closed-loop testing process adapted to the specific neural network model.
[0076] Example 3: Result Output and Engineering Application Implementation
[0077] After the sufficiency quantification score reaches a preset threshold, the multi-dimensional evaluation results and final judgment results of the target detection model are output. The multi-dimensional evaluation results include at least correctness evaluation results, generalization evaluation results, and robustness evaluation results; the final judgment results include at least sample sufficiency results, algorithm state coverage sufficiency results, test result convergence sufficiency results, and the Test Availability Quantification Score (ATA). When the ATA meets the preset requirements, the current evaluation results can be used as a technical basis for model version acceptance, model comparison evaluation, or engineering selection; when the ATA does not meet the preset requirements, the automated supplementary testing closed-loop operation continues until the current test has sufficient support at the sample layer, model layer, and conclusion layer.
[0078] Through the above-described embodiments, this invention enables comprehensive, automated, iterative, and closed-loop evaluation of target detection neural network models. Compared to methods that rely solely on single-shot accuracy statistics, this invention emphasizes the synergistic effects of image data processing, model internal state extraction, controlled perturbation generation, and automated supplementary testing control, thus more effectively serving the engineering deployment and reliable application of target detection models.
[0079] It should be noted that, without departing from the core concept of this invention, those skilled in the art can adjust or replace the specific organization of the above steps, the feature layer selection method, the perturbation generation method, the sample retrieval method, and the scoring calculation method. Any technical solution that uses the model encapsulation, internal state extraction, multi-dimensional evaluation, sufficiency quantification judgment, and automated supplementary testing closed-loop control proposed in this invention to test the target detection neural network model should fall within the protection scope of this invention.
Claims
1. A method for multi-dimensional intelligent evaluation and test sufficiency determination of an object detection model, characterized in that, Performed by a computer device, including the following steps: S1. Obtain the target detection neural network model to be tested, and encapsulate the target detection neural network model to be tested through a standardized input / output interface; load the initial test image dataset, which includes test images with real bounding box labels; S2. Perform forward inference and internal state extraction: Input the initial test image dataset into the encapsulated target detection neural network model, output the target prediction box, target category and corresponding confidence score, and simultaneously extract the internal neuron activation state feature map of the target detection neural network model during the inference process; S3. Perform controlled perturbation generation and perturbation condition inference: Apply parameterized controlled perturbation to the initial test image dataset to generate a perturbation test image sequence; The perturbation test image sequence is input into the target detection neural network model to obtain the target prediction result under the perturbation condition; S4. Calculate the multi-dimensional evaluation index matrix: Based on the spatial overlap matching relationship between the target prediction box and the real bounding box, calculate the correctness evaluation index; Based on the differences in prediction results under different scenario conditions, different modal conditions, and different sample variation conditions, a generalization evaluation index is calculated; based on the deviation of prediction results between the initial test image dataset and the perturbed test image sequence, a robustness evaluation index is calculated. S5. Quantitative determination of test adequacy: Calculate the quantitative score of test adequacy based on the internal neuron activation state feature map extracted in step S2, the multidimensional distribution statistical features of the test samples, and the multidimensional evaluation index matrix obtained in step S4. S6. Execute intelligent supplementary testing closed-loop control: When the test adequacy quantitative score is lower than the preset threshold, the intelligent supplementary testing process is triggered; based on the low-scoring dimension that causes the test adequacy quantitative score to fail, supplementary image samples for the low-scoring dimension are automatically retrieved from the candidate sample library and / or synthesized and injected into the current test set, and the process returns to step S2 until the test adequacy quantitative score meets the stop condition.
2. The method of claim 1, wherein, The standardized input / output interface in step S1 is used to uniformly encapsulate the input image format, output target box format, category label format, and confidence output format of the target detection neural network model.
3. The method of claim 1, wherein, The initial test image dataset includes at least one of visible light images and infrared images, and includes corresponding ground truth bounding box labels and category labels.
4. The method of claim 1, wherein, The internal neuron activation state feature map in step S2 includes the activation output of at least one type of network layer in the target detection neural network model during the inference process, including the backbone network layer, feature fusion layer, detection head layer, and attention layer.
5. The method of claim 1, wherein, The parameterized controlled perturbation in step S3 is applied in the pixel domain and / or frequency domain, and the controlled perturbation includes at least one of noise perturbation, blur perturbation, compression perturbation, occlusion perturbation, brightness change perturbation, and contrast change perturbation.
6. The method according to claim 1, characterized in that, The correctness evaluation metrics in step S4 include at least one of the following: true positives, false positives and false negatives obtained from the statistical analysis of the matching relationship between the predicted box and the true box, precision, recall, F1 score, mean precision, mean mean precision, false positive rate, and false negative rate.
7. The method according to claim 6, characterized in that, The matching relationship between the predicted bounding box and the ground truth bounding box is determined based on the intersection-union ratio (IU) and class consistency. When the IU of the predicted bounding box and the ground truth bounding box is not less than the preset IU threshold and the predicted class is consistent with the ground truth class, the predicted bounding box is counted as a ground truth instance. Otherwise, it is counted as a false detection or a missed detection based on the matching result.
8. The method according to claim 1, characterized in that, The generalization evaluation index in step S4 includes at least one of the following: scene change evaluation index, modality change evaluation index, and sample change evaluation index.
9. The method according to claim 1, characterized in that, The robustness evaluation metrics in step S4 include performance retention rate and / or performance degradation rate, which are determined based on the difference in prediction results between the original test sample and the perturbed test sample.
10. The method according to claim 1, characterized in that, The test adequacy quantification determination in step S5 includes: The sample sufficiency score is calculated based on the category coverage, scene coverage, modality coverage, sample quantity distribution, and boundary sample distribution of the test samples. Based on the neuron activation coverage, structural coverage, failure disclosure, and confidence reliability represented by the internal neuron activation state feature map, the algorithm's state coverage sufficiency score is calculated. Based on the convergence and support of the correctness evaluation index, generalization evaluation index, and robustness evaluation index in step S4, the convergence sufficiency score of the test results is calculated.
11. The method according to claim 10, characterized in that, The calculation of the Test Adequacy Quantification (ATA) score satisfies the following relationship: in, Indicates the sample sufficiency score. This represents the score for the sufficiency of state coverage in the algorithm. This indicates the convergence sufficiency score of the test results, and , , All are normalized to [0,1]. , and Let represent the corresponding weight, and satisfy:
12. The method according to claim 1, characterized in that, The intelligent supplementary testing process in step S6 includes: Analysis of low-scoring dimensions that lead to substandard test adequacy quantitative scores; Based on the aforementioned low-level dimension, retrieve and / or synthesize corresponding supplementary image samples from the candidate sample library; Inject the supplementary image samples into the current test set; Repeat steps S2 to S5 to update the multidimensional evaluation index matrix and the quantitative score for test adequacy.
13. The method according to claim 12, characterized in that, The supplementary image samples include at least one of the following: high uncertainty boundary samples, low confidence samples, high failure rate samples, specific scene samples, specific modal samples, and specific perturbation samples.
14. The method according to claim 1, characterized in that, The target detection neural network model includes at least one of a single-stage target detection network and a two-stage target detection network; wherein, the single-stage target detection network includes YOLO series models based on anchor boxes or without anchor boxes.
15. A multi-dimensional intelligent evaluation and sufficiency determination device for a target detection model, characterized in that, include: The model encapsulation module is used to acquire the target detection neural network model and encapsulate the target detection neural network model through a standardized input / output interface; The reasoning and state extraction module is used to perform forward reasoning and extract the feature maps of the activation states of internal neurons. The perturbation generation module is used to apply parameterized controlled perturbations to the initial test image dataset to generate a perturbation test image sequence. The multidimensional evaluation module is used to calculate the correctness evaluation index, generalization evaluation index, and robustness evaluation index. The sufficiency determination module is used to calculate the quantitative score of test sufficiency based on the internal neuron activation state feature map, the statistical characteristics of test sample distribution, and the multi-dimensional evaluation index matrix. The closed-loop supplementary testing control module is used to trigger the intelligent supplementary testing process and control the return to the execution of reasoning and evaluation steps when the quantitative score of test adequacy is lower than the preset threshold. The model encapsulation module, reasoning and state extraction module, disturbance generation module, multi-dimensional evaluation module, sufficiency determination module, and closed-loop supplementary testing control module are configured to perform the method described in any one of claims 1 to 14.
16. A computer device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method according to any one of claims 1 to 14.
17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 14.
