Automatic model verification and evaluation method and system based on intersection-union comparison

Abstract

The invention discloses an automatic model verification and evaluation method and system based on intersection-to-union comparison, and belongs to the technical field of model evaluation, and the method comprises the steps: collecting a target detection standard test image, and carrying out the marking, so as to form a test set; inputting the test set into a target evaluation model to carry out bounding box prediction so as to determine a target regression mode and a target prediction head of the model, and judging the detection correctness through an intersection-union ratio threshold value; and respectively calculating the detection accuracy rate, the recall rate and the false alarm rate, recording the feature information of the false detection image, and calculating the deviation value of the prediction frame and the real frame in each dimension so as to obtain the verification and evaluation data. Through the scheme of the invention, the requirements of different detection scenes can be met, and the actual performance of the model in a specific application scene can be reflected more objectively.

Description

Technical Field

[0001] This application relates to the field of model evaluation, and in particular to an automated model evaluation method and system based on intersection-union-comparison. Background Technology

[0002] With the rapid development of deep learning technology, research on object detection model evaluation methods has received increasing attention. Object detection model evaluation mainly includes a comprehensive assessment of performance indicators such as detection accuracy and localization precision. In practical applications, object detection requirements vary significantly across different scenarios, placing higher demands on model evaluation methods. Traditional object detection model evaluation methods typically employ a single evaluation criterion, such as a fixed Intersection over Union (IOU) threshold for accuracy assessment. This method struggles to comprehensively reflect the model's actual performance in different scenarios. Especially in complex scenarios, where the scale, shape, and density of targets differ significantly, using a single evaluation criterion can easily lead to discrepancies between the evaluation results and actual application effects. Furthermore, a common problem in existing technologies is the lack of targeted evaluation methods for object detection models with multiple regression approaches. Existing evaluation methods (such as COCO and VOC) employ fixed regression strategies (such as YOLO's center point prediction or Faster R-CNN's boundary offset prediction), resulting in a significant increase in localization errors in scenarios with large variations in target size (such as industrial part detection) or irregular shapes (such as natural objects). The limitations of this evaluation method make the direction of model optimization unclear, hindering the further development of object detection technology. Furthermore, in evaluating the performance of model prediction heads, existing technologies rarely consider the complementarity between prediction heads, resulting in evaluation results that cannot fully reflect the overall performance of the model.

[0003] To address the aforementioned issues, researchers have disclosed scene-adaptive evaluation methods that dynamically adjust the Intersection over Union (IOU) threshold to adapt to feature differences in different scenes. While this method improves evaluation accuracy to some extent, it still has shortcomings when handling multi-predictor models. Specifically, when the model employs multiple regression methods or multiple predictors, existing fusion evaluation methods often use simple averaging or weighting strategies, failing to fully leverage the strengths of each predictor. Although some researchers have disclosed feature-matching-based fusion evaluation methods that attempt to guide predictor selection through feature similarity, these methods still suffer from accuracy loss during feature extraction and matching. Furthermore, in determining predictor priority, existing techniques often rely on empirical rule settings, lacking in-depth analysis and understanding of image features. This can lead to inaccurate predictor selection in complex scenes, affecting the effectiveness of fusion evaluation. In addition, existing fusion evaluation methods are prone to feature confusion when dealing with highly similar targets or scenes with severe occlusion, impacting the reliability of evaluation results. More importantly, while current fusion evaluation methods improve evaluation accuracy, they often incur high computational costs, posing a significant limitation in applications with high real-time requirements.

[0004] Therefore, there is an urgent need for a technical solution that can adapt to the needs of different detection scenarios and more objectively reflect the actual performance of the model in specific application scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, embodiments of this application disclose an automated model evaluation method and system based on intersection-union-comparison (IUCN) comparison. This application solves the technical problem that existing technologies neglect the advantages of different regression methods in specific scenarios.

[0006] This application discloses an automated model evaluation method based on intersection-union comparison (IUCN) comparison, comprising: collecting standard test images for target detection and annotating them to form a test set; inputting the test set into a target evaluation model to predict bounding boxes, thereby determining the target regression method and target prediction head of the model, and determining the detection correctness by using an IUCN threshold; calculating the detection accuracy, recall, and false alarm rate respectively, recording the feature information of falsely detected images, and calculating the deviation values ​​between the predicted boxes and the ground truth boxes in each dimension to obtain evaluation data.

[0007] In one possible implementation, the process of acquiring and annotating standard test images for target detection to form a test set includes: acquiring standard test images for target detection and annotating each image by drawing a rectangular bounding box; recording the coordinate values ​​of the four vertices of the rectangular bounding box and establishing a correlation between the coordinate values ​​and the corresponding images in a database; for regular geometric targets, ensuring that the rectangular bounding box is parallel to the target contour; for irregularly shaped targets, selecting the smallest bounding rectangle as the annotation box; and using all the annotated images and their corresponding annotation data as the test set.

[0008] In one possible implementation, the test set is input into the target evaluation model for bounding box prediction to determine the model's target regression method and target prediction head. The detection correctness is then determined by the intersection-over-union (IoU) threshold. This includes: selecting center point offset prediction or four-boundary prediction method based on target scene features, calculating the offset and scaling factor, and obtaining the initial predicted position of the target bounding box; extracting multi-scale features from the input image, calculating the matching score between the features and one or more prediction heads, and weighted fusing the bounding box coordinates output by the prediction heads to obtain the optimized prediction result; and setting the IoU threshold hierarchically and adaptively optimizing the local neighborhood based on the edge complexity and local contrast index of the target region, and determining the detection result.

[0009] In one possible implementation, the accuracy, recall, and false alarm rate of the detection are calculated respectively, the feature information of the falsely detected images is recorded, and the deviation values ​​between the predicted bounding boxes and the ground truth bounding boxes in each dimension are calculated to obtain evaluation data. This includes: calculating the accuracy of the model, where the accuracy is the number of correctly detected objects divided by the total number of detections; calculating the recall of the model, where the recall is the number of correctly detected objects divided by the total number of standard bounding boxes; calculating the false alarm rate of the model, where the false alarm rate is the number of incorrectly detected objects divided by the total number of detections; recording information including image features, lighting conditions, and / or target size in the case of false detections; and calculating the deviation values ​​between the predicted bounding boxes and the ground truth bounding boxes in the dimensions of horizontal offset, vertical offset, width deviation, and / or height deviation.

[0010] In one possible implementation, the method involves selecting either center point offset prediction or four-boundary prediction based on the characteristics of the target scene, calculating the offset and scaling factor, and obtaining the initial predicted position of the target bounding box. This includes: using the center point offset prediction method, calculating the center coordinates and width and height values ​​of the original predicted box, and determining the target bounding box position based on the predicted offset and scaling factor; using the four-boundary prediction method, calculating the top, bottom, left, and right boundary coordinates of the original predicted box, and determining the target bounding box position based on the predicted four-boundary offset; calculating the intersection-union ratio (IUGR) between the bounding boxes and the ground truth boxes for both prediction methods; selecting the center point offset prediction method for scenes with large changes in target size, and selecting the four-boundary prediction method for scenes with irregular target shapes.

[0011] One possible implementation involves extracting multi-scale features from the input image, calculating the matching score between the features and one or more prediction heads, and weighted fusing the bounding box coordinates output by the prediction heads to obtain an optimized prediction result. This includes: extracting edge texture features, local structural features, and semantic features from the input image as multi-scale features; setting feature index weight coefficients for each prediction head; calculating the matching score between the image features and the prediction head; normalizing the matching score to obtain the weight coefficient for each prediction head; and weighted calculating the target bounding box coordinates output by each prediction head based on the weight coefficients to obtain the final bounding box position.

[0012] One possible implementation involves setting a graded Intersection over Union (IoU) threshold and performing adaptive optimization of the local neighborhood based on the edge complexity and local contrast index of the target region, and judging the detection results. This includes: calculating the edge complexity index based on the contour perimeter and area of ​​the target region to quantify and grade the scene; setting a baseline IoU threshold according to the edge complexity level and dynamically adjusting it using an adjustment coefficient; dividing the image into grid cells and calculating the local contrast index of each cell; marking low-contrast regions and performing adaptive optimization of the IoU threshold of the local neighborhood based on the normalized area ratio of the target region.

[0013] In one possible implementation, the image is divided into grid cells, and the local contrast index of each cell is calculated, including: obtaining grid division parameters according to the image resolution; dividing the input image into multiple grid cells according to the number of rows and columns; extracting the maximum and minimum pixel values ​​within each grid cell; and calculating the local contrast index of each grid cell by subtracting the minimum pixel value from the maximum pixel value and dividing by the sum of the maximum and minimum pixel values.

[0014] In one possible implementation, low-contrast regions are marked, and local neighborhood adaptive optimization of the cross-union ratio (CUP) threshold is performed based on the normalized area ratio of the target region. This includes: marking grid cells with local contrast indices less than a preset threshold as low-contrast regions; calculating the proportion of low-contrast regions in grid cells containing the detected target and their eight neighboring grid cells; adjusting the CUP threshold of the region when the proportion of low-contrast regions exceeds one-third based on the ratio of the local contrast index to the preset threshold; calculating the ratio of the target region area to the total image area as the normalized area ratio; increasing the CUP threshold based on the difference between the normalized area ratio and the first preset ratio when the normalized area ratio is less than a first preset ratio; decreasing the CUP threshold based on the difference between the normalized area ratio and the second preset ratio when the normalized area ratio is greater than a second preset ratio; calculating the ratio of the minimum distance between adjacent targets to the diagonal length of the larger target; and uniformly decreasing the CUP threshold of these targets when the ratio is less than a preset coefficient.

[0015] This application also discloses an evaluation system that applies the automated model evaluation method described in any of the above embodiments, comprising: an annotation unit, a model invocation unit, and an evaluation calculation unit; wherein, the annotation unit is used to collect and annotate standard test images for target detection to form a test set; the model invocation unit is used to invoke the target evaluation model through an interface; the evaluation calculation unit is used to input the test set into the target evaluation model for bounding box prediction to determine the target regression method and target prediction head of the model, and to determine the detection correctness by using the intersection-union threshold; the accuracy, recall, and false alarm rate of the detection are calculated respectively, the feature information of the falsely detected images are recorded, and the deviation values ​​between the predicted box and the real box in each dimension are calculated to obtain evaluation data.

[0016] In the automated model evaluation method and system based on intersection-union comparison (IUCN) disclosed above, the embodiments of this application, through optimization of regression methods and prediction heads, combined with dynamic threshold adjustment, can adapt to the needs of different detection scenarios and more objectively reflect the actual performance of the model in specific application scenarios. Furthermore, in some embodiments, through a dynamic selection mechanism of two regression methods—center point offset prediction and four-boundary prediction—the most suitable regression method can be selected in different scenarios, improving the accuracy and robustness of bounding box prediction. Furthermore, in some embodiments, through multi-level feature extraction and adaptive weight allocation, the contributions of each prediction head are integrated, making the final detection results more accurate and reliable. Furthermore, in some embodiments, through a hierarchical IOU threshold setting scheme, through quantification of the complexity of target features and adaptive optimization of local neighborhoods, dynamic adjustment of the threshold can be achieved. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an automated model verification method disclosed in an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating a method for determining the correctness of a detection by using an intersection-union comparison threshold, as disclosed in an embodiment of this application.

[0020] Figure 3 This is a schematic block diagram of a model verification and evaluation system disclosed in an embodiment of this application. Detailed Implementation

[0021] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0022] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them. It should also be understood that in the embodiments of this disclosure, "multiple" can refer to two or more, and "at least one" can refer to one, two, or more. It should also be understood that any component, data, or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless explicitly limited or given a contrary indication in the context. Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this disclosure generally indicates that the related objects before and after are in an "or" relationship. It should also be understood that the descriptions of the various embodiments in this disclosure emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be elaborated upon one by one.

[0023] Furthermore, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. Techniques, methods, and apparatus known to those skilled in the art will not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Figure 1This is a flowchart illustrating an automated model evaluation method disclosed in an embodiment of this application. It is important to understand that in existing object detection model evaluation practices, the evaluation methods for mainstream models such as the YOLO series, Faster R-CNN, and RetinaNet mainly focus on calculating the mAP metric in general scenarios, which is insufficient to meet the differentiated detection accuracy requirements of specific application scenarios. These models typically employ a single prediction method during bounding box prediction. For example, the YOLO series models primarily use a prediction method that combines center point coordinates with width and height, while Faster R-CNN uses a four-boundary offset prediction method based on anchors. This fixed prediction strategy is difficult to adapt to the changing characteristics of target shape and scale in different scenarios. Especially in high-precision scenarios such as industrial visual inspection, a single prediction method often leads to significant deviations between the detected bounding box and the actual target contour. Furthermore, existing models typically employ simple feature concatenation or weighted averaging methods for multi-scale feature fusion, such as the top-down feature fusion method in Feature Pyramid Network (FPN). While this improves detection performance to some extent, it fails to fully consider the adaptive matching relationship between features at different levels and specific detection targets, resulting in low feature utilization efficiency.

[0026] To address the aforementioned technical issues, the dual-path regression evaluation method disclosed in this application dynamically selects between center point offset prediction and four-boundary prediction, enabling it to better adapt to the needs of different detection scenarios. For example, when detecting industrial parts with regular shapes and large size variations, prioritizing center point offset prediction can more accurately capture the overall position and scale information of the target; while when detecting irregularly shaped natural objects, using four-boundary prediction can more accurately fit the contour features of the target. Compared with the fixed default box mechanism in the SSD model and the anchor-based prediction method in RetinaNet, this adaptive dual-path prediction strategy significantly improves the flexibility and accuracy of bounding box localization. Simultaneously, the feature adaptive selection mechanism disclosed in this application optimizes the utilization efficiency of multi-scale features by establishing a feature-prediction head matching scoring mechanism, and compared with the BiFPN feature fusion method used in models such as EfficientDet, it can more effectively utilize feature information at different levels.

[0027] Regarding the formulation of evaluation criteria, existing model evaluation methods, such as the COCO evaluation system, use a fixed IOU threshold sequence (0.5:0.05:0.95) to calculate the AP index, or, like the VOC evaluation, a single 0.5 IOU threshold. Such uniform evaluation criteria are difficult to adapt to the specific needs of different application scenarios. The hierarchical IOU threshold setting scheme disclosed in this application achieves dynamic adjustment of the evaluation criteria through target complexity quantification and local neighborhood adaptive optimization. For example, in the scenario of precision parts inspection, the model may perform well under general evaluation metrics, but still suffer from insufficient positioning accuracy in practical applications. By introducing edge complexity metrics and local contrast analysis, this scheme can dynamically adjust the evaluation criteria according to the characteristics of the specific inspection target, thereby more objectively reflecting the actual performance of the model in specific application scenarios. Compared with general evaluation systems such as COCO and VOC, this adaptive evaluation method can disclose more targeted performance evaluation results, providing more accurate guidance for model optimization.

[0028] Regarding optimization suggestions for bounding box prediction, the deviation feature-based analysis method disclosed in this application establishes a complete optimization guidance mechanism by systematically quantifying the deviations between the predicted and ground truth bounding boxes in dimensions such as position, scale, and deformation. This mechanism differs from simple strategies commonly used in existing detection models, such as NMS post-processing or score thresholding. Instead, it provides targeted correction suggestions for different types of prediction deviations through precise numerical analysis. For example, when a significant scale deviation is detected in the predicted bounding box, the system automatically calculates an appropriate scaling factor based on the relative error index; when a severe deformation problem is detected, the confidence of the corresponding detection box is reduced through corner distance analysis. This optimization mechanism based on specific deviation features can more effectively guide the improvement of model performance, making the model evaluation results more practically relevant.

[0029] like Figure 1 As shown, in step S101, standard test images for target detection are acquired and labeled to form a test set. This includes: acquiring standard test images for target detection and labeling each image by drawing a rectangular bounding box; recording the coordinate values ​​of the four vertices of the rectangular bounding box and establishing a correlation between the coordinate values ​​and the corresponding images in a database; for regular geometric targets, ensuring that the rectangular bounding box is parallel to the target contour; for irregularly shaped targets, selecting the smallest bounding rectangle as the labeling box; and using all the labeled images and their corresponding labeling data as the test set.

[0030] In this embodiment, standard test images for target detection are first acquired, and each image is annotated. Specifically, a rectangle is drawn on the image using a mouse to select the target region. After the rectangle is selected, the coordinates of its four vertices are recorded, and these coordinates are associated with the corresponding images and stored in the database. During the annotation process, for multiple target regions existing in the same image, rectangles are selected and their coordinates are recorded separately. Different annotation rules are set for target features in different scenarios: when the target is a regular geometric shape, the rectangle is kept parallel to the target outline; when the target is an irregular shape, the smallest bounding rectangle is selected as the annotation box. After annotation is completed, all images and their corresponding annotation data are used as a test set.

[0031] In step S102, the test set is input into the target evaluation model for bounding box prediction to determine the model's target regression method and target prediction head, and the detection correctness is determined by the intersection-over-union (IoU) threshold. This includes (e.g.) Figure 2 As shown in the diagram: Based on the characteristics of the target scene, either center point offset prediction or four-boundary prediction is selected, the offset and scaling factor are calculated, and the initial predicted position of the target bounding box is obtained; multi-scale features of the input image are extracted, the matching score between the features and one or more prediction heads is calculated, and the bounding box coordinates output by the prediction heads are weighted and fused to obtain the optimized prediction result; based on the edge complexity and local contrast index of the target region, the cross-union ratio threshold is set hierarchically and local neighborhood adaptive optimization is performed, and the detection result is judged.

[0032] Specifically, for the called object detection model, bounding box prediction evaluation is performed based on the characteristics of different regression methods. For the multi-predictor fusion evaluation of the called object detection model, an adaptive feature selection method is adopted. Next, the object detection model is validated and evaluated. First, images from the test set are input into the object detection model to be evaluated, and the model outputs detection results, including the location information of the detected target regions. Specifically, the model outputs the coordinates of the four vertices of each detection box and its confidence score. Then, the detection boxes output by the model are compared with manually labeled standard boxes. For each detection box, the intersection-union ratio (IOU) between it and the corresponding standard box is calculated. The IOU is calculated as the area of ​​the intersection of the two rectangles divided by the area of ​​their union. A pre-set IOU threshold is compared with the calculated actual IOU value. When the actual IOU value is greater than or equal to the threshold, the detection result is considered a correct detection; when the actual IOU value is less than the threshold, it is considered an incorrect detection.

[0033] In step S103, the accuracy, recall, and false alarm rate of the detection are calculated respectively. Feature information of the falsely detected images is recorded, and the deviation values ​​between the predicted bounding boxes and the ground truth bounding boxes in each dimension are calculated to obtain evaluation data. This includes: calculating the accuracy of the model, where accuracy is the number of correctly detected objects divided by the total number of detections; calculating the recall of the model, where recall is the number of correctly detected objects divided by the total number of standard bounding boxes; calculating the false alarm rate of the model, where false alarm rate is the number of incorrectly detected objects divided by the total number of detections; recording information including image features, lighting conditions, and / or target size in false detection cases; and calculating the deviation values ​​between the predicted bounding boxes and the ground truth bounding boxes in the dimensions of horizontal offset, vertical offset, width deviation, and / or height deviation.

[0034] In one implementation scenario, after evaluating the detection results for all test images, various performance metrics are calculated. First, accuracy is calculated (the number of correctly detected objects divided by the total number of detections). Second, recall is calculated (the number of correctly detected objects divided by the total number of bounding boxes). Finally, false positive rate (FPR) is calculated (the number of incorrectly detected objects divided by the total number of detections). Based on these metrics, the model performance is quantitatively evaluated. Specifically, when both accuracy and recall exceed 92.5% and the FPR is below 7.5%, the model is rated as excellent; when both accuracy and recall are between 85.0% and 92.5% and the FPR is between 7.5% and 15.0%, the model is rated as good; when accuracy or recall is below 85.0%, or the FPR is above 15.0%, the model is rated as needing optimization.

[0035] To address the issues identified during the evaluation process, a detailed problem analysis mechanism was established. For false positives, key information such as corresponding image features, lighting conditions, and target size was recorded. For false negatives, influencing factors such as target-background contrast and occlusion levels were analyzed. Statistical analysis of this information identified key factors affecting detection performance, providing a basis for model optimization. Simultaneously, the evaluation results were presented visually, including overlays of detection and standard bounding boxes and statistical charts of performance metrics, facilitating an intuitive understanding of the model's detection effectiveness. Specifically, in one embodiment, suggestions for improving bounding box prediction were disclosed based on the deviation characteristics between the predicted and ground truth bounding boxes.

[0036] Specifically, first, the deviation values ​​between the predicted box and the ground truth box in different dimensions are calculated: horizontal offset. Vertical offset Width deviation Height deviation Where p and g represent the coordinates of the predicted bounding box and the ground truth bounding box, respectively, and w and h are the width and height of the ground truth bounding box. When the horizontal offset d... x Greater than 0.328 or vertical offset d yWhen the value is greater than 0.312, it is recommended to increase the weight coefficient of the center point prediction branch; when the width deviation d w When the weighting factor is greater than 0.267 or the height deviation (dh) is greater than 0.283, it is recommended to increase the weighting factor of the scale prediction branch. For bounding box scale prediction, a relative error index should be introduced. Where A pred and A true These represent the areas of the predicted bounding box and the ground truth bounding box, respectively. When e > 0.236, the predicted bounding box is too large, and it is recommended to reduce its size to 0.862 times its original size during post-processing. When e < -0.248, the predicted bounding box is too small, and it is recommended to enlarge its size to 1.157 times its original size. For corner location, the Euclidean distance d between the four corners of the predicted bounding box and the corresponding corners of the ground truth bounding box is calculated. When the maximum corner distance dmax exceeds the preset threshold of 0.376, it indicates severe deformation of the bounding box, and it is recommended to lower the confidence score of that detection box. Based on these specific deviation characteristics and corresponding thresholds, targeted adjustments are made to the predicted bounding boxes in the detection results, thereby improving the accuracy of bounding box localization.

[0037] Figure 2 This is a flowchart illustrating a method for determining the correctness of a detection by using an intersection-union ratio threshold, as disclosed in an embodiment of this application.

[0038] like Figure 2 As shown, in step S201, a center point offset prediction or four-boundary prediction method is selected based on the target scene characteristics. The offset and scaling factor are calculated to obtain the initial predicted position of the target bounding box. This includes: using the center point offset prediction method, calculating the center coordinates and width and height values ​​of the original predicted box, and determining the target bounding box position based on the predicted offset and scaling factor; using the four-boundary prediction method, calculating the top, bottom, left, and right boundary coordinates of the original predicted box, and determining the target bounding box position based on the predicted four-boundary offset; calculating the intersection-union ratio (IUGR) between the bounding boxes and the ground truth boxes for both prediction methods; using the center point offset prediction method for scenes with large changes in target size, and using the four-boundary prediction method for scenes with irregular target shapes.

[0039] When the target model incorporates multiple regression methods, bounding box prediction and evaluation can be performed based on the characteristics of each method. In the bounding box regression process, two regression methods are employed: center point offset prediction and four-boundary prediction. For the center point offset prediction method, the offset d of the target center point relative to the center of the original predicted bounding box is predicted. x d y And the scaling factors d for width and height w d h To locate the target. Specifically, set the center coordinates of the original predicted bounding box to (p x p y), with width and height p respectively. w p h The final predicted bounding box coordinates are calculated as follows: x = p x +d x ×p w y = p y +d y ×p h w = p w ×exp(d w ), h = p h ×exp(d h For the four-boundary prediction method, the offset d of the target bounding box relative to the four boundaries of the original predicted box is directly predicted. t d b d l d r Specifically, if the coordinates of the top, bottom, left, and right boundaries of the original predicted bounding box are set to t, b, l, and r respectively, then the coordinates of the final predicted bounding box are calculated as follows: top = t + d t ×(bt),bottom=b+db×(bt),left=l+d l ×(rl), right=r+d r ×(rl). Calculate the intersection-union ratio (IUGR) between the bounding boxes predicted by the two regression methods and their corresponding ground truth boxes, and select the appropriate regression method based on the characteristics of different scenarios. For scenarios with large variations in target size, the center point offset prediction method is preferred; for scenarios with irregular target shapes, the four-boundary prediction method is preferred.

[0040] In step S202, multi-scale features of the input image are extracted, and the matching scores between the features and one or more prediction heads are calculated. The bounding box coordinates output by the prediction heads are then weighted and fused to obtain an optimized prediction result. This includes: extracting edge texture features, local structural features, and semantic features of the input image as multi-scale features; setting feature index weight coefficients for each prediction head; calculating the matching score between the image features and the prediction head; normalizing the matching score to obtain the weight coefficient for each prediction head; and weighting the target bounding box coordinates output by each prediction head according to the weight coefficients to obtain the final bounding box position.

[0041] When the target model contains multiple predictor heads, an adaptive feature selection approach can be employed. First, multi-scale features of the input image are extracted, including low-level edge texture features, mid-level local structure features, and high-level semantic features. For each predictor head, a corresponding feature index weight coefficient is set. Specifically, the matching score S between image features F and predictor head Pi is calculated. i =∑(w ij ×f j ), where wij f represents the weight coefficient of the i-th prediction head for the j-th class feature. j This represents the intensity value of the j-th feature. The matching score is normalized using the sigmoid function. Where β is the temperature coefficient, which is optimized on the validation set using a grid search method. Experiments show that a β value of 2.37 yields the best normalization effect for the feature matching score. Based on the normalized matching score, the detection results of different predictors are weighted and fused. Specifically, the target bounding box coordinates output by the i-th predictor are set as (x... i y i w i h i If the final bounding box coordinates are: Through a feature adaptive selection mechanism, the prediction head that best matches the image features is used preferentially during the detection process.

[0042] In step S203, based on the edge complexity and local contrast index of the target region, the intersection-union ratio (IU) threshold is set hierarchically and locally adaptively optimized, and the detection result is judged. This includes: calculating the edge complexity index based on the contour perimeter and area of ​​the target region to quantify and classify the scene; setting a baseline IU threshold according to the edge complexity level and dynamically adjusting it using an adjustment coefficient; dividing the image into grid cells and calculating the local contrast index of each cell; marking low-contrast regions and performing local neighborhood adaptive optimization of the IU threshold based on the normalized area ratio of the target region.

[0043] Specifically, to adapt to different scenarios and application requirements, a hierarchical IOU threshold setting scheme is adopted. First, the scene is quantified and graded according to the complexity of the target features. The target features are quantified by calculating the edge complexity index Ce of the target region in the image, where... P represents the perimeter of the target contour, and A represents the area of ​​the target region. Edge complexity index C e Borrowing from the circle degree metric, a higher value indicates a more irregular shape (circle C). e =1, polygon C e >1). For each target region, when C e When C < 1.237, it is classified as a regular objective; when 1.237 ≤ C e When C < 2.368, it is classified as a semi-regular objective. e Targets with an IOU value ≥2.368 are classified as irregular targets. Based on this, a baseline IOU threshold T0 = 0.75 is set. For target regions of different complexities, an adaptive adjustment coefficient K is used to dynamically adjust the IOU threshold. The adjusted IOU threshold is calculated as follows: T = T0 × (1 - κ × ln(C)).e Among them, when C e When <1.237, κ=0.132; when 1.237≤C e When C < 2.368, κ = 0.187; when C e When ≥2.368, κ=0.234. For scenarios with high accuracy requirements, such as precision parts inspection, a correction value of 0.118 is added to T; for target inspection in complex backgrounds, a correction value of 0.083 is subtracted from T.

[0044] The process of dividing the image into grid cells and calculating the local contrast index of each cell includes: obtaining grid division parameters based on the image resolution; dividing the input image into multiple grid cells according to the number of rows and columns; extracting the maximum and minimum pixel values ​​within each grid cell; and calculating the local contrast index of each grid cell by subtracting the minimum pixel value from the maximum pixel value and dividing by the sum of the maximum and minimum pixel values.

[0045] Specifically, when performing threshold calculation, the contour point set {(x} of the target region is first obtained. i y i )}, i = 1, 2, ..., n, calculate the perimeter of the target region based on the contour point set. Calculate the area of ​​the target region using Green's formula Substituting the calculated P and A into the calculation formula for Ce, the complexity level of the target is determined, and the corresponding K value is selected. Whether to add a correction value depends on the actual application scenario, ultimately obtaining the IOU threshold T for the current detection target. For scenarios with high accuracy requirements, such as precision parts inspection, the IOU threshold is set to 0.85; for general industrial scenarios, the IOU threshold is set to 0.75; and for target detection against complex backgrounds, the IOU threshold is set to 0.65. After determining the final IOU threshold, local neighborhood adaptive optimization is performed. First, the image is divided into M×N grid cells, where the values ​​of M and N are determined based on the image resolution. Where W and H are the width and height of the image, respectively.

[0046] The process includes marking low-contrast regions and performing local neighborhood adaptive optimization of the cross-union ratio (CUP) threshold based on the normalized area ratio of the target region. This includes: marking grid cells with local contrast indices less than a preset threshold as low-contrast regions; calculating the proportion of low-contrast regions in grid cells containing the detected target and their eight neighboring grid cells; adjusting the CUP threshold for the region when the proportion of low-contrast regions exceeds one-third based on the ratio of the local contrast index to the preset threshold; calculating the ratio of the target region area to the total image area as the normalized area ratio; increasing the CUP threshold based on the difference between the normalized area ratio and the first preset ratio when the normalized area ratio is less than a first preset ratio; decreasing the CUP threshold based on the difference between the normalized area ratio and the second preset ratio when the normalized area ratio is greater than a second preset ratio; calculating the ratio of the minimum distance between adjacent targets to the diagonal length of the larger target; and uniformly decreasing the CUP threshold for these targets when the ratio is less than a preset coefficient.

[0047] Specifically, for each grid cell, its local contrast index is calculated. Among them I max and I min These are the maximum and minimum pixel values ​​within the unit, respectively. The contrast threshold is set to λ = 0.386. When L... c When the value is less than λ, the grid cell is marked as a low-contrast region. For a grid cell containing a target, if more than 1 / 3 of itself or its eight adjacent grid cells are marked as low-contrast regions, the IOU threshold T for that region is adjusted to [value missing]. Furthermore, considering the impact of target scale on detection difficulty, a scale adjustment factor μ is introduced. Specifically, the normalized area ratio of the target region is calculated. Where A is the area of ​​the target region, and W and H are the image dimensions. When α < 0.047, the IOU threshold is adjusted to... When α > 0.328, the IOU threshold is adjusted to When performing multi-target detection, if the minimum distance between adjacent targets is less than 0.342 times the diagonal length of the larger target, the IOU threshold for these targets is uniformly adjusted to T. adjusted = T×(1-0.156). This method achieves local adaptive optimization of the IOU threshold, making it better suited to the detection difficulty of different regions. During the judgment process, the corresponding threshold standard is selected based on the specific application scenario.

[0048] Figure 3This is a schematic block diagram of a model verification system disclosed in an embodiment of this application. It should be understood that the system shown in the figure is exemplary and not restrictive. This means that the system architecture involved is not limited to a specific form or design, but is presented as an example. In other words, the architecture shown in the figure can be considered as a way of expressing related concepts and relationships clearly, and does not exclude other forms of architecture. Therefore, when interpreting the architecture in the figure, it should be understood that the model is flexible and diverse, and its purpose is to disclose an exemplary description, not a restrictive provision on a specific form.

[0049] Specifically, this application discloses an evaluation system that applies the automated model evaluation method described in any of the above embodiments, including: an annotation unit 301, a model calling unit 302, and an evaluation calculation unit 303; wherein, the annotation unit 301 is used to collect and annotate standard test images for target detection to form a test set; the model calling unit 302 is used to call the target evaluation model through an interface; the evaluation calculation unit 303 is used to input the test set into the target evaluation model for bounding box prediction to determine the target regression method and target prediction head of the model, and to determine the detection correctness by using the intersection-union threshold; the accuracy, recall, and false alarm rate of the detection are calculated respectively, the feature information of the falsely detected images are recorded, and the deviation values ​​between the predicted box and the real box in each dimension are calculated to obtain evaluation data.

[0050] In summary, traditional methods for evaluating bounding box regression in object detection models often employ a single regression approach, which cannot adapt to the differences in object features across various scenarios. The dual-path regression evaluation method disclosed in this application effectively solves this technical bottleneck through a dynamic selection mechanism of two regression methods: center point offset prediction and four-boundary prediction. Specifically, the center point offset prediction method predicts the offset d of the target center point relative to the center of the original predicted bounding box. x d y And the scaling factors d for width and height w d h This method, which locates the target, is particularly suitable for scenarios where the target size varies significantly. The four-boundary prediction method, on the other hand, directly predicts the offset d of the target bounding box relative to the four boundaries of the original predicted bounding box. t d b d l d r This approach is better suited for handling scenarios with irregularly shaped targets. By selecting the most suitable regression method for different scenarios, the accuracy and robustness of bounding box prediction are significantly improved. Furthermore, this method introduces a precise calculation method based on the predicted box coordinates; for center point offset prediction, x = p x +d x ×p w y = py +d y ×p h w = p w ×exp(d w ), h = p h ×exp(d h The calculation formula is as follows: For four-boundary prediction, top = t + d. t ×(bt), bottom=b+d b ×(bt), left=l+d l ×(rl), right=r+d r The calculation method of ×(rl) ensures the accuracy of bounding box positioning.

[0051] In multi-predictor fusion evaluation, existing technologies generally suffer from inaccurate feature matching and overly simplistic fusion strategies. The feature adaptive selection mechanism disclosed in this application effectively improves the quality of detection results through multi-level feature extraction and adaptive weight allocation. This mechanism first extracts multi-scale features, including low-level edge texture features, mid-level local structure features, and high-level semantic features. Then, it calculates the matching score S between each predictor and image features using feature index weight coefficients. i =∑(w ij ×f j The matching score is normalized using the sigmoid function. Where β is a temperature coefficient of 2.37, this approach ensures the smoothness and stability of the weight allocation. In the final fusion stage of the bounding box coordinates, a weighted average method is used: This fusion method fully considers the contribution of each prediction head, making the final detection results more accurate and reliable.

[0052] In setting IOU thresholds, traditional methods often use fixed thresholds, which are difficult to adapt to the differences in target features under different scenarios. The hierarchical IOU threshold setting scheme disclosed in this application achieves dynamic threshold adjustment through the quantification of target feature complexity and adaptive optimization of local neighborhoods. Firstly, it uses the edge complexity index... The target features are quantized, and an adaptive adjustment coefficient κ is set based on this, such that the IOU threshold T = T0 × (1 - κ × ln(C)). e It can dynamically adjust based on the complexity of the target. Furthermore, it improves the accuracy of threshold setting through a local neighborhood adaptive optimization mechanism. Specifically, this is achieved through M×N grid partitioning and local contrast index... The algorithm calculates and identifies low-contrast regions in the image, and locally adjusts the IOU threshold based on the characteristics of the grid cells. Simultaneously, considering the influence of target scale, a scale adjustment factor μ based on the normalized area ratio α is introduced to achieve adaptive threshold adjustment for targets of different scales. In multi-target detection scenarios, the impact of the distance between adjacent targets on the threshold setting is also considered, ensuring the reliability of detection results in complex scenes.

[0053] Furthermore, embodiments of this application also disclose an automated model verification and evaluation device based on intersection-union-comparison (IUCN) comparison, comprising: a processor, a memory, and a system bus; the processor and the memory are connected via the system bus; the memory is used to store one or more programs, the one or more programs comprising instructions, the instructions of which, when executed by the processor, cause the processor to perform any of the methods described above.

[0054] Furthermore, embodiments of this application also disclose a computer program product that, when run on a terminal device, causes the terminal device to execute any of the methods described above.

[0055] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0056] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0057] It should also be noted that, in the embodiments of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0058] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in the embodiments of this application may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown in this application, but is to be accorded the widest scope consistent with the principles and novel features disclosed in the embodiments of this application.

Claims

1. An automated model evaluation method based on intersection-union comparison, characterized in that, include: Collect and annotate target detection standard test images to form a test set; The test set is input into the target evaluation model to predict bounding boxes, in order to determine the target regression method and target prediction head of the model, and the detection correctness is determined by the intersection-over-union threshold. The accuracy, recall, and false alarm rate of the detection are calculated separately. The feature information of the falsely detected images is recorded, and the deviation values ​​between the predicted bounding box and the ground truth bounding box in each dimension are calculated to obtain the evaluation data.

2. The automated model evaluation method according to claim 1, characterized in that, in, Collect and annotate standard test images for target detection to form a test set, including: Collect standard test images for target detection and annotate each image by drawing a rectangular bounding box; Record the coordinates of the four vertices of the rectangle, and establish a relationship between the coordinates and the corresponding images and store them in the database; For regular geometric targets, the rectangle is kept parallel to the target outline; for irregular shaped targets, the smallest bounding rectangle is selected as the annotation box. All the images that have been annotated and their corresponding annotation data are used as the test set.

3. The automated model evaluation method according to claim 1, characterized in that, in, The test set is input into the target evaluation model for bounding box prediction to determine the model's target regression method and target prediction head. The correctness of the detection is then determined by the intersection-over-union (IoU) threshold, including: Based on the characteristics of the target scene, select the center point offset prediction or four-boundary prediction method, calculate the offset and scaling factor, and obtain the initial predicted position of the target bounding box. Extract multi-scale features from the input image, calculate the matching score between the features and one or more prediction heads, and perform weighted fusion of the bounding box coordinates output by the prediction heads to obtain the optimized prediction result; Based on the edge complexity and local contrast index of the target region, the cross-union ratio threshold is set in a hierarchical manner and locally adaptively optimized, and the detection results are judged.

4. The automated model evaluation method according to claim 1, characterized in that, in, Calculate the detection accuracy, recall, and false positive rate respectively, record the feature information of the falsely detected images, and calculate the deviation values ​​between the predicted bounding boxes and the ground truth bounding boxes in each dimension to obtain evaluation data, including: Calculate the accuracy of the model, where the accuracy is the number of correct detections divided by the total number of detections; Calculate the recall rate of the model, where the recall rate is the number of correctly detected objects divided by the total number of standard bounding boxes; Calculate the false alarm rate of the model, where the false alarm rate is the number of false detections divided by the total number of detections; Record information including image features, lighting conditions, and / or target size in cases of false detection; Calculate the deviation values ​​between the predicted bounding box and the ground truth bounding box in the dimensions of horizontal offset, vertical offset, width deviation, and / or height deviation.

5. The automated model evaluation method according to claim 3, characterized in that, in, Based on the characteristics of the target scene, either center point offset prediction or four-boundary prediction is selected. The offset and scaling factor are calculated to obtain the initial predicted position of the target bounding box, including: The center point offset prediction method is used to calculate the center coordinates and width and height values ​​of the original prediction box, and the target bounding box position is determined based on the predicted offset and scaling factor. The four-boundary prediction method is used to calculate the coordinates of the top, bottom, left, and right boundaries of the original prediction box, and the position of the target bounding box is determined based on the predicted four-boundary offset. Calculate the intersection-union ratio (IU) between the bounding boxes and the ground truth boxes for both prediction methods; For scenarios with significant changes in target size, a center point offset prediction method is used; for scenarios with irregular target shapes, a four-boundary prediction method is used.

6. The automated model evaluation method according to claim 3, characterized in that, in, Extract multi-scale features from the input image, calculate the matching score between the features and one or more prediction heads, and perform weighted fusion of the bounding box coordinates output by the prediction heads to obtain the optimized prediction result, including: Edge texture features, local structural features, and semantic features of the input image are extracted as multi-scale features; Set the feature index weight coefficients for each prediction head, and calculate the matching score between image features and prediction heads; The matching scores are normalized to obtain the weight coefficients for each prediction head; The target bounding box coordinates output by each prediction head are weighted according to the weighting coefficients to obtain the final bounding box position.

7. The automated model evaluation method according to claim 3, characterized in that, in, Based on the edge complexity and local contrast index of the target region, the cross-union ratio (CUI) threshold is set hierarchically and adaptively optimized in the local neighborhood. The detection results are then judged, including: The edge complexity index is calculated based on the contour perimeter and area of ​​the target region, and the scene is quantitatively classified. Set a baseline intersection-union ratio threshold based on the edge complexity level, and dynamically adjust it using an adjustment coefficient; The image is divided into grid cells, and the local contrast index of each cell is calculated. Low-contrast regions are marked, and local neighborhood adaptive optimization is performed based on the cross-union ratio threshold according to the normalized area ratio of the target region.

8. The automated model evaluation method according to claim 7, characterized in that, in, The image is divided into grid cells, and the local contrast index of each cell is calculated, including: Obtain grid division parameters based on image resolution; The input image is divided into multiple grid units according to the number of rows and columns; Extract the maximum and minimum pixel values ​​within each grid cell; The local contrast index of each grid cell is calculated by subtracting the minimum pixel value from the maximum pixel value and dividing by the sum of the maximum and minimum pixel values.

9. The automated model evaluation method according to claim 7, characterized in that, in, Low-contrast regions are marked, and local neighborhood adaptive optimization is performed on the cross-union ratio threshold based on the normalized area ratio of the target region, including: Grid cells with local contrast indices below a preset threshold are marked as low-contrast regions. The percentage of low-contrast regions in the grid cell containing the target and its eight adjacent grid cells is statistically analyzed. When the low-contrast area accounts for more than one-third, the cross-union ratio threshold of that area is adjusted according to the ratio of the local contrast index to the preset threshold. The ratio of the target region area to the total image area is calculated as the normalized area ratio. When the normalized area ratio is less than the first preset ratio, the crossover ratio threshold is increased based on the difference between the normalized area ratio and the first preset ratio. When the normalized area ratio is greater than the second preset ratio, the crossover ratio threshold is lowered based on the difference between the normalized area ratio and the second preset ratio. Calculate the ratio of the minimum distance between adjacent targets to the diagonal length of the larger target; When the ratio is less than a preset coefficient, the cross-union ratio threshold of these targets is uniformly lowered.

10. An evaluation system applying the automated model evaluation method as described in any one of claims 1-9, characterized in that, include: The unit comprises an annotation unit, a model invocation unit, and an evaluation calculation unit; among which, The annotation unit is used to collect target detection standard test images and annotate them to form a test set; The model invocation unit is used to invoke the target evaluation model through an interface; The verification calculation unit is used to input the test set into the target verification model to predict the bounding box, so as to determine the target regression method and target prediction head of the model, and to determine the detection correctness by the intersection-union threshold; calculate the detection accuracy, recall and false alarm rate respectively, record the feature information of the false detection image, and calculate the deviation value between the predicted box and the real box in each dimension to obtain verification data.

Images