Evaluation system for deep learning networks for object detection on image data

Abstract

The invention discloses an evaluation system for a deep learning network for target detection of image data, which belongs to the field of deep learning technology and detection technology. The existing credibility evaluation of deep learning networks for target detection of image data is not perfect. The system of the present invention provides a complete credibility evaluation system, including a false alarm and missed detection rate judgment subsystem and an optional index evaluation subsystem. The false alarm and missed detection rate judgment subsystem is used to determine the false alarm and missed detection situations, obtain the label file based on the prediction or recognition result of the target detection network, and then use the intersection-over-union ratio and the intersection-over-small ratio to match the target frame, so as to obtain the false alarm rate and missed detection rate; the optional index evaluation subsystem is used to calculate the corresponding index according to the option of the evaluation index input by the user and feed it back to the user. The subsystem includes a generalization evaluation module and a robustness evaluation module, which respectively evaluate the generalization and robustness of the target detection deep learning network.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology and detection technology, and specifically relates to an evaluation system for a deep learning network for single-target detection of image data. Background Art

[0002] In the 21st century, 80% of the information people obtain comes from images, so image and video processing is particularly important. In our daily lives and work, we generate a large amount of video and image data every day, such as dashcams, surveillance videos, and health checkups. This large amount of data needs to be processed manually, which cannot meet our needs. We can improve this situation through computers, and object detection is an indispensable technology.

[0003] Object detection technology has significant practical implications for reducing human capital consumption, protecting security information, and enabling intelligent living experiences. While object detection technology is widely used, accidents it causes can have a significant impact on human life and potentially lead to significant losses. For example, in 2016, a driver was killed in an accident while driving a self-driving car; that same year, a man in Florida was killed when his self-driving car collided with a train; and in 2018, a pedestrian in Arizona was fatally injured after a self-driving car collided with him. Therefore, testing and evaluating the reliability, feasibility, and safety of object detection technology is crucial and imperative.

[0004] However, current methods for evaluating the credibility of deep learning object detection networks are relatively limited. Research also lacks unified technical specifications and standards for credibility assessment. Typically, simple metrics such as accuracy and precision are used to evaluate detection results, or simply the coverage of neurons within the network is studied. These single metrics are incomplete and cannot serve as comprehensive indicators for credibility assessment. Furthermore, when studying object detection networks, the results presented are only obtained under ideal conditions or typical scenarios. Network performance cannot be guaranteed under non-ideal conditions or other scenarios, nor can it be guaranteed if attacked or interfered with. Therefore, relying on existing evaluation methods cannot comprehensively evaluate the credibility of object detection networks. Incompletely evaluated object detection networks have unknown generalization and robustness performance, as well as unknown internal neuron structure and utilization. This makes it impossible to fundamentally guarantee the credibility of object detection networks, leading to safety risks in practical applications and even major accidents. More importantly, current false alarm and missed detection rate assessments, using only the intersection-over-union ratio to determine whether an object box matches, are unreliable. Summary of the Invention

[0005] The present invention aims to solve the problem that the existing deep learning network evaluation for target detection on image data is unreliable in determining whether the target box matches only by using the intersection-over-union ratio, and the problem that the performance of the network under non-ideal conditions cannot be obtained.

[0006] The deep learning network evaluation system for object detection based on image data includes a false alarm and missed detection rate judgment subsystem and an optional indicator evaluation subsystem. The false alarm and missed detection rate judgment subsystem is used to determine false alarms and missed detections, and the optional indicator evaluation subsystem is used to calculate the corresponding indicators based on the evaluation indicator options entered by the user and provide feedback to the user.

[0007] The process of determining false alarms and missed detections by the false alarm and missed detection rate judgment subsystem comprises the following steps:

[0008] Based on the prediction or recognition results of the target detection network, obtain the label file; the target detection network can export two txt label files for each image, one original label file and one predicted label file; based on the two thresholds of the set intersection-over-union ratio, perform the following processing:

[0009] Of the two thresholds of the intersection-over-union ratio, the larger threshold is recorded as the first threshold, and the smaller threshold is recorded as the second threshold;

[0010] (A) When there are more original label target frames than predicted label target frames: take out a frame from the original label txt by line, compare it with each frame in the predicted label file, and calculate the intersection-over-union ratio;

[0011] When the IoU of two boxes is greater than the first threshold, it means that the two boxes match; after matching, the original label value and the predicted label value are read into two new txt files respectively;

[0012] If the IoU between the original box and each box in the predicted label txt is less than the second threshold, it is judged as a missed detection; the number of missed detection boxes and the total number of boxes are calculated, and finally the missed detection rate is calculated;

[0013] If the IoU of the original frame and each frame in the predicted label txt file is between the second threshold and the first threshold, calculate the intersection ratio (IoU): the area of the intersection of the two target frames / the target frame with the smaller area in the two frames. Select the one with the largest IoU, indicating that the two frames match. After matching, read the original label value and the predicted label value into two new txt files respectively.

[0014] (B) When the original label target box is less than the predicted label target box: take out a box from the original label txt file by line, compare it with each box in the original label file, and calculate the intersection-over-union ratio;

[0015] When the IoU of two boxes is greater than the first threshold, it means that the two boxes match; after matching, the original label and the predicted label are read into two new txt files respectively;

[0016] If the IoU calculated between the predicted box and each box in the original label txt is less than the second threshold, it is determined to be a false alarm, the number of false alarm boxes and the total number of boxes are calculated, and finally the false alarm rate is calculated;

[0017] If the IoU between the predicted box and each box in the original label txt is between the second threshold and the -first threshold, calculate the intersection-over-union ratio; select the one with the largest intersection-over-union ratio, indicating that the two boxes match; after matching, read the original label value and the predicted label value into two new txts respectively;

[0018] After the above operation is completed, you will get two txt files after label matching and two indicators: false alarm rate and missed detection rate. The label values in the two files are one-to-one corresponding.

[0019] Furthermore, the optional indicator evaluation subsystem includes a generalization evaluation module; the user selects and inputs one or more indicators in the generalization evaluation, and the optional indicator evaluation subsystem evaluates the corresponding indicators according to the corresponding indicators:

[0020] The generalization evaluation module includes a classification correctness evaluation unit and a positioning accuracy evaluation unit;

[0021] A. Classification Correctness Evaluation Unit:

[0022] The classification correctness evaluation unit is used to evaluate the generalization of the classification label. The generalization evaluation indicators of the classification label include one or more of the accuracy rate ACC, precision rate P, recall rate R, confusion matrix, average AP value, F1 value, ROC curve, AUC area, Kappa coefficient, Hamming distance, and Jaccard similarity coefficient;

[0023] The average AP value is the average AP value of multiple test set individuals;

[0024] B. Positioning accuracy evaluation unit:

[0025] The positioning accuracy evaluation unit is used to evaluate the generalization of the target box label. The generalization indicators of the target box label include one or more of IoU, GIoU, DIoU, CIoU, EIoU, root mean square error, mean square error, mean absolute error, and mean absolute error percentage.

[0026] Furthermore, the optional indicator evaluation subsystem further includes a robustness evaluation module; the user selects and inputs one or more indicators in the robustness evaluation, and the optional indicator evaluation subsystem evaluates the corresponding indicators according to the corresponding indicators:

[0027] The robustness evaluation module is used to evaluate the performance of the detection network after the data set is subjected to a series of interferences. The robustness evaluation module includes a classification correctness robustness evaluation unit and a positioning accuracy robustness evaluation unit;

[0028] a. Classification correctness robust evaluation unit:

[0029] The classification correctness robust evaluation unit evaluates the changes in the network's correctness evaluation indicators under different noise attacks. The correctness evaluation indicators corresponding to the classification correctness robust evaluation unit include the accuracy change indicator, the precision change indicator, and the recall change indicator.

[0030] b. Positioning accuracy robust evaluation unit:

[0031] The positioning accuracy robust evaluation unit evaluates the generalization evaluation indicators of the network under different noise attacks. The generalization evaluation indicators corresponding to the positioning accuracy robust evaluation unit include IoU change index, GIoU change index, DIoU change index, CIoU change index, and EIoU change index.

[0032] Furthermore, the process of evaluating the changes in the correctness evaluation indicators of the network under different noise attacks is as follows:

[0033] Before testing the network, the test data set is denoised, and different noise intensities are used. The changes in the correctness evaluation indicators are recorded as the noise intensity changes.

[0034] Furthermore, the noise includes Gaussian noise, salt and pepper noise, and random occlusion.

[0035] Furthermore, the robustness evaluation module also includes an other robustness index evaluation unit; the evaluation indexes corresponding to the other robustness index evaluation unit include: one or more indicators selected from the group consisting of noise capacity estimation, robustness to image compression, robustness to Gaussian blur, maximum boundary distance, average confidence, average confidence of correct categories, distortion against attack, average structural similarity, anti-motion sensitivity distance, and empirical noise sensitivity;

[0036] (1) Noise capacity estimation NTE:

[0037]

[0038] Among them, n represents the number of successful samples of all adversarial attacks, Indicates that the i-th sample is classified as class a, is the probability that the i-th sample is classified as class a;

[0039] (2) Robustness to image compression RIC:

[0040]

[0041]

[0042] Where UA represents undirected attack, TA represents directed attack, and IC(·) represents image compression processing. is the probability distribution of the i-th sample being classified as a, y i Indicates that the i-th sample is correctly classified, Indicates that the i-th sample is correctly classified after the targeted attack;

[0043] (3) Robustness to Gaussian blur RGB:

[0044]

[0045]

[0046] Where UA represents undirected attack, TA represents directed attack, and GB(·) represents Gaussian blur processing;

[0047] (4) Maximum boundary distance BD:

[0048]

[0049] Where V represents a randomly generated set, φ i (V) represents the RMS distance to the model decision boundary, d i Represents the maximum distance to the decision boundary;

[0050] (5) Average confidence ACAC:

[0051]

[0052] (6) Average confidence of correct category ACTC:

[0053]

[0054] Where, is the probability that the i-th sample is classified as the corresponding class, y i Indicates that the i-th sample is correctly classified as y i kind;

[0055] (7) Adversarial attack distortion degree ALDp:

[0056]

[0057] (8) Average structural similarity ASS:

[0058]

[0059] (9) Disturbance Sensitive Distance PSD:

[0060]

[0061] Where m is the total number of pixels, δ i,j represents the jth pixel of the i-th sample, R(x i,j ) represents x i,j The nearby square area, std represents the standard deviation function;

[0062] (10) Empirical noise sensitivity ENI:

[0063] Empirical noise sensitivity represents a test set of synthetic adversarial attacks and natural noise.

[0064] A computer storage medium stores at least one instruction, which is loaded and executed by a processor to implement the deep learning network evaluation system for target detection on image data.

[0065] A device for evaluating a deep learning network for target detection on image data, the device comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the deep learning network evaluation system for target detection on image data.

[0066] A method for determining missed detection rate and false alarm rate of a deep learning network for object detection in image data includes the following steps:

[0067] Based on the prediction or recognition results of the target detection network, obtain the label file; the target detection network can export two txt label files for each image, one original label file and one predicted label file; based on the two thresholds of the set intersection-over-union ratio, perform the following processing:

[0068] Of the two thresholds of the intersection-over-union ratio, the larger threshold is recorded as the first threshold, and the smaller threshold is recorded as the second threshold;

[0069] (A) When there are more original label target frames than predicted label target frames: take out a frame from the original label txt by line, compare it with each frame in the predicted label file, and calculate the intersection-over-union ratio;

[0070] When the IoU of two boxes is greater than the first threshold, it means that the two boxes match; after matching, the original label value and the predicted label value are read into two new txt files respectively;

[0071] If the IoU between the original box and each box in the predicted label txt is less than the second threshold, it is judged as a missed detection; the number of missed detection boxes and the total number of boxes are calculated, and finally the missed detection rate is calculated;

[0072] If the IoU of the original frame and each frame in the predicted label txt file is between the second threshold and the first threshold, calculate the intersection ratio (IoU): the area of the intersection of the two target frames / the target frame with the smaller area in the two frames. Select the one with the largest IoU, indicating that the two frames match. After matching, read the original label value and the predicted label value into two new txt files respectively.

[0073] (B) When the original label target box is less than the predicted label target box: take out a box from the original label txt file by line, compare it with each box in the original label file, and calculate the intersection-over-union ratio;

[0074] When the IoU of two boxes is greater than the first threshold, it means that the two boxes match; after matching, the original label and the predicted label are read into two new txt files respectively;

[0075] If the IoU calculated between the predicted box and each box in the original label txt is less than the second threshold, it is determined to be a false alarm, the number of false alarm boxes and the total number of boxes are calculated, and finally the false alarm rate is calculated;

[0076] If the IoU between the predicted box and each box in the original label txt is between the second threshold and the -first threshold, calculate the intersection-over-union ratio; select the one with the largest intersection-over-union ratio, indicating that the two boxes match; after matching, read the original label value and the predicted label value into two new txts respectively;

[0077] After the above operation is completed, you will get two txt files after label matching and two indicators: false alarm rate and missed detection rate. The label values in the two files are one-to-one corresponding.

[0078] A device for determining the missed detection rate and false alarm rate of a deep learning network for target detection in image data, the device comprising a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the method for determining the missed detection rate and false alarm rate of a deep learning network for target detection in image data.

[0079] Beneficial effects:

[0080] The present invention introduces two indicators, the intersection-in-union ratio and the intersection-in-small ratio, to determine whether the target frames match. This can effectively solve the problem that if only the intersection-in-union ratio is used, when one of the two target frames is particularly large and the other is particularly small, the intersection-in-union ratio value will be diluted and reduced, and the threshold value cannot be reached, which will lead to inaccurate results. Therefore, the present invention effectively improves the effect of false alarms and missed detections. At the same time, the present invention has a more comprehensive evaluation effect on the target detection deep learning network, and can comprehensively evaluate the credibility of the target detection network. In addition, the present invention also evaluates the changes in the correctness evaluation indicators of the network under different noise attacks, and makes a comprehensive evaluation from the perspective of generalization performance and robust performance. It can also enable the present invention to effectively level the performance of the network under non-ideal conditions, so that the performance of the network can be effectively guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 Schematic diagram of indicators for optional indicator subunits.

[0082] Figure 2 This is an image from the infrared small target dataset.

[0083] Figure 3 The effect diagram of different salt and pepper noise intensities.

[0084] Figure 4 The effect diagram under different Gaussian noise intensities.

[0085] Figure 5 These are the effects of different random occlusion sizes.

[0086] Figure 6 Evaluation process for generalization metrics of a single target detection network.

[0087] Figure 7(a) to Figure 7(c) Shows the changes in indicators after being processed with Gaussian noise of different intensities.

[0088] Figure 8(a) to Figure 8(c) The changes of indicators after random occlusion processing with different intensities.

[0089] Figure 9(a) to Figure 9(c) Shows the changes in indicators after being processed with salt and pepper noise of different intensities. DETAILED DESCRIPTION

[0090] In response to the problems in the background technology, when evaluating target detection networks, it is necessary to consider not only the black-box characteristics of the network but also the white-box characteristics, and not only the generalization ability of the network but also the robustness. Therefore, the present invention establishes a complete credibility assessment system for deep learning target detection networks to evaluate target detection networks. The present invention is described in detail below in conjunction with specific implementation methods. Specific implementation method one:

[0092] This embodiment is a deep learning network evaluation system for target detection on image data, including a false alarm and missed detection rate judgment subsystem and an optional indicator evaluation subsystem; among them, the false alarm and missed detection rate judgment subsystem is used to determine false alarms and missed detections, and the optional indicator evaluation subsystem is used to calculate the corresponding indicators according to the evaluation indicator options input by the user and feedback them to the user.

[0093] The false alarm and missed detection rate judgment subsystem comprises the following steps:

[0094] Based on the prediction or recognition results of the target detection network, obtain the label file; the target detection network can export two txt label files for each image, one original label file (that is, the label file corresponding to the original image) and one predicted label file (that is, the predicted label file);

[0095] The number of tags in the two files may be different, as shown below:

[0096]

[0097]

[0098] If the original label file has many target boxes and the predicted label file has few target boxes, the following may occur:

[0099] ① Missed detection. ② The original label txt file contains multiple target boxes, and the target detection network detects them as one target box.

[0100] If the original label file has a small number of target boxes and the predicted label file has a large number of target boxes, the following may occur:

[0101] ① False alarm. ② The original label txt file contains a target box, and the target detection network detects it as multiple target boxes.

[0102] As long as it is not a false alarm or missed detection, it doesn't matter whether it is split into multiple frames or merged into one frame. The original frame and the predicted frame will overlap. After research, the present invention found that if only the intersection-in-union ratio is used to evaluate the recognition effect of the target detection network, when one of the two target frames is particularly large and the other is particularly small, the intersection-in-union ratio value will be diluted and reduced, and the threshold cannot be reached, which will lead to inaccurate results. Therefore, the present invention introduces two indicators: intersection-in-union ratio and intersection-in-small ratio:

[0103] Intersection over Union (IoU): The area where two target boxes intersect / the area where two target boxes are combined.

[0104] Intersection ratio: the area of the intersection of the two target boxes / the target box with the smaller area of the two boxes.

[0105] In some embodiments, an IoU threshold is set, with 0.7 and 0.3 being used as examples for illustration:

[0106] (1) When the original label target box is more than the predicted label target box:

[0107]

[0108] Take out a box in the original label txt line by line, compare it with each box in the predicted label file, and calculate the intersection-over-union ratio.

[0109] When the IoU between two boxes exceeds a threshold of 0.7, they match. After matching, read the original label value and the predicted label value into two new text files. For example, if the calculated IoU value between the fifth line of the original file and the fourth line of the predicted file is greater than 0.7, save these two lines into two new text files.

[0110] b. If the IoU between the original box and each box in the predicted label txt is less than the threshold of 0.3, it is considered a missed detection. The number of missed detection boxes and the total number of boxes are calculated, and the missed detection rate is finally calculated.

[0111] c. If the IoU between the original box and each box in the predicted label txt file is between 0.3 and 0.7, calculate the intersection over union (IoU) ratio. Select the one with the largest IoU ratio, indicating that the two boxes match. After matching, read the original label value and the predicted label value into two new txt files.

[0112] (2) When the original label target box is less than the predicted label target box:

[0113]

[0114] Take out a box from the original label txt file line by line, compare it with each box in the original label file, and calculate the intersection and union ratio.

[0115] When the IoU between two boxes exceeds a threshold of 0.7, they match. After matching, read the original label and predicted label into two new txt files. For example, if the fourth line of the original label file and the fifth line of the predicted label file have an IoU greater than 0.7, store these two lines in two separate txt files.

[0116] b. If the calculated IoU between the predicted box and each box in the original label txt is less than the threshold of 0.3, it is judged as a false alarm. The number of false alarm boxes and the total number of boxes are calculated, and finally the false alarm rate is calculated.

[0117] c. If the IoU between the predicted box and each box in the original label text is between 0.3 and 0.7, calculate the intersection over union (IoU) ratio. Select the one with the largest IoU ratio, indicating that the two boxes match. After matching, read the original label value and the predicted label value into two new text files.

[0118] After the above operation is completed, you will get two txt files after the label matching, as well as two indicators: false alarm rate and missed detection rate. The label values in the two files are one-to-one corresponding.

[0119] The optional indicator evaluation subsystem includes a generalization evaluation module and a robustness evaluation module, such as Figure 1 As shown; the user can choose to input one or more indicators of generalization evaluation and / or robustness evaluation, and the optional indicator evaluation subsystem will evaluate the corresponding indicators according to the corresponding indicators:

[0120] (1) The generalization evaluation module includes a classification correctness evaluation unit and a positioning accuracy evaluation unit;

[0121] A. Classification Correctness Evaluation Unit

[0122] The classification accuracy evaluation unit is used to evaluate the generalization of classification labels. The generalization evaluation indicators of classification labels include 11 indicators such as accuracy, precision, recall, confusion matrix, mAP, F1 value, ROC curve, AUC area, Kappa coefficient, Hamming distance, and Jaccard similarity coefficient. The specific indicators are as follows:

[0123] (1) Accuracy

[0124] Accuracy refers to the ratio of samples that the model can still correctly classify among all samples after the adversarial attack.

[0125]

[0126] The number of samples that are labeled as positive and classified as positive is recorded as TP; the number of samples that are labeled as positive and classified as negative is recorded as FN; the number of samples that are labeled as negative and classified as positive is recorded as FP; the number of samples that are labeled as negative and classified as positive is recorded as TN.

[0127] (2) Precision

[0128] Precision refers to the proportion of samples that are truly positive among all samples that are judged to be positive.

[0129]

[0130] (3) Recall rate

[0131] Recall is a measure of coverage, which measures the proportion of samples that are correctly predicted to be positive to all samples that are actually positive.

[0132]

[0133] (4) Confusion Matrix

[0134] This is a standard format for expressing accuracy, represented as an n-row, n-column matrix. The horizontal axis of the confusion matrix represents the number of categories predicted by the model, while the vertical axis represents the number of true data labels. The diagonal line represents the number of cases where the model's predictions agree with the data labels. Therefore, the sum of the diagonal numbers divided by the total number of test sets gives the accuracy. Larger numbers on the diagonal are better, and darker colors in the visualization indicate higher model accuracy for that category. If viewed row by row, each row not on the diagonal represents an incorrectly predicted category. In general, it's best to have diagonal lines as high as possible and off-diagonal lines as low as possible.

[0135] (5) mAP

[0136] AP stands for Average Precision, and mAP stands for Average AP value, which is the average AP value of multiple test set individuals. The formula is as follows:

[0137]

[0138] (6) F1 value

[0139] The F1 value is an indicator used to measure the accuracy of a classification model by calculating the precision and recall rate.

[0140]

[0141] (7) ROC curve

[0142] The ROC curve provides a comprehensive analysis of the sensitivity and specificity of a classification model. First, multiple thresholds are defined for a continuous variable to calculate multiple sensitivity and specificity values. These values are then plotted against each other on a curve, with sensitivity on the y-axis and specificity on the x-axis.

[0143] (8)AUC area

[0144] The AUC value is equal to the area under the ROC curve. The larger the AUC area, the higher the classification accuracy.

[0145] (9) Kappa coefficient

[0146] The Kappa coefficient can be used to evaluate the stability of the classifier. The larger the Kappa coefficient, the more consistent the classification results of the classifier, which means the better the stability of the classifier.

[0147]

[0148] Where p o It represents the value of the correct sample ratio to the total sample. When the number of correct samples in each category is a1, a 2, ,...,a c , the number of prediction samples for each class is b1,b2,,...,b c , there are n samples in total, then there are

[0149] (10) Hamming distance

[0150] Hamming distance is used in scenarios where different categories of samples need to be classified. For a given sample i, is the prediction result of the jth label, y ij is the true result of the jth label, L is the number of labels, then with y i The Hamming distance between them is:

[0151]

[0152] (11) Jaccard similarity coefficient

[0153] Jaccard similarity coefficient is also used in scenarios where different categories of samples need to be classified. For a given sample i, is the prediction result, y i is the true result, L is the number of labels, then the Jaccard similarity coefficient of the i-th sample is:

[0154]

[0155] B. Positioning accuracy evaluation unit

[0156] The positioning accuracy evaluation unit is used to evaluate the generalization of the target box label. The generalization indicators of the target box label include 14 indicators such as IoU, GIoU, DIoU, CIoU, EIoU, root mean square error, mean square error, mean absolute error, and mean absolute error percentage.

[0157] (1)IoU

[0158] IoU stands for Intersection over Union. From this name, we can probably guess how IoU is calculated. IoU calculates the ratio of the intersection and union of the "predicted bounding box" and the "true bounding box". The calculation process is as follows:

[0159]

[0160] (2)GIoU

[0161] Since IoU is a ratio concept, it is insensitive to the scale of the target object. However, the optimization of the regression loss (MSE loss, l1-smooth loss, etc.) of the BBox in the detection task is not completely equivalent to the optimization of IoU, and the Ln norm is also sensitive to the scale of the object. IoU cannot directly optimize the non-overlapping parts. Therefore, it is proposed to directly set IoU as the regression loss. First calculate the minimum enclosed area of the two boxes, then calculate the IoU, and then calculate the proportion of the area in the enclosed area that does not belong to the two boxes in the enclosed area, and finally subtract this proportion from the IoU to get GIoU. The formula is as follows:

[0162]

[0163] (3)DIoU

[0164] DIoU is more consistent with the target box regression mechanism than GIou. It takes into account the distance, overlap rate and scale between the target and the anchor, making the target box regression more stable and not prone to divergence problems during training like IoU and GIoU. The formula is as follows:

[0165]

[0166] Among them, b and b gt represents the center point of the predicted box and the true box respectively, and ρ represents the Euclidean distance between the two center points. c represents the diagonal distance of the minimum closed area that can contain both the predicted box and the true box.

[0167] (4)CIoU

[0168] DIoU does not take into account the aspect ratio of the three elements of bbox regression. Therefore, CIoU is proposed based on DIoU. The formula is as follows:

[0169]

[0170] Among them, α is the weight function and ν is the similarity of aspect ratio.

[0171] (5)EIOU

[0172] Although CIOU considers the overlap area, center point distance, and aspect ratio of bounding box regression, the θ in its formula reflects the difference in aspect ratio, rather than the actual difference in width and height and their confidence, which sometimes hinders the model from effectively optimizing similarity. To address this issue, EIOU was proposed based on CIOU by breaking down the aspect ratio.

[0173]

[0174] Among them, c w and c h Is the width and height of the smallest bounding box that covers both boxes.

[0175] (6) Mean square error

[0176] The mean squared error is the expectation of the squared errors.

[0177]

[0178] (7) Root mean square error

[0179] RMS error is the square root of the mean squared error.

[0180]

[0181] (8) Mean absolute error

[0182] Mean absolute error is the expected value of the absolute error loss

[0183]

[0184] (9) Mean absolute error percentage

[0185] The mean absolute error percentage is the expected value of the relative error loss. The so-called relative error is the percentage of the absolute error and the true value.

[0186]

[0187] (10) Coefficient of determination

[0188] The coefficient of determination is the degree to which the independent variable x explains the variation in the dependent variable y. The closer the coefficient of determination is to 1, the better the model fit.

[0189]

[0190] in, It represents the degree of change in y and is proportional to the variance. It represents the residual between the model and the true value. It represents the model's prediction of the change in y.

[0191] (11) Median absolute error

[0192] The median absolute error is determined by taking the loss as the median of all absolute differences between the target and the prediction.

[0193]

[0194] (12) Hill inequality coefficient

[0195] The Hill inequality coefficient is also an indicator of regression. It represents the difference between the fitted value and the true value. The smaller the value, the smaller the difference and the higher the decision accuracy.

[0196]

[0197] (13) Variance rate

[0198] The variance ratio is a measure of the difference in standard deviations between the fitted values and the true values.

[0199]

[0200] (14) Explained variance fraction

[0201] The explained variance fraction is a measure of how closely the dispersion of the differences between the predicted values and the sample matches the dispersion of the sample itself. It's a measure of dispersion itself. Finally, it's calculated as 1 minus this value, with larger values indicating closer dispersion between the predicted and sample values.

[0202]

[0203] To verify the accuracy of the implemented metrics, they need to be verified. Therefore, this paper uses an open-source target detection project and replicates it using the network and dataset it provides. The replicated network is tested using the aforementioned metrics. The metrics are then compared with known metrics from the open-source network to verify their accuracy.

[0204] (2) the robustness evaluation module, which is used to evaluate the performance of the detection network after the data set is subjected to a series of interferences. The robustness evaluation module includes a classification correctness robustness evaluation unit and a positioning accuracy robustness evaluation unit, as well as an other robustness indicator evaluation unit;

[0205] Before going into details, let's first explain the parameters. In the following formula, n represents the number of successful samples of all adversarial attacks. Indicates that the i-th sample is classified as class a, is the probability that the i-th sample is classified as class a, is the probability distribution of the i-th sample being classified as a, y i Indicates that the i-th sample is correctly classified, It means that the i-th sample is correctly classified after the targeted attack.

[0206] a. Classification Correctness Robust Evaluation Unit

[0207] The classification correctness robust evaluation unit evaluates the changes in the network's correctness evaluation indicators under different noise attacks. The correctness evaluation indicators corresponding to the classification correctness robust evaluation unit include the accuracy change indicator, the precision change indicator, and the recall change indicator.

[0208] The evaluation of correctness evaluation indicators needs to be tested and evaluated for different noises; the different noises include Gaussian noise, salt and pepper noise, and random occlusion.

[0209] In some embodiments, before testing the network, the test data set needs to be denoised (using Gaussian noise, salt and pepper noise, and random occlusion to denoise the data set), and different noise intensities are used. The Gaussian noise intensity varies from 0.01 to 0.20, with an interval of 0.01. The intensity of the salt and pepper noise varies from 0.001 to 0.020, with an interval of 0.001. The size of the random occlusion varies from 0.01 to 0.20, with an interval of 0.01. Record the changes in the correctness evaluation index as the noise intensity changes.

[0210] b. Positioning accuracy robust evaluation unit

[0211] The positioning accuracy robust evaluation unit evaluates the generalization evaluation indicators of the network under different noise attacks. The generalization evaluation indicators corresponding to the positioning accuracy robust evaluation unit include IoU change index, GIoU change index, DIoU change index, CIoU change index, and EIoU change index;

[0212] The evaluation of generalization evaluation metrics also needs to be tested and evaluated against different noises;

[0213] In some embodiments, for attacks with different noises, changes in the generalization evaluation index as the noise intensity changes are recorded.

[0214] During the evaluation, we calculated how the indicators change under different noise attacks as the noise intensity changes, thereby evaluating the robustness of the classification labels of the object detection network. We also used the changes in indicators such as Intersection over Union (IoU), GIoU, DIoU, CIoU, and EIoU to evaluate the robustness of the object detection network's object box labels.

[0215] c. Other robustness indicator evaluation units

[0216] In addition to robustness assessments for classification accuracy and positioning precision, the robustness of the image itself must also be considered. Other robustness metrics evaluated in the evaluation unit include: noise capacity estimation, robustness to image compression, robustness to Gaussian blur, maximum boundary distance, average confidence, average confidence in the correct category, adversarial attack distortion, average structural similarity, motion-sensitive distance, and empirical noise sensitivity.

[0217] (1) Noise Capacity Estimation (NTE)

[0218] The noise capacity estimate reflects the amount of noise that can be tolerated by the adversarial sample while keeping the classification category unchanged. The formula is defined as follows:

[0219]

[0220] (2) Robustness to Image Compression (RIC)

[0221] Image compression robustness is often used to evaluate image denoising in computer vision algorithms. Normally, a highly robust adversarial example should maintain its misclassification effect after image compression. The formula is defined as follows:

[0222]

[0223]

[0224] Where UA represents undirected attack, TA represents directed attack, and IC function represents image compression processing.

[0225] (3) Robustness to Gaussian blur (RGB)

[0226] Robustness to Gaussian blur is often used to evaluate image denoising in computer vision algorithms. Normally, a highly robust adversarial example should maintain its misclassification effect after Gaussian blurring. The formula is defined as follows:

[0227]

[0228]

[0229] Where UA represents undirected attack, TA represents directed attack, and GB function represents Gaussian blur processing.

[0230] (4) Maximum boundary distance (BD)

[0231] The maximum boundary distance represents the distance between data points and the decision boundary, which measures the stability and robustness of the model in the worst case. The formula is defined as follows:

[0232]

[0233] Where V represents a randomly generated set, φ i (V) represents the RMS distance to the model decision boundary, d i Represents the maximum distance to the decision boundary.

[0234] (5) Average confidence level (ACAC)

[0235] The average confidence score represents the average prediction confidence score for the wrong category, which is the average probability of all misclassified categories for all successful adversarial examples after the adversarial attack. The formula is defined as follows:

[0236]

[0237] (6) Average confidence of correct category (ACTC)

[0238] The average confidence score for the correct category is calculated by averaging the predicted confidence scores for the true category of the adversarial attack sample. This is used to assess the extent to which the attack deviates from the true value. The formula is defined as follows:

[0239]

[0240] In the formula is the probability that the i-th sample is classified as the corresponding class, and yi indicates that the i-th sample is correctly classified as class yi.

[0241] (7) Adversarial Attack Distortion (ALDp)

[0242] The adversarial attack distortion is the average normalized Lp distortion of all successful adversarial examples. The formula is defined as follows:

[0243]

[0244] (8) Average structural similarity (ASS)

[0245] The average structural similarity is defined as the average similarity between all successfully attacked adversarial samples and their original samples. The formula is defined as follows:

[0246]

[0247] (9) Perturbation Sensitive Distance (PSD)

[0248] The disturbance sensitivity distance is used to evaluate human perception of disturbances. The formula is defined as follows:

[0249]

[0250] Where m is the total number of pixels, δ i,j represents the jth pixel of the i-th sample, R(x i,j ) represents x i,j The square area around it, std represents the standard deviation function.

[0251] (10) Empirical Noise Sensitivity (ENI)

[0252] Empirical noise sensitivity represents a test set of synthetic adversarial attacks and natural noise. Specific implementation method two:

[0254] This embodiment is a computer storage medium, which stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the deep learning network evaluation system for target detection on image data.

[0255] It should be understood that the instructions include computer program products, software, or computerized methods corresponding to any method described in the present invention; the instructions can be used to program a computer system or other electronic device. Computer storage media may include readable media on which instructions are stored, and may include but are not limited to magnetic storage media, optical storage media; magneto-optical storage media include read-only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM) and flash memory layers, or other types of media suitable for storing electronic instructions. Specific implementation method three:

[0257] This embodiment is a device for evaluating deep learning networks for object detection on image data. The device includes a processor and a memory. It should be understood that the device includes any device including a processor and a memory described in the present invention, and may also include other units and modules that perform display, interaction, processing, control, and other functions through signals or instructions.

[0258] At least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the target detection deep learning network evaluation system for image data. Specific implementation method four:

[0260] This embodiment is a method for determining the missed detection rate and false alarm rate of a deep learning network for object detection in image data, including the following steps:

[0261] Based on the prediction or recognition results of the target detection network, obtain the label file; the target detection network can export two txt label files for each image, one original label file and one predicted label file; based on the two thresholds of the set intersection-over-union ratio, perform the following processing:

[0262] Of the two thresholds of the intersection-over-union ratio, the larger threshold is recorded as the first threshold, and the smaller threshold is recorded as the second threshold;

[0263] (A) When there are more original label target frames than predicted label target frames: take out a frame from the original label txt by line, compare it with each frame in the predicted label file, and calculate the intersection-over-union ratio;

[0264] When the IoU of two boxes is greater than the first threshold, it means that the two boxes match; after matching, the original label value and the predicted label value are read into two new txt files respectively;

[0265] If the IoU between the original box and each box in the predicted label txt is less than the second threshold, it is judged as a missed detection; the number of missed detection boxes and the total number of boxes are calculated, and finally the missed detection rate is calculated;

[0266] If the IoU of the original frame and each frame in the predicted label txt file is between the second threshold and the first threshold, calculate the intersection ratio (IoU): the area of the intersection of the two target frames / the target frame with the smaller area in the two frames. Select the one with the largest IoU, indicating that the two frames match. After matching, read the original label value and the predicted label value into two new txt files respectively.

[0267] (B) When the original label target box is less than the predicted label target box: take out a box from the original label txt file by line, compare it with each box in the original label file, and calculate the intersection-over-union ratio;

[0268] When the IoU of two boxes is greater than the first threshold, it means that the two boxes match; after matching, the original label and the predicted label are read into two new txt files respectively;

[0269] If the IoU calculated between the predicted box and each box in the original label txt is less than the threshold of 0.3, it is judged as a false alarm, the number of false alarm boxes and the total number of boxes are calculated, and finally the false alarm rate is calculated;

[0270] If the IoU between the predicted box and each box in the original label txt is between the second threshold and the -first threshold, calculate the intersection-over-union ratio; select the one with the largest intersection-over-union ratio, indicating that the two boxes match; after matching, read the original label value and the predicted label value into two new txts respectively;

[0271] After the above operation is completed, you will get two txt files after label matching and two indicators: false alarm rate and missed detection rate. The label values in the two files are one-to-one corresponding. Specific implementation method five:

[0273] This embodiment is a device for determining the missed detection rate and false alarm rate of a deep learning network for object detection in image data. The device includes a processor and a memory. It should be understood that the device includes any device including a processor and a memory described in the present invention, and may also include other units and modules that perform display, interaction, processing, control, and other functions through signals or instructions;

[0274] At least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the method for determining the missed detection rate and false alarm rate of the deep learning network for target detection of image data.

[0275] Example

[0276] The system for evaluating deep learning networks for object detection in image data described in this embodiment can evaluate the network's results (e.g., reliability) for single-target recognition, as well as for single-class and multi-class target recognition. This embodiment evaluates the dataset generation of a single object detection network.

[0277] This detection is for single infrared small target detection, and the YOLOv5 algorithm is used for testing and training. The dataset is the AntiUAV dataset. The original data format of this dataset is an MP4 video file. In this paper, the dataset is processed by frame extraction, and a total of 9214 frames of images are extracted, of which 8292 frames are used for training the target detection network and 922 frames are used for testing the network. The test images are as follows Figure 2 shown.

[0278] When evaluating the robustness test indicators, the data set was denoised. Three different noise addition processes were performed, namely, adding salt and pepper noise of different intensities, Gaussian noise, and random occlusion. The noise intensity of salt and pepper noise is 0.01-0.20, with an interval of 0.01. The noise intensity of Gaussian noise is 0.001-0.020, with an interval of 0.001. The proportion of random occlusion is 0.01-0.20, with an interval of 0.01. Taking an image as an example, the effects of the image after salt and pepper noise, Gaussian noise, and random occlusion are as follows: Figure 3-Figure 5We present the effects of salt and pepper noise with intensities of 0.01, 0.05, 0.10, 0.15, and 0.20, Gaussian noise with intensities of 0.001, 0.005, 0.010, 0.015, and 0.020, and random occlusion with scales of 0.01, 0.05, 0.10, 0.15, and 0.20.

[0279] Results of the generalization test indicators for single target detection based on YOLOv5.

[0280] We test the trained yolov5 infrared small target detection network, export the original labels and test labels of the test set, and use the classification labels and target box labels in the label set to calculate the generalization test index. The specific operation process is as follows Figure 6 shown.

[0281] Through statistical analysis and evaluation of classification labels, this implementation selected eight metrics: accuracy, precision, recall, F1 index, Kappa, Hamming distance, and Jaccard similarity coefficient. Table 1 shows this. Through statistical analysis and evaluation of target box labels, this implementation selected four metrics: IoU_Loss, GIoU_Loss, DIoU_Loss, and CIoU_Loss. Taking one sample as an example, its metrics are shown in Table 2.

[0282] Table 1 Classification label evaluation results of infrared small target detection network

[0283]

[0284] Table 2 Target box label evaluation results of infrared small target detection network

[0285]

[0286] Robustness test results of single target detection based on YOLOv5

[0287] This paper tests the infrared single-target network using a dataset that has been processed with Gaussian noise, salt and pepper noise, and random occlusion. The original labels and test labels are derived, and robustness test metrics are calculated using the classification labels and target box labels in the label set.

[0288] By processing the classification labels of the test results, we can obtain the changes in accuracy, precision, and recall under the influence of noise intensity. Figure 7(a) to Figure 7(c) 、 Figure 8(a) to Figure 8(c) and Figure 9(a) to Figure 9(c) shown. Figure 7(a) to Figure 7(c) Shows the changes in indicators after being processed with Gaussian noise of different intensities. Figure 8(a) to Figure 8(c)The changes of indicators after random occlusion processing with different intensities. Figure 9(a) to Figure 9(c) Shows the changes in indicators after being processed with salt and pepper noise of different intensities.

[0289] By processing the target box labels in the test results, we obtain the four metrics of IoU_Loss, GIoU_Loss, DIoU_Loss, and CIoU_Loss under the influence of noise intensity. Taking one sample in the dataset as an example, we only take the changes in the four metrics after it undergoes random occlusion of a certain intensity, as shown in Table 3.

[0290] Table 3 Changes of the four indicators after random occlusion of partial intensity

[0291]

[0292] The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. A deep learning network evaluation system for object detection on image data, characterized by: It includes a false alarm and missed detection rate judgment subsystem and an optional index evaluation subsystem; wherein, the false alarm and missed detection rate judgment subsystem is used to determine the false alarm and missed detection situations, and the optional index evaluation subsystem is used to calculate the corresponding index according to the evaluation index options input by the user and feedback it to the user; The process of determining false alarms and missed detections by the false alarm and missed detection rate judgment subsystem comprises the following steps: Based on the prediction or recognition results of the target detection network, obtain the label file; the target detection network can export two txt label files for each image, one original label file and one predicted label file; based on the two thresholds of the set intersection-over-union ratio, perform the following processing: Of the two thresholds of the intersection-over-union ratio, the larger threshold is recorded as the first threshold, and the smaller threshold is recorded as the second threshold; (A) When there are more original label target frames than predicted label target frames: take out a frame from the original label txt by line, compare it with each frame in the predicted label file, and calculate the intersection-over-union ratio; When the IoU of two boxes is greater than the first threshold, it means that the two boxes match; after matching, the original label value and the predicted label value are read into two new txt files respectively; If the IoU between the original box and each box in the predicted label txt is less than the second threshold, it is judged as a missed detection; the number of missed detection boxes and the total number of boxes are calculated, and finally the missed detection rate is calculated; If the IoU of the original frame and each frame in the predicted label txt file is between the second threshold and the first threshold, calculate the intersection ratio (IoU): the area of the intersection of the two target frames / the target frame with the smaller area in the two frames. Select the one with the largest IoU, indicating that the two frames match. After matching, read the original label value and the predicted label value into two new txt files respectively. (B) When the original label target box is less than the predicted label target box: take out a box from the original label txt file by line, compare it with each box in the original label file, and calculate the intersection-over-union ratio; When the IoU of two boxes is greater than the first threshold, it means that the two boxes match; after matching, the original label and the predicted label are read into two new txt files respectively; If the IoU calculated between the predicted box and each box in the original label txt is less than the second threshold, it is determined to be a false alarm, the number of false alarm boxes and the total number of boxes are calculated, and finally the false alarm rate is calculated; If the IoU between the predicted box and each box in the original label txt is between the second threshold and the -first threshold, calculate the intersection-over-union ratio; select the one with the largest intersection-over-union ratio, indicating that the two boxes match; after matching, read the original label value and the predicted label value into two new txts respectively; After the above operation is completed, you will get two txt files after label matching and two indicators: false alarm rate and missed detection rate. The label values in the two files are one-to-one corresponding.

2. The deep learning network evaluation system for target detection based on image data according to claim 1, characterized in that: The optional indicator evaluation subsystem includes a generalization evaluation module; the user selects and inputs one or more indicators in the generalization evaluation, and the optional indicator evaluation subsystem evaluates the corresponding indicators according to the corresponding indicators: The generalization evaluation module includes a classification correctness evaluation unit and a positioning accuracy evaluation unit; A. Classification Correctness Evaluation Unit: The classification correctness evaluation unit is used to evaluate the generalization of the classification label. The generalization evaluation indicators of the classification label include one or more of the accuracy rate ACC, precision rate P, recall rate R, confusion matrix, average AP value, F1 value, ROC curve, AUC area, Kappa coefficient, Hamming distance, and Jaccard similarity coefficient; The average AP value is the average AP value of multiple test set individuals; B. Positioning accuracy evaluation unit: The positioning accuracy evaluation unit is used to evaluate the generalization of the target box label. The generalization indicators of the target box label include one or more of IoU, GIoU, DIoU, CIoU, EIoU, root mean square error, mean square error, mean absolute error, and mean absolute error percentage.

3. The deep learning network evaluation system for target detection based on image data according to claim 2, characterized in that: The optional indicator evaluation subsystem also includes a robustness evaluation module; the user selects and inputs one or more indicators in the robustness evaluation, and the optional indicator evaluation subsystem evaluates the corresponding indicators according to the corresponding indicators: The robustness evaluation module is used to evaluate the performance of the detection network after the data set is subjected to a series of interferences. The robustness evaluation module includes a classification correctness robustness evaluation unit and a positioning accuracy robustness evaluation unit; a. Classification correctness robust evaluation unit: The classification correctness robust evaluation unit evaluates the changes in the network's correctness evaluation indicators under different noise attacks. The correctness evaluation indicators corresponding to the classification correctness robust evaluation unit include the accuracy change indicator, the precision change indicator, and the recall change indicator. b. Positioning accuracy robust evaluation unit: The positioning accuracy robust evaluation unit evaluates the generalization evaluation indicators of the network under different noise attacks. The generalization evaluation indicators corresponding to the positioning accuracy robust evaluation unit include IoU change index, GIoU change index, DIoU change index, CIoU change index, and EIoU change index.

4. The deep learning network evaluation system for target detection based on image data according to claim 3, characterized in that: The process of evaluating the changes in the correctness evaluation indicators of the network under different noise attacks is as follows: Before testing the network, the test data set is denoised, and different noise intensities are used. The changes in the correctness evaluation indicators are recorded as the noise intensity changes.

5. The deep learning network evaluation system for target detection based on image data according to claim 4, characterized in that: The noise includes Gaussian noise, salt and pepper noise, and random occlusion.

6. The deep learning network evaluation system for target detection based on image data according to claim 3, 4 or 5, characterized in that: The robustness evaluation module further includes an other robustness index evaluation unit; the evaluation indexes corresponding to the other robustness index evaluation unit include: one or more indicators selected from the group consisting of noise capacity estimation, robustness to image compression, robustness to Gaussian blur, maximum boundary distance, average confidence, average confidence of correct categories, distortion against attack, average structural similarity, motion-sensitive distance, and empirical noise sensitivity; (1) Noise capacity estimation NTE: Among them, n represents the number of successful samples of all adversarial attacks, Indicates that the i-th sample is classified as class a, is the probability that the i-th sample is classified as class a; (2) Robustness to image compression RIC: Where UA represents undirected attack, TA represents directed attack, and IC(·) represents image compression processing. is the probability distribution of the i-th sample being classified as a, y i Indicates that the i-th sample is correctly classified, Indicates that the i-th sample is correctly classified after the targeted attack; (3) Robustness to Gaussian blur RGB: Where UA represents undirected attack, TA represents directed attack, and GB(·) represents Gaussian blur processing; (4) Maximum boundary distance BD: Where V represents a randomly generated set, φ i (V) represents the RMS distance to the model decision boundary, d i Represents the maximum distance to the decision boundary; (5) Average confidence ACAC: (6) Average confidence of correct category ACTC: Where, is the probability that the i-th sample is classified as the corresponding class, y i Indicates that the i-th sample is correctly classified as y i kind; (7) Adversarial attack distortion degree ALDp: (8) Average structural similarity ASS: (9) Disturbance Sensitive Distance PSD: Where m is the total number of pixels, δ i,j represents the jth pixel of the i-th sample, R(x i,j ) represents x i,j The nearby square area, std represents the standard deviation function; (10) Empirical noise sensitivity ENI: Empirical noise sensitivity represents a test set of synthetic adversarial attacks and natural noise.

7. A computer storage medium, characterized in that The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the deep learning network evaluation system for target detection on image data as described in any one of claims 1 to 6.

8. A device for evaluating deep learning networks for object detection in image data, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the target detection deep learning network evaluation system for image data as described in any one of claims 1 to 6.

9. A method for determining missed detection rate and false alarm rate of a deep learning network for object detection based on image data, characterized in that: The following steps are involved: Based on the prediction or recognition results of the target detection network, obtain the label file; the target detection network can export two txt label files for each image, one original label file and one predicted label file; based on the two thresholds of the set intersection-over-union ratio, perform the following processing: Of the two thresholds of the intersection-over-union ratio, the larger threshold is recorded as the first threshold, and the smaller threshold is recorded as the second threshold; (A) When there are more original label target frames than predicted label target frames: take out a frame from the original label txt by line, compare it with each frame in the predicted label file, and calculate the intersection-over-union ratio; When the IoU of two boxes is greater than the first threshold, it means that the two boxes match; after matching, the original label value and the predicted label value are read into two new txt files respectively; If the IoU between the original box and each box in the predicted label txt is less than the second threshold, it is judged as a missed detection; the number of missed detection boxes and the total number of boxes are calculated, and finally the missed detection rate is calculated; If the IoU of the original frame and each frame in the predicted label txt file is between the second threshold and the first threshold, calculate the intersection ratio (IoU): the area of the intersection of the two target frames / the target frame with the smaller area in the two frames. Select the one with the largest IoU, indicating that the two frames match. After matching, read the original label value and the predicted label value into two new txt files respectively. (B) When the original label target box is less than the predicted label target box: take out a box from the original label txt file by line, compare it with each box in the original label file, and calculate the intersection-over-union ratio; When the IoU of two boxes is greater than the first threshold, it means that the two boxes match; after matching, the original label and the predicted label are read into two new txt files respectively; If the IoU calculated between the predicted box and each box in the original label txt is less than the second threshold, it is determined to be a false alarm, the number of false alarm boxes and the total number of boxes are calculated, and finally the false alarm rate is calculated; If the IoU between the predicted box and each box in the original label txt is between the second threshold and the -first threshold, calculate the intersection-over-union ratio; select the one with the largest intersection-over-union ratio, indicating that the two boxes match; after matching, read the original label value and the predicted label value into two new txts respectively; After the above operation is completed, you will get two txt files after label matching and two indicators: false alarm rate and missed detection rate. The label values in the two files are one-to-one corresponding.

10. A device for determining missed detection rate and false alarm rate of a deep learning network for target detection in image data, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method for determining the missed detection rate and false alarm rate of the deep learning network for target detection of image data as described in claim 9.

Images