Black box test method and system for infrared image target detection algorithm

Abstract

The invention discloses a black box testing method and system for an infrared image target detection algorithm, and belongs to the technical field of computer vision and algorithm testing. According to the method, the comprehensiveness, robustness and generalization of the detection algorithm are tested on the premise that the internal structure and code logic of the algorithm are not obtained. The method comprises the following steps: firstly, on the basis of an original data set with normal gray level, generating aspect data required by a test by using a condition coverage and metamorphic test metamorphic relation method and a mode of collecting other data sets; secondly, inputting the generated data set into an infrared image target detection algorithm, and drawing an ROC curve, a PR curve and a confusion matrix through an IoU value between an algorithm prediction frame and a real frame; then, according to a performance test result, evaluation indexes such as Precise, Recall and F1 values are calculated, the test performance of the algorithm on a corresponding data set is obtained, and the comprehensiveness, reliability, generalization and efficiency of the algorithm are verified through a test result.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and algorithm testing technology, and relates to the performance evaluation of infrared image processing and target detection algorithms. Specifically, it is a black-box testing method and system for infrared image target detection algorithms. Without obtaining the internal structure of the algorithm, it can systematically evaluate the comprehensiveness, robustness and generalization of the algorithm, and provide a basis for the improvement and optimization of the algorithm. Background Technology

[0002] Most optical measurement devices currently possess target detection and tracking capabilities, and infrared image target detection is one of the technologies used in this field, determining whether the optical measurement device can correctly capture the target. Infrared image target detection is an important research direction in computer vision. Its core principle is to locate and identify targets in complex scenes by analyzing the physical characteristics of objects captured by infrared cameras. Compared to visible light images, infrared images used in infrared target detection methods (such as Chinese patents CN118196369A and CN109784336A) are more resistant to light interference and can penetrate smoke, enabling all-weather operation. However, they also suffer from drawbacks such as low signal-to-noise ratio, limited texture detail, and poor contrast, making target detection more difficult. Therefore, it is necessary to test the detection performance of infrared image target detection methods to ensure that the algorithm, when configured in optical measurement devices, can correctly capture targets.

[0003] Currently, most testing methods for infrared image target detection methods employ white-box testing, which evaluates the logical correctness of the algorithm by analyzing the source code structure, execution path coverage, and boundary condition verification. However, these methods can only judge the rationality of the algorithm based on the completeness of the code logic and cannot objectively demonstrate the algorithm's detection performance. Furthermore, existing testing methods have limitations in the construction of test datasets. For example, conventional test datasets often come from limited scenarios and lack coverage of special cases such as edge targets, small targets, and low-contrast targets, failing to truly reflect the algorithm's performance in complex real-world environments. Especially when facing factors such as changes in target position, number of targets, image brightness variations, and noise interference, existing testing systems lack targeted design and cannot systematically measure the algorithm's comprehensiveness, robustness, and generalization ability.

[0004] In summary, existing performance testing methods for infrared image target detection algorithms suffer from problems such as incomplete test coverage, strong subjectivity in evaluation results, and a lack of standardized testing procedures, making it impossible to objectively and comprehensively evaluate the detection performance of algorithms in real-world application environments. Therefore, how to construct a testing method that does not rely on the internal logic of the algorithm and can scientifically, objectively, and comprehensively measure the performance of infrared image target detection algorithms is a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0005] (I) Purpose of the Invention

[0006] This invention aims to propose a black-box testing method and system for infrared image target detection methods. It performs black-box testing on infrared image target detection algorithms in optical measurement equipment without considering the algorithm code logic. Test cases are generated using metamorphic testing and conditional coverage methods. The algorithm output results are statistically analyzed, and evaluation indicators are used to objectively demonstrate the performance of the infrared image target detection method. This verifies the algorithm's performance and limitations, determines the direction for algorithm improvement, and ensures the detection capability of optical measurement equipment.

[0007] (II) Technical Solution

[0008] To achieve the objective of this invention and solve its technical problems, the present invention adopts the following technical solution:

[0009] The first objective of this invention is to propose a black-box testing method for infrared image target detection algorithms. This method systematically tests and quantitatively evaluates the comprehensiveness, robustness, and generalization ability of infrared image target detection algorithms under different input perturbation conditions without obtaining the algorithm's internal structure and code logic. The method comprises three parts: test case generation, performance testing, and test evaluation. Specifically:

[0010] S100. Test Case Generation:

[0011] Based on the evaluation requirements of the infrared image target testing algorithm in terms of comprehensiveness, robustness and generalization, the conditional coverage method is used to construct test cases of the target in different regions and at different scales of the infrared image. The metamorphic test method is used to construct test cases under different perturbation conditions according to the set metamorphic relationship. And / or heterogeneous infrared image datasets are introduced through data collection to expand the test range.

[0012] S200 Algorithm Performance Test:

[0013] Test cases are input into the target detection algorithm of the infrared image under test. The detection results under different test scenarios are obtained by calculating the intersection over union (IoU) between the predicted bounding box and the ground truth bounding box. Based on the preset IoU threshold, the number of true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN) in the test results are counted. A confusion matrix is ​​generated based on the statistical results. At the same time, the precision-recall (PR) curve and the receiver operating characteristic (ROC) curve are plotted to reflect the response behavior of the algorithm under different thresholds.

[0014] S300. Test Result Evaluation:

[0015] The statistical results obtained during the performance testing phase are quantitatively analyzed, and quantitative evaluation indicators are calculated, including at least precision, recall, F1 score and / or average precision. The performance of the infrared image target detection algorithm in three dimensions—comprehensiveness, robustness and generalization—is comprehensively evaluated, and the output results of the black-box test framework for the infrared image target detection algorithm are obtained.

[0016] The second objective of this invention is to provide a black-box testing system for infrared image target detection algorithms, corresponding to the aforementioned black-box testing method of this invention, which includes at least:

[0017] The test case generation module generates multi-dimensional test cases based on the evaluation requirements of the infrared image target detection algorithm in terms of comprehensiveness, robustness, and generalization. This includes: a conditional coverage submodule, which generates test images with preset positional distribution characteristics of targets located in different regions of the image (including image edges) to cover various spatial layout scenarios by setting sliding windows of different sizes and step sizes; a metamorphic testing submodule, which performs mutation processing on the original image based on set metamorphic relationships (including changes in the number of targets, brightness perturbations, noise interference, scale transformations, etc.) to form test cases under different input perturbation conditions; and a data collection submodule, which introduces heterogeneous infrared image datasets independent of the training data distribution to expand the sample domain and evaluate the model's generalization performance.

[0018] The algorithm performance testing module is used to input the generated test cases into the infrared image target detection algorithm under test, and to collect and statistically analyze the output results generated by the algorithm. This includes: an IoU calculation submodule, used to calculate the IoU value based on the coordinate information of the predicted bounding box and the ground truth bounding box; a confusion matrix construction submodule, used to count the number of TP, FP, FN, and TN based on the set IoU threshold and construct the corresponding confusion matrix; and a performance curve plotting submodule, used to iteratively plot the PR curve and ROC curve based on the confusion matrix under different thresholds, showing the trend of algorithm performance changes under different fault tolerance levels.

[0019] The results evaluation module is used to quantitatively analyze the statistical results output by the performance testing module and calculate evaluation metrics such as precision, recall, F1 score and / or average precision.

[0020] The multidimensional evaluation module is used to evaluate and comprehensively assess the algorithm performance in three aspects: comprehensiveness, robustness, and generalization, and output the evaluation results to assist in algorithm optimization and deployment decisions.

[0021] (III) Technical Effects

[0022] Compared with existing technologies, the black-box testing method and system for infrared image target detection methods of the present invention have the following beneficial and significant technical effects: The present invention designs a black-box testing framework for target detection algorithms, testing the comprehensiveness, robustness, and generalization of the algorithm without modifying the code or affecting the equipment. The testing framework includes three parts: generating test cases, performance testing, and evaluating test results. In the test case generation part, conditional coverage, metamorphic testing, and collection methods are used to generate test cases corresponding to the aspects of the algorithm to be tested. The conditional coverage method is used to generate a dataset of targets at edge positions to test the algorithm's ability to detect edge targets. The metamorphic testing method is used to generate datasets with different quantities, shapes, brightness, and noise interference to test the algorithm's detection performance under these complex backgrounds. The collection method is used to collect a set of infrared image datasets to test the detection performance of the target detection algorithm based on grayscale integral in color images. In the performance testing part, the test cases are input into the detection algorithm to generate test results. The confusion matrix, PR curve, and ROC curve are calculated by the intersection-union ratio of the test results and the real results to obtain the performance test results of the algorithm. In the test result evaluation section, the performance metrics obtained from the performance testing section are used to evaluate the performance of the target detection algorithm, identify the limitations of the current algorithm, and provide directions for improving the algorithm's performance. Attached Figure Description

[0023] Figure 1 The diagram shows the black-box testing flowchart for infrared image target detection algorithms.

[0024] Figure 2The diagram shown illustrates the IoU calculation.

[0025] Figure 3 The diagram shown is a schematic of a confusion matrix;

[0026] Figure 4 The diagram shows the distribution of the initial dataset, where (a) represents the distribution of the x-coordinate of the target, (b) represents the distribution of the y-coordinate of the target, and (c) represents the distribution of the coordinates of the target.

[0027] Figure 5 The diagram shows the distribution of the edge dataset, where: (a) the distribution of the x-coordinate of the target being measured, (b) the distribution of the y-coordinate of the target being measured, and (c) the distribution of the coordinates of the target being measured.

[0028] Figure 6 The diagram shows the distribution of target quantities, including: (a) distribution of a small number of target datasets, (b) distribution of a medium number of target datasets, and (c) distribution of a large number of target datasets.

[0029] Figure 7 The diagram shows some test cases under different brightness levels, where: (a) original brightness, (b) alpha = 0.25, (c) alpha = 0.50, (d) alpha = 0.70;

[0030] Figure 8 The image shown is a schematic diagram of the pixel value distribution of the test cases (original image), where: (a) is the overall pixel value distribution histogram, (b) is the pixel value distribution of the target area, and (c) is the pixel value distribution of the background area;

[0031] Figure 9 The diagram shows the pixel value distribution of the test cases (pixel distribution under 0.25alpha), where: (a) is the overall pixel value distribution histogram, (b) is the pixel value distribution of the target area, and (c) is the pixel value distribution of the background area;

[0032] Figure 10 The diagram shows the pixel value distribution of the test cases (pixel distribution under 0.50alpha), where: (a) is the overall pixel value distribution histogram, (b) is the pixel value distribution of the target area, and (c) is the pixel value distribution of the background area;

[0033] Figure 11 The diagram shows the pixel value distribution of the test cases (pixel distribution under 0.70alpha), where: (a) is the overall pixel value distribution histogram, (b) is the pixel value distribution of the target area, and (c) is the pixel value distribution of the background area;

[0034] Figure 12 The following are schematic diagrams of some test cases under different signal-to-noise ratios, where: (a) original image, (b) 25 < PSNR < 30, (c) 15 < PSNR < 20, (d) 5 < PSNR < 10;

[0035] Figure 13 The following is a schematic diagram of the pixel value distribution of the test case (original image), where: (a) is the overall pixel value distribution histogram, (b) is the pixel value distribution of the target area, and (c) is the pixel value distribution of the background area;

[0036] Figure 14 The following is a schematic diagram of the pixel value distribution of the test case (pixel distribution under 25 < PSNR < 30), where: (a) is the overall pixel value distribution histogram, (b) is the pixel value distribution of the target area, and (c) is the pixel value distribution of the background area;

[0037] Figure 15 The following is a schematic diagram of the pixel value distribution of the test case (pixel distribution under 15 < PSNR < 20), where: (a) is the overall pixel value distribution histogram, (b) is the pixel value distribution of the target area, and (c) is the pixel value distribution of the background area;

[0038] Figure 16 The following is a schematic diagram of the pixel value distribution of the test case (pixel distribution under 5 < PSNR < 10), where: (a) is the overall pixel value distribution histogram, (b) is the pixel value distribution of the target area, and (c) is the pixel value distribution of the background area;

[0039] Figure 17 The following is a schematic diagram of the test results of the test case for the target position, where: (a) is the confusion matrix, (b) is the PR curve, and (c) is the ROC curve;

[0040] Figure 18 The following is a schematic diagram of the test results of the test case for the target edge position, where: (a) is the confusion matrix, (b) is the PR curve, and (c) is the ROC curve;

[0041] Figure 19 The following is a schematic diagram of the test results of the test case for a small number of target datasets, where: (a) is the confusion matrix, (b) is the PR curve, and (c) is the ROC curve;

[0042] Figure 20 The following is a schematic diagram of the test results of the test case for a medium number of target datasets, where: (a) is the confusion matrix, (b) is the PR curve, and (c) is the ROC curve;

[0043] Figure 21The figure shows the schematic diagram of the test results of a large number of target dataset test cases, where: (a) is the confusion matrix, (b) is the PR curve, and (c) is the ROC curve;

[0044] Figure 22 The figure shows the test results of the test cases with 25 < PSNR < 30, where: (a) is the confusion matrix, (b) is the PR curve, and (c) is the ROC curve;

[0045] Figure 23 The figure shows the test results of the test cases with 15 < PSNR < 20, where: (a) is the confusion matrix, (b) is the PR curve, and (c) is the ROC curve;

[0046] Figure 24 The figure shows the test results of the test cases with 5 < PSNR < 10, where: (a) is the confusion matrix, (b) is the PR curve, and (c) is the ROC curve;

[0047] Figure 25 The figure shows the test results of the test cases with alpha = 0.25, where: (a) is the confusion matrix, (b) is the PR curve, and (c) is the ROC curve;

[0048] Figure 26 The figure shows the test results of the test cases with alpha = 0.5, where: (a) is the confusion matrix, (b) is the PR curve, and (c) is the ROC curve;

[0049] Figure 27 The figure shows the test results of the test cases with alpha = 0.7, where: (a) is the confusion matrix, (b) is the PR curve, and (c) is the ROC curve. Detailed implementation mode

[0050] The present invention aims to propose a black-box test method and system for an infrared image target detection method, which is used to systematically test and quantitatively evaluate the comprehensiveness, robustness and generalization ability of an infrared image target detection algorithm under different input perturbation conditions without obtaining the internal structure and code logic of the algorithm. To make the purpose, technical solution and advantages of the implementation of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in more detail below with reference to the accompanying drawings in the embodiments of the present invention. The described embodiments are part of the embodiments of the present invention, rather than all of the embodiments, and the described embodiments are exemplary and are intended to explain the present invention, and should not be construed as a limitation to the present invention.

[0051] Embodiment 1: Black-box test experimental method

[0052] As a specific embodiment, the following combines with the attached Figure 1The working principle and implementation process of the black-box testing method for infrared image target detection algorithm of the present invention are described in detail. The method is mainly divided into three steps: test case generation, algorithm performance testing, and test structure evaluation.

[0053] S100. Test Case Generation

[0054] Based on the evaluation requirements of the infrared image target testing algorithm in terms of comprehensiveness, robustness and generalization, the conditional coverage method is used to construct test cases of the target under different regions and scales of infrared images, the metamorphic testing method is used to construct test cases under different perturbation conditions according to the set metamorphic relationship, and / or heterogeneous infrared image datasets are introduced through data collection to expand the test range.

[0055] For the conditional coverage method, a sliding window approach is used to generate target regions within the image area. The size of the sliding window, win_size, and the step size, step_size, are adaptively set based on the height, img_h, and width, img_w, of the input infrared image. win_size includes at least the size of the small target window and the size of the large target window. step_size is calculated based on the greatest common divisor of the corresponding window sizes. The window slides across the entire infrared image to generate small target datasets and large target datasets that cover the entire image area. At the same time, to verify the algorithm's performance in detecting the field of view covering the range of the infrared binary image and detecting edge targets in the binary image, the conditional coverage method is used to generate a binary image dataset that meets the edge conditions.

[0056] Furthermore, when generating binary image datasets with edge conditions, the conditional coverage method constructs binary images with black (pixel value 0) as the background and white (pixel value 255) as the target points. It generates targets for image edge regions by setting window indices. A sliding window with a small target window size and a fixed window index is used to generate binary image datasets with small targets at the image edges, while a sliding window with a large target window size and a fixed window index is used to generate binary image datasets with large targets at the image edges. The target coordinates in the edge binary image datasets are mainly concentrated at the initial and final positions of the image. After edge-mapping, the dataset retains only test cases where the target is at the image edge, specifically for testing the infrared image target detection algorithm's ability to detect targets in image boundary regions and verifying the algorithm's performance in edge target detection scenarios.

[0057] More specifically, in this embodiment, the main objective of generating test cases for the black-box testing method of the infrared target detection algorithm is to cover as many scenes as possible in order to fully evaluate the performance of the detection algorithm. In the testing experiment, to verify the algorithm's ability to detect the field of view covering the range of the infrared binary image and its performance in detecting edge targets in the binary image, a conditional coverage method will be used to generate a binary image dataset that meets the edge conditions. The conditions upon which the conditional coverage method generates test cases are shown in Table 1.

[0058] Table 1. Conditions for generating test cases using the condition coverage method.

[0059]

[0060] For the metamorphosis test method, test cases with input perturbation features are constructed based on the preset metamorphosis relationship to indirectly infer the output stability and correctness of the target detection algorithm. The metamorphosis relationship reflects the input-output mapping relationship between the original image and the mutated image. Its types include at least brightness transformation, scale scaling, random noise addition and target position perturbation. Test cases under different perturbation conditions are constructed by transforming the overall brightness of the original image, the target scale ratio, introducing salt and pepper noise of different intensities, and translating the target position based on a sliding window.

[0061] Specifically, in the black-box testing framework for infrared image target detection algorithms, metamorphic testing is used to generate test cases. Metamorphic testing is primarily used to verify whether the model output meets expectations when the input data to the target detection algorithm changes. The core idea is to construct metamorphic relationships when the performance of the detection algorithm cannot be judged by a single input-output result, and then test the algorithm's performance based on the test cases generated from these relationships. The formula for the metamorphic relationship in metamorphic testing is as follows:

[0062] r(x1,x2,...,x n )>>r f (f(x1),f(x2),...,f(x n ))

[0063] In the formula, x1, x2, ..., x n (n>1) are the variables of the function f, f(x1), f(x2), ..., f(x... n The input variables are the output of the function, representing the correspondence between function variables and function output. If x1, x2, ..., x... n When relation r is satisfied, f(x1), f(x2), ..., f(x) n Satisfying relation r f Then (r,r) is called f ) represents the metamorphic relationship in the metamorphic test.

[0064] In the black-box testing framework for infrared image target detection algorithms, the transformation relationship formula for generating corresponding test cases is: r(m1,m2,...,m n )>>r_f(P(m1),P(m2),...,P(m n P is the device target detection algorithm, where n sets of variables m1, m2, ..., m n For each original image with different brightness, scale, and noise levels generated in the corresponding test cases, the relationship (r, r) is then established. f Given an image with varying brightness, scale, and noise levels within a certain range, the relationship between the generated test cases and the original image can be determined. The conditions for generating test cases can be constructed as follows: using an image processing algorithm to process images satisfying the relationship r, the brightness, scale, and noise levels at different values ​​m1, m2, ..., m within a certain range are... n The image is processed to obtain P(m1), P(m2), ..., P(m n Test cases. If we want to verify P(m1), P(m2), ..., P(m... n ) is composed of m1, m2, ..., m n If the original image m1, m2, ..., m is generated according to certain rules, then it proves that the original image m1, m2, ..., m n With test cases P(m1), P(m2), ..., P(m n Satisfying the metamorphic relation (r, r) f According to the comprehensiveness, robustness, and generalization performance of the algorithm to be tested in the test, the metamorphic relationships used in the test are shown in Table 2.

[0065] Table 2. Metamorphic Relationships Used in the Experiment

[0066]

[0067] These metamorphic relationships are designed based on the main challenges faced by infrared image target detection algorithms in practical applications. The target location metamorphic relationship systematically alters the spatial distribution of targets in the image to verify the algorithm's detection coverage of targets at different locations; the brightness metamorphic relationship simulates imaging environments under different lighting conditions to test the algorithm's adaptability to lighting changes; the scale metamorphic relationship evaluates the algorithm's detection performance for multi-scale targets by changing the size ratio of the target or image; and the noise metamorphic relationship introduces random interference of varying intensities to verify the algorithm's robustness in complex noisy environments. Through the combined application of these metamorphic relationships, diverse test cases comprehensively covering real-world application scenarios can be constructed.

[0068] Table 3 shows the external libraries required to implement the testing framework.

[0069]

[0070] S200 Algorithm Performance Test:

[0071] The test cases are input into the target detection algorithm of the infrared image under test. The detection results under different test scenarios are obtained by calculating the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. Based on the preset IoU threshold, the number of true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN) in the test results are counted. A confusion matrix is ​​generated based on the statistical results. At the same time, the precision-recall (PR) curve and the receiver operating characteristic (ROC) curve are plotted to reflect the response behavior of the algorithm under different thresholds.

[0072] Specifically, after generating test cases that meet the testing requirements, the test cases are input into the object detection algorithm. Based on the performance information corresponding to the confusion matrix generated under the IoU threshold, ROC and PR curves are plotted. The confusion matrix can be plotted using the U-value results, and the performance changes of the algorithm under different thresholds can be determined by calculating the changes in the two curves. In performance testing, after the test cases are input into the object detection algorithm to generate predicted bounding boxes, the IoU value between the predicted bounding boxes and the ground truth bounding boxes is calculated. The IoU value measures the detection performance of the algorithm by calculating the degree of overlap between the predicted bounding boxes of the object detection algorithm and the labeled ground truth bounding boxes. Using this method to calculate algorithm performance is relatively intuitive. The IoU calculation diagram is shown below. Figure 2 . Figure 2 The numerator represents the area of ​​the intersection region between the predicted bounding box and the ground truth bounding box detected by the algorithm, and the denominator represents the area of ​​the union region between the predicted bounding box and the ground truth bounding box. The IoU value corresponding to a detected bounding box is obtained by comparing the areas of the two regions. The pseudocode in Table 4 below details the execution process of calculating the IoU value using test cases.

[0073] Table 4. Method and Flow for Calculating IoU Values ​​of Test Cases

[0074]

[0075] When the Intersection over Union (IoU) value between the predicted bounding box and the ground truth bounding box output by the infrared image target detection algorithm is sufficiently large, it can be considered that the detection algorithm has correctly detected the target. During the testing process, the IoU value ranged from (0,1). A calculated output value close to 1 indicated a higher degree of overlap between the predicted and ground truth bounding boxes, demonstrating a better detection result. Initially, the IoU threshold was set to 0.5; an IoU value exceeding the threshold indicated correct detection by the algorithm. Preferably, the IoU threshold was set as an adjustable parameter and adjusted step-by-step within the range of [0.3, 0.7]. The threshold was gradually decreased or increased at set intervals (preferably a step size of 0.05) for multi-threshold detection performance evaluation. At each threshold level, IoU matching, confusion matrix generation, and subsequent index calculations were performed again to form a performance profile analysis under multiple thresholds, quantifying the changes in the detection capability of the infrared image target detection algorithm under different tolerance conditions.

[0076] In performance testing, the IoU values ​​of all test cases were calculated using an infrared image target detection algorithm, and then categorized according to rules. In target detection, four cases were identified: true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). The confusion matrices for these four cases are as follows: Figure 3 The confusion matrix illustrates the temporal relationship between predicted and ground truth bounding boxes in the tested infrared image target detection algorithm. Figure 3 True positives (TP) represent the number of positive samples correctly detected within the ground truth bounding boxes. First, the IoU value of the predicted bounding box output by the object detection algorithm must be greater than the IoU threshold; second, the predicted bounding box type must match the ground truth bounding box type. False positives (FP) represent the number of negative samples incorrectly detected by the object detection algorithm. Detection errors include two situations: the IoU value between the predicted and ground truth bounding boxes is less than the IoU threshold, and the ground truth bounding box category does not match. False negatives (FN) represent the number of positive samples detected as negative samples, and true negatives (TN) represent the number of negative samples correctly detected as negative samples. By calculating the number of samples in the four categories in the confusion matrix, further statistical analysis of the samples can be performed to measure the performance of the algorithm in different aspects.

[0077] After obtaining the confusion matrix, the performance of the model under different thresholds is displayed by calculating and plotting PR curves and ROC curves. The PR curve is plotted with precision and recall as variables, where recall is on the x-axis and precision is on the y-axis. The detailed steps for plotting the PR curve are shown in Table 5 below:

[0078] Table 5. Methods for plotting PR curves

[0079]

[0080] The ROC curve is plotted using true positives (TP) and false positives (FP) as variables. Since plotting the PR curve requires precision, which is easily affected by the distribution of test case samples, the PR curve cannot accurately reflect the algorithm's true detection performance when the sample distribution is uneven. The ROC curve, however, uses only recall and is unaffected by sample distribution, thus compensating for the shortcomings of the PR curve and providing a more stable display of the test algorithm's performance. Detailed steps for plotting the ROC curve are shown in Table 6 below.

[0081] Table 6. Methods for plotting ROC curves

[0082]

[0083] S300. Test Result Evaluation:

[0084] The statistical results obtained during the performance testing phase are quantitatively analyzed, and quantitative evaluation indicators are calculated, including at least precision, recall, F1 score and / or average precision. The performance of the infrared image target detection algorithm in three dimensions—comprehensiveness, robustness and generalization—is comprehensively evaluated, and the output results of the black-box test framework for the infrared image target detection algorithm are obtained.

[0085] Specifically, the main factors reflecting the performance of target detection algorithms include detection accuracy, detection quantity, and detection speed. In the black-box testing framework for infrared image target detection algorithms, metrics such as precision, recall, average precision, and F1 score are used to evaluate the algorithm's performance in terms of detection accuracy and detection quantity. Precision represents the proportion of truly positive samples among all detected positive samples. In the testing framework, this is represented by inputting test cases into the device's target detection algorithm and generating the number of results that truly contain the target. The precision calculation formula is Pr = (TP / (TP+FP)) × 100%, where TP is the number of correctly detected positive samples and FP is the number of incorrectly detected positive samples. Average precision represents the area formed by the precision corresponding to different recall rates, consistent with the area covered by the PR curve. This metric is a comprehensive evaluation of the algorithm's overall accuracy and reliability. The average precision calculation formula is... Recall represents the number of positive targets detected by the detection algorithm out of all targets included in the test cases. It is calculated as Recall = (TP / (TP+FN)) × 100%, where FN is the number of false negatives. The black-box testing framework for infrared image target detection algorithms uses this metric to evaluate the algorithms. Furthermore, the F1 score is the harmonic mean of precision and recall, comprehensively reflecting the detection performance of the algorithm.

[0086] As a preferred approach, the generalization evaluation involves cross-domain testing using heterogeneous infrared image datasets to verify the detection performance of gray-scale integral-based target detection algorithms in color images. The comprehensive performance judgment generates a performance evaluation report based on quantitative indicators including at least precision, recall, and F1 score. By comparing the performance indicator trends under different test conditions, the performance bottlenecks and limitations of the algorithm are identified, providing clear technical directions for algorithm performance improvement. The test framework outputs confusion matrices, PR curves, ROC curves, and quantitative evaluation indicator statistics tables for various test cases, forming a complete black-box testing and evaluation system.

[0087] Example 2: Application Case

[0088] In Example 1 above, the working principle and implementation process of the black-box testing method for infrared image target detection algorithms were described in detail, including three key steps: test case generation, algorithm performance testing, and test result evaluation. To further illustrate the practical application effect of this invention, Example 2, using a CTIR infrared dataset, will provide a detailed description of the application example of this black-box testing method to verify its feasibility and effectiveness in real-world scenarios.

[0089] S100. Test case generation.

[0090] The test case design uses the CTIR infrared dataset as the original dataset, which contains 9551 data points. The dataset includes five categories: Car, Pedestrian, Cyclist, Bus, and Truck. The number of data points for each category is shown in Table 7.

[0091]

[0092] Table 7 Attributes of the Original Dataset

[0093] The metamorphic testing method was used to generate test cases based on the three aspects of comprehensiveness, robustness, and generalization required to be tested in the infrared image target detection algorithm, according to the corresponding metamorphic relationships. The test cases generated by the metamorphic testing are set as shown in Table 8.

[0094] Table 8 Experimental Setup for Generating Test Cases in Metamorphosis Testing

[0095]

[0096] The experiment generated test cases according to the principles of comprehensiveness, robustness, and generalization. First, comprehensive test cases were generated based on the target position transformation relationship. Considering the difficulty in changing the positions of existing targets in the CTIR dataset, and the challenge of directly altering their positions, a binary image was created when generating the dataset using the target position transformation relationship. This image used a black background (pixel value 0) and white target points (pixel value 255), and a sliding window method was used to generate test cases with determinable positions. Initially, a uniformly distributed initial dataset of the tested targets was generated, where targets were generated by sliding from the upper left corner of the background to the lower right corner, totaling 8281 images. The distribution of target coordinates in the statistical dataset was then analyzed. Figure 4 .like Figure 4 As shown, the initial dataset contains all possible positions of the target in the image, which can verify the comprehensiveness of the target detection algorithm to a certain extent. Meanwhile, detecting targets at image edges is a challenge for target detection algorithms; therefore, edge processing is performed on the initial dataset to generate an edge target dataset, and the distribution of target coordinates in the statistical dataset is analyzed. Figure 5 The distribution of edge datasets Figure 5It can be seen that the distribution of the target's x and y coordinates is most frequent at the initial and end positions in the image, indicating that the initial dataset after edge-mapping only retains the data of the target at the image edges.

[0097] To test the detection performance of infrared image target detection algorithms under different numbers and categories of targets, the CTIR dataset was used as the original dataset. The dataset was classified according to the number and category of targets. The original dataset was divided into three categories according to the number of targets, as shown in Table 9.

[0098] Table 9. Classification by Target Quantity

[0099]

[0100] The test cases containing 1-4 targets are considered a small number of targets, totaling 1970 data points; the test cases containing 5-8 targets are considered a medium number of targets, totaling 4950 data points; and the test cases containing 9-21 targets are considered a large number of targets, totaling 3071 data points. These three types of data are input into the target detection algorithm for testing to verify its comprehensiveness. Statistical analysis of the three datasets divided by the number of targets shows the target distribution in the data. Figure 6 The data distribution map shows that the targets in the three types of datasets are mainly distributed in the center of the image and gradually spread to both sides of the image. Furthermore, the density of distribution points in the map reveals that the number of targets is the least in the small-target dataset, and the number of targets increases sequentially in the large-target dataset and the medium-target dataset.

[0101] To verify the robustness of the infrared image target detection algorithm, this embodiment generates test cases with different perturbation characteristics by using the brightness and noise degradation relationship, and systematically tests the stability and anti-interference ability of the detection algorithm under complex environmental conditions.

[0102] Infrared sensor imaging produces binary images. Adjusting the brightness of a binary image essentially means changing the black-and-white thresholds in the image, making previously black areas even darker (pixel values ​​closer to 0), and previously white areas even whiter (pixel values ​​closer to 255). Therefore, the experiment uses the `cv2.addWeighted()` function to adjust the brightness, setting all white pixels in the current binary image to a new, higher value, while keeping the black pixels unchanged. Different test cases are generated by adjusting the brightness using different alpha parameter values ​​(0.25, 0.50, 0.70), such as... Figure 7As shown. Gradually increase the parameter value of the brightness alpha parameter. The target features with higher brightness in the binary image will become more obvious. It may be that there is no significant change in the data after the brightness change when observed with the naked eye. Therefore, use a histogram to count the pixel value distribution in the image, and the statistical results are as Figures 8-11 shown. It can be seen from the statistical results that as the alpha parameter gradually increases, the number of corresponding white pixel points in the generated test cases gradually increases, making the target brightness in the test cases increase. By changing the brightness of the image, infrared images under different lighting conditions can be simulated, so as to evaluate the robustness of the algorithm to lighting changes.

[0103] Noise is an important factor affecting image quality. Noise may cause the image to become blurred, making details difficult to identify, reducing the overall visual effect and aesthetic feeling. At the same time, high-intensity noise may cover key features in the image, making some objects, edges or textures invisible, resulting in the loss of useful information. During the target detection process, noise will interfere with the extraction and matching of features by the algorithm, reducing the accuracy of detection and recognition. To test the anti-interference ability of the infrared image target detection algorithm, in this embodiment, different signal-to-noise ratios of interference are added to the original data set and detected, and the robustness of the algorithm in a noisy environment is verified through the results. Set three PSNR parameter ranges according to the noise intensity: 25 < PSNR < 30 corresponds to mild noise interference, 15 < PSNR < 20 corresponds to moderate noise interference, 5 < PSNR < 10 corresponds to severe noise interference. Different test cases generated by adding salt-and-pepper noise to increase the signal-to-noise ratio are as Figure 12 shown.

[0104] It can be seen from the generated test cases that the larger the signal-to-noise ratio of the test case, the more similar it is to the original image and the fewer the noise points, while the test cases with a smaller signal-to-noise ratio have more noise points, which more severely tests the detection performance of the target detection algorithm. The pixel value distribution of the test cases under different signal-to-noise ratios is Figures 13-16 shown.

[0105] Salt-and-pepper noise adds random white and black noise points to the original test cases. It can be seen from the pixel value distribution diagram that as the PSNR decreases, the number of pixels corresponding to pixel values of 0 and 255 in the image gradually exceeds the number of target pixels in the original image, increasing the detection difficulty of the target detection algorithm. By adding salt-and-pepper noise of different intensities, infrared images under different sensor noise environments can be simulated, so as to evaluate the robustness of the target detection algorithm to noise interference.

[0106] S200. Performance test.

[0107] Use the test cases to test and verify the comprehensiveness, robustness and generalization of the infrared image target detection algorithm respectively, and comprehensively analyze the detection ability and adaptability of the algorithm through multi-dimensional performance evaluation.

[0108] For the target position test of the comprehensiveness of the algorithm, a dataset with custom target positions is used. The target generation rule in the dataset is to sequentially generate white rectangular regions according to a fixed sliding window and step size, with a black background, ensuring that all possible positions of the image are covered. The test cases in the dataset are used to test the comprehensiveness of the position detection of the algorithm, and the test results are Figure 17 Then, the test cases in the custom dataset where the target only appears at the edge are screened and input into the infrared image target detection algorithm to test the detection effect of the algorithm on the target at the edge, verify the performance of the algorithm in the edge target detection scenario, and the test results are Figure 18 .

[0109] It can be seen from the test result confusion matrix that the algorithm detected 1798 positive samples and 132 misdetected samples, and there was no missed detection. Therefore, the recall rate of the algorithm in the PR curve and the PRC curve is 100%, and it is drawn as a straight line. The test results show that the infrared image target detection algorithm has high detection accuracy and recall rate for the targets in the custom dataset, which can verify the comprehensiveness of the algorithm in one aspect. However, considering that the custom dataset is only composed of a black background and white targets, the content of the test cases is relatively simple and single, while the binary images generated by devices in the real environment have a complex background, and the target positions, quantities, and shapes are uncertain, with many random factors. Therefore, the CTIR gray dataset is used to further verify the comprehensiveness of the algorithm through test cases with different numbers and shapes of targets in the dataset.

[0110] The CTIR dataset is classified according to different target numbers in the test cases into a small target dataset (1 - 4 targets), a medium target dataset (5 - 8 targets), and a large target dataset (9 - 21 targets). The infrared image target detection algorithm is used to test the dataset respectively to verify the detection performance of the algorithm in different target density scenarios. The test results of using the algorithm for the small target dataset, the medium target dataset, and the large target dataset are Figure 19 , Figure 20 , Figure 21 . By comparing and analyzing the confusion matrix, PR curve, and ROC curve at different target number levels, the detection stability and accuracy of the algorithm in complex multi - target scenarios are evaluated.

[0111] To verify the robustness of the algorithm, test cases with added noise and changed brightness are input into the algorithm to obtain the corresponding test results. Different levels of salt - and - pepper noise are added to the CTIR dataset, so that the peak signal - to - noise ratios of the test cases are maintained at three levels: 25 < PSNR < 30 (mild noise), 15 < PSNR < 20 (moderate noise), and 5 < PSNR < 10 (severe noise). The test results of using the target detection algorithm are Figure 22 , Figure 23 , Figure 24By analyzing the impact of noise intensity on the algorithm's detection accuracy, recall, and other indicators, the algorithm's ability to resist noise interference is evaluated.

[0112] To verify the algorithm's adaptability to changes in lighting conditions, the brightness of the CTIR dataset was then increased to different levels: 0.25alpha (slight brightness enhancement), 0.5alpha (moderate brightness enhancement), and 0.7alpha (severe brightness enhancement). The results of testing with the object detection algorithm were as follows: Figure 25 , Figure 26 , Figure 27 By comparing the detection performance differences between the original image and the image after brightness adjustment, the detection stability and robustness of the algorithm under different illumination intensities are analyzed.

[0113] S300. Test Evaluation.

[0114] Based on the results generated by the algorithm using various test cases, the following evaluation indicators are calculated. Based on this indicator table, the algorithm performance can be objectively evaluated and the direction for further improvement can be determined.

[0115] Table 10 Performance Indicators of Target Detection Algorithm

[0116]

[0117] The objectives of this invention have been fully and effectively achieved through the above embodiments. Those skilled in the art will understand that this invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments described above. Although the invention has been described with reference to what is currently considered the most practical and preferred embodiments, it should be understood that the invention is not limited to the disclosed embodiments, and any modifications that do not depart from the functional and structural principles of the invention will be included within the scope of the claims.

Claims

1. A black-box testing method for infrared image target detection algorithms, characterized in that, It should include at least the following implementation steps: S100. Based on the evaluation requirements of the test algorithm in terms of comprehensiveness, robustness and generalization, the conditional coverage method is used to construct test cases of the target in different regions and at different scales of infrared images. The metamorphic test method is used to construct test cases under different perturbation conditions according to the set metamorphic relationship. The heterogeneous infrared image dataset is introduced through data collection to expand the test range. S200. Input the test cases into the target detection algorithm of the infrared image under test, obtain the detection results under different test scenarios by calculating the IoU value between the predicted box and the real labeled box, and count the number of TP, FP, TN and FN in the test results based on the preset IoU threshold. Generate a confusion matrix based on the statistical results, and draw PR curve and ROC curve to reflect the response behavior of the algorithm under different thresholds. S300. Quantitatively analyze the statistical results obtained during the performance testing phase, calculate quantitative evaluation indicators, including at least precision, recall, F1 score, and average accuracy, and comprehensively evaluate the performance of the infrared image target detection algorithm in three dimensions: comprehensiveness, robustness, and generalization.

2. The black-box testing method for infrared image target detection algorithms according to claim 1, characterized in that, In step S100, the conditional coverage method generates target regions within the image area using a sliding window approach. The size and step size of the sliding window are adaptively set based on the height and width of the input infrared image. The window size includes at least a small target window size and a large target window size. The sliding step size is calculated based on the greatest common divisor of the corresponding window sizes. The window slides across the entire infrared image to generate small target datasets and large target datasets that cover the entire image area. Simultaneously, to verify the algorithm's performance in detecting the field of view covering the range of the infrared binary image and detecting edge targets in the binary image, the conditional coverage method is used to generate a binary image dataset that meets the edge conditions.

3. The black-box testing method for infrared image target detection algorithms according to claim 2, characterized in that, In step S100, when generating the edge conditional binary image dataset, the conditional coverage method constructs a binary image with black as the background and white as the target point. It generates targets for the image edge region by setting a window index. It generates a binary image dataset with small targets at the image edge by using a sliding window with a small target window size and a fixed window index, and generates a binary image dataset with large targets at the image edge by using a sliding window with a large target window size and a fixed window index.

4. The black-box testing method for infrared image target detection algorithms according to claim 1, characterized in that, In step S100, the metamorphosis test method constructs test cases with input perturbation features based on a preset metamorphosis relationship to indirectly infer the output stability and correctness of the target detection algorithm. The metamorphosis relationship reflects the input-output mapping relationship between the original image and the mutated image. Its types include at least brightness transformation, scale scaling, random noise addition and target position perturbation. Test cases under different perturbation conditions are constructed by transforming the overall brightness of the original image, the target scale ratio, introducing salt-and-pepper noise of different intensities, and translating the target position based on a sliding window.

5. The black-box testing method for infrared image target detection algorithms according to claim 1, characterized in that, In step S200, the calculation of the IoU value includes the following sub-steps: S210. Coordinate Acquisition and Initialization: Input the test case into the target detection algorithm to obtain the position coordinates p = [xp1, yp1, xp2, yp2] of the predicted bounding box and the position coordinates q = [xg1, yg1, xg2, yg2] of the corresponding ground truth bounding box in the test case. Here, (xp1, yp1) and (xg1, yg1) represent the upper left corner coordinates of the predicted bounding box and the ground truth bounding box, respectively, and (xp2, yp2) and (xg2, yg2) represent the lower right corner coordinates of the predicted bounding box and the ground truth bounding box, respectively. S211. Intersection Region Calculation: By comparing the coordinates of the predicted bounding box and the ground truth bounding box, the intersection region between the predicted and ground truth bounding boxes is determined. The coordinates of the upper left corner of the intersection region are the maximum value of the upper left corner coordinates of the predicted and ground truth bounding boxes, a = (ax, ay) = max{(xp1, yp1), (xg1, yg1)}. The coordinates of the lower right corner of the intersection region are the minimum value of the lower right corner coordinates of the predicted and ground truth bounding boxes, b = (bx, by) = min{(xp2, yp2), (xg2, yg2)}. The intersection area S is calculated according to the rectangular definition of intersection. i = (bx-ax)×(by-ay), if there is no intersection, then S i =0; S212. Union Region Calculation: Determine the union region of the predicted bounding box and the ground truth bounding box. Calculate the area of ​​the predicted bounding box A = (xp2 - xp1) × (yp2 - yp1) and the area of ​​the ground truth bounding box B = (xg2 - xg1) × (yg2 - yg1). Subtract the intersection area from the sum of the two areas to obtain the union region area S. u =A+BS i ; S213. Calculate the IoU value: Calculate the area S of the intersection region. i Divide by the area S of the union region u The IoU value between the current predicted bounding box and the ground truth bounding box is obtained. The IoU value ranges from (0,1). The closer the value is to 1, the higher the degree of matching. This value serves as the basis for subsequent confusion matrix statistics and performance evaluation.

6. The black-box testing method for infrared image target detection algorithms according to claim 1, characterized in that, In step S200, the IoU threshold is adjusted step by step within the range of [0.3, 0.7], and is gradually decreased or increased at set intervals for multi-threshold detection performance evaluation. At each threshold level, IoU matching judgment, confusion matrix generation and subsequent index calculation are performed again to form a performance profile analysis under multiple thresholds, and to quantify the change in detection capability of the detection algorithm under different tolerance conditions.

7. The black-box testing method for infrared image target detection algorithms according to claim 1, 5, or 6, characterized in that, In step S200, the generation rules for the confusion matrix are as follows: TP represents the number of positive samples correctly detected in the ground truth bounding box, requiring the IoU value of the predicted bounding box output by the detection algorithm to be greater than a preset threshold and the type of the predicted bounding box to match the type of the ground truth bounding box; FP represents the number of negative samples incorrectly detected by the target detection algorithm, including two cases: the IoU value between the predicted bounding box and the ground truth bounding box is less than the threshold and the predicted bounding box does not match the category of the ground truth bounding box; FN represents the number of positive samples detected as negative samples; TN represents the number of negative samples correctly detected as negative samples by the detection algorithm; the initial IoU threshold is set to 0.

5.

8. The black-box testing method for infrared image target detection algorithms according to claim 7, characterized in that, In step S200, when plotting the PR curve and ROC curve, a threshold iteration mechanism is used to perform multiple rounds of calculation on the detection results. In each round, a certain threshold step size is gradually decreased or increased to regenerate the confusion matrix. Precision p = TP / (TP+FP) and recall r = TP / (TP+FN), as well as true positive rate tpr = TP / (TP+FN) and false positive rate fpr = FP / (FP+TN), are iteratively calculated. The PR curve is plotted with recall on the horizontal axis and precision on the vertical axis, and the ROC curve is plotted with false positive rate on the horizontal axis and true positive rate on the vertical axis.

9. The black-box testing method for infrared image target detection algorithms according to claim 7, characterized in that, In step S300, the calculation formulas for the quantitative evaluation metrics include: Precision Pr = (TP / (TP+FP)) × 100%, representing the proportion of truly positive examples when the detection algorithm detects them as positive examples; Recall = (TP / (TP+FN)) × 100%, representing the number of truly positive samples detected by the detection algorithm among all targets included in the test cases; Average Precision The F1 value represents the area formed by the precision corresponding to the detection algorithm at different recall rates, and is used to comprehensively evaluate the accuracy and reliability of the detection algorithm; the F1 value is the harmonic mean of precision and recall.

10. The black-box testing method for infrared image target detection algorithms according to claim 1, characterized in that, In step S300, the detection algorithm's ability to detect targets at different locations is comprehensively evaluated using a custom target location dataset generated through conditional coverage. The detection algorithm's performance in detecting edge targets in images is tested based on an edge target dataset. The detection coverage ability of the detection algorithm in different target density scenarios is verified using three datasets divided according to the number of targets: a small number of targets, a medium number of targets, and a large number of targets. The robustness evaluation is performed using test cases generated based on metamorphosis testing to verify the stability and anti-interference ability of the detection algorithm under complex environmental conditions.

11. The black-box testing method for infrared image target detection algorithms according to claim 1 or 10, characterized in that, In step S300, the generalization evaluation is carried out by introducing a heterogeneous infrared image dataset for cross-domain testing to verify the detection performance of the gray-scale integral-based target detection algorithm in color images; the comprehensive performance judgment generates a performance evaluation report based on quantitative indicators including at least precision, recall and F1 score, and identifies the performance bottlenecks and limitations of the algorithm by comparing the changing trends of performance indicators under different test conditions; the test framework output results include confusion matrices, PR curves, ROC curves and quantitative evaluation indicator statistics tables for various test cases.

12. A black-box testing system for infrared image target detection algorithms, characterized in that, The black-box testing system includes at least: The test case generation module is used to generate multi-dimensional test cases based on the evaluation requirements of the infrared image target detection algorithm in terms of comprehensiveness, robustness, and generalization. This includes: a conditional coverage submodule, which generates test images with preset positional distribution characteristics of targets located in different regions of the image by setting sliding windows of different sizes and step sizes to cover various spatial layout scenarios; a metamorphosis test submodule, which performs mutation processing on the original image based on a set metamorphosis relationship to form test cases under different input perturbation conditions; and a data collection submodule, which introduces heterogeneous infrared image datasets independent of the training data distribution to expand the sample domain and evaluate the model's generalization performance. The algorithm performance testing module is used to input the generated test cases into the infrared image target detection algorithm under test, and to collect and statistically analyze the output results generated by the algorithm. This includes: an IoU calculation submodule, used to calculate the IoU value based on the coordinate information of the predicted bounding box and the ground truth bounding box; a confusion matrix construction submodule, used to count the number of TP, FP, FN, and TN based on the set IoU threshold and construct the corresponding confusion matrix; and a performance curve plotting submodule, used to iteratively plot the PR curve and ROC curve based on the confusion matrix under different thresholds, showing the trend of algorithm performance changes under different fault tolerance levels. The results evaluation module is used to quantitatively analyze the statistical results output by the performance testing module and calculate evaluation metrics such as precision, recall, F1 score and / or average precision. The multidimensional evaluation module is used to evaluate and comprehensively assess the algorithm performance in three aspects: comprehensiveness, robustness, and generalization, and output the evaluation results to assist in algorithm optimization and deployment decisions.

Images