Multi-dimensional performance evaluation method based on visual inspection model

Abstract

The invention discloses a multi-dimensional performance evaluation method based on a visual detection model. The method comprises the following steps: firstly, constructing an original remote sensing evaluation sample set, a corresponding adversarial sample set, a noise sample set and a disturbance sample set; then performing model interpretability evaluation based on interpretation consistency, interpretation causality and interpretation sufficiency; carrying out model robustness evaluation from three aspects of adversarial robustness, noise robustness and interference robustness; and finally, safety evaluation of the visual detection model is completed. According to the invention, the problems of non-uniform evaluation standards, incomplete evaluation dimensions and poor expandability in the prior art are solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of model evaluation, and particularly relates to a multi-dimensional performance evaluation method based on a visual detection model. BACKGROUND

[0002] Model evaluation is an important part of the model development process, aiming to evaluate the performance of the trained model in multiple aspects. When evaluating the performance of the model, the explainability, robustness and other comprehensive performance of the model are mainly considered, then a corresponding evaluation scheme and index are formulated for each performance, and a solution to improve the performance is provided according to the index.

[0003] The explainability evaluation index is a core index for measuring the transparency of the internal decision-making process and the clarity of the decision-making basis of the model, and is an important index for humans to understand the internal operation mechanism of the model. As the internal decision-making logic of the model becomes more and more complex, the problem of opacity gradually becomes the main obstacle to the wide application of intelligent models, so it is an urgent need for the development of artificial intelligence technology to establish a systematic explainability evaluation system. In the model explainability technology, local explainable model and Shapley value are two mainstream methods. The local explainable model attempts to simulate the decision-making process of the model by constructing a local proxy model near the prediction point, while the Shapley value calculates the contribution of each feature in the model to the prediction result and the model decision based on game theory, which has a high theoretical basis.

[0004] Robustness is a core element for measuring whether the model can maintain stable output and reliable performance under uncertain conditions such as input disturbance, abnormal environmental change, distribution drift, etc. It not only requires the model to perform well under normal input, but also verifies whether the model can adapt well and maintain the expected performance under abnormal, noise, attack or offset input. In the model robustness evaluation system, adversarial attack, perturbation test and sensitivity analysis are several commonly used methods. Adversarial attack aims to construct corresponding adversarial samples to interfere with model decision-making through various attack methods such as black box and white box, in order to test the performance decline of the model on these samples; perturbation test adds different degrees of perturbation to the input data to observe the output decay of the model; sensitivity analysis is used to analyze the response strength of the model to input changes, and then judge its stability and robustness level.

[0005] Although the existing model performance evaluation technology has made significant progress, there are still limitations such as non-uniform evaluation standards, non-comprehensive evaluation dimensions, and poor scalability. In addition, unlike traditional model evaluation tasks, visual detection models have both classification and regression capabilities due to their unique structure and function, making it difficult to transfer traditional explainability and robustness evaluation methods. Therefore, it is urgent to build a systematic, scalable, and sound performance evaluation framework for visual detection models to fully quantify the performance of visual detection models under different conditions and ensure the reliability and security of the models in practical applications. SUMMARY

[0006] The purpose of the present application is to provide a multi-dimensional performance evaluation method based on a visual detection model, which solves the problems of non-uniform evaluation standards, non-comprehensive evaluation dimensions, and poor scalability in the prior art.

[0007] The technical solution adopted by the present application is a multi-dimensional performance evaluation method based on a visual detection model, characterized by implementing the following steps: Step 1, construction of the original remote sensing evaluation sample set and the corresponding adversarial sample set, noise sample set, and perturbation sample set; Step 2, model explainability evaluation based on explanation consistency, explanation causality, and explanation sufficiency; Step 3, model robustness evaluation based on adversarial robustness, noise robustness, and interference robustness; Step 4, completion of the safety evaluation of the visual detection model.

[0008] The present application is also characterized by, Step 1 is implemented according to the following steps: A visual detection model yolov13 and a substitute model are constructed, and DOTA data set and FAIR1M data set for the evaluation of the visual detection model are constructed, and a total of pictures are selected from the two data sets as the original remote sensing evaluation sample set, and a projection gradient descent method is used to randomly sample a starting point from around each picture to generate an adversarial sample set; at the same time, Gaussian noise with levels of , and is added to each picture in the original evaluation data set, where , to generate three different noise sample sets; in addition, the clipping ratio and the occlusion ratio are set, and random clipping and occlusion are performed on each picture in the original remote sensing evaluation sample set, and the clipped pictures and the occluded pictures are used as two different perturbation sample sets.

[0009] Step 2: Input the generated perturbation sample set into the visual detection model YOLOv13 to obtain the corresponding output results, and calculate the heterogeneity ratio of the output results. The explanatory consistency score was obtained. ; Divide each image in the original remote sensing evaluation sample set in step 1 into Using a feature attribution method based on class activation mapping, the causality of each region in the input image to the model output is calculated, thereby obtaining the corresponding... Zhang heatmap, and used the heatmap to derive relevant quality scores. and related ranking accuracy And calculate the explanatory causality score. Simultaneously, the target intersection-union ratio of the heatmap is calculated. and hit rate And calculate the sufficiency of explanation score. Complete the interpretability evaluation of the visual inspection model; Step 2 involves calculating the heterogeneity ratio of the model output to obtain the explanatory consistency score, which is implemented according to the following steps: Step 201: Calculate the total frequency of the variable values ​​in the model output results. Frequency of the modal array ; Step 202: Calculate the difference between the total frequency of the variable values ​​and the frequency of the mode in the model output results. Then calculate the ratio of this difference to the total frequency of the variable values, and use the obtained ratio as the heterogeneity ratio of the output results. ; Step 203: Subtract the heterogeneity ratio of the output from 1 to obtain the interpretability score. .

[0010] Step 2 uses heatmaps to derive the relevance quality score and relevance ranking accuracy, and calculates the explanatory causality score, specifically including the following steps: Step 204: Using a counting function And give the target area ,statistics Activation value of heat map within the target region Total activation value It calculates the proportion of activation values ​​in the target area of ​​each heatmap to the total number of activation values, and uses this proportion as the relevant quality score. ; Step 205: Sort all pixels in each heatmap by activation value and select the top 10% of high-scoring pixels. Calculate the proportion that falls within the target area, and use this proportion as the accuracy of the relevant ranking. ; Step 206, average the calculated correlation quality and correlation ranking accuracy as the explanatory causality score .

[0011] Target intersection over union of the heat map calculated in step 2 and point hit rate , and calculate the explanatory sufficiency score , specifically including the following steps: Step 207, extract the high activation region in each heat map , and combine the given target region , calculate the ratio of the intersection and union of the two as the target intersection over union of each heat map ; Step 208, get the highest activation point in each heat map , determine whether the highest activation point falls within the target region, and take the result as the point hit rate ; Step 209, take the average of the target intersection over union and the point hit rate as the explanatory sufficiency score .

[0012] In step 3, the original remote sensing evaluation sample set in step 1 and the generated adversarial sample set, noise sample set and perturbation sample set are respectively input into the visual detection model yolov13 for target detection, and the detection accuracy of the original remote sensing evaluation sample set, adversarial sample set, noise sample set and perturbation sample set is obtained in turn to calculate the adversarial robustness score , noise robustness score and perturbation robustness score ; complete the robustness evaluation of the visual detection model.

[0013] Step 3 is implemented according to the following steps: Step 301, calculate the ratio of the detection accuracy of the generated adversarial sample set and the original remote sensing evaluation sample set in step 1, and take this ratio value as the adversarial robustness score ; Step 302, calculate the relative change rate of the detection accuracy of each noise sample set and the original remote sensing evaluation sample set in step 1 respectively, and take the average value as the noise robustness score ; Step 303, obtain the detection accuracy of the two perturbation sample sets and the original remote sensing evaluation sample set in step 1 in turn, take the average of the two as the average detection accuracy of the perturbation sample set, calculate the ratio of the average detection accuracy of the perturbation sample set and the detection accuracy of the original remote sensing evaluation sample set, and use this ratio value as the perturbation robustness score .

[0014] Step 4 is specifically implemented according to the following steps: Using the substitute model in step 1, the original remote sensing evaluation sample set is respectively input into the visual detection model yolov13 and the substitute model, the average cosine similarity of the output results of the visual detection model yolov13 and the substitute model is calculated, and the result is taken as the extractability risk score .

[0015] The beneficial effects of the present application are: 1. Diversification, high integration: the present application integrates the performance types of multiple dimensions including but not limited to explainability and robustness in the visual detection model, as well as various evaluation indicators under each performance dimension, effectively unifying the evaluation methods and standards under different remote sensing scenes and different performance evaluation tasks, thereby alleviating the problem that the existing evaluation scheme focuses on a single performance while ignoring the overall reliability, and further enhancing the effectiveness and rationality of the evaluation. 2. High reliability and high efficiency: the present application proposes a small number of reliable evaluation indicators for each type of performance, and since each indicator is realized through simple steps and clear calculation formulas, the complexity and resource cost of the evaluation process are significantly reduced while ensuring the accuracy of the evaluation, thereby improving the evaluation efficiency of the visual detection model. 3. Practical and scalable: the present application is applicable to various remote sensing data sets, covering multiple data sources such as high-resolution satellites, Google Earth platforms, and Chinese resource satellite data, and includes multiple scene distributions such as Asia, America, and Europe, including multiple target types such as airplanes, ships, bridges, vehicles, and helicopters, which fully demonstrates the practicality and scalability of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is the overall flowchart of the multi-dimensional performance evaluation method of the visual detection model based on the present application. DETAILED DESCRIPTION

[0017] The present application will be described in detail below in conjunction with the drawings and specific embodiments.

[0018] Embodiment 1 The multi-dimensional performance evaluation method of the visual detection model based on the present application has a flowchart as shown in Figure 1 The specific implementation is as follows: Step 1, construction of the original remote sensing evaluation sample set and the corresponding adversarial sample set, noise sample set and perturbation sample set; Step 1 is specifically implemented according to the following steps: A visual detection model yolov13 and a substitute model are constructed, DOTA data set and FAIR1M data set for evaluation of the visual detection model are constructed, and The pictures are taken as original remote sensing evaluation sample sets, and a projection gradient descent method is used to randomly sample a starting point from the surroundings in each picture to generate an adversarial sample set; at the same time, Gaussian noise with levels of 、 and is added to each picture in the original evaluation data set in turn, where , to generate three different noise sample sets; in addition, a clipping ratio and an occlusion ratio are set, and each picture in the original remote sensing evaluation sample set is randomly clipped and occluded, and the clipped pictures and the occluded pictures are taken as two different perturbation sample sets.

[0019] Step 2, model explainability evaluation based on explanation consistency, explanation causality and explanation sufficiency; Step 2 inputs the generated perturbation sample set into the visual detection model yolov13 to obtain the corresponding output result, calculates the heterogeneity ratio of the output result, and obtains the explanation consistency score ; each picture in the original remote sensing evaluation sample set in step 1 is divided into equal-sized regions, a feature attribution method based on class activation mapping is used to calculate the causality of each region of the input picture to the model output, thereby obtaining corresponding heat maps, and the related quality degree and the related ranking accuracy are obtained by using the heat maps, and the explanation causality score is calculated; at the same time, the target intersection over union and the point hit rate of the heat map are calculated, and the explanation sufficiency score is calculated; the explainability evaluation of the visual detection model is completed; In step 2, for calculating the heterogeneity ratio of the model output result, the explanation consistency score is obtained, which is implemented according to the following steps: Step 201, respectively, the total frequency and the frequency of the majority group of the variable value of the model output result are counted; Step 202, the difference between the total frequency and the frequency of the majority group of the variable value of the model output result is calculated, and the ratio of the difference to the total frequency of the variable value is calculated, and the obtained ratio is taken as the heterogeneity ratio of the output result; Step 203, subtracting the heterogeneity ratio of the output result from 1 obtains the explanation consistency score .

[0020] The correlation quality and correlation ranking accuracy are obtained by using the heat map in step 2, and the explanatory causality score is calculated, specifically including the following steps: Step 204, using the counting function And give the target area , statistics The activation value of each heat map in the target area And the total activation value , and calculate the proportion of the number of activation values of each heat map in the target area to the total number of activation values, and take this proportion value as the correlation quality ; Step 205, sort all pixels in each heat map by activation value, and take the top 10% high-score pixels , calculate the proportion falling within the target area, and take this proportion value as the correlation ranking accuracy ; Step 206, take the average of the calculated correlation quality and correlation ranking accuracy as the explanatory causality score .

[0021] Step 2, calculate the target intersection-over-union of the heat map And the point hit rate , and calculate the explanatory sufficiency score , specifically including the following steps: Step 207, extract the high activation area in each heat map , and combine the given target area , calculate the ratio of the intersection and union of the two, as the target intersection-over-union of each heat map ; Step 208, get the highest activation point in each heat map , judge whether the highest activation point falls within the target area, and take the judgment result as the point hit rate ; Step 209, take the average of the target intersection-over-union and the point hit rate as the explanatory sufficiency score .

[0022] Step 3, based on the model robustness evaluation from the aspects of adversarial robustness, noise robustness and interference robustness; In step 3, the original remote sensing evaluation sample set in step 1 and the generated adversarial sample set, noise sample set and perturbation sample set are respectively input into the visual detection model yolov13 for target detection, and the detection accuracy of the original remote sensing evaluation sample set, the adversarial sample set, the noise sample set and the perturbation sample set is obtained in turn, to calculate the adversarial robustness score , noise robustness score And the perturbation robustness score ; complete the robustness evaluation of the visual detection model.

[0023] Step 3 is specifically implemented according to the following steps: Step 301, calculate the ratio of the detection accuracy of the generated adversarial sample set and the original remote sensing evaluation sample set in step 1, and take this ratio value as the adversarial robustness score ; Step 302, calculate the relative change rate of the detection accuracy of each noise sample set and the original remote sensing evaluation sample set in step 1 respectively, and take the average value as the noise robustness score ; Step 303, the detection accuracy of the two perturbation sample sets and the original remote sensing evaluation sample set in step 1 is obtained in turn, and the average of the two is taken as the average detection accuracy of the perturbation sample set, and the ratio of the average detection accuracy of the perturbation sample set and the original remote sensing evaluation sample set is calculated. The ratio value is used as the perturbation robustness score .

[0024] Step 4, complete the security evaluation of the visual detection model.

[0025] Step 4 is specifically implemented according to the following steps: Using the substitute model in step 1, the original remote sensing evaluation sample set is input into the visual detection model yolov13 and the substitute model respectively, the average cosine similarity of the output results of the visual detection model yolov13 and the substitute model is calculated, and the result is taken as the extractability risk score .

[0026] Embodiment 2 The multi-dimensional performance evaluation method of the visual detection model based on the present application, the flow chart is as Figure 1 shown, specifically implemented according to the following steps: Step 1, construction of the original remote sensing evaluation sample set and the corresponding adversarial sample set, noise sample set and perturbation sample set; Step 2, model interpretability evaluation based on explanation consistency, explanation causality and explanation sufficiency; Step 3, model robustness evaluation based on adversarial robustness, noise robustness and interference robustness; Step 4, complete the security evaluation of the visual detection model.

[0027] Embodiment 3 The multi-dimensional performance evaluation method of the visual detection model based on the present application, the flow chart is as Figure 1 shown, specifically implemented according to the following steps: Step 1, construction of the original remote sensing evaluation sample set and the corresponding adversarial sample set, noise sample set and perturbation sample set; Step 2, model explainability evaluation based on explanation consistency, explanation causality and explanation sufficiency; Step 3, model robustness evaluation based on adversarial robustness, noise robustness and interference robustness; Step 4, complete the safety evaluation of the visual detection model.

[0028] Step 1 is implemented according to the following steps: The visual detection model yolov13 and an alternative model are constructed, the DOTA dataset and the FAIR1M dataset for evaluating the visual detection model are constructed, and a total of 1000 pictures are selected from the two datasets as the original remote sensing evaluation sample set, and an adversarial sample set is generated by randomly sampling a starting point from the surrounding of each picture using the projected gradient descent method; at the same time, Gaussian noise with levels of , , and is added to each picture in the original evaluation dataset in turn, wherein , three different noise sample sets are generated; in addition, the clipping ratio and the occlusion ratio are set, and random clipping and occlusion are performed on each picture in the original remote sensing evaluation sample set, and pictures after clipping and pictures after occlusion are taken as two different perturbation sample sets.

[0029] Embodiment 4 The multi-dimensional performance evaluation method of the visual detection model is implemented according to the following steps according to the flow chart shown in Figure 1 . Step 1, construction of the original remote sensing evaluation sample set and the corresponding adversarial sample set, noise sample set and perturbation sample set; Step 1 is implemented according to the following steps: The visual detection model yolov13 and an alternative model are constructed, the DOTA dataset and the FAIR1M dataset for evaluating the visual detection model are constructed, and a total of 1000 pictures are selected from the two datasets as the original remote sensing evaluation sample set, and an adversarial sample set is generated by randomly sampling a starting point from the surrounding of each picture using the projected gradient descent method; at the same time, Gaussian noise with levels of , , and is added to each picture in the original evaluation dataset in turn, wherein , three different noise sample sets are generated; in addition, the clipping ratio and the occlusion ratio , and the occluded images in the original remote sensing evaluation sample set as two different perturbation sample sets.

[0030] Step 2, based on the explanation consistency, the model explainability evaluation of explanation causality and explanation sufficiency; Step 2 inputs the generated perturbation sample set into the visual detection model yolov13 to obtain the corresponding output result, calculates the heterogeneity ratio of the output result , and obtains the explanation consistency score ; divide each image in the original remote sensing evaluation sample set in step 1 into equal size regions, calculate the causality of each region of the input image to the model output by using the feature attribution method based on class activation mapping, thereby obtaining the corresponding heat map, and obtaining the related quality degree and the related ranking accuracy by using the heat map, and calculating the explanation causality score ; at the same time, calculate the target intersection over union and the point hit rate of the heat map, and calculate the explanation sufficiency score ; complete the explainability evaluation of the visual detection model; In step 2, for calculating the heterogeneity ratio of the model output result, obtaining the explanation consistency score, the following steps are implemented: Step 201, respectively, count the total frequency and the frequency of the majority of the variable value of the model output result ; Step 202, calculate the difference between the total frequency and the frequency of the majority of the variable value of the model output result, then calculate the ratio of the difference to the total frequency of the variable value, and take the obtained ratio as the heterogeneity ratio of the output result ; Step 203, subtract 1 from the heterogeneity ratio of the output result to obtain the explanation consistency score .

[0031] In step 2, the related quality degree and the related ranking accuracy are obtained by using the heat map, and the explanation causality score is calculated, which includes the following steps: Step 204, use the counting function and give the target region , count the activation value and the total activation value of the heat map in the target region ​​and the proportion of the number of activation values in the target region is taken as the correlation quality degree ; Step 205: sort all pixels in each heat map according to activation values, and take the top 10% high-score pixels , calculate the proportion falling in the target region, and take the proportion value as the correlation ranking accuracy ; Step 206: take the average of the calculated correlation quality degree and correlation ranking accuracy as the explanatory causality score .

[0032] Step 2: calculate the target intersection-over-union of the heat map and the point hit rate , and calculate the explanatory sufficiency score , which specifically includes the following steps: Step 207: extract the high-activation region in each heat map , and combine the given target region , calculate the ratio of the intersection to the union as the target intersection-over-union of each heat map ; Step 208: obtain the highest activation point in each heat map , judge whether the highest activation point falls within the target region, and take the judgment result as the point hit rate ; Step 209: take the average of the target intersection-over-union and the point hit rate as the explanatory sufficiency score .

[0033] Step 3: based on the model robustness evaluation from the aspects of adversarial robustness, noise robustness and interference robustness; Step 4: complete the safety evaluation of the visual detection model.

[0034] Embodiment 5 The multi-dimensional performance evaluation method of the visual detection model provided by the application is shown in the flow chart Figure 1 , and is implemented according to the following steps: Step 1: construction of the original remote sensing evaluation sample set and the corresponding adversarial sample set, noise sample set and perturbation sample set; Step 1 is implemented according to the following steps: A visual detection model yolov13 and an alternative model are constructed, a DOTA data set and a FAIR1M data set for evaluating the visual detection model are constructed, and a total of The pictures are taken as original remote sensing evaluation sample sets, and a projection gradient descent method is used to randomly sample a starting point from the surroundings in each picture to generate an adversarial sample set; at the same time, Gaussian noise with levels of 、 and is added to each picture in the original evaluation data set in turn, wherein , three different noise sample sets corresponding to the three levels of noise are generated; in addition, a cropping ratio and an occlusion ratio are set, and each picture in the original remote sensing evaluation sample set is randomly cropped and occluded, and the cropped pictures and the occluded pictures are taken as two different perturbation sample sets.

[0035] Step 2, model explainability evaluation based on explanation consistency, explanation causality and explanation sufficiency; Step 2 inputs the generated perturbation sample set into the visual detection model yolov13 to obtain the corresponding output result, calculates the heterogeneity ratio of the output result, and obtains the explanation consistency score ; each picture in the original remote sensing evaluation sample set in step 1 is divided into equal-sized regions, a feature attribution method based on class activation mapping is used to calculate the causality of each region of the input picture to the model output, thereby obtaining corresponding heat maps, and the related quality degree and the related ranking accuracy are obtained by using the heat maps, and the explanation causality score is calculated; at the same time, the target intersection over union and the point hit rate of the heat map are calculated, and the explanation sufficiency score is calculated; the explainability evaluation of the visual detection model is completed; In step 2, for calculating the heterogeneity ratio of the model output result, the explanation consistency score is obtained, which is implemented according to the following steps: Step 201, the total frequency and the frequency of the majority of the variable value of the model output result are counted respectively; Step 202, the difference between the total frequency and the frequency of the majority of the variable value of the model output result is calculated, and the ratio of the difference to the total frequency of the variable value is calculated, and the obtained ratio is taken as the heterogeneity ratio of the output result; Step 203, the heterogeneity ratio of the output result is subtracted by 1 to obtain the explanation consistency score .

[0036] The correlation quality and correlation ranking accuracy are obtained by using the heat map in step 2, and the explanatory causality score is calculated, specifically including the following steps: Step 204, using the counting function And give the target area , statistics The activation value of each heat map in the target area And the total activation value , and calculate the proportion of the number of activation values of each heat map in the target area to the total number of activation values, and take this proportion value as the correlation quality ; Step 205, sort all pixels in each heat map by activation value, and take the top 10% high-score pixels , calculate the proportion falling within the target area, and take this proportion value as the correlation ranking accuracy ; Step 206, take the average of the calculated correlation quality and correlation ranking accuracy as the explanatory causality score .

[0037] Step 2, calculate the target intersection-over-union of the heat map And the point hit rate , and calculate the explanatory sufficiency score , specifically including the following steps: Step 207, extract the high activation area in each heat map , and combine the given target area , calculate the ratio of the intersection and union of the two, as the target intersection-over-union of each heat map ; Step 208, get the highest activation point in each heat map , judge whether the highest activation point falls within the target area, and take the judgment result as the point hit rate ; Step 209, take the average of the target intersection-over-union and the point hit rate as the explanatory sufficiency score .

[0038] Step 3, based on the model robustness evaluation from the aspects of adversarial robustness, noise robustness and interference robustness; In step 3, the original remote sensing evaluation sample set in step 1 and the generated adversarial sample set, noise sample set and perturbation sample set are respectively input into the visual detection model yolov13 for target detection, and the detection accuracy of the original remote sensing evaluation sample set, the adversarial sample set, the noise sample set and the perturbation sample set is obtained in turn, to calculate the adversarial robustness score , noise robustness score And the perturbation robustness score ; complete the robustness evaluation of the visual detection model.

[0039] Step 3 is specifically implemented according to the following steps: Step 301, calculate the ratio of the detection accuracy of the generated adversarial sample set and the original remote sensing evaluation sample set in step 1, and take this ratio value as the adversarial robustness score ; Step 302, calculate the relative change rate of the detection accuracy of each noise sample set and the original remote sensing evaluation sample set in step 1 respectively, and take the average value as the noise robustness score ; Step 303, in turn, the detection accuracy of the two perturbation sample sets and the original remote sensing evaluation sample set in step 1 is obtained, and the average of the two is taken as the average detection accuracy of the perturbation sample set. Calculate the ratio of the average detection accuracy of the perturbation sample set and the detection accuracy of the original remote sensing evaluation sample set, and use this ratio value as the perturbation robustness score .

[0040] Step 4, complete the security evaluation of the visual detection model.

[0041] Example 6 Referring Figure 1 , the application is based on the multi-dimensional performance evaluation method of the visual detection model, which includes the construction of the remote sensing evaluation data set and the corresponding generation of the adversarial sample, noise sample and perturbation sample, the model explainability evaluation based on explanation consistency, explanation causality and explanation sufficiency, and the model robustness evaluation based on three aspects of adversarial robustness, noise robustness and interference robustness. A series of processing procedures. Specifically, it includes the following steps: Step 1, construct the visual detection model yolov13 and an alternative model, and the DOTA data set and FAIR1M data set for the evaluation of the visual detection model, and select pictures from the two data sets as the original remote sensing evaluation sample set, and use the projected gradient descent method to randomly sample a starting point from around each picture to generate an adversarial sample set; At the same time, add pictures in the original evaluation data set Gaussian noise of different levels to generate the corresponding noise sample set; In addition, set the clipping ratio , randomly clip the pictures in the original remote sensing evaluation sample set, and take pictures after clipping as the perturbation sample set; Step 2, input the generated perturbation sample set into the visual detection model yolov13 to obtain the corresponding output result, calculate the heterogeneity ratio of the output result, and obtain the explanation consistency score ; meanwhile, divide each picture in the original remote sensing evaluation sample set into an equal number of regions, calculate the causality of each region of the input picture to the model output using a feature attribution method based on class activation mapping, thereby obtaining a corresponding heat map, and derive a relevant quality degree and a relevant ranking accuracy from the heat map, and calculate an explanation causality score ; meanwhile, calculate the target intersection over union and the point hit rate of the heat map, and calculate an explanation sufficiency score ; complete the explainability evaluation of the visual detection model; wherein, for the calculation of the heterogeneity ratio of the model output result, an explanation consistency score is derived, specifically including the following steps: Step 201, respectively, count the total frequency of the variable value of the model output result and the frequency of the majority group ; Step 202, calculate the ratio of the difference between the total frequency of the variable value and the frequency of the majority group to the total frequency of the variable value, and take it as the heterogeneity ratio of the output result , that is: ; Step 203, subtract 1 from the heterogeneity ratio of the output result to obtain the explanation consistency score , that is: .

[0042] wherein, for deriving the relevant quality degree and the relevant ranking accuracy from the heat map and calculating the explanation causality score, specifically including the following steps: Step 204, use the counting function and give the target region , count the activation values and the total activation values of the heat map in the target region, and calculate the proportion of the number of activation values in the target region in the total number of activation values for each heat map, and take this proportion value as the relevant quality degree , that is:

[0043] Step 205, sort all pixels in each heat map according to the activation value, take the top 10% high-score pixels , calculate the proportion falling within the target region, and take this proportion value as the relevant ranking accuracy , that is: ​​

[0044] Step 206, average the calculated correlation quality and correlation ranking accuracy as the score of explaining causality That is,

[0045] Wherein, for the target intersection over union and point hit rate of the heat map, the score of explaining sufficiency is calculated, which specifically includes the following steps: Step 207, extract the high activation region in each heat map And combine the given target region Calculate the ratio of the intersection and the union of the two as the target intersection over union of each heat map That is,

[0046] Step 208, get the highest activation point in each heat map Determine whether it falls within the target region, and take the determination result as the point hit rate That is,

[0047] Step 209, take the average value of the target intersection over union and the point hit rate as the score of explaining sufficiency That is,

[0048] Step 3, input the original sample in step 1 and the generated adversarial sample, noise sample and perturbation sample into the visual detection model yolov13 for target detection, and obtain the detection accuracy of the original sample, adversarial sample, noise sample and perturbation sample in turn, to calculate the adversarial robustness score , noise robustness score And perturbation robustness score ; complete the robustness evaluation of the visual detection model.

[0049] Wherein, the detection accuracy of the obtained original sample, adversarial sample, noise sample and perturbation sample is used to calculate the adversarial robustness score, noise robustness score and perturbation robustness score, which specifically includes the following steps: Step 301, calculate the ratio of the detection accuracy of the generated adversarial sample and the original sample, and take this ratio value as the adversarial robustness score ; Step 302, calculate the relative change rate of the detection accuracy of each noise sample and the original sample respectively, and take the average value as the noise robustness score ; Step 303, the detection accuracies of the cropped and occluded perturbation sample and the original sample are obtained in turn, and the average of the two is taken as the average detection accuracy of the perturbation sample, and the ratio of the original sample detection accuracy is calculated as the perturbation robustness score .

[0050] Step 4, using the substitute model in step 1, input the original sample into the visual detection model yolov13 and the substitute model respectively, according to the output results of the two models, calculate the model extractability risk score , complete the safety evaluation of the visual detection model.

[0051] Among them, the model extractability risk score is calculated, which specifically includes the following steps: Step 401, calculate the average cosine similarity of the output results of the visual detection model yolov13 and the substitute model, and take this result as the extractability risk score , that is:

[0052] In summary, the present application realizes the multi-dimensional performance evaluation of the visual detection model in various application scenarios such as military and civilian, including the construction of remote sensing evaluation data set and the corresponding generation of adversarial samples, noise samples and perturbation samples, model explainability evaluation based on explanation consistency, causality and sufficiency, and model robustness evaluation based on three angles of adversarial robustness, noise robustness and interference robustness.

[0053] The present application effectively integrates various data sources, various real scene distributions and various performance evaluation indexes in the general visual detection model, and has good unified model performance evaluation standard. The present application has the advantages of perfect evaluation system, various angles, unified degree and the like, greatly improves the efficiency and reliability of model evaluation, and plays a very key role in ensuring the practicability and credibility of the visual detection model.

Claims

1. A multi-dimensional performance evaluation method based on a visual inspection model, characterized in that, The specific steps are as follows: Step 1: Construction of the original remote sensing evaluation sample set and corresponding adversarial sample set, noise sample set and disturbance sample set; Step 2: Based on the consistency of explanation, evaluate the interpretability of the model in explaining causality and adequacy of explanation; Step 3: Evaluate the model's robustness based on three aspects: adversarial robustness, noise robustness, and interference robustness; Step 4: Complete the security assessment of the visual inspection model.

2. The multi-dimensional performance evaluation method based on a visual detection model according to claim 1, characterized in that, Step 1 is implemented in the following steps: We construct a visual detection model, YOLOv13, and an alternative model. We also build the DOTA and FAIR1M datasets for evaluating the visual detection model and select data from both datasets. The original remote sensing evaluation sample set was generated using 100 images. An adversarial example set was generated by randomly sampling a starting point from the surrounding area of ​​each image using the projective gradient descent method. Simultaneously, each image in the original evaluation dataset was sequentially assigned a rating level. , and Gaussian noise, where This generates three different noise sample sets; additionally, it sets the cropping ratio. and occlusion ratio Each image in the original remote sensing assessment sample set was randomly cropped and occluded, and the cropped images were then processed separately. Zhang's picture and the image after being covered The images serve as two different sets of perturbation samples.

3. The multi-dimensional performance evaluation method based on a visual detection model according to claim 2, characterized in that, Step 2 involves inputting the generated perturbation sample set into the visual detection model YOLOv13 to obtain the corresponding output results, and then calculating the heterogeneity ratio of the output results. The explanatory consistency score was obtained. ; Each image in the original remote sensing evaluation sample set in step 1 is divided into... Using a feature attribution method based on class activation mapping, the causality of each region in the input image to the model output is calculated, thereby obtaining the corresponding... Heatmaps were generated, and relevant quality scores were derived from the heatmaps. and related ranking accuracy And calculate the explanatory causality score. Simultaneously, the target intersection-union ratio of the heatmap is calculated. And point hit rate And calculate the sufficiency of explanation score. ; Complete the interpretability evaluation of the visual inspection model.

4. The multi-dimensional performance evaluation method based on a visual detection model according to claim 3, characterized in that, In step 2, the heterogeneity ratio of the calculated model output is used to derive the explanatory consistency score, which is implemented according to the following steps: Step 201: Calculate the total frequency of the variable values ​​in the model output results. Frequency of the modal array ; Step 202: Calculate the difference between the total frequency of the variable values ​​and the frequency of the mode in the model output results. Then calculate the ratio of this difference to the total frequency of the variable values, and use the obtained ratio as the heterogeneity ratio of the output results. ; Step 203: Subtract the heterogeneity ratio of the output from 1 to obtain the interpretability score. .

5. The multi-dimensional performance evaluation method based on a visual detection model according to claim 4, characterized in that, Step 2 uses heatmaps to derive the relevance quality score and relevance ranking accuracy, and calculates the explanatory causality score, specifically including the following steps: Step 204: Using a counting function And give the target area ,statistics Activation value of heat map within the target region Total activation value It calculates the proportion of activation values ​​in the target area of ​​each heatmap to the total number of activation values, and uses this proportion as the relevant quality score. ; Step 205: Sort all pixels in each heatmap by activation value and select the top 10% of high-scoring pixels. Calculate the proportion that falls within the target area, and use this proportion as the accuracy of the relevant ranking. ; Step 206: Average the calculated relevance quality score and relevance ranking accuracy to obtain the explanatory causality score. .

6. The multi-dimensional performance evaluation method based on a visual detection model according to claim 5, characterized in that, In step 2, the target crossover ratio of the heatmap is calculated. And point hit rate And calculate the sufficiency of explanation score. Specifically, it includes the following steps: Step 207: Extract the highly activated regions from each heatmap. and combined with the given target area Calculate the ratio of the intersection to the union of the two sets, and use it as the target intersection-union ratio for each heatmap. ; Step 208: Obtain the highest activation point in each heatmap. Determine whether the highest activation point falls within the target area, and use the result as the point hit rate. ; Step 209: Take the average of the target intersection-union ratio and the point hit rate as the sufficiency score. .

7. The multi-dimensional performance evaluation method based on a visual detection model according to claim 6, characterized in that, In step 3, the original remote sensing evaluation sample set from step 1, as well as the generated adversarial sample set, noise sample set, and perturbation sample set, are input into the visual detection model YOLOv13 for target detection. The detection accuracy of the original remote sensing evaluation sample set, adversarial sample set, noise sample set, and perturbation sample set is obtained sequentially to calculate the adversarial robustness score. Noise robustness score and perturbation robustness score ; Complete the robustness evaluation of the visual inspection model.

8. The multi-dimensional performance evaluation method based on a visual detection model according to claim 7, characterized in that, Step 3 is implemented in the following steps: Step 301: Calculate the ratio of the detection accuracy of the generated adversarial example set to that of the original remote sensing evaluation sample set in Step 1, and use this ratio as the adversarial robustness score. ; Step 302: Calculate the relative rate of change of detection accuracy of each noise sample set compared to the original remote sensing evaluation sample set in Step 1, and take the average value as the noise robustness score. ; Step 303: Calculate the detection accuracy of the two perturbation sample sets and the original remote sensing evaluation sample set from Step 1, and average them as the average detection accuracy of the perturbation sample set. Calculate the ratio of the average detection accuracy of the perturbation sample set to the detection accuracy of the original remote sensing evaluation sample set, and use this ratio as the perturbation robustness score. .

9. The multi-dimensional performance evaluation method based on a visual detection model according to claim 8, characterized in that, Step 4 is implemented in the following steps: Using the alternative model from step 1, the original remote sensing assessment sample set is input into the visual detection model YOLOv13 and the alternative model, respectively. The mean cosine similarity of the outputs of the visual detection model YOLOv13 and the alternative model is calculated, and this result is used as the extractability risk score. .

Images