Target detection model performance evaluation method and system, storage medium and equipment

Abstract

The invention belongs to the technical field of artificial intelligence, and provides a target detection model performance evaluation method and system, a storage medium and equipment in order to solve the technical problems that a traditional evaluation method cannot reflect the generalization ability of a target detection model and cannot specifically evaluate the feature expression ability of the model. The target detection model performance evaluation method comprises the steps of obtaining a feature map set and a target recognition result corresponding to each image based on a to-be-evaluated target detection model and an initial image data set; calculating to obtain a feature space analysis index of the to-be-evaluated target detection model; according to the method, the image enhancement data set under each set factor scene is generated, the sample content representation identification index of the to-be-evaluated target detection model is obtained, the performance evaluation result of the to-be-evaluated target detection model is obtained by combining the feature space analysis index on the initial image data set, and more accurate instructive suggestions are provided for users.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a target detection model performance evaluation method, system, storage medium and device. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In the life cycle of target detection tasks, the performance evaluation of target detection models is an essential link to ensure model quality, improve the practical value of models and promote technological innovation. The traditional target detection model performance evaluation method uses indicators such as accuracy, recall, F1 score and average precision for evaluation. However, the inventors found that the traditional target detection model performance evaluation indicators are closely related to the distribution of the data set, and cannot reflect the generalization ability of the target detection model, nor do they fully consider the impact of adversarial attacks and various environmental factors on the performance of the target detection model caused by slight changes in input data; in addition, the traditional target detection model performance evaluation method only regards the target detection model as a black box, lacks analysis of the model feature space, and cannot understand in detail the model's decision-making process for sample expression, nor can it specifically evaluate the model's feature expression ability.

[0004] In summary, the traditional target detection model performance evaluation method cannot reflect the generalization ability of the target detection model and cannot specifically evaluate the model's feature expression ability. As a result, it cannot provide users with target detection model selection suggestions for target detection needs in different scenarios, nor can it provide R&D personnel with suggestions for optimizing the target detection model. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a target detection model performance evaluation method, system, storage medium and device, which can realize the analysis and evaluation of the model feature space level, and evaluate the sample content representation and recognition ability of the target detection model on the basis of greatly improving the capacity and generalization of the sample set, fully reflecting the generalization ability of the target detection model in complex environments, and providing users with more accurate target detection model selection suggestions for target detection needs in different scenarios, and also providing R&D personnel with suggestions for optimizing the target detection model.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A first aspect of the present invention provides a method for evaluating the performance of a target detection model.

[0008] In one or more embodiments, a method for evaluating performance of a target detection model is provided, comprising:

[0009] Obtain an initial image data set, and use the target detection model to be evaluated to process each image in the initial image data set to obtain a feature map set and a target recognition result corresponding to each image;

[0010] According to the feature map set and the target recognition result, a feature activation map corresponding to each correct target in the target recognition result is calculated, and then the feature activation map is compared with the corresponding annotation true value of the initial image data set to obtain the feature space analysis index of the target detection model to be evaluated on the initial image data set;

[0011] Simulating at least two setting factors and combining the initial image data set to generate image enhancement data sets under each of the setting factor scenarios, and then combining the target detection model to be evaluated to obtain feature space analysis indicators of the target detection model to be evaluated on the image enhancement data sets under each of the setting factor scenarios and summarize them to obtain sample content characterization and recognition indicators of the target detection model to be evaluated;

[0012] The performance of the target detection model to be evaluated is evaluated using the sample content characterization recognition index of the target detection model to be evaluated and the feature space analysis index on the initial image data set.

[0013] As an implementation method, the calculation steps of the feature activation map are:

[0014] According to the comparison result of the target recognition result of each image in the initial image data set and its corresponding annotation true value, the correct target is screened out from the target recognition results of each image;

[0015] For any correct target of each image in the initial image data set, use coordinates and category probability back propagation to obtain the gradient of the correct target on the corresponding feature map set, perform global average pooling on the value of each channel in the gradient, and obtain a channel weight factor;

[0016] Based on the channel weight factor, the value of each channel of the correct target in the corresponding feature map set is weighted and accumulated to obtain a feature activation map corresponding to the correct target.

[0017] The advantage of the above technical solution is that, by utilizing each channel of the gradient of the feature map set and its corresponding weight factors, the feature activation map corresponding to the correct target is calculated by weighted summation, thereby realizing the target detection model to analyze in the feature space, and providing a more accurate data basis for subsequent true value comparison.

[0018] As an implementation method, the step of comparing the feature activation map with the corresponding annotation true value of the initial image data set is:

[0019] The feature activation map is scaled to the original size of the feature map in the corresponding feature map set and normalized, and then the value of the feature activation map outside the limit is set to zero using the set size of the corresponding detection target frame as the limit, and the feature activation map is binarized;

[0020] The minimum positive circumscribed rectangle of the maximum connected area whose binary value is 1 is screened out, and then the overlap between the minimum positive circumscribed rectangle and the true value frame corresponding to the current detection target is calculated and used as the feature space analysis score of the current detection target.

[0021] The advantage of the above technical solution is that by activating the feature Figure 2 The smallest regular circumscribed rectangle corresponding to the value of 1 is compared with the true value box corresponding to the current detection target, and the feature space analysis score of the current detection target is calculated, which realizes the analysis and evaluation of the model feature space level, improves the model's understanding of the decision-making process of sample expression and the feature expression ability of the specific evaluation model.

[0022] As an implementation method, all detection targets in all images in the initial image data set are traversed and their feature space analysis scores are calculated. For each detection type, the mean feature space analysis score of all targets of the corresponding type is calculated and used as the corresponding detection type score. All detection type scores are combined to obtain the feature space analysis index of the target detection model to be evaluated on the initial image data set.

[0023] The advantage of the above technical solution is that it realizes the analysis and evaluation of the model feature space level, improves the model's understanding of the decision-making process of sample expression and the feature expression ability of the specific evaluation model.

[0024] As an implementation manner, the setting factors include rain, snow, fog, noise interference, motion blur and compression loss factors.

[0025] The advantage of the above technical solution is that it fully considers interference factors such as rain, snow, fog, noise interference, motion blur and compression loss, greatly improving the capacity and generalization of the sample set; on this basis, the sample content representation and recognition capabilities of the target detection model are evaluated, which fully reflects the generalization ability of the target detection model in complex environments.

[0026] A second aspect of the present invention provides a target detection model performance evaluation system.

[0027] In one or more embodiments, a target detection model performance evaluation system includes:

[0028] An image processing module is used to obtain an initial image data set, and use the target detection model to be evaluated to process each image in the initial image data set to obtain a feature map set and a target recognition result corresponding to each image;

[0029] A feature space analysis index calculation module, which is used to calculate the feature activation map corresponding to each correct target in the target recognition result according to the feature map set and the target recognition result, and then compare the feature activation map with the corresponding annotation true value of the initial image data set to obtain the feature space analysis index of the target detection model to be evaluated on the initial image data set;

[0030] A sample content characterization and recognition index calculation module, which is used to simulate at least two set factors and combine the initial image data set to generate an image enhancement data set under each of the set factor scenarios, and then combine the target detection model to be evaluated to obtain the feature space analysis index of the target detection model to be evaluated on the image enhancement data set under each of the set factor scenarios and summarize them to obtain the sample content characterization and recognition index of the target detection model to be evaluated;

[0031] The model performance evaluation result summary module is used to evaluate the performance of the target detection model to be evaluated by using the sample content characterization recognition index of the target detection model to be evaluated and the feature space analysis index on the initial image data set.

[0032] As an implementation manner, in the feature space analysis index calculation module, the calculation steps of the feature activation map are:

[0033] According to the comparison result of the target recognition result of each image in the initial image data set and its corresponding annotation true value, the correct target is screened out from the target recognition results of each image;

[0034] For any correct target of each image in the initial image data set, use coordinates and category probability back propagation to obtain the gradient of the correct target on the corresponding feature map set, perform global average pooling on the value of each channel in the gradient, and obtain a channel weight factor;

[0035] Based on the channel weight factor, the value of each channel of the correct target in the corresponding feature map set is weighted and accumulated to obtain a feature activation map corresponding to the correct target.

[0036] The advantage of the above technical solution is that, by utilizing each channel of the gradient of the feature map set and its corresponding weight factors, the feature activation map corresponding to the correct target is calculated by weighted summation, thereby realizing the target detection model to analyze in the feature space, and providing a more accurate data basis for subsequent true value comparison.

[0037] As an implementation mode, in the feature space analysis index calculation module, all detection targets in all images in the initial image data set are traversed and their feature space analysis scores are calculated. For each detection type, the mean feature space analysis score of all targets of the corresponding type is calculated and used as the corresponding detection type score. All detection type scores are combined to obtain the feature space analysis index of the target detection model to be evaluated on the initial image data set.

[0038] The advantage of the above technical solution is that it realizes the analysis and evaluation of the model feature space level, improves the model's understanding of the decision-making process of sample expression and the feature expression ability of the specific evaluation model.

[0039] A third aspect of the present invention provides a computer device.

[0040] In one or more embodiments, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the target detection model performance evaluation method as described above are implemented.

[0041] A fourth aspect of the present invention provides a computer-readable storage medium.

[0042] In one or more embodiments, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the target detection model performance evaluation method as described above.

[0043] A fifth aspect of the present invention provides a computer program product.

[0044] In one or more embodiments, a computer program product includes a computer program, which, when executed by a processor, implements the steps of the target detection model performance evaluation method as described above.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) The present invention uses the target detection model to be evaluated to process each image in the initial image data set, and combines the feature activation map corresponding to each correct target in the target recognition result with the corresponding annotation true value of the initial image data set to realize the spatial level analysis and evaluation of the target detection model to be evaluated, so as to provide users with more accurate target detection model selection suggestions for target detection needs in different scenarios, and also provide R&D personnel with suggestions for optimizing the target detection model.

[0047] (2) The present invention constructs an image enhancement data set for a complex environment by simulating at least two setting factors, thereby calculating the sample content representation and recognition index of the target detection model to be evaluated, thereby improving the capacity and generalization of the sample set. On this basis, the sample content representation and recognition capability of the target detection model to be evaluated is improved, fully reflecting the generalization capability of the model in a complex environment, providing users with more accurate target detection model selection suggestions for target detection needs in different scenarios, and also providing R&D personnel with suggestions for optimizing the target detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0049] Figure 1 is a flow chart of a method for evaluating performance of a target detection model according to an embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of the structure of a target detection model performance evaluation system according to an embodiment of the present invention;

[0051] Figure 3 It is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0053] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0054] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0055] Terminology explanation:

[0056] Accuracy: Accuracy usually refers to how many objects detected by the target detection model are actually real objects; mathematically, accuracy is defined as:

[0057]

[0058] Among them, TP (True Positive) represents the number of correctly detected targets, and FP (False Positive) represents the number of false positives, that is, non-target objects are mistakenly marked as target objects.

[0059] Recall: The recall rate reflects the model's ability to identify all real targets, that is, how many targets the target detection model finds out of all the actual targets. In target detection, the recall rate refers to the proportion of real targets successfully found by the model to all actual targets. Its mathematical expression is:

[0060]

[0061] Here, FN (False Negative) refers to the actual number of targets that were not correctly identified.

[0062] F1 score: F1 is the harmonic mean of precision and recall, which can concisely represent the overall performance of the model.

[0063] Average Precision (AP) and Mean Average Precision (mAP): Since there is a trade-off between precision and recall in practical applications, a single indicator is difficult to fully reflect the performance of the model. Therefore, the average precision (AP) is generally used as a comprehensive evaluation indicator in the field of object detection. AP is calculated based on the area under the precision-recall curve, which reflects the average precision of a certain detection type at different recall levels; for multi-type object detection models, the mean of AP of all categories is usually calculated, that is, mAP. mAP can reflect the average detection performance of the object detection model in each category and is the gold standard for measuring the overall performance of the object detection model.

[0064] Embodiment 1

[0065] Figure 1 is a flow chart of a method for evaluating the performance of a target detection model in an embodiment of the present invention. Figure 1 The target detection model performance evaluation method in the embodiment shown may include:

[0066] S101, obtaining an initial image data set, and processing each image in the initial image data set using the target detection model to be evaluated to obtain a feature map set and a target recognition result corresponding to each image;

[0067] S102, calculating a feature activation map corresponding to each correct target in the target recognition result according to the feature map set and the target recognition result, and then comparing the feature activation map with the corresponding annotation true value of the initial image data set to obtain a feature space analysis index of the target detection model to be evaluated on the initial image data set;

[0068] S103, simulating at least two set factors and combining the initial image data set to generate image enhancement data sets under each of the set factor scenarios, and then combining the target detection model to be evaluated to obtain feature space analysis indicators of the target detection model to be evaluated on the image enhancement data sets under each of the set factor scenarios and summarize them to obtain sample content representation and recognition indicators of the target detection model to be evaluated;

[0069] S104, evaluating the performance of the target detection model to be evaluated by using the sample content characterization identification index of the target detection model to be evaluated and the feature space analysis index on the initial image data set.

[0070] This embodiment realizes the spatial level analysis and evaluation of the target detection model to be evaluated, and solves the problems that the existing evaluation methods lack analysis of the model feature space, regard the model as a "black box", cannot understand in detail the decision-making process of the model for sample expression, and cannot specifically evaluate the feature expression ability of the model. It also improves the capacity and generalization of the sample set, and on this basis, improves the sample content representation and identification ability of the evaluated target detection model, fully reflects the generalization ability of the model in complex environments, and provides users with more accurate target detection model selection suggestions for target detection needs in different scenarios, and also provides R&D personnel with suggestions for optimizing target detection models.

[0071] In the specific implementation process of step S101, taking the field of intelligent inspection as an example:

[0072] The most commonly used method of intelligent inspection is to use target detection technology to analyze the images taken by intelligent inspection to identify important targets such as power transmission and transformation equipment, defects, hidden dangers, etc. The intelligent inspection method can accurately locate defects and hidden dangers and provide efficient early warning, reduce labor and maintenance costs, and ensure the safety, reliability, economy and environmental protection of power supply.

[0073] In step S101, the image data set may be a power inspection image data set. In other embodiments, based on different fields or actual application scenarios, image data sets corresponding to the fields or actual application scenarios may be specifically set.

[0074] Below, a single image in the initial image dataset is taken as an example. It is processed using the target detection model to be evaluated, and the target recognition result obtained is denoted as R, which usually contains the coordinates and confidence information of multiple targets.

[0075] The output of the last layer of the backbone (feature extraction network) structure of the target detection model to be evaluated when processing the single image is denoted as F. F is a set of feature maps with a shape of 1×c×h×w, where c, h, and w represent the number of channels, height, and width of the feature map, respectively.

[0076] It should be noted here that c, h, and w are determined by the backbone (feature extraction network) of the target detection model to be evaluated.

[0077] In the specific implementation process of step S102, the step of calculating the feature activation map M corresponding to each correct target in the target recognition result is:

[0078] Step S1021: According to the comparison result between the target recognition result of each image in the initial image data set and its corresponding annotation true value, the correct target is screened out from the target recognition results of each image.

[0079] Among them, the target detection result R of each image is compared with the true value of the annotation of the image, and the correct target, false alarm target and missed target of the image detection result are judged according to the set IoU (overlap) threshold; for false alarm targets and missed targets, their feature space analysis score is recorded as 0.

[0080] Step S1022: For any correct target in each image in the initial image data set, use coordinate and category probability back propagation to obtain the gradient of the correct target on the corresponding feature map set, perform global average pooling on the value of each channel in the gradient, and obtain the channel weight factor.

[0081] Specifically, for any correct target in any image, use the coordinates and category probabilities to back-propagate to obtain its gradient G on the feature map set F, perform global average pooling on the value of each channel k in G, and obtain the weight W of each channel k in G k , the formula is as follows:

[0082]

[0083] Among them, G k is the gradient of channel k on the feature map set F; i,j is the gradient G k The width and height of the .

[0084] Step S1023: Based on the channel weight factor, weighted accumulation is performed on the value of each channel of the correct target in the corresponding feature map set to obtain a feature activation map corresponding to the correct target.

[0085] The weight set of all channels in G is W. With W as the weight, the value of each channel in the feature map set F is weightedly accumulated to obtain the feature activation map M corresponding to the target:

[0086]

[0087] Among them, F k is the feature map set of channel k.

[0088] In this way, the feature activation map corresponding to the correct target is calculated by weighted summing up the gradient channels of the feature map set and their corresponding weight factors, which enables the target detection model to analyze in the feature space and provides a more accurate data basis for subsequent true value comparison.

[0089] In a specific implementation process, the steps of comparing the feature activation map with the corresponding annotation true value of the initial image data set are:

[0090] The feature activation map is scaled to the original size of the feature map in the corresponding feature map set and normalized, and then the value of the feature activation map outside the limit is set to zero using the set size of the corresponding detection target frame as the limit, and the feature activation map is binarized;

[0091] The minimum positive circumscribed rectangle of the maximum connected area whose binary value is 1 is screened out, and then the overlap between the minimum positive circumscribed rectangle and the true value frame corresponding to the current detection target is calculated and used as the feature space analysis score of the current detection target.

[0092] Specifically, the feature activation map M is scaled to the size of the original image and normalized, and then the size of the corresponding detection target box is set as the boundary, so that the value of the activation map M outside the boundary is set to zero, and the activation map M is binarized using the Otsu method. The minimum positive circumscribed rectangle of the maximum connected area that is binarized to 1 is recorded as box, and the IoU (overlap) between the box and the corresponding true value box of the detection target is calculated as the feature space analysis score of the target.

[0093] So through feature activation Figure 2 The smallest positive circumscribed rectangle corresponding to the value of 1 is compared with the true value box corresponding to the current detection target, and the feature space analysis score of the current detection target is calculated, which realizes the analysis and evaluation of the model feature space level, improves the model's understanding of the decision-making process of sample expression and the feature expression ability of the specific evaluation model.

[0094] In the specific implementation process, all detection targets in all images in the initial image data set are traversed and their feature space analysis scores are calculated. For each detection type, the mean feature space analysis score of all targets of the corresponding type is calculated and used as the corresponding detection type score. All detection type scores are combined to obtain the feature space analysis index of the target detection model to be evaluated on the initial image data set. This realizes the analysis and evaluation of the model feature space level, improves the model's understanding of the decision-making process of sample expression and the feature expression ability of the specific evaluation model.

[0095] In step S103, the set factors include but are not limited to rain, snow, fog, noise interference, motion blur and compression loss factors. In this way, the interference factors such as rain, snow, fog, noise interference, motion blur and compression loss are fully considered, which greatly improves the capacity and generalization of the sample set; on this basis, the sample content representation and recognition ability of the target detection model is evaluated, which fully reflects the generalization ability of the target detection model in complex environments.

[0096] For example, existing algorithms are used to simulate factors such as rain, snow, fog, noise interference, motion blur, and compression loss, to generate enhanced data sets for corresponding scenarios, and the above steps S101 and S102 are repeatedly performed.

[0097] Specifically, for each image in the initial image data set, a preset image processing algorithm or a preset image generation model is used to generate an image of rain, snow, and fog to form an enhanced data set A;

[0098] Using a preset image processing algorithm, randomly add a Gaussian noise, Laplace noise or Poisson noise of a specified intensity to each image of the initial image dataset to obtain an enhanced dataset B;

[0099] Use a preset image filtering algorithm to perform motion blur of a specified vector (with direction and length) on each image of the initial image data set to obtain an enhanced data set C;

[0100] Use compression formats such as JPEG, AVIF, WEBP, etc. to compress each image of the initial image dataset to a set size (referring to data size, number of bytes), and then restore it to obtain the enhanced dataset D;

[0101] Using enhanced data sets A, B, C and D respectively, repeat steps S101 and S102 to obtain feature space analysis indicators of the target detection model on the enhanced data sets A, B, C and D. The collection of these indicators is the sample content representation and recognition indicator of the target detection model, which reflects the sample content representation and recognition ability of the target detection model in different environments.

[0102] For example, the feature space analysis indicators of the target detection model to be evaluated on the initial image dataset are shown in Table 1:

[0103] Table 1 Feature space analysis indicators for evaluating target detection models on the initial image dataset

[0104] Tower Crane crane excavator bulldozer Pile Driver Dump Truck Cement Pump Truck average Score 80.7 80.1 74 73.4 77.9 77.3 66.3 75.7

[0105] The content representation and identification indicators of the target detection model samples to be evaluated are shown in Table 2:

[0106] Table 2 Content representation and identification indicators of target detection model samples to be evaluated

[0107]

[0108] Referring to the performance evaluation results of the above target detection models can help users select the optimal target detection model or provide guidance for developers on the direction of optimizing the target detection model.

[0109] When the user is selecting the optimal target detection model, the target detection model with the highest weighted sum is selected as the optimal target detection model based on the weighted sum of the sample content characterization identification index and feature space analysis index of the target detection model to be evaluated and the preset weights.

[0110] Specifically, the weights of the sample content characterization identification index and the feature space analysis index of the above-mentioned target detection model to be evaluated can be set according to the actual situation of the device and the scene.

[0111] For example, the field of power transmission visualization is basically outdoor scenes, so we can pay more attention to the item related to "rain, snow and fog weather" in the sample content characterization identification index. Similarly, if it is located in a mountainous area and the fog is particularly severe, we need to pay more attention to this indicator;

[0112] For example, when the device has poor anti-shake capability or is located in an area with high wind force level, motion blur is more likely to occur. In this case, the weight of "motion blur" in the sample content characterization and identification index can be increased when selecting the target detection model. The target detection model performance evaluation index obtained by the method of the present invention takes into account various factors in the power field scenario more comprehensively. By referring to the weighted sum of the sample content characterization and identification index and the feature space analysis index of the target detection model to be evaluated and the preset weights, the user can more easily and accurately select the optimal target detection model that meets the actual scenario.

[0113] When considering the optimization of the target detection model, we can compare the differences between feature space analysis indicators and common indicators (such as AP, F1, etc.) in various types to determine whether the backbone (feature extraction network) of the model has poor feature extraction capabilities for a certain type of target, and then perform targeted optimization; we can also analyze the sample content characterization identification indicators to find that the target detection model has poor generalization capabilities for certain environmental scenes. For example, the results described above show that under the interference of "rain, snow and fog", the generalization capability of the model is significantly lower, and we can consider increasing this type of training samples for optimization. The target detection model performance evaluation indicators obtained by the method of the present invention can expose the detailed problems of the target detection model in various aspects. By referring to the target detection model performance evaluation results of the embodiments of the present invention, R&D personnel can find a more reasonable target detection model optimization direction.

[0114] Embodiment 2

[0115] Figure 2 is a schematic diagram of the structure of a target detection model performance evaluation system in an embodiment of the present invention. Figure 1 The target detection model performance evaluation method corresponds to Figure 2 As shown, the target detection model performance evaluation system in this embodiment may include:

[0116] An image processing module 201 is used to obtain an initial image data set, and process each image in the initial image data set using the target detection model to be evaluated to obtain a feature map set and a target recognition result corresponding to each image;

[0117] A feature space analysis index calculation module 202 is used to calculate the feature activation map corresponding to each correct target in the target recognition result according to the feature map set and the target recognition result, and then compare the feature activation map with the corresponding annotation true value of the initial image data set to obtain the feature space analysis index of the target detection model to be evaluated on the initial image data set;

[0118] The sample content characterization identification index calculation module 203 is used to simulate at least two set factors and combine the initial image data set to generate an image enhancement data set under each of the set factor scenarios, and then combine the target detection model to be evaluated to obtain the feature space analysis index of the target detection model to be evaluated on the image enhancement data set under each of the set factor scenarios and summarize them to obtain the sample content characterization identification index of the target detection model to be evaluated;

[0119] The model performance evaluation result summary module 204 is used to evaluate the performance of the target detection model to be evaluated by using the sample content characterization identification index of the target detection model to be evaluated and the feature space analysis index on the initial image data set.

[0120] In a specific implementation process, in the feature space analysis index calculation module 202, the calculation steps of the feature activation map are:

[0121] According to the comparison result of the target recognition result of each image in the initial image data set and its corresponding annotation true value, the correct target is screened out from the target recognition results of each image;

[0122] For any correct target of each image in the initial image data set, use coordinates and category probability back propagation to obtain the gradient of the correct target on the corresponding feature map set, perform global average pooling on the value of each channel in the gradient, and obtain a channel weight factor;

[0123] Based on the channel weight factor, the value of each channel of the correct target in the corresponding feature map set is weighted and accumulated to obtain a feature activation map corresponding to the correct target.

[0124] In this way, the feature activation map corresponding to the correct target is calculated by weighted summing up the gradient channels of the feature map set and their corresponding weight factors, which enables the target detection model to analyze in the feature space and provides a more accurate data basis for subsequent true value comparison.

[0125] In the feature space analysis index calculation module 202, the step of comparing the feature activation map with the corresponding annotation true value of the initial image data set is:

[0126] The feature activation map is scaled to the original size of the feature map in the corresponding feature map set and normalized, and then the value of the feature activation map outside the limit is set to zero using the set size of the corresponding detection target frame as the limit, and the feature activation map is binarized;

[0127] The minimum positive circumscribed rectangle of the maximum connected area whose binary value is 1 is screened out, and then the overlap between the minimum positive circumscribed rectangle and the true value frame corresponding to the current detection target is calculated and used as the feature space analysis score of the current detection target.

[0128] So through feature activation Figure 2 The smallest positive circumscribed rectangle corresponding to the value of 1 is compared with the true value box corresponding to the current detection target, and the feature space analysis score of the current detection target is calculated, which realizes the analysis and evaluation of the model feature space level, improves the model's understanding of the decision-making process of sample expression and the feature expression ability of the specific evaluation model.

[0129] In the feature space analysis index calculation module 202, all detection targets in all images in the initial image data set are traversed and their feature space analysis scores are calculated. For each detection type, the mean of the feature space analysis scores of all targets of the corresponding type is calculated and used as the corresponding detection type score. All detection type scores are combined to obtain the feature space analysis index of the target detection model to be evaluated on the initial image data set. This realizes the analysis and evaluation of the model feature space level, improves the model's understanding of the decision-making process of sample expression and the feature expression ability of the specific evaluation model.

[0130] In some specific implementations, in the sample content representation and identification index calculation module 203, the set factors include rain, snow, fog, noise interference, motion blur and compression loss factors. In this way, interference factors such as rain, snow, fog, noise interference, motion blur and compression loss are fully considered, and the capacity and generalization of the sample set are greatly improved; on this basis, the sample content representation and identification ability of the target detection model is evaluated, which fully reflects the generalization ability of the target detection model in complex environments.

[0131] It should be noted here that the various modules in the target detection model performance evaluation system in this embodiment correspond to the various modules in the target detection model performance evaluation method in Example 1, and their specific implementation processes are the same, which will not be repeated here.

[0132] The target detection model performance evaluation system of the embodiment of the present invention deeply evaluates the feature space analysis indicators and sample representation and recognition indicators of the target detection model, making up for the shortcomings of the existing evaluation methods; this method can complement the advantages of the existing model evaluation methods, and provide a scientific basis for comprehensively evaluating the actual performance of the target model in a complex environment.

[0133] Embodiment 3

[0134] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the following is achieved: Figure 1 The steps of the object detection model performance evaluation method shown.

[0135] Reference Figure 3 , a schematic diagram of a computer device is given. It should be noted that Figure 3 The computer device 300 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0136] like Figure 3As shown, the computer device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303. In the RAM 303, various programs and data required for system operation are also stored. The central processing unit 301, the ROM 302 and the RAM 103 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0137] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a local area network (LAN) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0138] When the central processing unit 301 in the computer device of this embodiment executes the program, the following is achieved: Figure 1 Steps in the method for evaluating the performance of an object detection model shown.

[0139] Embodiment 4

[0140] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, Figure 1 The steps of the object detection model performance evaluation method shown.

[0141] in, Figure 1 The computer program instructions corresponding to the method shown may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0143] Embodiment 5

[0144] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. This embodiment provides a computer program product, which includes a computer program carried on a computer readable medium, the computer program including a computer program for executing Figure 1 In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit 301, various functions defined in the apparatus of the present application are executed.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for evaluating the performance of a target detection model, characterized in that: include: Obtain an initial image data set, and use the target detection model to be evaluated to process each image in the initial image data set to obtain a feature map set and a target recognition result corresponding to each image; According to the feature map set and the target recognition result, a feature activation map corresponding to each correct target in the target recognition result is calculated, and then the feature activation map is compared with the corresponding annotation true value of the initial image data set to obtain the feature space analysis index of the target detection model to be evaluated on the initial image data set; Simulating at least two setting factors and combining the initial image data set to generate image enhancement data sets under each of the setting factor scenarios, and then combining the target detection model to be evaluated to obtain feature space analysis indicators of the target detection model to be evaluated on the image enhancement data sets under each of the setting factor scenarios and summarize them to obtain sample content characterization and recognition indicators of the target detection model to be evaluated; The performance of the target detection model to be evaluated is evaluated using the sample content characterization recognition index of the target detection model to be evaluated and the feature space analysis index on the initial image data set.

2. The target detection model performance evaluation method according to claim 1, characterized in that: The calculation steps of the feature activation map are: According to the comparison result of the target recognition result of each image in the initial image data set and its corresponding annotation true value, the correct target is screened out from the target recognition results of each image; For any correct target of each image in the initial image data set, use coordinates and category probability back propagation to obtain the gradient of the correct target on the corresponding feature map set, perform global average pooling on the value of each channel in the gradient, and obtain a channel weight factor; Based on the channel weight factor, the value of each channel of the correct target in the corresponding feature map set is weighted and accumulated to obtain a feature activation map corresponding to the correct target.

3. The target detection model performance evaluation method according to claim 2, characterized in that: The steps of comparing the feature activation map with the corresponding annotation true value of the initial image data set are: The feature activation map is scaled to the original size of the feature map in the corresponding feature map set and normalized, and then the value of the feature activation map outside the limit is set to zero using the set size of the corresponding detection target frame as the limit, and the feature activation map is binarized; The minimum positive circumscribed rectangle of the maximum connected area whose binary value is 1 is screened out, and then the overlap between the minimum positive circumscribed rectangle and the true value frame corresponding to the current detection target is calculated and used as the feature space analysis score of the current detection target.

4. The target detection model performance evaluation method according to claim 1 or 3, characterized in that: Traverse all detected targets in all images in the initial image dataset and calculate their feature space Analyze the scores, calculate the mean of the feature space analysis scores of all targets of the corresponding type for each detection type and use it as the score of the corresponding detection type, and combine all the detection type scores to obtain the feature space analysis index of the target detection model to be evaluated on the initial image data set.

5. The target detection model performance evaluation method according to claim 1, characterized in that: The setting factors include rain, snow, fog, noise interference, motion blur and compression loss factors.

6. A target detection model performance evaluation system, characterized in that: include: An image processing module is used to obtain an initial image data set, and use the target detection model to be evaluated to process each image in the initial image data set to obtain a feature map set and a target recognition result corresponding to each image; A feature space analysis index calculation module, which is used to calculate the feature activation map corresponding to each correct target in the target recognition result according to the feature map set and the target recognition result, and then compare the feature activation map with the corresponding annotation true value of the initial image data set to obtain the feature space analysis index of the target detection model to be evaluated on the initial image data set; A sample content characterization and recognition index calculation module, which is used to simulate at least two set factors and combine the initial image data set to generate an image enhancement data set under each of the set factor scenarios, and then combine the target detection model to be evaluated to obtain the feature space analysis index of the target detection model to be evaluated on the image enhancement data set under each of the set factor scenarios and summarize them to obtain the sample content characterization and recognition index of the target detection model to be evaluated; The model performance evaluation result summary module is used to evaluate the performance of the target detection model to be evaluated by using the sample content characterization recognition index of the target detection model to be evaluated and the feature space analysis index on the initial image data set.

7. The target detection model performance evaluation system according to claim 6, characterized in that: In the feature space analysis index calculation module, the calculation steps of the feature activation map are: According to the comparison result of the target recognition result of each image in the initial image data set and its corresponding annotation true value, the correct target is screened out from the target recognition results of each image; For any correct target of each image in the initial image data set, use coordinates and category probability back propagation to obtain the gradient of the correct target on the corresponding feature map set, perform global average pooling on the value of each channel in the gradient, and obtain a channel weight factor; Based on the channel weight factor, weighted accumulation is performed on the value of each channel of the correct target in the corresponding feature map set to obtain a feature activation map corresponding to the correct target; Or / and, in the feature space analysis index calculation module, all detection targets in all images in the initial image data set are traversed and their feature space analysis scores are calculated, for each detection type, the mean feature space analysis score of all targets of the corresponding type is calculated and used as the corresponding detection type score, and all detection type scores are combined to obtain the feature space analysis index of the target detection model to be evaluated on the initial image data set.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the target detection model performance evaluation method as described in any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the target detection model performance evaluation method as described in any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the target detection model performance evaluation method as described in any one of claims 1 to 5 are implemented.

Images