A multi-dimensional multi-layer intelligent identification algorithm evaluation index system construction method

Abstract

This specification discloses a method for constructing a multi-dimensional, multi-layered intelligent recognition algorithm evaluation index system, belonging to the field of intelligent perception algorithm performance verification and evaluation technology. It includes: constructing traditional evaluation indices based on target recognition accuracy, target false alarm rate, target false alarm rate, and latency; constructing generalization evaluation indices based on target adaptability and environmental adaptability indices; constructing security evaluation indices based on adversarial example security and poisoning example security indices; constructing learning ability evaluation indices based on inversion learning ability, few-shot learning ability, and incremental learning ability indices; and calculating a comprehensive evaluation result based on the traditional evaluation indices, generalization evaluation indices, security evaluation indices, learning ability evaluation indices, and their corresponding weights. This method aims to address the problems of inaccurate, incomplete, and unreliable evaluations in current intelligent perception algorithm performance verification and evaluation methods.

Description

Technical Field

[0001] This invention relates to the field of performance verification and evaluation technology for intelligent sensing algorithms, and in particular to a method for constructing an evaluation index system for multi-dimensional and multi-layer intelligent recognition algorithms. Background Technology

[0002] Current performance evaluation methods for intelligent sensing algorithms typically focus solely on accuracy, measuring performance through metrics such as precision, recall, and accuracy. This accuracy-based approach fails to consider the data-driven nature of intelligent sensing algorithms and lacks evaluation of their generalization ability, security, and learning capacity, resulting in inaccurate, incomplete, and unreliable evaluations. Effective performance evaluation of intelligent sensing algorithms is crucial for improving their performance and a prerequisite for their practical application, yet current technologies cannot fully address this. Existing methods, which primarily evaluate accuracy, lack a comprehensive evaluation metric system, hindering effective performance assessment. Summary of the Invention

[0003] The purpose of this invention is to provide a method for constructing a multi-dimensional, multi-layer intelligent recognition algorithm evaluation index system, so as to solve the problems of inaccurate, incomplete, and unreliable evaluation in the current performance verification and evaluation methods of intelligent perception algorithms.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] On the one hand, this specification provides a method for constructing a multi-dimensional, multi-layer intelligent recognition algorithm evaluation index system, including:

[0006] Step 102: Based on target recognition accuracy, target false alarm rate, target false alarm rate, and latency, construct traditional evaluation indicators;

[0007] Step 104: Construct generalization assessment indicators based on target adaptability indicators and environmental adaptability indicators;

[0008] Step 106: Construct a security assessment index based on the security index of adversarial examples and the security index of poisoned samples;

[0009] Step 108: Construct a learning ability assessment index based on the inversion learning ability index, the few-shot learning ability index, and the incremental learning ability index;

[0010] Step 110: Calculate the comprehensive evaluation result based on traditional evaluation indicators, generalization evaluation indicators, security evaluation indicators, learning ability evaluation indicators, and their corresponding weights.

[0011] On the other hand, this specification provides a device for constructing a multi-dimensional, multi-layer intelligent recognition algorithm evaluation index system, including:

[0012] The traditional evaluation index construction module is used to construct traditional evaluation indexes based on target recognition accuracy, target false detection rate, target false alarm rate, and latency.

[0013] The generalization assessment index construction module is used to construct generalization assessment indices based on target adaptability indices and environmental adaptability indices.

[0014] The security assessment metric construction module is used to construct security assessment metrics based on adversarial sample security metrics and poisoning sample security metrics.

[0015] The learning ability assessment index construction module is used to construct learning ability assessment indices based on inverted learning ability indices, few-shot learning ability indices, and incremental learning ability indices.

[0016] The comprehensive evaluation result calculation module is used to calculate the comprehensive evaluation result based on traditional evaluation indicators, generalization evaluation indicators, security evaluation indicators, learning ability evaluation indicators and their corresponding weights.

[0017] Based on the above technical solution, this specification can achieve the following technical effects:

[0018] This method first constructs a verification and evaluation index system, including 4 primary indicators and 11 secondary indicators, from the perspectives of traditional indicators, generalization, security, and learning ability. Then, it quantitatively calculates the secondary indicators and calculates the primary indicators by average weighting. Finally, it performs weighted fusion of the evaluation results from different dimensions to achieve multi-dimensional and comprehensive verification and evaluation of intelligent perception algorithms, support the optimized design of intelligent perception algorithms, and improve the performance of intelligent perception algorithms. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a method for constructing an evaluation index system for a multidimensional and multi-layer intelligent recognition algorithm according to an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of a minimum entropy pruning framework in one embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of a device for constructing an evaluation index system for a multidimensional and multi-layer intelligent recognition algorithm according to an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of an electronic device according to the present invention. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer from the following description and claims. It should be noted that the drawings are all in a very simplified form and are not to a precise scale, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0024] It should be noted that, in order to clearly illustrate the content of this invention, several embodiments are provided to further explain different implementations of the invention. These embodiments are enumerated rather than exhaustive. Furthermore, for the sake of brevity, content mentioned in the preceding embodiments is often omitted in the following embodiments. Therefore, content not mentioned in the later embodiments can be referred to in the preceding embodiments. Example 1

[0025] Please refer to Figure 1 , Figure 1 The illustration shows a method for constructing an evaluation index system for a multi-dimensional, multi-layer intelligent recognition algorithm provided in this embodiment. In this embodiment, the method includes:

[0026] Step 102: Based on target recognition accuracy, target false alarm rate, target false alarm rate, and latency, construct traditional evaluation indicators;

[0027] In this embodiment, the target recognition accuracy is used to characterize the ability of the intelligent perception algorithm to correctly identify targets. The higher the target recognition accuracy, the stronger the ability of the intelligent perception algorithm to correctly identify targets. The target false negative rate is used to characterize the ability of the intelligent perception algorithm to correctly predict the purity of negative samples. The lower the target false negative rate, the stronger the ability of the intelligent perception algorithm to identify each type of target sample. The target false alarm rate is used to characterize the ability of the intelligent perception algorithm to correctly predict the purity of positive samples. The lower the target false alarm rate, the stronger the ability of the intelligent recognition algorithm to identify positive samples. The latency is the time required to process images of the same size.

[0028] In this embodiment, one implementation of step 102 is as follows:

[0029] Step 202: Calculate the target recognition accuracy based on the number of positive samples that are predicted as positive samples and negative samples respectively;

[0030] Step 204: Calculate the target false negative rate based on the number of positive samples predicted as positive samples and the total number of negative samples.

[0031] Step 206: Calculate the false alarm rate of the target based on the number of positive samples predicted as negative samples and the number of negative samples predicted as positive samples;

[0032] Step 208: Calculate the time delay based on the time required to process the image and the total number of negative samples;

[0033] Step 210: Based on target recognition accuracy, target false alarm rate, target false alarm rate, time delay and their respective weights, calculate the traditional evaluation index.

[0034] Step 104: Construct generalization assessment indicators based on target adaptability indicators and environmental adaptability indicators;

[0035] In this embodiment, the target adaptability index is used to reflect the intelligent recognition algorithm's ability to adapt to samples of various complex target information; the target information includes target scale, target category, target perspective, and target signal strength; the environmental adaptability index is used to reflect the intelligent recognition algorithm's ability to adapt to samples of various complex environmental information; the environmental information includes weather, climate, lighting, and natural environment.

[0036] In this embodiment, one implementation of step 104 is as follows:

[0037] Step 302: Construct a sample set containing various complex target information and input it into the intelligent recognition algorithm for target recognition to obtain the first target recognition accuracy and the first target recall rate;

[0038] Step 304: Calculate the target adaptability index based on the first target recognition accuracy and the first target recall rate;

[0039] Step 306: Construct a sample set containing various complex environmental information and input it into the intelligent recognition algorithm to perform target recognition, thereby obtaining the second target recognition accuracy and the second target recall rate;

[0040] Step 308: Based on the second target recognition accuracy and the second target recall rate, calculate the environmental adaptability index.

[0041] Step 310: Calculate the generalization assessment index based on the target adaptability index, the environmental adaptability index, and the corresponding weights.

[0042] Step 106: Construct a security assessment index based on the security index of adversarial examples and the security index of poisoned samples;

[0043] In this embodiment, the adversarial sample security index is used to characterize the ability of the intelligent recognition algorithm to resist noise attacks; the poisoned sample security index is used to characterize the ability of the intelligent recognition algorithm to effectively remove erroneous samples during model training.

[0044] In this embodiment, one implementation of step 106 is as follows:

[0045] Step 402: Construct an adversarial sample set, and input the original test dataset and the adversarial sample set into the intelligent recognition algorithm for target recognition to obtain the experimental results corresponding to the original test dataset and the adversarial sample set;

[0046] Step 404: Compare the experimental results corresponding to the original test dataset and the adversarial sample set to obtain the adversarial sample security index;

[0047] Step 406: Construct a poisoning dataset and use the poisoning dataset to train an intelligent recognition algorithm to obtain the poisoning-trained intelligent recognition algorithm;

[0048] Step 408: Construct a verification dataset that has not been poisoned and input it into the intelligent recognition algorithm trained after poisoning to perform target recognition and obtain the results of the poisoning experiment;

[0049] Step 410: Based on the results of the poisoning experiment, calculate and obtain the safety index of the poisoned sample;

[0050] Step 412: Calculate the security assessment index based on the adversarial sample security index, the poisoning sample security index, and the corresponding weights.

[0051] Step 108: Construct a learning ability assessment index based on the inversion learning ability index, the few-shot learning ability index, and the incremental learning ability index;

[0052] In this embodiment, the inversion learning capability index is used to characterize the performance retention capability of the intelligent recognition algorithm when trained on low-complexity data and tested on high-complexity data; the few-shot learning capability index is used to characterize the performance retention capability of the intelligent recognition algorithm when trained on few-shot data and tested on a large-sample data; and the incremental learning capability index is used to characterize the performance retention capability of the intelligent recognition algorithm when trained on a small number of real samples and a large number of simulated samples and then tested on real targets.

[0053] In this embodiment, one implementation of step 108 is as follows:

[0054] Step 502: Construct a training dataset for typical scenarios and input it into an intelligent recognition algorithm for target recognition to obtain the target performance change curve;

[0055] Step 504: Gradually reduce the amount of high-complexity data in the training dataset and record the performance change curve of the intelligent recognition algorithm in real time to obtain the first performance change curve.

[0056] Step 506: Calculate the ratio of the area of ​​the first performance change curve to the area of ​​the target performance change curve to obtain the inversion learning ability index.

[0057] Step 508: Gradually reduce the number of samples in the training dataset and record the performance change curve of the intelligent recognition algorithm in real time to obtain the second performance change curve;

[0058] Step 510: Calculate the ratio of the area of ​​the second performance change curve to the area of ​​the target performance change curve to obtain the few-sample learning ability index.

[0059] Step 512: Gradually reduce the number of real samples in the training dataset and record the performance change curve of the intelligent recognition algorithm in real time to obtain the third performance change curve.

[0060] Step 514: Calculate the ratio of the area of ​​the third performance change curve to the area of ​​the target performance change curve to obtain the incremental learning ability index.

[0061] Step 516: Calculate the learning ability assessment index based on the inversion learning ability index, the few-shot learning ability index, the incremental learning ability index, and the corresponding weights.

[0062] Step 110: Calculate the comprehensive evaluation result based on traditional evaluation indicators, generalization evaluation indicators, security evaluation indicators, learning ability evaluation indicators, and their corresponding weights.

[0063] Specifically, a method for constructing an evaluation index system for multi-dimensional and multi-layer intelligent recognition algorithms, referencing... Figure 1 The system consists of four primary evaluation indicators: traditional indicators, generalization ability, security, and learning ability, and eleven corresponding secondary evaluation indicators. It includes the following steps:

[0064] The first step is to design and construct traditional primary evaluation indicators.

[0065] Traditional evaluation metrics consist of four secondary evaluation metrics: target recognition accuracy, false alarm rate, false alarm rate, and latency. These secondary evaluation metrics can be quantitatively calculated, and the traditional primary evaluation metrics are calculated through weighted fusion.

[0066] (1) Target recognition accuracy: This reflects the ability of the intelligent recognition model to correctly identify targets. The higher the target recognition rate, the stronger the recognition algorithm's ability to identify targets. The specific calculation formula for each category is as follows:

[0067]

[0068] TP represents the number of positive samples predicted as positive samples, FP represents the number of positive samples predicted as negative samples, and i=1,2,…,K represents the number of classes.

[0069] The overall target recognition accuracy is calculated as follows:

[0070]

[0071] (2) False negative rate: This reflects the ability of the intelligent recognition model to correctly predict the purity of negative samples. The lower the false negative rate, the stronger the recognition algorithm's ability to identify each type of target sample. The false negative rate for each category is calculated as follows:

[0072]

[0073] N represents the total number of negative samples.

[0074] The overall false negative rate is calculated as follows:

[0075]

[0076] (3) False alarm rate: This reflects the ability of the intelligent recognition model to correctly predict the purity of positive samples. The lower the false alarm rate, the stronger the recognition algorithm's ability to identify positive samples. The calculation method for the false alarm rate for each category is as follows:

[0077]

[0078] TN represents the number of negative samples that are predicted as positive samples.

[0079] The overall false alarm rate is calculated as follows:

[0080]

[0081] (4) Latency: The time required to process images of the same size is used as the latency evaluation index, and its calculation method is as follows:

[0082]

[0083] (5) Weighted fusion calculation of traditional primary evaluation indicators

[0084]

[0085] for The weights corresponding to the indicators.

[0086] The second step is to design and construct generalization evaluation indicators.

[0087] The generalization assessment index consists of two secondary assessment indicators: target adaptability and environmental adaptability. The secondary assessment indicators can be quantitatively calculated, and the generalization primary assessment index is calculated through weighted fusion.

[0088] (1) Target adaptability: reflects the ability of intelligent recognition models to adapt to samples of various complexities under different conditions such as target scale, target category, target perspective, and signal strength.

[0089] Taking target scale and type adaptability as an example, a sample set containing multiple types and target scales is constructed to record key performance indicators such as recognition accuracy and recall rate of the intelligent model on different target types and target scales.

[0090] (2) Environmental adaptability: reflects the ability of the intelligent recognition model to adapt to samples in different scenarios such as different weather, weather conditions and natural environment.

[0091] Construct multi-environment datasets under different lighting and background conditions, and record key performance indicators such as accuracy and recall of the intelligent model on the multi-environment datasets.

[0092] (3) Weighted fusion calculation of generalization level 1 evaluation index

[0093] The third step is to design and construct security assessment indicators.

[0094] The safety assessment index consists of two secondary assessment indicators: the safety of adversarial samples and the safety of poisoned samples. The secondary assessment indicators can be quantitatively calculated, and the primary safety assessment index is calculated through weighted fusion.

[0095] (1) Adversarial sample security: reflects the ability of intelligent recognition models to resist noise attacks.

[0096] Construct a set of adversarial examples and record key performance indicators such as accuracy and recall of the intelligent model on the set. By comparing the performance differences of the model on the original test dataset and adversarial examples, evaluate the adversarial example security of the model.

[0097] (2) Safety of poisoned samples: reflects the ability of the intelligent recognition model to effectively remove erroneous samples during training.

[0098] Construct a poisoning dataset, use the poisoning dataset to train an intelligent model, and record key performance indicators such as accuracy and recall of the intelligent model on a validation dataset that has not been poisoned.

[0099] (3) Level 1 security assessment index of weighted fusion computing

[0100] The fourth step is to design and construct learning ability assessment indicators.

[0101] The learning ability assessment index consists of five secondary assessment indicators: training efficiency, inversion learning ability, few-shot learning ability, memory ability, and incremental learning ability. These secondary assessment indicators can be quantitatively calculated, and the primary assessment indicators of learning ability are calculated through weighted fusion.

[0102] (1) Inversion learning ability: reflects the ability of intelligent recognition models to maintain performance when trained on low-complexity data and tested on high-complexity data.

[0103] Construct a training dataset for typical scenarios, record the performance changes of the intelligent model as the amount of high-complexity data in the training set decreases, and calculate the ratio of the area under the performance curve to the area under the ideal performance curve.

[0104] (2) Few-sample learning ability: reflects the ability of intelligent recognition models to maintain performance when trained with a small number of sample data and tested with a large number of sample data.

[0105] Construct a training dataset for typical scenarios, record the performance changes of the intelligent model as the number of training set samples changes, and calculate the ratio of the area under the performance curve to the area under the ideal performance curve.

[0106] (3) Incremental learning ability: reflects the ability of intelligent recognition models to maintain performance on real targets when trained with a small number of real samples and a large number of simulated samples.

[0107] Construct a training dataset for typical scenarios, record the performance changes of the intelligent model as the number of real samples in the training set changes, calculate the area between the curve and the training cycle axis by integration, and calculate the ratio of the area of ​​the performance curve to that under ideal conditions.

[0108] (4) Weighted fusion computing learning ability primary assessment indicators

[0109] Step 5: Calculate the comprehensive evaluation indicators

[0110] The four primary evaluation indicators—traditional indicators, generalization ability, security, and learning ability—are weighted to obtain a comprehensive evaluation result.

[0111] The formula is as follows:

[0112]

[0113] in, These are the evaluation results based on traditional indicators, generalization ability, security, and learning ability. The weights are assigned to the traditional indicator evaluation results, the generalization evaluation results, the security evaluation results, and the learning ability evaluation results, respectively.

[0114] In summary, this method establishes a verification and evaluation index system for intelligent sensing algorithms, including 4 primary evaluation indicators and 11 secondary evaluation indicators. It verifies and evaluates intelligent sensing algorithms from four dimensions: traditional indicators, generalization, security, and learning ability. It develops a quantitative calculation method for secondary indicators and a comprehensive calculation method for primary indicators. By weighted and fused evaluation results from different dimensions, it achieves a comprehensive verification and evaluation of intelligent sensing algorithms. This solves the problem that current methods relying solely on accuracy indicators for intelligent sensing algorithm evaluation are inaccurate and unreliable. It achieves beneficial technical effects through quantitative evaluation and analysis based on the intelligent sensing algorithm evaluation index system, demonstrating outstanding substantive features and significant progress.

[0115] Example 2

[0116] Please refer to Figure 3 , Figure 3 The image shows a device for constructing a multi-dimensional, multi-layer intelligent recognition algorithm evaluation index system provided in this embodiment. In this embodiment, the device includes:

[0117] The traditional evaluation index construction module is used to construct traditional evaluation indexes based on target recognition accuracy, target false detection rate, target false alarm rate, and latency.

[0118] The generalization assessment index construction module is used to construct generalization assessment indices based on target adaptability indices and environmental adaptability indices.

[0119] The security assessment metric construction module is used to construct security assessment metrics based on adversarial sample security metrics and poisoning sample security metrics.

[0120] The learning ability assessment index construction module is used to construct learning ability assessment indices based on inverted learning ability indices, few-shot learning ability indices, and incremental learning ability indices.

[0121] The comprehensive evaluation result calculation module is used to calculate the comprehensive evaluation result based on traditional evaluation indicators, generalization evaluation indicators, security evaluation indicators, learning ability evaluation indicators and their corresponding weights.

[0122] Optionally, the target recognition accuracy is used to characterize the ability of the intelligent perception algorithm to correctly identify targets. The higher the target recognition accuracy, the stronger the ability of the intelligent perception algorithm to correctly identify targets. The target false negative rate is used to characterize the ability of the intelligent perception algorithm to correctly predict the purity of negative samples. The lower the target false negative rate, the stronger the ability of the intelligent perception algorithm to identify each type of target sample. The target false alarm rate is used to characterize the ability of the intelligent perception algorithm to correctly predict the purity of positive samples. The lower the target false alarm rate, the stronger the ability of the intelligent recognition algorithm to identify positive samples. The latency is the time required to process images of the same size.

[0123] Optional, traditional evaluation metric construction modules include:

[0124] The target recognition accuracy calculation unit is used to calculate the target recognition accuracy based on the number of positive samples that are predicted as positive samples and negative samples respectively.

[0125] The target false negative rate calculation unit is used to calculate the target false negative rate based on the number of positive samples that are predicted as positive samples and the total number of negative samples.

[0126] The target false alarm rate calculation unit is used to calculate the target false alarm rate based on the number of positive samples predicted as negative samples and the number of negative samples predicted as positive samples.

[0127] The delay calculation unit is used to calculate the delay based on the time required to process the image and the total number of negative samples;

[0128] The traditional evaluation index calculation unit is used to calculate traditional evaluation indexes based on target recognition accuracy, target false alarm rate, target false alarm rate, time delay and their respective weights.

[0129] Optionally, the target adaptability index is used to reflect the intelligent recognition algorithm's ability to adapt to samples with various complex target information; the target information includes target scale, target category, target perspective, and target signal strength; the environmental adaptability index is used to reflect the intelligent recognition algorithm's ability to adapt to samples with various complex environmental information; the environmental information includes weather, climate, lighting, and natural environment.

[0130] Optional generalization evaluation metric construction modules include:

[0131] The first target recognition unit is used to construct a sample set containing various complex target information and input it into an intelligent recognition algorithm to perform target recognition, thereby obtaining the first target recognition accuracy and the first target recall rate.

[0132] The target adaptability index calculation unit is used to calculate the target adaptability index based on the first target recognition accuracy and the first target recall rate.

[0133] The second target recognition unit is used to construct a sample set containing various complex environmental information and input it into the intelligent recognition algorithm to perform target recognition, thereby obtaining the second target recognition accuracy and the second target recall rate.

[0134] An environmental adaptability index calculation unit is used to calculate environmental adaptability indexes based on the second target identification accuracy and the second target recall rate.

[0135] The generalization assessment index calculation unit is used to calculate the generalization assessment index based on the target adaptability index, the environmental adaptability index, and the corresponding weights.

[0136] Optionally, the adversarial sample security index is used to characterize the ability of the intelligent recognition algorithm to resist noise attacks; the poisoned sample security index is used to characterize the ability of the intelligent recognition algorithm to effectively remove erroneous samples during model training.

[0137] Optional, the security assessment metrics building module includes:

[0138] The adversarial test unit is used to construct an adversarial sample set and input the original test dataset and the adversarial sample set into the intelligent recognition algorithm for target recognition to obtain the test results corresponding to the original test dataset and the adversarial sample set.

[0139] The adversarial example security index calculation unit is used to compare the experimental results corresponding to the original test dataset and the adversarial example set to obtain the adversarial example security index.

[0140] The poisoning training unit is used to construct a poisoning dataset and use the poisoning dataset to train an intelligent recognition algorithm, thereby obtaining the intelligent recognition algorithm after poisoning training.

[0141] The poisoning test unit is used to construct a verification dataset that has not been poisoned and input it into the intelligent recognition algorithm trained after poisoning to identify the target and obtain the poisoning test results;

[0142] The poisoned sample safety index calculation unit is used to calculate the safety index of the poisoned sample based on the results of the poisoning test.

[0143] The security assessment index calculation unit is used to calculate the security assessment index based on the adversarial sample security index, the poisoning sample security index, and the corresponding weights.

[0144] Optionally, the inversion learning capability index is used to characterize the performance retention capability of the intelligent recognition algorithm when trained on low-complexity data and tested on high-complexity data; the few-shot learning capability index is used to characterize the performance retention capability of the intelligent recognition algorithm when trained on few-shot data and tested on a large-sample data; and the incremental learning capability index is used to characterize the performance retention capability of the intelligent recognition algorithm when trained on a small number of real samples and a large number of simulated samples and then tested on real targets.

[0145] Optional, the learning ability assessment indicator construction module includes:

[0146] Ideally, the target recognition unit is used to construct a training dataset for typical scenarios and input it into an intelligent recognition algorithm to perform target recognition and obtain the target performance change curve.

[0147] The first performance change curve calculation unit is used to gradually reduce the amount of high-complexity data in the training dataset and record the performance change curve of the intelligent recognition algorithm in real time to obtain the first performance change curve.

[0148] The inversion learning ability index calculation unit is used to calculate the ratio of the area of ​​the first performance change curve to the area of ​​the target performance change curve to obtain the inversion learning ability index.

[0149] The second performance change curve calculation unit is used to gradually reduce the number of samples in the training dataset and record the performance change curve of the intelligent recognition algorithm in real time to obtain the second performance change curve.

[0150] The few-shot learning ability index calculation unit is used to calculate the ratio of the area of ​​the second performance change curve to the area of ​​the target performance change curve to obtain the few-shot learning ability index.

[0151] The third performance change curve calculation unit is used to gradually reduce the number of real samples in the training dataset and record the performance change curve of the intelligent recognition algorithm in real time to obtain the third performance change curve.

[0152] The incremental learning ability index calculation unit is used to calculate the ratio of the area of ​​the third performance change curve to the area of ​​the target performance change curve to obtain the incremental learning ability index.

[0153] The learning ability assessment index calculation unit is used to calculate the learning ability assessment index based on the inverted learning ability index, the few-shot learning ability index, the incremental learning ability index, and the corresponding weights.

[0154] Based on this, a verification and evaluation index system for intelligent sensing algorithms is established, including 4 primary evaluation indicators and 11 secondary evaluation indicators. The intelligent sensing algorithms are verified and evaluated from four dimensions: traditional indicators, generalization, security, and learning ability. A quantitative calculation method for secondary indicators and a comprehensive calculation method for primary indicators are formed. By weighted and fused evaluation results from different dimensions, a comprehensive verification and evaluation of intelligent sensing algorithms is achieved. This solves the problem that current evaluation of intelligent sensing algorithms based solely on accuracy indicators is inaccurate and unreliable. It achieves beneficial technical effects of quantitative evaluation and analysis based on the intelligent sensing algorithm evaluation index system, and has outstanding substantive features and significant progress.

[0155] Example 3

[0156] Please refer to Figure 4This embodiment provides an electronic device including a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it, forming a method for constructing a multi-dimensional, multi-layer intelligent recognition algorithm evaluation index system at the logical level. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0157] Network interfaces, processors, and memory can be interconnected via a bus system. These buses can be categorized as address buses, data buses, control buses, etc.

[0158] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include read-only memory and random access memory, and provides instructions and data to the processor.

[0159] The processor is used to execute the program stored in the aforementioned memory, and specifically perform the following:

[0160] Step 102: Based on target recognition accuracy, target false alarm rate, target false alarm rate, and latency, construct traditional evaluation indicators;

[0161] Step 104: Construct generalization assessment indicators based on target adaptability indicators and environmental adaptability indicators;

[0162] Step 106: Construct a security assessment index based on the security index of adversarial examples and the security index of poisoned samples;

[0163] Step 108: Construct a learning ability assessment index based on the inversion learning ability index, the few-shot learning ability index, and the incremental learning ability index;

[0164] Step 110: Calculate the comprehensive evaluation result based on traditional evaluation indicators, generalization evaluation indicators, security evaluation indicators, learning ability evaluation indicators, and their corresponding weights.

[0165] A processor may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method can be completed through the processor's integrated hardware logic circuits or software instructions.

[0166] Based on the same invention, embodiments of this specification also provide a computer-readable storage medium that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform... Figures 1-2 A corresponding embodiment provides a method for constructing an evaluation index system for multidimensional and multi-layer intelligent recognition algorithms.

[0167] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-readable storage media containing computer-usable program code.

[0168] Furthermore, the specific implementation of the above system is basically similar to the method implementation, so the description is relatively simple. For relevant details, please refer to the description of the method implementation. Moreover, it should be noted that in the various modules of the system of this application, the components are logically divided according to the functions they are to perform. However, this application is not limited to this and can re-divide or combine the components as needed.

[0169] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences between it and other embodiments.

[0170] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and still achieve the desired result. Furthermore, the specific order or sequential order shown in the drawings is not necessarily required to achieve the desired result; in some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0171] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for constructing a multi-dimensional multi-layer intelligent identification algorithm evaluation index system, characterized in that, The application comprises the following steps: Based on target recognition accuracy, target missed detection rate, target false alarm rate and time delay, traditional evaluation indexes are constructed; Based on target adaptability indexes and environment adaptability indexes, generalization evaluation indexes are constructed; Based on adversarial sample safety indexes and poisoned sample safety indexes, safety evaluation indexes are constructed; Based on inversion learning ability indexes, few-shot learning ability indexes and incremental learning ability indexes, learning ability evaluation indexes are constructed; Based on traditional evaluation indexes, generalization evaluation indexes, safety evaluation indexes, learning ability evaluation indexes and corresponding weights, comprehensive evaluation results are calculated. The traditional evaluation indexes are constructed based on target recognition accuracy, target missed detection rate, target false alarm rate and time delay, which comprises the following steps: Based on the number of positive samples predicted as positive samples and negative samples respectively, target recognition accuracy is calculated; Based on the number of positive samples predicted as positive samples and the total number of negative samples, target missed detection rate is calculated; Based on the number of positive samples predicted as negative samples and the number of negative samples predicted as positive samples, target false alarm rate is calculated; Based on the time required for image processing and the total number of negative samples, time delay is calculated; Based on target recognition accuracy, target missed detection rate, target false alarm rate, time delay and their respective corresponding weights, traditional evaluation indexes are calculated. The learning ability evaluation indexes are constructed based on inversion learning ability indexes, few-shot learning ability indexes and incremental learning ability indexes, which comprises the following steps: Training data sets in typical scenarios are constructed and input into intelligent recognition algorithms for target recognition to obtain target performance change curves; The number of high complexity data in the training data set is gradually reduced, and the performance change curves of the intelligent recognition algorithm are recorded in real time to obtain the first performance change curve; The ratio of the area of the first performance change curve to the area of the target performance change curve is calculated to obtain the inversion learning ability index; The number of samples in the training data set is gradually reduced, and the performance change curves of the intelligent recognition algorithm are recorded in real time to obtain the second performance change curve; The ratio of the area of the second performance change curve to the area of the target performance change curve is calculated to obtain the few-shot learning ability index; The number of real samples in the training data set is gradually reduced, and the performance change curves of the intelligent recognition algorithm are recorded in real time to obtain the third performance change curve; The ratio of the area of the third performance change curve to the area of the target performance change curve is calculated to obtain the incremental learning ability index; Based on inversion learning ability indexes, few-shot learning ability indexes, incremental learning ability indexes and corresponding weights, learning ability evaluation indexes are calculated.

2. The method of claim 1, wherein, The target recognition accuracy is used to represent the ability of the intelligent perception algorithm to correctly identify targets. The higher the target recognition accuracy, the stronger the ability of the intelligent perception algorithm to correctly identify targets. The target missed detection rate is used to represent the ability of the intelligent perception algorithm to correctly predict the purity of negative samples. The lower the target missed detection rate, the stronger the ability of the intelligent perception algorithm to identify each type of target sample. The target false alarm rate is used to represent the ability of the intelligent perception algorithm to correctly predict the purity of positive samples. The lower the target false alarm rate, the stronger the ability of the intelligent recognition algorithm to identify positive samples. The time delay is the time required to process images of the same size.

3. The method of claim 1, wherein, The target adaptability index is used to reflect the sample adaptability of the intelligent identification algorithm to various complex target information; the target information includes target scale, target category, target view angle, and target signal strength; the environment adaptability index is used to reflect the sample adaptability of the intelligent identification algorithm to various complex environment information; the environment information includes time, climate, illumination, and natural environment.

4. The method of claim 3, wherein, The generalization evaluation index is constructed based on the target adaptability index and the environment adaptability index, and includes: A sample set containing various complex target information is input into the intelligent identification algorithm for target identification, to obtain a first target identification precision and a first target recall rate; Based on the first target identification precision and the first target recall rate, the target adaptability index is calculated and obtained; A sample set containing various complex environment information is input into the intelligent identification algorithm for target identification, to obtain a second target identification precision and a second target recall rate; Based on the second target identification precision and the second target recall rate, the environment adaptability index is calculated and obtained; Based on the target adaptability index, the environment adaptability index, and corresponding weights, the generalization evaluation index is calculated and obtained.

5. The method of claim 1, wherein, The adversarial sample safety index is used to represent the ability of the intelligent identification algorithm to resist noise attacks; the poisoned sample safety is used to represent the ability of the intelligent identification algorithm to effectively eliminate error samples during model training.

6. The method of claim 5, wherein, The safety evaluation index is constructed based on the adversarial sample safety index and the poisoned sample safety index, and includes: An adversarial sample set is constructed, and the original test data set and the adversarial sample set are respectively input into the intelligent identification algorithm for target identification, to obtain the test results corresponding to the original test data set and the adversarial sample set; The test results corresponding to the original test data set and the adversarial sample set are compared, to obtain the adversarial sample safety index; A poisoned data set is constructed, and the intelligent identification algorithm is trained using the poisoned data set, to obtain a poisoned trained intelligent identification algorithm; An unpoisoned verification data set is constructed and input into the poisoned trained intelligent identification algorithm for target identification, to obtain a poisoned test result; Based on the poisoned test result, the poisoned sample safety index is calculated and obtained; Based on the adversarial sample safety index, the poisoned sample safety index, and corresponding weights, the safety evaluation index is calculated and obtained.

7. The method of claim 1, wherein, The inversion learning ability index is used to represent the performance maintenance ability of the intelligent identification algorithm when trained under low complexity data and tested under high complexity data; the few-sample learning ability index is used to represent the performance maintenance ability of the intelligent identification algorithm when trained under few-sample data and tested under large-sample data; the incremental learning ability index is used to represent the performance maintenance ability of the intelligent identification algorithm when trained under a small amount of real samples and a large amount of simulation samples and tested on real targets.

8. A multi-dimensional multi-layer intelligent identification algorithm evaluation index system construction device, characterized in that, It includes: A traditional evaluation index construction module is configured to construct a traditional evaluation index based on target identification precision, target miss detection rate, target false alarm rate, and time delay; A generalization evaluation index construction module is configured to construct a generalization evaluation index based on a target adaptability index and an environment adaptability index; A safety evaluation index construction module is configured to construct a safety evaluation index based on an adversarial sample safety index and a poisoned sample safety index; The learning capability evaluation index construction module is configured to construct a learning capability evaluation index based on an inversion learning capability index, a few-sample learning capability index, and an incremental learning capability index. The comprehensive evaluation result calculation module is configured to calculate a comprehensive evaluation result based on the traditional evaluation index, the generalization evaluation index, the security evaluation index, the learning capability evaluation index, and corresponding weights. The traditional evaluation index is constructed based on the target recognition accuracy, the target missing detection rate, the target false alarm rate, and the time delay, and includes: The target recognition accuracy is calculated based on the number of positive samples predicted as positive samples and the number of positive samples predicted as negative samples. The target missing detection rate is calculated based on the number of positive samples predicted as positive samples and the total number of negative samples. The target false alarm rate is calculated based on the number of positive samples predicted as negative samples and the number of negative samples predicted as positive samples. The time delay is calculated based on the time required for processing the image and the total number of negative samples. The traditional evaluation index is calculated based on the target recognition accuracy, the target missing detection rate, the target false alarm rate, the time delay, and corresponding weights. The learning capability evaluation index is constructed based on the inversion learning capability index, the few-sample learning capability index, and the incremental learning capability index, and includes: A target performance change curve is obtained by constructing a training data set in a typical scenario and inputting the intelligent recognition algorithm for target recognition. A first performance change curve is obtained by gradually reducing the number of high-complexity data in the training data set and recording the performance change curve of the intelligent recognition algorithm in real time. The inversion learning capability index is obtained by calculating the ratio of the area of the first performance change curve to the area of the target performance change curve. A second performance change curve is obtained by gradually reducing the number of samples in the training data set and recording the performance change curve of the intelligent recognition algorithm in real time. The few-sample learning capability index is obtained by calculating the ratio of the area of the second performance change curve to the area of the target performance change curve. A third performance change curve is obtained by gradually reducing the number of real samples in the training data set and recording the performance change curve of the intelligent recognition algorithm in real time. The incremental learning capability index is obtained by calculating the ratio of the area of the third performance change curve to the area of the target performance change curve. The learning capability evaluation index is calculated based on the inversion learning capability index, the few-sample learning capability index, the incremental learning capability index, and corresponding weights.

Images