Multi-dimensional multi-layer intelligent identification algorithm evaluation index system construction method

Abstract

The invention discloses a multi-dimensional multi-layer intelligent identification algorithm evaluation index system construction method, and belongs to the technical field of intelligent perception algorithm performance verification evaluation, and the method comprises the steps: constructing a traditional evaluation index based on the target identification precision, the target omission rate, the target false alarm rate and the time delay; constructing a generalization evaluation index based on the target adaptability index and the environmental adaptability index; constructing a safety evaluation index based on the confrontation sample safety index and the poisoning sample safety index; constructing a learning ability evaluation index based on the inversion learning ability index, the few-sample learning ability index and the incremental learning ability index; based on a traditional evaluation index, a generalization evaluation index, a safety evaluation index, a learning ability evaluation index and corresponding weights, a comprehensive evaluation result is obtained through calculation, so that the problem that a performance verification evaluation method of an intelligent sensing algorithm at the present stage is inaccurate, incomplete and unreliable in evaluation is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent perception algorithm performance verification and evaluation, and in particular to a method for constructing a multi-dimensional and multi-layer intelligent recognition algorithm evaluation index system. Background Art

[0002] At present, the performance verification and evaluation methods of intelligent perception algorithms usually only evaluate the performance of intelligent algorithms from the perspective of accuracy, that is, by calculating indicators such as accuracy, recall, and precision to measure the performance of intelligent perception algorithms. This accuracy-based verification and evaluation method does not take into account the data-driven characteristics of intelligent perception algorithms, lacks verification and evaluation of the generalization, security, and learning ability of intelligent algorithms, and has problems such as inaccurate, incomplete, and unreliable verification and evaluation of intelligent perception algorithms. Effective evaluation of the performance of intelligent perception algorithms is a key link in improving the performance of intelligent perception algorithms and a prerequisite for supporting the implementation of intelligent perception algorithms. Existing technologies cannot yet achieve this. Existing methods usually only evaluate intelligent perception algorithms from the perspective of accuracy, lack a comprehensive intelligent perception verification and evaluation indicator system, and are difficult to effectively evaluate the performance of intelligent perception algorithms. Summary of the invention

[0003] The purpose of the present invention is to provide a method for constructing a multi-dimensional and multi-layer intelligent recognition algorithm evaluation index system to solve the problems of inaccurate, incomplete and unreliable evaluation in the performance verification and evaluation methods of intelligent perception algorithms at this stage.

[0004] To achieve the above object, the present invention adopts the following technical solution: On the one hand, this specification provides a method for constructing a multi-dimensional and multi-layer intelligent recognition algorithm evaluation index system, including: Step 102, constructing a traditional evaluation index based on target recognition accuracy, target missed detection rate, target false alarm rate and delay; Step 104, constructing a generalization evaluation index based on the target adaptability index and the environment adaptability index; Step 106, constructing a security assessment index based on the adversarial sample security index and the poisoned sample security index; Step 108, constructing a learning ability evaluation index based on the inversion learning ability index, the few-sample learning ability index and the incremental learning ability index; Step 110, based on the traditional evaluation index, the generalization evaluation index, the safety evaluation index, the learning ability evaluation index and the corresponding weights, a comprehensive evaluation result is calculated.

[0005] On the other hand, this specification provides a device for constructing a multi-dimensional and multi-layer intelligent recognition algorithm evaluation index system, comprising: Traditional evaluation index construction module, used to construct traditional evaluation indicators based on target recognition accuracy, target missed detection rate, target false alarm rate and delay; A generalization evaluation index construction module is used to construct a generalization evaluation index based on the target adaptability index and the environment adaptability index; A security evaluation index construction module is used to construct security evaluation indicators based on adversarial sample security indicators and poisoned sample security indicators; A learning ability evaluation indicator construction module is used to construct a learning ability evaluation indicator based on an inverse learning ability indicator, a few-sample learning ability indicator, and an incremental learning ability indicator; The comprehensive evaluation result calculation module is used to calculate the comprehensive evaluation results based on traditional evaluation indicators, generalization evaluation indicators, safety evaluation indicators, learning ability evaluation indicators and corresponding weights.

[0006] Based on the above technical solution, this specification can achieve the following technical effects: This method first starts from the evaluation perspectives of traditional indicators, generalization, security, and learning ability, and constructs a verification and evaluation indicator system including 4 first-level indicators and 11 second-level indicators. It then quantitatively calculates the second-level indicators and calculates the first-level indicators by average weighted calculation. Finally, it performs weighted fusion on the evaluation results of different dimensions to achieve multi-dimensional and comprehensive verification and evaluation of the intelligent perception algorithm, support the optimal design of the intelligent perception algorithm, and improve the performance of the intelligent perception algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 The present invention is a flowchart of a method for constructing a multi-dimensional and multi-layer intelligent recognition algorithm evaluation index system in one embodiment of the present invention.

[0008] Figure 2 Schematic diagram of a minimum entropy pruning framework in one embodiment of the present invention.

[0009] Figure 3 It is a structural schematic diagram of a device for constructing a multi-dimensional and multi-layer intelligent recognition algorithm evaluation index system in one embodiment of the present invention.

[0010] Figure 4 The figure is a schematic diagram of an electronic device according to the present invention. DETAILED DESCRIPTION

[0011] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description and claims. It should be noted that the drawings are all in very simplified form and are not in precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0012] It should be noted that, in order to clearly explain the content of the present invention, the present invention specifically cites multiple embodiments to further illustrate different implementations of the present invention, wherein the multiple embodiments are enumerated rather than exhaustive. In addition, for the sake of brevity of explanation, the contents mentioned in the previous embodiments are often omitted in the subsequent embodiments. Therefore, the contents not mentioned in the subsequent embodiments can refer to the previous embodiments accordingly. Example 1

[0013] Please refer to Figure 1 , Figure 1 The figure shows a method for constructing a multi-dimensional and multi-layer intelligent recognition algorithm evaluation index system provided by this embodiment. In this embodiment, the method includes: Step 102, constructing a traditional evaluation index based on target recognition accuracy, target missed detection rate, target false alarm rate and delay; In this embodiment, the target recognition accuracy is used to characterize the ability of the intelligent perception algorithm to correctly identify the target. The higher the target recognition accuracy, the stronger the ability of the intelligent perception algorithm to correctly identify the target; the target missed detection rate is used to characterize the ability of the intelligent perception algorithm to correctly predict the purity of negative samples. The lower the target missed detection rate, the stronger the ability of the intelligent perception algorithm to identify each type of target sample; the target false alarm rate is used to characterize the ability of the intelligent perception algorithm to correctly predict the purity of positive samples. The lower the target false alarm rate, the stronger the intelligent recognition algorithm's recognition ability for positive samples; the delay is the time required to process images of the same size.

[0014] In this embodiment, one implementation of step 102 is: Step 202, based on the number of positive samples predicted as positive samples and negative samples, the target recognition accuracy is calculated; Step 204, based on the number of positive samples predicted as positive samples and the total number of negative samples, a target missed detection rate is calculated; Step 206, calculating a target false alarm rate based on the number of positive samples predicted as negative samples and the number of negative samples predicted as positive samples; Step 208, calculating the time delay based on the time required to process the image and the total number of negative samples; Step 210, based on the target recognition accuracy, the target missed detection rate, the target false alarm rate, the delay and their corresponding weights, a traditional evaluation index is calculated and obtained.

[0015] Step 104, constructing a generalization evaluation index based on the target adaptability index and the environment adaptability index; In this embodiment, the target adaptability index is used to reflect the sample adaptability of the intelligent recognition algorithm to a variety of complex target information; the target information includes target scale, target category, target viewing angle, and target signal strength; the environmental adaptability index is used to reflect the sample adaptability of the intelligent recognition algorithm to a variety of complex environmental information; the environmental information includes time of day, climate, light, and natural environment.

[0016] In this embodiment, one implementation of step 104 is: Step 302, constructing a sample set containing a variety of complex target information and inputting it into an intelligent recognition algorithm for target recognition, thereby obtaining a first target recognition accuracy and a first target recall rate; Step 304, calculating and obtaining a target adaptability index based on the first target recognition accuracy and the first target recall rate; Step 306, constructing a sample set containing a variety of complex environmental information and inputting it into an intelligent recognition algorithm for target recognition, thereby obtaining a second target recognition accuracy and a second target recall rate; Step 308, calculating and obtaining an environmental adaptability index based on the second target recognition accuracy and the second target recall rate; Step 310: Based on the target adaptability index, the environment adaptability index and the corresponding weights, a generalization evaluation index is calculated.

[0017] Step 106, constructing a security assessment index based on the adversarial sample security index and the poisoned sample security index; In this embodiment, the adversarial sample security index is used to characterize the ability of the intelligent recognition algorithm to resist noise attacks; the poisoned sample security is used to characterize the ability of the intelligent recognition algorithm to effectively eliminate erroneous samples during the model training process.

[0018] In this embodiment, one implementation of step 106 is: Step 402, construct an adversarial sample set, and input the original test data set and the adversarial sample set into the intelligent recognition algorithm for target recognition, and obtain the test results corresponding to the original test data set and the adversarial sample set; Step 404, comparing the test results corresponding to the original test data set and the adversarial sample set to obtain the adversarial sample security index; Step 406, constructing a poisoning data set and using the poisoning data set to train an intelligent recognition algorithm to obtain an intelligent recognition algorithm after poisoning training; Step 408, constructing a validation data set that has not been poisoned and inputting it into the intelligent recognition algorithm trained with poisoning to perform target recognition and obtain the poisoning test results; Step 410, based on the poisoning test results, calculate and obtain the poisoning sample safety index; Step 412, based on the adversarial sample security index, the poisoned sample security index and the corresponding weights, calculate and obtain the security assessment index.

[0019] Step 108, constructing a learning ability evaluation index based on the inversion learning ability index, the few-sample learning ability index and the incremental learning ability index; In this embodiment, the inversion learning ability index is used to characterize the performance retention ability of the intelligent identification algorithm when it is trained with low-complexity data and tested with high-complexity data; the few-sample learning ability index is used to characterize the performance retention ability of the intelligent recognition algorithm when it is trained with few-sample data and tested with a large amount of sample data; the incremental learning ability index is used to characterize the performance retention ability of the intelligent recognition algorithm when it is tested on real targets after being trained with a small number of real samples and a large number of simulated samples.

[0020] In this embodiment, one implementation of step 108 is: Step 502, constructing a training data set under a typical scenario and inputting it into an intelligent recognition algorithm to perform target recognition, and obtaining a target performance change curve; Step 504, gradually reducing the amount of high-complexity data in the training data set, and recording the performance change curve of the intelligent recognition algorithm in real time to obtain a first performance change curve; Step 506, calculating the ratio of the area of ​​the first performance change curve to the area of ​​the target performance change curve to obtain an inversion learning ability index; Step 508, gradually reducing the number of samples in the training data set, and recording the performance change curve of the intelligent recognition algorithm in real time to obtain a second performance change curve; Step 510, calculating the ratio of the area of ​​the second performance change curve to the area of ​​the target performance change curve to obtain a few-sample learning ability index; Step 512, gradually reducing the number of real samples in the training data set, and recording the performance change curve of the intelligent recognition algorithm in real time to obtain a third performance change curve; Step 514, calculating the ratio of the area of ​​the third performance change curve to the area of ​​the target performance change curve to obtain an incremental learning ability index; Step 516, based on the inverse learning ability index, the few-sample learning ability index, the incremental learning ability index and the corresponding weights, calculate and obtain the learning ability evaluation index.

[0021] Step 110, based on the traditional evaluation index, the generalization evaluation index, the safety evaluation index, the learning ability evaluation index and the corresponding weights, a comprehensive evaluation result is calculated.

[0022] Specifically, a method for constructing a multi-dimensional and multi-layer intelligent recognition algorithm evaluation index system is described in detail. Figure 1The system consists of four first-level evaluation indicators, namely traditional indicators, generalization, security, and learning ability, and corresponding 11 second-level evaluation indicators, including the following steps: The first step is to design and construct the traditional indicators and first-level evaluation indicators The traditional evaluation index consists of four secondary evaluation indicators: target recognition accuracy, missed detection rate, false alarm rate, and delay. The secondary evaluation indicators can be quantitatively calculated by weighted fusion to calculate the traditional primary evaluation indicators.

[0023] (1) Target recognition accuracy: reflects the ability of the intelligent recognition model to correctly identify the target. The higher the target recognition rate, the stronger the recognition algorithm's ability to identify the target. The specific calculation formula for each category is:

[0024] TP represents the number of positive samples predicted as positive samples, FP represents the number of positive samples predicted as negative samples, and i=1,2,…,K represents the number of categories.

[0025] The overall target recognition accuracy is calculated as follows:

[0026] (2) Missed detection rate: reflects the ability of the intelligent recognition model to correctly predict the purity of negative samples. The lower the missed detection rate, the stronger the recognition algorithm's ability to identify each type of target sample. The missed detection rate for each category is calculated as follows:

[0027] N represents the total number of negative samples.

[0028] The total missed detection rate is calculated as follows:

[0029] (3) False alarm rate: reflects the ability of the intelligent recognition model to correctly predict the purity of positive samples. The lower the false alarm rate, the stronger the recognition algorithm's ability to recognize positive samples. The false alarm rate for each category is calculated as follows:

[0030] TN represents the number of negative samples predicted as positive samples.

[0031] The total false alarm rate is calculated as follows:

[0032] (4) Latency: The time required to process images of the same size is used as the latency evaluation indicator, and its calculation method is as follows:

[0033] (5) Weighted fusion calculation of traditional first-level evaluation indicators

[0034] for Indicators correspond to weights.

[0035] The second step is to design and construct generalization evaluation indicators The generalization evaluation index consists of two secondary evaluation indicators: target adaptability and environmental adaptability. The secondary evaluation indicators can be quantitatively calculated, and the generalization primary evaluation index is calculated through weighted fusion.

[0036] (1) Target adaptability: reflects the ability of the intelligent recognition model to adapt to samples of various complexities under different conditions such as target scale, target category, target perspective, and signal strength.

[0037] Taking target scale and type adaptability as an example, a sample set containing multiple types and target scales is constructed to record key performance indicators such as recognition accuracy and recall rate of the intelligent model on different target types and target scales.

[0038] (2) Environmental adaptability: reflects the ability of the intelligent recognition model to adapt to samples in different scenarios such as time of day, weather, and natural environment.

[0039] Construct multi-environment data sets under different environmental conditions such as lighting and background, and record key performance indicators such as accuracy and recall rate of intelligent models on multi-environment data sets.

[0040] (3) Weighted fusion calculation generalization level 1 evaluation index

[0041] The third step is to design and construct security evaluation indicators The security evaluation index consists of two secondary evaluation indicators: adversarial sample security and poisoned sample security. The secondary evaluation indicators can be quantitatively calculated, and the primary security evaluation indicators are calculated through weighted fusion.

[0042] (1) Adversarial sample security: reflects the ability of intelligent recognition models to resist noise attacks.

[0043] Build a set of generative adversarial samples, record key performance indicators such as accuracy and recall of the intelligent model on the adversarial sample set, and evaluate the adversarial sample security of the model by comparing the performance differences between the model on the original test data set and the adversarial samples.

[0044] (2) Safety of poisoned samples: reflects the ability to effectively eliminate erroneous samples during the training process of the intelligent recognition model.

[0045] Construct a poisoned dataset, use it to train the intelligent model, and record key performance indicators such as accuracy and recall of the intelligent model in the non-poisoned validation dataset.

[0046] (3) Weighted fusion computing security first-level evaluation index

[0047] Step 4: Design and construct learning ability assessment indicators The learning ability evaluation index consists of five secondary evaluation indicators: training efficiency, inversion learning ability, few-sample learning ability, memory ability, and incremental learning ability. The secondary evaluation indicators can be quantitatively calculated, and the first-level evaluation indicators of learning ability are calculated through weighted fusion.

[0048] (1) Inversion learning ability: reflects the ability of the intelligent recognition model to maintain performance during low-complexity data training and high-complexity data testing.

[0049] Construct training data sets for typical scenarios, record the changes in intelligent model performance as the high-complexity data in the training set decreases, and calculate the ratio of the performance curve area to the performance curve area under ideal conditions.

[0050] (2) Few-sample learning ability: reflects the ability of the intelligent recognition model to maintain performance when trained with a small amount of sample data and tested with a large amount of sample data.

[0051] Construct training data sets under typical scenarios, record the changes in intelligent model performance as the number of training set samples changes, and calculate the ratio of the performance curve area to the performance curve area under ideal conditions.

[0052] (3) Incremental learning capability: reflects the ability of the intelligent recognition model to maintain performance on real targets when trained with a small number of real samples and a large number of simulated samples.

[0053] Construct a training data set for a typical scenario, record the performance changes of the intelligent model as the number of real samples in the training set changes, obtain the area between the curve and the training cycle axis through integration, and calculate the ratio of the performance curve area to that in the ideal case.

[0054] (4) First-level evaluation index of weighted fusion computing learning ability

[0055] The fifth step is to calculate the comprehensive evaluation index The four first-level evaluation indicators, namely traditional indicators, generalization, security, and learning ability, are weighted to obtain a comprehensive evaluation result.

[0056] The formula is as follows:

[0057] in, They are the traditional indicator evaluation results, generalization evaluation results, safety evaluation results, and learning ability evaluation results. They are the weights of traditional indicator evaluation results, generalization evaluation results, safety evaluation results, and learning ability evaluation results.

[0058] In summary, this method establishes a verification and evaluation index system for intelligent perception algorithms, including 4 first-level evaluation indicators and 11 second-level evaluation indicators. It verifies and evaluates the intelligent perception algorithm from four dimensions: traditional indicators, generalization, security, and learning ability. It forms a quantitative calculation method for the second-level indicators and a comprehensive calculation method for the first-level indicators. By weighted fusion of the evaluation results of different dimensions, it realizes the comprehensive verification and evaluation of the intelligent perception algorithm, solves the current problem of inaccurate and unreliable evaluation of the intelligent perception algorithm based on precision indicators, and achieves the beneficial technical effect of quantitative evaluation and analysis based on the intelligent perception algorithm evaluation index system, which has outstanding substantive characteristics and significant progress.

[0059] Example 2 Please refer to Figure 3 , Figure 3 The figure shows a device for constructing a multi-dimensional and multi-layer intelligent recognition algorithm evaluation index system provided by this embodiment. In this embodiment, the device includes: Traditional evaluation index construction module, used to construct traditional evaluation indicators based on target recognition accuracy, target missed detection rate, target false alarm rate and delay; A generalization evaluation index construction module is used to construct a generalization evaluation index based on the target adaptability index and the environment adaptability index; A security evaluation index construction module is used to construct security evaluation indicators based on adversarial sample security indicators and poisoned sample security indicators; A learning ability evaluation indicator construction module is used to construct a learning ability evaluation indicator based on an inverse learning ability indicator, a few-sample learning ability indicator, and an incremental learning ability indicator; The comprehensive evaluation result calculation module is used to calculate the comprehensive evaluation results based on traditional evaluation indicators, generalization evaluation indicators, safety evaluation indicators, learning ability evaluation indicators and corresponding weights.

[0060] Optionally, the target recognition accuracy is used to characterize the ability of the intelligent perception algorithm to correctly identify the target. The higher the target recognition accuracy, the stronger the ability of the intelligent perception algorithm to correctly identify the target; the target missed detection rate is used to characterize the ability of the intelligent perception algorithm to correctly predict the purity of negative samples. The lower the target missed detection rate, the stronger the ability of the intelligent perception algorithm to identify each type of target sample; the target false alarm rate is used to characterize the ability of the intelligent perception algorithm to correctly predict the purity of positive samples. The lower the target false alarm rate, the stronger the intelligent recognition algorithm's recognition ability of positive samples; the delay is the time required to process images of the same size.

[0061] Optional, traditional evaluation indicator building blocks include: A target recognition accuracy calculation unit, used for calculating the target recognition accuracy based on the number of positive samples predicted as positive samples and negative samples respectively; A target missed detection rate calculation unit, used to calculate the target missed detection rate based on the number of positive samples predicted as positive samples and the total number of negative samples; A target false alarm rate calculation unit, used to calculate a target false alarm rate based on the number of positive samples predicted as negative samples and the number of negative samples predicted as positive samples; A time delay calculation unit, used for calculating the time delay based on the time required for processing the image and the total number of negative samples; The traditional evaluation index calculation unit is used to calculate and obtain the traditional evaluation index based on the target recognition accuracy, the target missed detection rate, the target false alarm rate, the delay and their corresponding weights.

[0062] Optionally, the target adaptability index is used to reflect the sample adaptability of the intelligent recognition algorithm to a variety of complex target information; the target information includes target scale, target category, target viewing angle, and target signal strength; the environmental adaptability index is used to reflect the sample adaptability of the intelligent recognition algorithm to a variety of complex environmental information; the environmental information includes time of day, climate, light and natural environment.

[0063] Optional, generalization evaluation metric building blocks include: A first target recognition unit is used to construct a sample set containing a variety of complex target information and input it into an intelligent recognition algorithm for target recognition, thereby obtaining a first target recognition accuracy and a first target recall rate; A target adaptability index calculation unit, used to calculate and obtain a target adaptability index based on a first target recognition accuracy and a first target recall rate; The second target recognition unit is used to construct a sample set containing a variety of complex environmental information and input it into the intelligent recognition algorithm for target recognition to obtain the second target recognition accuracy and the second target recall rate; An environmental adaptability index calculation unit, used to calculate and obtain the environmental adaptability index based on the second target recognition accuracy and the second target recall rate; The generalization evaluation index calculation unit is used to calculate the generalization evaluation index based on the target adaptability index, the environment adaptability index and the corresponding weights.

[0064] Optionally, the adversarial sample security index is used to characterize the ability of the intelligent recognition algorithm to resist noise attacks; the poisoned sample security is used to characterize the ability of the intelligent recognition algorithm to effectively eliminate erroneous samples during the model training process.

[0065] Optionally, the security assessment indicator building blocks include: The adversarial test unit is used to construct an adversarial sample set, and input the original test data set and the adversarial sample set into the intelligent recognition algorithm for target recognition, and obtain the test results corresponding to the original test data set and the adversarial sample set; An adversarial sample security index calculation unit, used to compare the test results corresponding to the original test data set and the adversarial sample set to obtain the adversarial sample security index; A poisoning training unit, used to construct a poisoning data set and use the poisoning data set to train an intelligent recognition algorithm to obtain an intelligent recognition algorithm after poisoning training; The poisoning test unit is used to construct a validation data set that has not been poisoned and input it into the intelligent recognition algorithm trained by poisoning to perform target recognition and obtain the poisoning test results; A poisoning sample safety index calculation unit, used to calculate the poisoning sample safety index based on the poisoning test results; The security evaluation index calculation unit is used to calculate the security evaluation index based on the adversarial sample security index, the poisoned sample security index and the corresponding weights.

[0066] Optionally, the inversion learning ability index is used to characterize the performance retention ability of the intelligent discrimination algorithm when trained with low-complexity data and tested with high-complexity data; the few-sample learning ability index is used to characterize the performance retention ability of the intelligent recognition algorithm when trained with few-sample data and tested with a large amount of sample data; the incremental learning ability index is used to characterize the performance retention ability of the intelligent recognition algorithm when tested on real targets after training with a small number of real samples and a large number of simulated samples.

[0067] Optionally, the learning ability assessment indicator building blocks include: Ideally, the target recognition unit is used to construct a training data set under typical scenarios and input it into the intelligent recognition algorithm for target recognition to obtain the target performance change curve; A first performance change curve calculation unit, used to gradually reduce the amount of high-complexity data in the training data set, and record the performance change curve of the intelligent recognition algorithm in real time to obtain a first performance change curve; An inversion learning ability index calculation unit, used to calculate the ratio of the area of ​​the first performance change curve to the area of ​​the target performance change curve to obtain an inversion learning ability index; A second performance change curve calculation unit, used to gradually reduce the number of samples in the training data set, and record the performance change curve of the intelligent recognition algorithm in real time to obtain a second performance change curve; A few-sample learning ability index calculation unit, used to calculate the ratio of the area of ​​the second performance change curve to the area of ​​the target performance change curve to obtain the few-sample learning ability index; A third performance change curve calculation unit, used to gradually reduce the number of real samples in the training data set, and record the performance change curve of the intelligent recognition algorithm in real time to obtain a third performance change curve; an incremental learning capability index calculation unit, used to calculate the ratio of the area of ​​the third performance change curve to the area of ​​the target performance change curve to obtain an incremental learning capability index; The learning ability evaluation index calculation unit is used to calculate the learning ability evaluation index based on the inversion learning ability index, the few-sample learning ability index, the incremental learning ability index and the corresponding weights.

[0068] Based on this, a verification and evaluation index system for intelligent perception algorithms is established, including 4 first-level evaluation indicators and 11 second-level evaluation indicators. The intelligent perception algorithms are verified and evaluated from four dimensions: traditional indicators, generalization, security, and learning ability. A quantitative calculation method for the second-level indicators and a comprehensive calculation method for the first-level indicators are formed. By weighted fusion of the evaluation results of different dimensions, a comprehensive verification and evaluation of the intelligent perception algorithms is achieved, which solves the problem of inaccurate and unreliable evaluation of intelligent perception algorithms that currently only relies on precision indicators. The beneficial technical effect of quantitative evaluation and analysis based on the intelligent perception algorithm evaluation index system is achieved, which has outstanding substantive characteristics and significant progress.

[0069] Example 3 Please refer to Figure 4, this embodiment provides an electronic device, which includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a method for constructing a multi-dimensional and multi-layer intelligent recognition algorithm evaluation index system at the logical level. Of course, in addition to software implementation methods, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0070] The network interface, processor and memory can be connected to each other through a bus system. The above bus can be divided into an address bus, a data bus, a control bus, etc.

[0071] The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor.

[0072] The processor is used to execute the program stored in the above memory, and specifically perform: Step 102, constructing a traditional evaluation index based on target recognition accuracy, target missed detection rate, target false alarm rate and delay; Step 104, constructing a generalization evaluation index based on the target adaptability index and the environment adaptability index; Step 106, constructing a security assessment index based on the adversarial sample security index and the poisoned sample security index; Step 108, constructing a learning ability evaluation index based on the inversion learning ability index, the few-sample learning ability index and the incremental learning ability index; Step 110, based on the traditional evaluation index, the generalization evaluation index, the safety evaluation index, the learning ability evaluation index and the corresponding weights, a comprehensive evaluation result is calculated.

[0073] The processor may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method may be completed by an integrated logic circuit of the processor hardware or by instructions in the form of software.

[0074] Based on the same invention, the embodiment of this specification also provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes Figure 1-Figure 2 The corresponding embodiment provides a method for constructing a multi-dimensional and multi-layer intelligent recognition algorithm evaluation index system.

[0075] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-readable storage media containing computer-usable program code.

[0076] In addition, for the above-mentioned specific implementation of the system, since it is basically similar to the method implementation, the description is relatively simple, and the relevant parts can be referred to the partial description of the method implementation. Moreover, it should be noted that in each module of the system of the present application, the components therein are logically divided according to the functions to be implemented, but the present application is not limited thereto, and the components can be re-divided or combined as needed.

[0077] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between it and other embodiments.

[0078] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the specific order or sequential order shown in the process depicted in the drawings is not necessarily required to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0079] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for constructing a multi-dimensional and multi-layer intelligent recognition algorithm evaluation index system, characterized in that: include: Construct traditional evaluation indicators based on target recognition accuracy, target missed detection rate, target false alarm rate and delay; Based on the target adaptability index and the environmental adaptability index, a generalization evaluation index is constructed; Construct security evaluation indicators based on adversarial sample security indicators and poisoned sample security indicators; Construct learning ability evaluation indicators based on inverse learning ability indicators, few-sample learning ability indicators and incremental learning ability indicators; Based on traditional evaluation indicators, generalization evaluation indicators, safety evaluation indicators, learning ability evaluation indicators and corresponding weights, the comprehensive evaluation results are calculated.

2. The method according to claim 1, characterized in that The target recognition accuracy is used to characterize the ability of the intelligent perception algorithm to correctly identify the target. The higher the target recognition accuracy, the stronger the ability of the intelligent perception algorithm to correctly identify the target; the target missed detection rate is used to characterize the ability of the intelligent perception algorithm to correctly predict the purity of negative samples. The lower the target missed detection rate, the stronger the ability of the intelligent perception algorithm to identify each type of target sample; the target false alarm rate is used to characterize the ability of the intelligent perception algorithm to correctly predict the purity of positive samples. The lower the target false alarm rate, the stronger the intelligent recognition algorithm's recognition ability for positive samples; the delay is the time required to process images of the same size.

3. The method according to claim 2, characterized in that The traditional evaluation indicators constructed based on target recognition accuracy, target missed detection rate, target false alarm rate and delay include: The target recognition accuracy is calculated based on the number of positive samples predicted as positive samples and negative samples respectively; Based on the number of positive samples predicted as positive samples and the total number of negative samples, the target missed detection rate is calculated; Based on the number of positive samples predicted as negative samples and the number of negative samples predicted as positive samples, the target false alarm rate is calculated; The time delay is calculated based on the time required to process the image and the total number of negative samples; Traditional evaluation indicators are calculated based on target recognition accuracy, target missed detection rate, target false alarm rate, delay and their corresponding weights.

4. The method according to claim 1, characterized in that: The target adaptability index is used to reflect the sample adaptability of the intelligent recognition algorithm to a variety of complex target information; the target information includes target scale, target category, target viewing angle, and target signal strength; the environmental adaptability index is used to reflect the sample adaptability of the intelligent recognition algorithm to a variety of complex environmental information; the environmental information includes time of day, climate, light, and natural environment.

5. The method according to claim 4, characterized in that The generalization evaluation index constructed based on the target adaptability index and the environment adaptability index includes: Construct a sample set containing a variety of complex target information and input it into the intelligent recognition algorithm for target recognition to obtain the first target recognition accuracy and the first target recall rate; Based on the first target recognition accuracy and the first target recall rate, a target adaptability index is calculated; Construct a sample set containing a variety of complex environmental information and input it into the intelligent recognition algorithm for target recognition, and obtain the second target recognition accuracy and second target recall rate; Based on the second target recognition accuracy and the second target recall rate, the environmental adaptability index is calculated; Based on the target adaptability index, the environmental adaptability index and the corresponding weights, the generalization evaluation index is calculated.

6. The method according to claim 1, characterized in that The adversarial sample security index is used to characterize the ability of the intelligent recognition algorithm to resist noise attacks; the poisoned sample security is used to characterize the ability of the intelligent recognition algorithm to effectively eliminate erroneous samples during the model training process.

7. The method according to claim 6, characterized in that The security evaluation index constructed based on the adversarial sample security index and the poisoned sample security index includes: Construct an adversarial sample set, and input the original test data set and the adversarial sample set into the intelligent recognition algorithm for target recognition, and obtain the experimental results corresponding to the original test data set and the adversarial sample set; Compare the test results of the original test data set and the adversarial sample set to obtain the adversarial sample security index; Construct a poisoning data set and use it to train an intelligent recognition algorithm to obtain an intelligent recognition algorithm after poisoning training; Construct a validation data set that has not been poisoned and input it into the intelligent recognition algorithm trained with poisoning to perform target recognition and obtain the poisoning test results; Based on the results of the poisoning test, the safety index of the poisoning sample is calculated; Based on the adversarial sample security index, poisoned sample security index and the corresponding weights, the security evaluation index is calculated.

8. The method according to claim 1, characterized in that: The inverse learning ability index is used to characterize the performance retention ability of the intelligent identification algorithm when it is trained with low-complexity data and tested with high-complexity data; the few-sample learning ability index is used to characterize the performance retention ability of the intelligent recognition algorithm when it is trained with few-sample data and tested with a large amount of sample data; the incremental learning ability index is used to characterize the performance retention ability of the intelligent recognition algorithm when it is tested on real targets after being trained with a small number of real samples and a large number of simulated samples.

9. The method according to claim 8, characterized in that The construction of learning ability evaluation indicators based on the inversion learning ability indicator, the few-sample learning ability indicator and the incremental learning ability indicator includes: Construct training data sets under typical scenarios and input them into intelligent recognition algorithms for target recognition to obtain target performance change curves; Gradually reduce the amount of high-complexity data in the training data set, and record the performance change curve of the intelligent recognition algorithm in real time to obtain a first performance change curve; Calculating the ratio of the area of ​​the first performance change curve to the area of ​​the target performance change curve to obtain an inversion learning ability index; Gradually reduce the number of samples in the training data set, and record the performance change curve of the intelligent recognition algorithm in real time to obtain a second performance change curve; Calculate the ratio of the area of ​​the second performance change curve to the area of ​​the target performance change curve to obtain a few-sample learning ability index; Gradually reduce the number of real samples in the training data set, and record the performance change curve of the intelligent recognition algorithm in real time to obtain a third performance change curve; Calculating the ratio of the area of ​​the third performance change curve to the area of ​​the target performance change curve to obtain an incremental learning ability index; Based on the inverse learning ability index, the few-sample learning ability index, the incremental learning ability index and the corresponding weights, the learning ability evaluation index is calculated.

10. A device for constructing a multi-dimensional and multi-layer intelligent recognition algorithm evaluation index system, characterized in that: include: Traditional evaluation index construction module, used to construct traditional evaluation indicators based on target recognition accuracy, target missed detection rate, target false alarm rate and delay; A generalization evaluation index construction module is used to construct a generalization evaluation index based on the target adaptability index and the environment adaptability index; A security evaluation index construction module is used to construct security evaluation indicators based on adversarial sample security indicators and poisoned sample security indicators; A learning ability evaluation indicator construction module is used to construct a learning ability evaluation indicator based on an inverse learning ability indicator, a few-sample learning ability indicator, and an incremental learning ability indicator; The comprehensive evaluation result calculation module is used to calculate the comprehensive evaluation results based on traditional evaluation indicators, generalization evaluation indicators, safety evaluation indicators, learning ability evaluation indicators and corresponding weights.

Images