A confidence evaluation method for artificial intelligence image recognition results

Abstract

The present invention provides a confidence evaluation method for artificial intelligence image recognition results. The method performs quantitative confidence analysis on the sample conditions of algorithm model training and the corresponding generated recognition results. The confidence value is calculated through two strongly correlated indicators, sample conformity and algorithm capability. The obtained confidence value allows the user to clearly understand the distribution of training sample features in the iterative training process of the artificial intelligence image recognition algorithm model. The credibility of the final recognition result, thereby further improving the training samples and clarifying the specific indicators that need to be optimized in the model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence image recognition, and in particular to a confidence evaluation method for artificial intelligence image recognition results. Background Art

[0002] Since the 1970s, with the successful launch of the first land satellite, people have begun to use computers to interpret satellite remote sensing images. Initially, digital image processing software was used to perform geometric correction and position registration on satellite digital images, and on this basis, human-computer interaction was used to obtain relevant geological information from remote sensing images. The essence of this method is still visual interpretation of remote sensing images, which relies on the experience and level of business personnel in viewing images. It has not made any new breakthroughs in remote sensing image interpretation methods. With the application of artificial intelligence and deep learning technology in computer graphics and image processing, under certain conditions, the level of computer image recognition has reached the level of manual recognition, which greatly improves the efficiency of remote sensing image interpretation.

[0003] In addition, the quality of remote sensing images is also affected by many factors, such as shooting time, resolution, weather and other single factors or the combined influence of multiple factors. Under different quality conditions, there are different differences in image recognition results.

[0004] Artificial intelligence image recognition algorithm models need to be trained first and reach certain indicators, such as a detection rate of 85% or more, before they can be put into practical use. However, during the training process, there are problems such as large single files (over 2GB), long model training time (in weeks), and a lot of computing resources required. In order to improve training efficiency, it is impossible to put all sample data into training, but how much sample data and what characteristics of sample data to select can improve training efficiency while making the indicator results obtained by actual reasoning credible. This problem has no solution in the current fields of mathematics and artificial intelligence. Summary of the invention

[0005] The purpose of the present invention is to provide a confidence evaluation method for artificial intelligence image recognition results, which uses a scientific and quantitative method to evaluate the credibility of image recognition result indicators, i.e., the confidence level, in the field of artificial intelligence image recognition when the training samples and test samples are incomplete. The confidence level calculated can be used to guide how to improve the training samples and to clarify the specific indicators that need to be optimized in the model.

[0006] Therefore, the technical solution adopted in the present invention is as follows:

[0007] A confidence evaluation method for artificial intelligence image recognition results is to evaluate and analyze the confidence of the recognition results by using two indicators: sample conformity and algorithm capability of the artificial intelligence image recognition algorithm model training. The expression of confidence is:

[0008] Z=1-W1*M 算法 +W2*F 任务

[0009] Among them, Z is the confidence level; F 任务 is the sample conformity; M 算法 is the algorithm capability; W1 and W2 are weight values, the value range of W1 and W2 are both between 0 and 1, W1+W2=1, the assignment principle of W1 and W2 is that if the capability of one indicator is high, the weight value of the indicator is correspondingly small, and the weight value of the other indicator is correspondingly small.

[0010] The sample compliance calculation method is:

[0011]

[0012] n: number of inherent attributes of the sample;

[0013] Represents the number of sample features in the inherent attributes of the i-th sample in the test task scenario;

[0014] The sum of the sample features representing the inherent attributes of the i-th sample in the application scenario;

[0015] α i : The weight of the intrinsic attribute of the i-th sample,

[0016] The calculation method of the algorithm capability is:

[0017]

[0018] l: indicates the number of target categories;

[0019] Indicates the number of subcategories that a target category algorithm claims to support;

[0020] Indicates the number of all subcategories of a target category in the application scenario.

[0021] When the algorithm data is iterated, a threshold of the algorithm capability is set. When the recognition rate of a subcategory does not reach the threshold, The value is the count value supported by the subcategory minus 1; when the recognition rate of a subcategory is greater than or equal to the threshold, The value of is the count value supported by this subcategory.

[0022] The higher the confidence value, the more reliable the algorithm is, and the lower the confidence value, the less reliable the algorithm is, and it is necessary to reselect appropriate training samples and retrain.

[0023] The advantages of the present invention are:

[0024] 1. This method can be used to perform quantitative confidence analysis on the sample conditions of algorithm model training and the corresponding recognition results in the field of artificial intelligence image recognition. It can allow users to clearly understand the distribution of training sample features and the degree of credibility of the final recognition results during the iterative training of the algorithm, thereby guiding users to further optimize training samples and clarify which model indicators are weak or strong points and make corresponding optimizations.

[0025] 2. This method has been empirically verified to work well in simulation tests and actual applications, reducing the amount of computing resources required for training artificial intelligence image recognition models, reducing training time, and accelerating model iteration and optimization. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is described in detail below through specific embodiments.

[0027] In the confidence judgment method of artificial intelligence image recognition results, through quantitative analysis of confidence, users can clearly understand the distribution of training sample features and the degree of credibility of the final recognition results during the iterative training of the artificial intelligence image recognition algorithm model, thereby further optimizing the training samples and optimizing the model indicators.

[0028] The evaluation results are reflected by the confidence index value. The corresponding confidence index value has a strong correlation with the following two indicators, namely sample conformity and algorithm capability. The confidence is calculated by sample conformity and algorithm capability.

[0029] 1. Calculation of sample conformity

[0030] Sample conformity describes the difference between the sample characteristics of each sample in the test task scenario and the sample characteristics in the application scenario. The index value of sample conformity is F 任务 , the calculation formula is as follows:

[0031]

[0032] Where:

[0033] n: number of inherent attributes of the sample; i = 1, 2…, n;

[0034] Represents the number of sample features in the inherent attributes of the i-th sample in the test task scenario;

[0035] The sum of the sample features representing the inherent attributes of the i-th sample in the application scenario;

[0036] α i : The weight of the inherent attribute of the i-th sample; α i ∈[0,1], each α i Independent of each other, the weights are set to Include yes A subset of

[0037] For example, if n=5 (attributes are time, weather, season, location, and resolution);

[0038] The number of sample features in the location attribute of the test task scenario sample set is 2, then

[0039] In the application scenario, the sum of the application sample features of the location attribute in the sample set is 3, then

[0040] It is to map the inherent attribute feature difference of the i-th sample to the interval [0,1).

[0041] Sample conformity index value F 任务 The size of represents the degree of conformity of the training samples in the algorithm model.

[0042] 2. Calculation of algorithm capability

[0043] The algorithm capability describes the algorithm's ability to detect different target categories (lakes, rivers, buildings, ground vehicles, etc.) and its ability to identify subcategories under different target categories. The algorithm capability index value is M 算法 , the calculation formula is as follows:

[0044]

[0045] Where:

[0046] l: indicates the number of target categories;

[0047] Indicates the number of subcategories that a target category algorithm claims to support;

[0048] Indicates the number of all subcategories of a target category in the application scenario;

[0049] For example: It means that in the second target category, the number of subcategories supported by the algorithm is 4; It means that the number of all subcategories of the second target category in the application scenario is 6.

[0050] Initially, the algorithm capability is calculated by sub-category counts supported by the algorithm description. When the algorithm data is iterated, the algorithm capability threshold is set to A (e.g., 85%) according to the iterative data. When the recognition rate of a sub-category target does not reach the threshold, it is considered that the recognition rate of the sub-category does not meet the requirement. In the formula The value of is the count value corresponding to the subcategory minus 1, that is, At this time, it should be 4-1=3; when the recognition rate of a subcategory target is greater than or equal to 85%, then The value of remains unchanged.

[0051] 3. Confidence calculation

[0052] Confidence describes the reliability of the algorithm recognition indicator based on the distribution of sample characteristics. 算法 and F 任务 Related, expressed as:

[0053] Z=1-W1*M 算法 +W2*F 任务 (3)

[0054] The value range of Z is between 0 and 1. The higher the confidence value, the more reliable the algorithm capability index is. The lower the confidence value, the less reliable the index is. It is necessary to reselect appropriate training samples and retrain.

[0055] W1 and W2 are weight values, and the range of W1 and W2 is between 0 and 1, W1+W2=1; their assignment principle is that if the capability of a certain indicator is high, the weight of the indicator is small, and the weight of another indicator with relatively low capability is large. As shown in Table 1 below, application type refers to different algorithm usage scenarios, and the values ​​are assigned in the following two cases:

[0056] Table 1: Weight value table

[0057]

Claims

1. A confidence evaluation method for artificial intelligence image recognition results is to evaluate and analyze the confidence of the recognition results by using two indicators: sample conformity and algorithm capability of the artificial intelligence image recognition algorithm model training. The expression of confidence is: Z=1-W1*M 算法 +W2*F 任务 in, Z is the confidence level; F 任务 is the sample conformity; M 算法 is the algorithm capability; W1 and W2 are weight values, and the value range of W1 and W2 is between 0 and 1, W1+W2=1, and the principle of assigning values ​​of W1 and W2 is that if the capability of one indicator is high, the weight value of this indicator is correspondingly small, while the weight value of the other indicator is correspondingly large; The sample compliance calculation method is: Where n: number of inherent attributes of the sample; Represents the number of sample features in the inherent attributes of the i-th sample in the test task scenario; Represents the sum of the sample features of the inherent attributes of the i-th sample in the application scenario; α i : The weight of the intrinsic attribute of the i-th sample, The calculation method of the algorithm capability is: Among them, l: represents the number of target categories; Indicates the number of subcategories that a target category algorithm claims to support; Indicates the number of all subcategories of a target category in the application scenario.

2. The confidence evaluation method according to claim 1, characterized in that: When calculating the algorithm capability, when the algorithm data is iterated, a threshold of the algorithm capability is set. When the recognition rate of a certain subcategory does not reach the threshold, The value is the count value supported by the subcategory minus 1; when the recognition rate of a subcategory is greater than or equal to the threshold, The value of is the count value supported by this subcategory.

Images