Efficient estimation method for picture classification confidence
By combining multi-directional benchmark sampling and Gaussian sphere sampling, the resource and time overhead issues of image classification confidence estimation in label-only scenarios are solved, and efficient and accurate confidence calculation is achieved.
Patent Information
- Application Number
- CN202410292958.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-14
- Publication Date
- 2025-09-16
AI Technical Summary
Existing technologies for estimating image classification confidence in label-only scenarios suffer from excessive resource and time overhead. In particular, excessive number of sampling points in high-dimensional space leads to a sharp increase in cost, and the randomness and sparsity of the Gaussian sampling method lead to low efficiency.
A multi-directional benchmark sampling method is adopted to generate several orthogonal benchmark sampling directions, use Gaussian sphere sampling points to judge the classification accuracy, and combine binary search and Minkowski distance calculation to reduce the number of sampling times and improve accuracy.
It achieves efficient estimation of image classification confidence in label-only scenarios, reduces the number of sampling times, saves resources, while maintaining calculation accuracy in high-dimensional space and improving detection efficiency.
Smart Images
Figure CN120656111A_ABST
Abstract
Description
Technical Field
[0001] The present invention proposes an efficient estimation method for image classification confidence in label-only scenarios, which can be used to provide auxiliary conditions for user decision-making and model generation quality detection, and belongs to the field of artificial intelligence. Background Art
[0002] With the advancement of artificial intelligence and computer vision, the demand for image detection is growing. Image detection can be applied to areas such as autonomous driving and object recognition. In real-world applications, image detection results are often used to make decisions. However, with the advancement of artificial intelligence and computer vision, technologies such as image recognition and object detection are being applied in a variety of scenarios, including everyday life, industry, and transportation. In these scenarios, the output of AI models is used as a criterion for determining whether to execute a step or decision. However, some model providers do not provide users with the posterior vector of the model output, leaving users only with the image classification label. Therefore, when making decisions, users cannot determine their specific confidence level and struggle to assess the risks of different decisions. Because different users have different risk tolerances, and more conservative decision makers are reluctant to make high-risk decisions, there is an urgent need for a method that can determine model classification confidence based on the model classification label, or in label-only scenarios.
[0003] Classification confidence has the following two main functions:
[0004] 1) Serves as a proxy for confidence in decision-making. The essence of model classification confidence is the posterior vector of the model's logit vector (representing the probability of an event occurring; odds is the ratio of the probability of an event occurring to the probability of it not occurring, and the logit vector is the natural logarithm of odds) mapped by the softmax function. The softmax function maps the logit vector to a range between (0, 1), and the sum of the outputs is 1. For a classification a that the model is relatively confident about, the value corresponding to that classification in the posterior vector is larger than that for other less confident classifications. Therefore, this vector can be directly regarded as the probability of each output category. When making a decision, the size of the corresponding score in this vector can be used as the basis for the decision.
[0005] 2) The confidence level of a large number of samples can reflect the quality of model training. In AI model evaluation, it is generally believed that if the confidence level of a large number of samples is low, the learned sample features are not clear and universal enough. This is because the model has not learned clear distinguishing features between categories, resulting in low discrimination between different categories. Low discrimination can lead to similar confidence levels across multiple categories: the sum of the confidence levels is 1, and when multiple categories are similar, each category will have a low confidence level. Therefore, sampling multiple sample points and measuring their confidence levels can be used to measure the effectiveness of model training.
[0006] Existing methods for estimating classification confidence mainly include adversarial sample methods and Gaussian sampling methods. Both of them perturb the target sample, measure the approximate distance between the target sample and the decision boundary, and use this distance to represent the classification confidence.
[0007] Adversarial example methods generate adversarial examples with a small distance to the target image and are classified into the wrong category. The distance between the adversarial example and the target example is then used as an approximate estimate of the distance from the target image to the decision boundary, and this approximate distance is used as an estimate of the confidence level. Generally, the farther the target example is from the decision boundary, the greater the model's confidence in its classification result. The method follows these steps: First, an image of a different category from the target image is found as the adversarial example. Then, a binary search is used between the two images to find an image close to the classification boundary. At the classification boundary, Monte Carlo sampling is used to estimate the gradient at that location. Finally, the adversarial example is updated in the direction of the updated gradient. After a large number of iterative updates, the adversarial example with the smallest distance (closeness) to the target image is found. The distance from this adversarial example to the target point is used as the distance from the target image to the boundary, which is also an estimate of the confidence level. The key feature of this method is to find the optimal perturbation direction, that is, the direction that produces the shortest distance from the target image to the decision boundary. This shortest distance is used as the estimate of the confidence level.
[0008] The Gaussian sampling method performs Gaussian sampling around a target sample to obtain nearby points. The sampling of nearby points is completely controlled by the parameters of the Gaussian distribution. These nearby points are fed into the target model for classification. The classification accuracy of the nearby points is approximated as the distance from the target sample to the model boundary, and the confidence level is expressed as an approximate value of this distance. In other words, higher accuracy indicates that the point is farther from the boundary, and the higher the confidence level.
[0009] Existing adversarial sample methods and Gaussian sampling methods both have certain shortcomings.
[0010] Adversarial samples use Monte Carlo sampling when estimating gradients, which requires calling the model for prediction thousands to tens of thousands of times, consuming a lot of resources and time, and is not suitable for scenarios that require rapid detection.
[0011] The Gaussian sampling method is random when sampling, meaning the direction of the sampling point within the target sample is uncertain, as is the distance. Therefore, this method requires a large number of samples to ensure that the classification accuracy of the sampling points approximates the confidence level. However, for higher-dimensional targets, Gaussian sampling points become sparse, and further increasing the number of sampling points leads to a sharp increase in cost. Furthermore, the spatial distribution of Gaussian sampling points can be approximated as a hypersphere, with the sampling points very densely distributed near the center of the sphere. These sampling points, which are closer to the target sample, are often classified by the target model as the same category as the target image, making it difficult to provide distance information between the target image and the decision boundary. This makes the boundary determination process very inefficient.
[0012] Both of these approaches share a common problem: excessive sampling points lead to excessive resource consumption in terms of time and computing power. This is especially true in high-dimensional spaces, where higher dimensions mean greater overhead, ultimately making the cost unbearable. Summary of the Invention
[0013] In response to the problems existing in the prior art, the purpose of the present invention is to provide an efficient estimation method for image classification confidence in label-only scenarios, which can greatly reduce the overhead of time and hardware resources.
[0014] The technical solution of the present invention includes:
[0015] A method for quickly detecting image classification confidence in a label-only scenario, characterized by comprising the following steps:
[0016] Step 1: Determine several reference sampling directions of the image to be detected;
[0017] Step 2: Generate a reference point according to each reference sampling direction;
[0018] Step 3: For all reference points, a Gaussian sphere is sampled around each reference point;
[0019] Step 4: Use the classification accuracy of the sampling points on the Gaussian sphere as the discrimination index to determine the sampling distance of the corresponding reference point; where L i is the sampling distance of the reference point corresponding to the reference sampling direction i; i = 1 to n, n is the number of reference sampling directions;
[0020] Step 5: Synthesize the sampling distances of each reference point to obtain an approximate distance, and map the approximate distance to the confidence score of the image to be detected.
[0021] Furthermore, the number of reference sampling directions is determined according to the usage scenario. When accurate detection is required, the number of reference sampling directions can be appropriately increased.
[0022] Furthermore, the reference sampling directions are required to be orthogonal to each other. When calculating distance, the distances in different directions are linearly independent. Calculating distance when the reference directions are orthogonal can avoid redundant information caused by non-orthogonal directions.
[0023] Furthermore, the method for determining the reference sampling direction is as follows: generating a zero matrix of the same size as the image to be detected; changing the value of each position on the zero matrix to 1 with a certain probability p, changing to -1 with a probability p, and remaining unchanged with a probability 1-2p, and calling the changed zero matrix a sampling matrix; using the number of elements in the sampling matrix as the number of dimensions, and the element values as the values under the corresponding dimensions, the coordinate points uniquely determined by the sampling matrix are called matrix points; using the point determined by the zero matrix as the coordinate origin, and the direction from the coordinate origin to the matrix point as the reference sampling direction.
[0024] Furthermore, for a reference sampling direction i, the position of the image to be detected is taken as the origin, and a sampling point with a target distance value γ from the origin is selected in the direction of the reference sampling direction i as the farther point. The image category corresponding to the farther point is different from the image category corresponding to the origin; then the binary search algorithm is used to iteratively search for the middle point between the origin and the farther point. When the found middle point i m When approaching the classification decision boundary of the image classification model, the intermediate point i m As the reference point corresponding to the reference sampling direction i.
[0025] Furthermore, the target distance value γ is selected as follows: first, the maximum value max among the pixel values on all channels of the target image is selected, and then the far point coefficient α is determined, γ=α*max.
[0026] Furthermore, the far-point coefficient α should be selected between 10 and 100, and 10 can usually be selected.
[0027] Furthermore, for the generation of the Gaussian sphere, the relative orientation between the sampling points and the reference point in the sphere is determined by the Gaussian sampling, while the distance between the sampling points and the reference point is determined by the simulation accuracy. The higher the simulation accuracy required, the more accurate the distance approximation. After obtaining the Gaussian sampling points, the sphere radius r is uniformly scaled to a size predefined by the simulation accuracy.
[0028] Furthermore, the reference point i corresponding to the reference sampling direction i is obtained d Sampling distance L i The method is:
[0029] 41) For reference point i dThe corresponding Gaussian sphere is sampled, and the image corresponding to each sampling point is input into the image classification model to obtain the category of each sampling point. If the classification accuracy acc is within the set threshold range (λ, 1-λ), the sampling distance L i is the reference point i d Minkowski distance from the origin;
[0030] 42) If the classification accuracy acc>1-λ, then the reference point i d and the Gaussian sphere sampling points are moved toward the origin along the reference direction i, and the Minkowski distance of the movement is the set value ξ; then step 41 is executed;
[0031] 43) If the classification accuracy acc <λ, then the reference point i d The sampling points on the Gaussian sphere are all moved in the opposite direction of the origin along the reference direction i, and the Minkowski distance of the movement is the set value ξ; then step 41 is executed).
[0032] Furthermore, the sampling accuracy is obtained. For the calculation of the approximate distance, when the reference points are in multiple directions, the Minkowski distance is used to synthesize the distances of multiple reference sampling directions into the overall approximate distance.
[0033] Furthermore, the overall approximate distance is mapped to a value between (0, 1) using a function as a confidence score. This score can be considered as the model's confidence in the correctness of the output. The approximate function should have the following characteristics:
[0034] 1) The domain is (0,+∞) and the range is (0,1).
[0035] 2) It has a monotonically increasing property.
[0036] The present invention also provides a server, characterized in that it includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the above method.
[0037] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.
[0038] The advantages of the present invention are as follows:
[0039] 1) Using multi-directional benchmark sampling instead of Gaussian sampling in the original sampling method, on the one hand, reduces the randomness of the sampling direction, and on the other hand, uses the same set of sampling directions for all images, making the metric of approximate distance uniform;
[0040] 2) After using the binary search method to determine the initial boundary, Gaussian sphere sampling is used for fine adjustment, reducing the number of sampling times;
[0041] 3) Using Gaussian sphere sampling instead of Gaussian sampling eliminates a large number of unnecessary points inside the high-dimensional Gaussian sampling, further reducing the number of sampling times.
[0042] 4) Compared with using only the approximate distance in a single direction to express confidence, the Minkowski distance is calculated using the approximate distance obtained in multiple directions, which can solve the problem of accuracy of expressing confidence in a single-direction distance estimation due to the randomness of direction selection.
[0043] 5) Combining the above advantages, we achieve fewer sampling times compared to existing adversarial sample methods and Gaussian sampling methods, thereby improving measurement efficiency and saving measurement resources, while maintaining the accuracy of confidence calculation in high-dimensional space. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 This is a flow chart of the efficient estimation method for image classification confidence of the present invention.
[0045] Figure 2 This is the flowchart for binary search. DETAILED DESCRIPTION
[0046] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.
[0047] The process of the present invention comprises:
[0048] 1) Determine the reference sampling direction
[0049] After sampling a reference sampling direction, the subsequent reference sampling directions should be orthogonal to the existing reference directions. Orthogonality can ensure that different directions do not affect each other, so as to obtain more information from different directions.
[0050] The specific method for determining the reference sampling direction is to generate a zero matrix of the same size as the target image. The value of each position in the zero matrix changes to 1 with probability p, changes to -1 with probability p, and remains unchanged with probability 1-2p. This matrix is called the sampling matrix. With the number of elements in the sampling matrix as the number of dimensions and the element values as the values at the corresponding dimensions, the sampling matrix can uniquely identify a coordinate point, called a matrix point. The point determined by the zero matrix is used as the coordinate origin, and the reference direction is the direction from the coordinate origin to the matrix point.
[0051] The number of baseline sampling directions for the model of this invention depends on the model's application scenario. If the application requires high real-time performance, the number of baseline sampling directions should be small, as it is directly related to detection time. If the application requires less real-time performance, the number of baseline sampling directions can be appropriately increased to improve the accuracy of the results.
[0052] 2) Generate benchmark points
[0053] After the reference sampling direction is determined, for each reference sampling direction, a point is selected at a farther point, requiring that the predicted category of the farther point and the origin in the AI model is different; for the reference sampling direction i, the sampling matrix corresponding to the reference sampling direction i is input as the image into the image classification model (AI model) for classification, and the obtained category is the category of the matrix point corresponding to the sampling matrix. Use binary search between the origin and the farther point to select the middle point between the two points, so that the middle point approaches the classification decision boundary of the AI model, and the midpoint after multiple iterations is the reference point. This step only requires determining the approximate distance, the purpose is to reduce the sampling steps and number of sampling points caused by parameter adjustment. There is no need to perform Gaussian sphere sampling on the points during binary search. The maximum number of iterations max_iter should be set as the termination condition during binary search. The binary search process is as follows Figure 2 shown.
[0054] 3) Generate Gaussian sphere
[0055] The purpose of generating a Gaussian sphere is to eliminate the internal points within the Gaussian hypersphere that have little contribution to the results. When approaching the classification boundary, the boundary position can be determined based on the classification accuracy acc of the surface points (that is, the ratio of correctly classified sampling points to the total number of sampling points). When the classification accuracy acc is within the specified threshold range, it can be considered that the Gaussian sampling center, that is, the reference point, is near the classification boundary. The threshold range is: (λ, 1-λ). The sampling distance L can be estimated using the parameters of the reference point sampling i , where i is the i-th axis of the baseline sampling direction, and i ranges from 1, 2, …, n. Since the number of sampling points inside the sampling point hypersphere is much greater than the number of surface points under normal Gaussian sampling, we eliminate points with little or no contribution, reducing the number of queries and improving efficiency.
[0056] 4) Calculate distance
[0057] After Gaussian sampling, when the classification accuracy acc is not within the threshold range:
[0058] If the classification accuracy acc>1-λ, both the reference point and the Gaussian sphere sampling point are moved toward the origin along the reference direction, and the Minkowski distance of the movement is the set value ξ.
[0059] If the classification accuracy acc < λ, both the reference point and the Gaussian sphere sampling point are moved in the opposite direction of the origin along the reference direction, and the Minkowski distance of the movement is the set value ξ.
[0060] After the boundary is roughly determined, the distance is fine-tuned using Gaussian sphere sampling. For each reference point, the above steps are taken and the sampling distance L is recorded. i , the sampling distance is the Minkowski distance between the reference point and the origin. Where i is the i-th direction axis of the reference sampling, i is 1, 2...n.
[0061] During benchmark sampling, the benchmark point sampling distance is L i , where i is the i-th direction axis of the reference sampling, i is 1, 2...n; the Minkowski distance is used for this overall distance synthesis method:
[0062]
[0063] Where p is the order of the Minkowski norm.
[0064] Because the dimensions of the axes are the same, the Minkowski distance is appropriate. Usually, p can be set to 1, which is the Manhattan distance.
[0065] 5) Calculate confidence score
[0066] The confidence estimation function maps the approximate distance to between (0, 1). The mapping function should be a monotonically increasing function in the interval (0, +∞), for example:
[0067] s=1-e -d
[0068] The confidence after this mapping can be regarded as the relative confidence of the current sample compared with the most confident classification.
[0069] While specific embodiments of the present invention have been disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiments, and the scope of protection claimed in the present invention shall be determined by the scope of the claims.
Claims
1. An efficient method for estimating confidence in image classification, comprising the following steps: 1) Determine several reference sampling directions of the image to be detected; 2) generating a reference point according to each of the reference sampling directions; 3) Sampling around each of the reference points to generate a Gaussian sphere; 4) Using the classification accuracy of the sampling points on the Gaussian sphere as the discrimination index, determine the sampling distance of the corresponding reference point; where L i is the sampling distance of the reference point corresponding to the reference sampling direction i; i = 1 to n, n is the number of reference sampling directions; 5) The sampling distances of the reference points are synthesized to obtain an approximate distance, and the approximate distance is mapped to a confidence score of the image to be detected.
2. The method according to claim 1, characterized in that The reference sampling directions are orthogonal to each other.
3. The method according to claim 1, characterized in that For a reference sampling direction i, the position of the image to be detected is taken as the origin, and a sampling point with a target distance value γ from the origin is selected in the direction of the reference sampling direction i as the farther point. The image category corresponding to the farther point is different from the image category corresponding to the origin; then the binary search algorithm is used to iteratively find the middle point between the origin and the farther point. When the middle point i is found m When approaching the classification decision boundary of the image classification model, the intermediate point i m As the reference point corresponding to the reference sampling direction i.
4. The method according to claim 3, characterized in that Get the reference point i corresponding to the reference sampling direction i d Sampling distance L i The method is: 41) For reference point i d The corresponding Gaussian sphere is sampled, and the image corresponding to each sampling point is input into the image classification model. Get the category of each sampling point. If the classification accuracy acc is within the set threshold range (λ, 1-λ), the sampling distance L i is the reference point i d Minkowski distance from the origin; 42) If the classification accuracy acc>1-λ, then the reference point i d and the Gaussian sphere sampling points are moved toward the origin along the reference direction i, and the Minkowski distance of the movement is the set value ξ; then step 41 is executed; 43) If the classification accuracy acc <λ, then the reference point i d The sampling points on the Gaussian sphere are all moved in the opposite direction of the origin along the reference direction i, and the Minkowski distance of the movement is the set value ξ; then step 41 is executed).
5. The method according to claim 3, characterized in that The method for selecting the target distance value γ is as follows: first, the maximum value max among the pixel values on all channels of the image to be detected is selected, and then the far point coefficient α is determined, and the target distance value γ=α*max.
6. The method according to claim 1, 2 or 3, characterized in that: The sampling distances of the reference points are synthesized using the Minkowski distance to obtain the approximate distance.
7. The method according to claim 6, characterized in that The approximate distance is Where p is the order of the Minkowski norm.
8. The method according to claim 1, 2 or 3, characterized in that: The method for determining the reference sampling direction is as follows: generating a zero matrix of the same size as the image to be detected; changing the value of each position on the zero matrix to 1 with a certain probability p, changing to -1 with a probability p, and remaining unchanged with a probability 1-2p, and the changed zero matrix is called a sampling matrix; using the number of elements in the sampling matrix as the number of dimensions, the element values as the values under the corresponding dimensions, and the coordinate points uniquely determined by the sampling matrix are called matrix points; using the point determined by the zero matrix as the coordinate origin, and the direction from the coordinate origin to the matrix point as the reference sampling direction.
9. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the steps of the method according to any one of claims 1 to 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.