An image recognition classification method based on two-stage active learning

Abstract

This invention discloses an image recognition and classification method based on two-stage active learning, comprising the following steps: performing a first-stage active sampling using conventional active learning methods; clustering samples in each view; calculating the consistency between two views; performing a second-stage active sampling using expressiveness and stability; training the neural network; and iteratively optimizing the model. This invention is a novel active learning method independent of specific learning tasks and decoupled from existing active learning methods. It can be applied to various active learning scenarios and can be combined with existing active learning methods. Furthermore, this invention considers both the expressiveness and stability of the samples, resulting in highly reliable samples for annotation. This invention achieves an accurate and scalable two-stage active learning method based on multi-view clustering to calculate sample expressiveness and stability, thus possessing high practical value.

Description

Technical Field

[0001] This invention relates to an image recognition method, and more particularly to an image recognition and classification method based on two-stage active learning. Background Technology

[0002] In the field of computer vision, high-quality labeled data is indispensable for handling complex tasks and the neural networks designed to address them. However, in many application scenarios, it is difficult to obtain large quantities of high-quality labeled data. Active Learning (AL) aims to achieve the same results as fully supervised training by using a small number of labeled samples with sufficient information, while controlling the cost of data labeling. In traditional pooling-based active learning scenarios, a large number of unlabeled samples form a candidate sample pool (called the unlabeled pool), while the training set is finite. The model continuously selects key samples from the unlabeled pool through a specific sampling strategy, requesting manual annotation to expand the training set and iteratively optimize the current model.

[0003] The current mainstream approach in algorithmic analysis (AL) is to design different active sampling strategies while adhering to the aforementioned framework. For example, in classification tasks, classic algorithms such as Least Confidence (LC), Margin, and Entropy all use the prediction uncertainty of the current model as the basis for sampling. In object detection tasks, some methods directly borrow the approach from classification tasks, sampling only from the classification branch, while others focus on the regression branch, using the stability of the regression box predictions as the basis for sampling.

[0004] However, the AL sampling strategies in the above methods are task-specific. Although they can be adapted to other tasks with appropriate modifications, these methods often perform poorly on new tasks. In recent years, researchers have begun to explore and design task-agnostic AL methods, hoping to provide a general sampling strategy. For example, Yoo et al. proposed a task-agnostic loss prediction module that directly predicts sample loss to guide sampling. Sener et al. proposed a method for active sampling by measuring data distribution—Coreset. Unfortunately, the sampling criteria of the above methods still have certain limitations. Yoo et al. only considered the model's feedback, ignoring the characteristics of the data, while Sener et al. only considered the macroscopic feature distribution of the data. Summary of the Invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide an image recognition and classification method based on two-stage active learning, which addresses the shortcomings of the existing technology.

[0006] To address the aforementioned technical problems, this invention discloses an image recognition and classification method based on two-stage active learning, comprising the following steps:

[0007] Step 1: Determine the amount of data that needs to be labeled in the dataset used to train the image recognition and classification model;

[0008] Step 2: Use an active learning method to perform the first stage of active sampling to obtain the first stage samples;

[0009] Step 3: Use the multi-view clustering method to cluster the samples from the first stage;

[0010] Step 4: Calculate the consistency between any two views;

[0011] Step 5: Calculate the sampling score as the sampling strategy, perform the second stage of active sampling, and manually label the second stage samples obtained through the second stage of active sampling.

[0012] Step 6: Train the image recognition and classification model;

[0013] Step 7: Repeat steps 1 to 6 to iteratively optimize the image recognition classification model. When the number of manually labeled samples reaches the amount of data determined in step 1, skip steps 1 to 5 and use only the task loss described in step 6 as the loss function to train the image recognition classification model.

[0014] Step 8: Apply the optimized image recognition and classification model to perform image recognition and classification.

[0015] Beneficial effects:

[0016] This invention employs an active learning method for image recognition and classification that is independent of specific learning tasks and decoupled from existing active learning methods. Its independence from specific learning tasks means that this method is applicable not only to classification tasks but also to any deep learning task, provided there are a large number of unlabeled samples and the learning architecture uses neural networks to extract features from these samples. Its decoupling from existing active learning methods is primarily reflected in the fact that the core of this invention is a highly encapsulated sample sampling module, without significantly altering the overall model architecture. Therefore, it can be arbitrarily combined with other sample sampling strategies to leverage their strengths and compensate for their weaknesses, resulting in powerful effects. In summary, this invention has strong adaptability and can be applied to various active learning scenarios. Furthermore, this invention considers both the expressiveness and stability of the samples, ensuring high reliability of the sampled samples used for annotation. Therefore, the two-stage active learning method proposed in this invention, based on multi-view clustering to calculate sample expressiveness and stability, achieves better results in image recognition and classification. Attached Figure Description

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0018] Figure 1 This is a flowchart of the present invention.

[0019] Figure 2 A schematic diagram illustrating the clustering of each view sample.

[0020] Figure 3 Flowchart for calculating consistency between two views.

[0021] Figure 4 The flowchart shows the process of calculating the expression level of the sample.

[0022] Figure 5 This is a flowchart for calculating sample stability.

[0023] Figure 6 Flowchart for model iterative optimization.

[0024] Figure 7 A flowchart illustrating how to use this invention to complete a racial classification task. Detailed Implementation

[0025] like Figure 1 As shown, this invention discloses an image recognition and classification method based on two-stage active learning, comprising the following steps:

[0026] Step 1: Determine the amount of data that needs to be labeled;

[0027] Step 2: Perform the first stage of active sampling using conventional active learning methods;

[0028] Step 3: Cluster the samples in each view;

[0029] Step 4: Calculate the consistency between the two views;

[0030] Step 5: Perform second-stage active sampling using expression level and stability.

[0031] Step 6: Train the neural network;

[0032] Step 7: Iteratively optimize the model.

[0033] Step 1 specifically includes the following steps:

[0034] Determine the total amount of data to be labeled based on the labeling cost of the dataset. For example, consider training a racial classification model where the input is a face image and the output is the racial group to which that face belongs. Suppose we have crawled 1 million face images from around the world, but the racial groups corresponding to these images are unknown. To complete the training, we need to manually label the racial groups to which these faces belong.

[0035] However, manually labeling ethnic data is costly. For example, Huawei Cloud Store charges 48 yuan per 10,000 images for general image classification labeling, while the cost is even higher for images in specific domains such as faces. For instance, the price for labeling facial attributes, which includes gender, expression, and head posture, reaches 6,000 yuan per 10,000 images. Assuming the price for ethnic labeling in this application scenario is 500 yuan per 10,000 images, and the total budget for data labeling is 5,000 yuan, then 100,000 images can be labeled. Clearly, compared to labeling all 1 million images and using them all for model training, labeling only 100,000 images will yield poorer training results. The problem that active learning aims to solve is to select the 100,000 most valuable images from the 1 million unlabeled images, and use these 100,000 images to train the model, achieving the best possible training results.

[0036] The active learning method proposed here, when selecting samples, considers not only the features of the samples themselves but also certain output values ​​of the classification model. After training the classification model to a certain extent, the selected samples will be more accurate. Therefore, this invention divides the sample sampling process into several rounds, so that, except for the first round, the classification model has been trained to a certain extent in each round of sampling. For example, the sampling process can be divided into 10 rounds, with 10,000 images sampled in each round.

[0037] Step 2 specifically includes the following steps:

[0038] The unlabeled dataset is called the unlabeled pool. A round of sampling can be divided into two stages, with the second stage sampling based on the results of the first stage. For example, for the aforementioned racial classification model, the first stage could sample 20,000 images from the unlabeled pool, and the second stage could select 10,000 images from these 20,000. In this step, conventional active learning methods such as the minimum confidence (LC) algorithm, learning loss (LL) algorithm, and coreset algorithm are used as the active sampling strategy for the data in the unlabeled pool in the first stage, yielding the sampling results of the first stage, which are a subset of the unlabeled pool.

[0039] The LC algorithm selects samples from the unlabeled pool whose probability distributions in the model output have low confidence. Specifically, let the input sample (for the aforementioned racial classification task, this refers to face images) be x, the model parameters (for the aforementioned racial classification task, this refers to the racial classification model) be θ, and the model output be p(x|θ). Taking a classification task as an example, p(x|θ) can be specifically written as [p1, p2, ..., p...]. C ], where C is the number of categories, p c Let x represent the probability that x belongs to class c (1≤c≤C), which requires the following formula: [p1,p2,…,p...] C Calculate the score S of sample x. LC The higher the score, the higher the priority during sampling. Here is S... LC There are two calculation methods. The first method examines the probability corresponding to the classification result. The lower the probability, the lower the model's confidence in the classification result, and the more likely it is to be sampled. The second method examines [p1, p2, ..., p C The entropy of a model is such that a larger entropy indicates greater uncertainty in the model's output, and therefore the model should be sampled more frequently. During the sampling process, the samples with the highest scores are selected as the sampling results.

[0040] The LL algorithm uses a loss predictor to predict the loss function value for each sample in the unlabeled pooling. The larger this value, the greater the reduction in loss function when using these samples for training, and therefore, the more samples should be sampled. Specifically, the loss predictor L... loss The features of a sample x are extracted from several layers in a neural network. Each feature is then passed through an independent global average pooling layer, a fully connected layer, and a ReLU activation layer. All features after these three layers are concatenated, and finally, a fully connected layer is used to map the features to a scalar. This scalar is the predicted value of the loss function for this sample. During the sampling process, the samples with the largest predicted values ​​of the loss function are selected as the sampling results.

[0041] The Coreset algorithm also selects samples from the unlabeled pool whose loss function is closest to the loss function of all samples. Through theoretical analysis, it transforms this into the following problem: using the sampled sample as the center point, examine the distances of all other samples to the nearest center point, aiming to minimize the maximum value of these distances. Specifically, let x be the sample in the set X consisting of all samples. i(1≤i≤N, where N is the total number of samples), the set of samples with labels is s0 (this algorithm is applicable when some samples have labels in the initial state; if all samples are unlabeled, then s0 is an empty set), the number of samples to be sampled in this step is b (for example, the above racial classification task should be 20,000), and the following steps are executed:

[0042] S1: Let s represent the set of all samples with existing labels in the current state. At this time, it is obvious that s = s0.

[0043] S2: Order Where d(x) i ,x j ) represents sample x i and x j The distance between them;

[0044] S3: Add x to s, that is, let s = s∪{x};

[0045] S4: Repeat S2 and S3 until |s|=|s0|+b, where ‖·‖ represents the number of elements in a set;

[0046] S5: Take s-s0 as the result of sampling in this step.

[0047] To enhance the robustness of the Coreset algorithm, further optimizations can be made. Here, the hyperparameter Ξ needs to be specified, representing the number of outliers in the algorithm. Outliers are samples that are far from all center points; they are no longer considered when calculating the maximum distance. While introducing outliers may disrupt the definition of the problem, it can further improve the algorithm's computational results. The algorithm performs the following steps:

[0048] S1: The preliminary sampling result obtained using the above basic algorithm is denoted as s. g ,calculate Let lb = δ / 2, ub = δ;

[0049] S2: Set intermediate variables Determine whether the following conditions are satisfied: (1)∑ j u j =|s0|+b;(2)∑ i,j ξ i,j ≤Ξ;(3) (4) (5) (6) If d(x) i ,x j If )>δ, then w i,j =ξi,j If all six conditions above are met simultaneously, then execute. Otherwise execute

[0050]

[0051] S3: Repeat S2 until ub = lb;

[0052] S4: Execute δ←ub;

[0053] S5: Substitute δ into the six conditions in S2, and solve for u based on these six conditions. i ξ i,j and w i,j (The operations in S2 and S3 have already guaranteed that the solution here is unique. In actual calculation, it is difficult to satisfy ub = lb in S3. Generally, the iteration of S2 can be stopped when the value of ub - lb is a very small positive number ε. The consequence of doing so is that there may be multiple solutions here. However, as long as ε is sufficiently small, the difference between these solutions will be very small, and they can be approximated as the same solution.)

[0054] S6: Set {x} i |u i =1} is the final result of sampling in this step.

[0055] Step 3 specifically includes the following steps:

[0056] From this step onwards, the second stage of sampling will begin, which is the formal entry into the multi-view clustering algorithm. First, as... Figure 2 As shown, features of each sample are extracted from several layers in the model (for the above racial classification task, this refers to the racial classification model). Here, the features of all samples at the same layer are called a view. Suppose there are U layers used for feature extraction, which means there are a total of U views.

[0057] Then, a Gaussian Mixture Model (GMM) is used to model the distribution of samples in each view, and clustering results are generated based on this. In the GMM, a probability density function is used. This describes the distribution of samples within the same view. Here, x represents the input sample, K represents the number of Gaussian models in the GMM model, and φ(x|μ k ,σ k ) indicates that μ k For the mean, σ k Let α be the k-th Gaussian model with variance. k It is the weight, which can be understood as the probability that the current sample belongs to the k-th Gaussian model.

[0058] The specific operation method of GMM is described below. Let the set of all samples be . Where x (i) (1≤i≤N) represents the i-th sample in the first stage, introducing latent variables. in Then perform the following steps:

[0059] S1: Randomly initialize θ (including α) k μ k and σ k (1≤k≤K);

[0060] S2: Calculation The estimated value

[0061] S3: Calculate the estimated values ​​of the model parameters and in

[0062] S4: Repeat S2 and S3 until the model converges. Finally, use... and As α k μ k and σ k Approximate value.

[0063] After the above operations, for U views, we can obtain U distributions p1(x|θ1), p2(x|θ2), ..., p U (x|θ U Based on these distributions, each sample can be classified into its respective category on each view. As a result of clustering.

[0064] Step 4 specifically includes the following steps:

[0065] For a fully trained neural network, the clustering results on each view should be as similar as possible. Therefore, this step uses the Rand statistic to calculate the clustering results for the two views V. m and V n Consistency R(V) m V n (1≤m≤U, 1≤n≤U). For example... Figure 3 As shown.

[0066] Specifically, sample x i In view V m The clustering results on are denoted as Let the total number of samples be denoted as s. Therefore, all samples can form a total of s(s-1) / 2 groups of the form (x i ,x j Consider sample pairs (i ≠ j). Examine the element x in the sample pair.i and x j In view V m and V n The clustering results show that they can be divided into the following four cases: (1) simultaneously satisfying and (2) Simultaneously satisfy and (3) Simultaneously satisfy and (4) Simultaneously satisfy and For all s(s-1) / 2 sample pairs, each sample pair belongs to one and only one of these cases. Let s be the set of all sample pairs that satisfy (1) or (2). p (i.e., view V) m and V n For the consensus on whether the two samples in a sample pair belong to the same class, the set of all sample pairs that satisfy (3) or (4) is denoted as s. n (i.e., view V) m and V n Opinions differ regarding whether two samples in a sample pair belong to the same class.

[0067] In obtaining s p and s n Then, V can be calculated. m and V n Consistency R(V) m V n )=||s p || / ||s p +s n ||, where ‖·‖ represents the number of elements in a set.

[0068] Step 5 specifically includes the following steps:

[0069] This step calculates a sampling score based on the probability density function p(x|θ) obtained from the GMM algorithm and the consistency R between the two views, serving as the active sampling strategy for the second stage. This score incorporates considerations of sample expressiveness and stability.

[0070] For the expression level of the sample, such as Figure 4 As shown, the probability density at the point where the sample is located in the typical view is used as the evaluation criterion. A "typical view" refers to a view whose clustering results are similar to most other views. Specifically, the probability density of a view V is calculated... m Consistency Then the view V with the highest consistency oAs a typical view. Based on this, the expression level Rep(x) of the sample is... i ) is defined as V o The probability density function value obtained by the GMM algorithm is Rep(x) i ) = p o (x i |θ o ),in The sample expressivity calculated above reflects the distance of a sample to the cluster center. The higher the sample expressivity, the closer the sample is to the cluster center, indicating that samples with similar characteristics to the sample appear more frequently, and that the sample can represent more samples.

[0071] Regarding the stability of the sample, such as Figure 5 As shown, the similarity of a sample to other samples of the same category appearing in each view is used as the evaluation criterion. Specifically, sample x... i In view V m The set of samples of the same category is denoted as First, calculate sample x. i In view V m and V n Stability on Where ||·|| represents the number of elements in a set. Then, x is calculated based on this. i Stability of Stab(x) i ),have

[0072] Finally, sample x i Sampling score S(x) i S(x) is a weighted sum of expressiveness and stability. i ) = Rep(x i )+λStab(x i ), where λ is a hyperparameter satisfying λ>0, used to balance the model's emphasis on expressiveness and stability, and needs to be repeatedly tested and adjusted in real-world application scenarios. After obtaining the sampling scores of all samples, the highest-scoring samples from the samples obtained in the first stage of active sampling are selected as the results of the second stage of active sampling and submitted for manual annotation. The proportion of the sampled samples needs to be determined based on the scale available for manual annotation; for example, in the aforementioned racial classification task, the 10,000 highest-scoring face images should be selected.

[0073] Step 6 specifically includes the following steps:

[0074] This step trains the model based on the labels obtained from a subset of samples. During training, the parameters of the neural network are optimized to improve both the model's ability to perform the specified task and its feature extraction capabilities. For this purpose, the loss function consists of the task loss (TL). Multi-view clustering loss (MVCL) It consists of two parts. Among them, The specific task determines the calculation method. For example, in classification tasks, cross-entropy loss can be used, and only labeled samples are used in the calculation process; while MVCL is determined by sample consistency, and the specific calculation method is as follows. All samples are used in the calculation process.

[0075] Ultimately, the total loss function of the training process Where μ is a hyperparameter, satisfying μ>0. When the cost of manual annotation is limited and only a small number of samples can be annotated, the value of μ can be appropriately increased to focus on monitoring the consistency of samples, so as to ensure that each sampled sample is as carefully selected as possible. Conversely, the value of μ can be appropriately decreased.

[0076] Furthermore, it should be noted that if the method selected for active sampling in the first stage needs to be iteratively optimized during training, then a corresponding term needs to be added to the loss function. For example, when selecting the LL algorithm for active sampling in the first stage, the loss predictor L needs to be optimized based on the labeled data. Specifically, a loss function value is introduced. During training, the batch size is set to an even number. In each iteration, samples in the batch are paired up in pairs as shown in the figure (x...). i ,x j The sample pairs of x i and x j After being input into the neural network, the corresponding output is denoted as... and x i and x j The actual labels marked manually are denoted as y(x) i ) and y(x j ), calculate the actual loss At the same time, according to x i and x j In a neural network, feature calculation of prediction loss across several layers Then the sample pair (x i ,x j The loss function corresponding to the predicted loss value. Where ξ is a hyperparameter. It should be noted that in numerical calculations, l(x) i )=l(x jThe scenario described is highly unlikely and will therefore be ignored here. Total Predicted Loss in It is the set of all sample pairs in the current batch. During training, it will... As the total loss function One of the factors, and uses the coefficient as a hyperparameter to balance its relationship with The importance of the relationship.

[0077] Step 7 specifically includes the following steps:

[0078] Repeat steps 1 through 5 to iteratively optimize the model, such as... Figure 6 As shown. In each iteration, all unlabeled samples form an unlabeled pool. Two-stage active sampling is performed on the samples in the unlabeled pool. The sampled samples are then manually labeled, and the model is trained using all samples after labeling. As the number of iterations increases, the number of labeled samples grows. When the number of labeled samples reaches a certain scale, the iteration process skips active sampling and only trains the model. Furthermore, the loss function used during training only uses... For example, in the above race classification task, after 10 iterations, the number of labeled samples reached 100,000, at which point only the model was trained.

[0079] After the model is trained, it is tested. Once the test results meet the expectations, it can be used in real-world tasks.

[0080] Example

[0081] This embodiment takes the racial classification task as an example. It crawls facial images from the internet and uses the active learning method proposed in this invention to train the racial classification model. The overall process is as follows: Figure 7 As shown. Includes the following parts:

[0082] Step 1: Determine the amount of data that needs to be labeled.

[0083] Based on the annotation cost of the dataset, determine the total amount of data that needs to be labeled. This section uses training a racial classification model as an example. The input of this model is a face image, and the output is the racial category to which the face belongs, with a total of 5 categories. Now assume that 1 million face images from around the world have been crawled from the Internet, but the corresponding racial categories are unknown. To complete the training, these faces need to be manually labeled with their racial categories.

[0084] However, manually labeling ethnic data is costly. For example, Huawei Cloud Store charges 48 yuan per 10,000 images for general image classification labeling, while the cost is even higher for images in specific domains such as faces. For instance, the price for labeling facial attributes, which includes gender, expression, and head posture, reaches 6,000 yuan per 10,000 images. Assuming the price for ethnic labeling in this application scenario is 500 yuan per 10,000 images, and the total budget for data labeling is 5,000 yuan, then 100,000 images can be labeled. Clearly, compared to labeling all 1 million images and using them all for model training, labeling only 100,000 images will yield poorer training results. The problem that active learning aims to solve is to select the 100,000 most valuable images from the 1 million unlabeled images, and use these 100,000 images to train the model, achieving the best possible training results.

[0085] The active learning method proposed here, when selecting samples, considers not only the features of the samples themselves but also certain output values ​​of the classification model. After training the classification model to a certain extent, the selected samples will be more accurate. Therefore, this invention divides the sample sampling process into several rounds, so that, except for the first round, the classification model has been trained to a certain extent in each round of sampling. For example, the sampling process can be divided into 10 rounds, with 10,000 images sampled in each round.

[0086] To facilitate image processing by the model, the images are scaled here, and the resolution of all images is adjusted to 224×224.

[0087] Step 2: Perform the first stage of active sampling using conventional active learning methods.

[0088] The unlabeled dataset is called the unlabeled pool. A sampling round can be divided into two stages, with the second stage sampling based on the results of the first stage. For example, for the aforementioned racial classification model, we can set the first stage to sample 20,000 images from the unlabeled pool, and the second stage to select 10,000 images from these 20,000. In this step, we use three schemes as the active sampling strategy for the data in the unlabeled pool in the first stage: the minimum confidence (LC) algorithm, the learning loss (LL) algorithm, and the coreset algorithm. After sampling is completed, we obtain the sampling results of the first stage, which is a subset of the unlabeled pool.

[0089] The LC algorithm selects samples with lower confidence levels in the probability distribution of the model's output from the unlabeled pooling. Specifically, let the face image be x, the parameters of the race classification model be θ, and the model's output be p(x|θ). Since the task of p(x|θ) is to classify all x images into 5 categories, p(x|θ) can be specifically written as [p1, p2, ..., p5], where p... cLet x represent the probability that x belongs to class c (1≤c≤5). We need to calculate the score S of sample x based on [p1,p2,…,p5]. LC The higher the score, the higher the priority during sampling. Here is S... LC There are two calculation methods. The first method examines the probability corresponding to the classification result. The lower the probability, the lower the model's confidence in the classification result, and the more likely it is to be sampled. The second method examines [p1, p2, ..., p C The entropy of a model is such that a larger entropy indicates greater uncertainty in the model's output, and therefore the model should be sampled more frequently. After calculating the results, the 20,000 images with the highest scores were selected as the sampling results.

[0090] The LL algorithm uses a loss predictor to predict the loss function value for each sample in the unlabeled pooling. The larger this value, the greater the reduction in loss function when using these samples for training, and therefore, the more samples should be sampled. Specifically, the loss predictor... The features of a sample x are extracted from several layers in a neural network. Each feature is then passed through an independent global average pooling layer, a fully connected layer, and a ReLU activation layer. All features after these three layers are concatenated, and finally, a fully connected layer is used to map the features to a scalar. This scalar is the predicted value of the loss function for this sample. During the sampling process, the 20,000 images with the highest predicted values ​​of the loss function were selected as the sampling results.

[0091] The Coreset algorithm also selects samples from the unlabeled pool whose loss function is closest to the loss function of all samples. Through theoretical analysis, it transforms this into the following problem: using the sampled sample as the center point, examine the distances of all other samples to the nearest center point, with the goal of minimizing the maximum value of these distances. In a face image dataset X, let the sample be x. i (1≤i≤10000), the set of samples with labels is s0 (s0 is an empty set if all samples are unlabeled). The planned sample size for this step is 20,000. The following steps will be performed:

[0092] S1: Let s represent the set of all samples with existing labels in the current state. At this time, it is obvious that s = s0.

[0093] S2: Order Where d(x) i ,x j ) represents sample x i and x j The distance between them can be expressed, for example, using Euclidean distance ||x i -x j||2, the x obtained in this step is the selected sample to be labeled. It is relatively close to its sample and has strong representativeness.

[0094] S3: Add x to s, that is, let s = s∪{x};

[0095] S4: Repeat S2 and S3 until |s|=|s0|+20000, where ‖·‖ represents the number of elements in a set. When this condition is met, the number of selected samples to be labeled has reached the predetermined 20,000.

[0096] S5: Take s-s0 as the result of sampling in this step.

[0097] Since our face images are obtained from web scraping, some anomalous images are inevitable, such as faces wearing masks or cartoon images. These are not helpful for model training. Therefore, to make the Coreset algorithm more robust, it can be further optimized. Here, these anomalous images are considered "outliers," which are far from all center points, and a hyperparameter Ξ is introduced to represent the number of outliers. Outliers are no longer considered when calculating the maximum distance. Although the introduction of outliers violates the definition of the problem, it can further improve the algorithm's calculation results. The algorithm performs the following steps:

[0098] S1: The preliminary sampling result obtained using the above basic algorithm is denoted as s. g ,calculate Let lb = δ / 2, ub = δ;

[0099] S2: Set intermediate variables Determine whether the following conditions are satisfied: (1)∑ j u j =|s0|+b;(2)∑ i,j ξ i,j ≤Ξ;(3) (4) (5) (6) If d(x) i ,x j If )>δ, then w i,j =ξ i,j If all six conditions above are met simultaneously, then execute. Otherwise execute

[0100] S3: Repeat S2 until ub = lb;

[0101] S4: Execute δ←ub;

[0102] S5: Substitute δ into the six conditions in S2, and solve for u based on these six conditions. i ξ i,j and w i,j (The operations in S2 and S3 have already guaranteed that the solution here is unique. In actual calculation, it is difficult to satisfy ub = lb in S3. Generally, the iteration of S2 can be stopped when the value of ub - lb is a very small positive number ε. The consequence of doing so is that there may be multiple solutions here. However, as long as ε is sufficiently small, the difference between these solutions will be very small, and they can be approximated as the same solution.)

[0103] S6: Set {x} i |u i =1} is the final result of sampling in this step.

[0104] Step 3: Cluster the samples in each view.

[0105] From this step onwards, the second stage of sampling begins, which is the formal entry into the multi-view clustering algorithm. First, the 20,000 face image samples obtained in the first stage are fed into the neural network model, and features at each layer are extracted. Features from all samples at the same layer are referred to as a view. For example, using a ViT network with a 12-layer Transformer structure, features are extracted from the 2nd, 4th, 6th, 8th, 10th, and 12th Transformer layers (where the features from the 12th Transformer layer are the features fed into the classification network head), resulting in a total of 6 views.

[0106] Then, a Gaussian Mixture Model (GMM) was used to model the distribution of samples in the six views, and clustering results were generated based on this. In the GMM, a probability density function was used. This describes the distribution of samples within the same view. Here, x represents the input sample, K represents the number of Gaussian models in the GMM model, and φ(x|μ k ,σ k ) indicates that μ k For the mean, σ k Let α be the k-th Gaussian model with variance. k It is the weight, which can be understood as the probability that the current sample belongs to the k-th Gaussian model.

[0107] The specific operation method of GMM is described below. Let the set of all samples be . Where x (i) (1≤i≤N) represents the i-th sample in the first stage, introducing latent variables. in Then perform the following steps:

[0108] S1: Randomly initialize θ (including α) k μ k and σ k (1≤k≤K);

[0109] S2: Calculation The estimated value

[0110] S3: Calculate the estimated values ​​of the model parameters and in

[0111] S4: Repeat S2 and S3 until the model converges. Finally, use... and As α k μ k and σ k Approximate value.

[0112] After the above operations, for the six views, we can obtain six distributions: p1(x|θ1), p2(x|θ2), ..., p6(x|θ6). Based on these distributions, we can classify each sample in each view into its respective category. As a result of clustering.

[0113] Step 4: Calculate the consistency between the two views.

[0114] For a fully trained neural network, the clustering results on each view should be as similar as possible. Therefore, this step uses the Rand statistic to calculate the clustering results for the two views V. m and V n Consistency R(V) m V n (1≤m≤U,1≤n≤U).

[0115] Specifically, sample x i In view V m The clustering results on are denoted as Let the total number of samples be denoted as s. Therefore, all samples can form a total of s(s-1) / 2 groups of the form (x i ,x j Consider sample pairs (i ≠ j). Examine the element x in the sample pair. i and x j In view V m and V n The clustering results show that they can be divided into the following four cases: (1) simultaneously satisfying and (2) Simultaneously satisfy and (3) Simultaneously satisfy and (4) Simultaneously satisfy and For all s(s-1) / 2 sample pairs, each sample pair belongs to one and only one of these cases. Let s be the set of all sample pairs that satisfy (1) or (2). p (i.e., view V) m and V n For the consensus on whether the two samples in a sample pair belong to the same class, the set of all sample pairs that satisfy (3) or (4) is denoted as s. n (i.e., view V) m and V n Opinions differ regarding whether two samples in a sample pair belong to the same class.

[0116] In obtaining s p and s n Then, V can be calculated. m and V n Consistency R(V) m V n )=||s p || / ||s p +s n ||, where ‖·‖ represents the number of elements in a set, that is, how many sample pairs in V out of all 20,000 sample pairs are in V. m and V n The partitioning results are consistent. Since we have a total of 6 views, we can get a total of 6 × 5 / 2 = 15 consistency scores.

[0117] Step 5: Use expression level and stability to perform the second stage of active sampling.

[0118] This step calculates a sampling score based on the probability density function p(x|θ) obtained from the GMM algorithm and the consistency R between the two views, serving as the active sampling strategy for the second stage. This score incorporates considerations of sample expressiveness and stability.

[0119] The expressiveness of a sample is evaluated using the probability density at the point where the sample is located in the typical view. A "typical view" is a view whose clustering results are similar to most other views. Specifically, the expression density of a view V is calculated... m Consistency Then the view V with the highest consistency oAs a typical view, the fifth view in our ViT network becomes the typical view. Intuitively, during the forward propagation of the network, random noise in the samples is gradually removed, while essential features are preserved. Therefore, views located later in the model are more likely to have higher consistency with most views, which is consistent with the actual calculation results. Based on this, the expression level Rep(x) of the samples is... i ) is defined as V o The probability density function value obtained by the GMM algorithm is Rep(x) i ) = p o (x i |θ o ),in The sample expressiveness calculated above reflects the distance of a sample to the cluster center. A higher sample expressiveness indicates a closer distance to the cluster center, meaning samples with similar features appear more frequently, and the sample can represent a larger sample pool. Specifically, after clustering face images, each ethnicity tends to be assigned to different categories (when there are many categories, different regional appearances within the same ethnicity will also be assigned to different categories. For example, when there are 5 categories, each ethnicity is roughly assigned to the same category, meaning most people of a particular ethnicity will be in the same category; however, when there are 30 categories, different sub-ethnicities within that ethnicity are likely to be assigned to different categories). Each ethnicity contains some "common faces." Highly labeling these "common faces" allows the model to learn the main characteristics of that ethnicity. Conversely, if the labeled data contains a large number of "niche faces," the model may use the accidental features of these "niche faces" as the basis for judging ethnicity, leading to significant deviations in the classification results. Samples with high expressiveness correspond to "common faces."

[0120] For sample stability, the similarity of the sample to other samples of the same category appearing in each view is used as the evaluation criterion. Specifically, sample x... i In view V m The set of samples of the same category is denoted as First, calculate sample x. i In view V m and V n Stability on Where ||·|| represents the number of elements in a set. Then, x is calculated based on this. i Stability of Stab(x) i ),have Specifically, facial images may contain images of "mixed-race" individuals. For example, an image of a person of mixed AB descent might have brown skin, but their facial features are more characteristic of race A. In the model, one view might be more concerned with skin color, classifying the image as race B, while another view might be more concerned with the shape of the facial features, classifying the image as race A. If this image is manually labeled, labeling it "race A" or "race B" would be inappropriate, as the features in the image that do not match the race can mislead the model's training. Therefore, such samples should be avoided as much as possible. Stability is precisely about controlling the samples from this perspective.

[0121] Finally, sample x i Sampling score S(x) i S(x) is a weighted sum of expressiveness and stability. i ) = Rep(x i )+λStab(x i ), where λ is a hyperparameter satisfying λ>0, used to balance the model's emphasis on expressiveness and stability, and needs to be repeatedly tested and adjusted in real-world applications. In our racial classification model, a λ value of 0.05 is ideal. After obtaining the sampling scores of all samples, from the 20,000 face images obtained in the first stage of active sampling, the 10,000 images with the highest scores are selected as the results of the second stage of active sampling and submitted for manual annotation.

[0122] Step 6: Train the neural network.

[0123] This step trains the model based on the labels obtained from a subset of samples. During training, the parameters of the neural network are optimized to improve both the model's ability to perform ethnic classification tasks and its feature extraction capabilities. To this end, the loss function consists of task loss (TL). Multi-view clustering loss (MVCL) It consists of two parts. Among them, Cross-entropy loss is used, and only labeled samples are used in the calculation process; while MVCL is determined by sample consistency, and the specific calculation method is as follows: The entire dataset is used in the calculation. Ultimately, the total loss function for the training process is... Where μ is 0.1.

[0124] During training, we set the learning rate to 1×10. -3 The optimizer uses Adam (parameters α1 = 0.9, α2 = 0.99).

[0125] When the active sampling strategy in the first stage is the LL algorithm, the loss predictor L needs to be optimized based on the labeled data. Specifically, a loss function value loss is introduced. During training, the batch size is set to 64. In one iteration, samples in the batch are paired up as follows: (x... i ,x j The sample pairs of x i and x j After being input into the neural network, the corresponding output is denoted as... and x i and x j The actual labels marked manually are denoted as y(x) i ) and y(x j ), calculate the actual loss At the same time, according to x i and x j In a neural network, feature calculation of prediction loss across several layers Then the sample pair (x i ,x j The loss function corresponding to the predicted loss value. The hyperparameter ξ is set to 1. It should be noted that in numerical calculations, l(x) i )=l(x j The scenario described is highly unlikely and will therefore be ignored here. Total Predicted Loss in It is the set of all sample pairs in the current batch. During training, it will... As the total loss function One of the factors, and uses the coefficient as a hyperparameter to balance its relationship with To determine the importance between them, we set the value of this coefficient to 0.1, that is...

[0126] Step 7: Iteratively optimize the model.

[0127] Repeat steps 1 through 5 to iterate and optimize the model for 10 rounds. In each iteration, all unlabeled samples form an unlabeled pool. Two-stage active sampling is performed on the samples in the unlabeled pool to obtain 10,000 face images, which are then manually labeled. After labeling, the ViT model is trained using all samples. After 10 iterations, only the model is trained, and the loss function used in the training process is only...

[0128] After the model training was completed, it was tested by crawling 1000 face images with similar styles to the training data from the internet. The top-1 classification accuracies achieved using the LC, LL, and Coreset algorithms in the first-stage sampling strategy were 89.7%, 90.2%, and 86.3%, respectively. Therefore, the model using the LL algorithm in the first-stage sampling strategy was ultimately selected for the face classification task.

[0129] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the inventive content of the image recognition method based on two-stage active learning provided by this invention, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0130] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0131] This invention provides an idea and method for image recognition based on two-stage active learning. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. An image recognition classification method based on two-stage active learning, characterized in that, Comprising the following steps: Step 1, in the data set for training the image recognition classification model, determine the amount of data that needs to be labeled; Step 2, use the active learning method to perform first-stage active sampling to obtain first-stage samples; Step 3, use a multi-view clustering method to cluster the first-stage samples; Step 4, calculate the consistency between any two views; Step 5, calculate the sampling score as a sampling strategy, perform second-stage active sampling, and manually label the second-stage samples obtained through the second-stage active sampling; Step 6, train the image recognition classification model; Step 7, repeat steps 1 to 6 to iteratively optimize the image recognition classification model, when the number of manually labeled samples reaches the amount of data determined in step 1, skip steps 1 to 5, and only use the task loss used in the process of training the image recognition classification model in step 6 as the loss function to train the image recognition classification model; Step 8, apply the optimized image recognition classification model to perform image recognition classification; Wherein, step 3 clusters the first-stage samples, specifically including: Step 3-1, extract the features of each first-stage sample in each layer of the image recognition classification model, define the features of all samples in the same layer as a view; suppose there are layers in the image recognition classification model for extracting features, i.e. there are views; Step 3-2, use a Gaussian Mixture Model (GMM) to model the distribution of first-stage samples in each view to obtain a GMM model and form a clustering result; Step 4 involves calculating the consistency between any two views, i.e., using the Rand statistic to calculate the consistency between the two views. A view and Consistency between ,in, , The specific method is as follows: The first stage samples The clustering results on the view The total number of the first stage samples is All samples together make up sample pairs, wherein the sample pair composed of sample and sample is , wherein ; the elements and in the sample pair are clustered on the view and , which are divided into the following four cases:​ Case 1: both of and are satisfied; Case 2: both of the following are satisfied and ; Case 3: both of the following are satisfied and ; Case 4, both of and ; For all Let there be n sample pairs, each belonging to one and only one of the above cases. The set of all sample pairs satisfying either case 1 or case 2 is denoted as . The set of all sample pairs that satisfy condition 3 or condition 4 is denoted as ;calculate and Consistency between The method is as follows: ； wherein denotes the number of elements in the set; The sampling score in step 5 is calculated as a sampling strategy, i.e. the sampling score is calculated according to the sample expression degree and the sample stability, and the sampling score is used as the second-stage active sampling strategy; wherein the first stage samples have sampling scores that are a weighted sum of expressiveness and stability, i.e.: ； wherein, is a hyperparameter satisfying , is the sample expressiveness of the sample , is the sample stability of the sample ; The second-stage active sampling strategy is to select the sample with the highest score from all first-stage samples as the result of the second-stage active sampling after obtaining the sampling score of all first-stage samples; The sample expression degree in step 5 is calculated as follows: Computational view Consistency The method is as follows: ； the view with the highest agreement sample expressiveness is The probability density function value obtained by using the Gaussian Mixture Model (GMM) in step 3, i.e.: ； wherein , is the number of typical views; The sample stability in step 5 is calculated as follows: The samples are denoted by The set of samples of the same category in view is denoted by The set of samples of the same category in view is denoted by The stability of the samples in view and is calculated first, as follows: ； The stability of is then calculated as follows: 。 2.The image recognition classification method based on two-stage active learning of claim 1, wherein, Step 2 uses the active learning method to perform first-stage active sampling, including: Use the active learning method on the data in the original data set, i.e. the unlabeled pool, as the first-stage active sampling strategy to obtain the first-stage sampling result, i.e. the first-stage samples, and the set of first-stage samples is a subset of the unlabeled pool. 3.The image recognition classification method based on two-stage active learning of claim 2, wherein, Step 3-2 in step 3 uses a Gaussian Mixture Model (GMM) to model the distribution of first-stage samples in each view to form a clustering result, specifically including: In Gaussian mixture models (GMMs), the probability density function is used. To depict the distribution of various samples in the same view, where, Indicates the input sample. The parameters of the GMM model , This indicates the number of Gaussian models in the Gaussian mixture model. Indicated by For the mean, The variance of the first A Gaussian model, It is the weight, that is, the current sample belongs to the first... The probability of a Gaussian model; The specific operation method of using Gaussian mixture model (GMM) to form clustering results is described as follows: let the set of all first-stage samples be wherein is the number of first-stage samples, wherein wherein, represents the first first-stage sample; Introducing a latent variable wherein, , , then performing the following steps: Step 3-2-1: Random initialization , including weights , mean , and variance ; Step 3-2-2: Compute the latent variables of the estimate as follows: ； Step 3-2-3: Calculate the GMM model parameters the estimated values of , and , as follows: ； ； ； and , and as , and ​ Step 3-2-4: Repeat Step 3-2-2 and Step 3-2-3 until the model converges, i.e. for a pre-given threshold , and is satisfied: , and ; finally using , and as an approximation of , and ; Step 3-2-5: For the first view, obtain distributions , ,..., , where represents the first parameter; according to the above distributions, respectively, in each view for each first stage sample divided into the category to which it belongs as a result of clustering.

4. The image recognition classification method based on two-stage active learning according to claim 3, characterized in that, Step 6 trains the image recognition classification model, i.e. uses the second-stage samples labeled by manual annotation in step 5 to train the image recognition classification model; During training, the loss function is composed of two parts: task loss and multi-view clustering loss , where task loss uses cross-entropy loss in classification task and only uses labeled samples in calculation; multi-view clustering loss is determined by sample consistency and uses all samples in calculation, and the specific calculation method is: ； Finally, the total loss function of the training process is L = Lr + Lc + Ld ； wherein is a hyperparameter satisfying .

5. The image recognition classification method based on two-stage active learning according to claim 4, characterized in that, The total loss function of the training process described in step 6 When the active learning method described in step 2 is the learning loss LL algorithm, the loss function value loss is introduced , the total loss function is ; the sample pair corresponding to the prediction loss value The loss function is : ； wherein, is a hyperparameter, is a real loss, is a prediction loss; The loss function value loss is: ； wherein, is the set of all sample pairs in the current training batch.

Images