Small sample image classification method and device based on feature decoupling and storage medium
This few-shot image classification method, which decouples features, utilizes a dual-path feature extraction network to decouple semantic and label information, thus solving the problem of insufficient label information extraction capability in few-shot learning. This achieves efficient image classification and network convergence, and improves classification accuracy.
Patent Information
- Application Number
- CN202510079229.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-17
- Publication Date
- 2025-10-17
- Estimated Expiration
- 2045-01-17
AI Technical Summary
Existing few-shot learning methods have limited ability to extract label information, making it difficult to achieve efficient classification with limited data, especially in fields such as military remote sensing detection, disease diagnosis, and defective product detection in industrial production, where data acquisition and labeling are difficult.
A small sample image classification method based on feature decoupling is adopted. Through a dual-path feature extraction network, one path detects semantic information and the other detects label information. Label propagation is performed through a graph neural network, ultimately achieving decoupling of label information from semantic information to enhance the extraction of label information.
It improves the accuracy of image classification and the convergence speed of the network, enhances classification performance under small sample conditions, and reaches the advanced level of current small sample classification models.
Smart Images
Figure CN119992193B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, in particular to a small sample image classification method based on feature decoupling, a device and a storage medium. BACKGROUND
[0002] In the field of computer vision research, deep learning technology has become the main method to solve problems. However, deep learning needs a large amount of labeled data for training, and it is very laborious to obtain a large amount of data and label them, and in some special scenarios, it is impossible to obtain a large amount of data. Therefore, it is crucial to adapt deep learning to small sample data. At present, the research of small sample learning is still in its infancy, and the performance gap under the condition of large sample is still large. Among many related tasks, the small sample image classification task refers to the technology of classifying images through machine learning and deep learning methods under limited training data. In some specific application scenarios, some scarce categories may be difficult to obtain and label due to privacy, security and high labeling cost, such as military remote sensing detection, disease diagnosis and defective product detection in industrial production. Therefore, this technology has wide application in medical imaging, object detection and bioinformatics, and is an important research direction in small sample learning. SUMMARY
[0003] In order to solve the problems existing in the prior art, the present application provides a small sample image classification method based on feature decoupling, a device and a storage medium, which solves the technical problem that the existing small sample learning method has limited extraction ability of label information.
[0004] The present application designs a small sample image classification method based on feature decoupling, a device and a storage medium, which deeply focuses on the label specific information into the mechanism of the classifier, and decouples it from the semantic information. The present application designs a feature decoupling network that can realize double-channel feature extraction. One channel detects label information, and the other channel detects semantic information. The two channels of information are transmitted to each other, and finally the label information and the semantic information are decoupled to strengthen the extraction of label information. Through this method, the accuracy of classification is effectively improved, which has high scientific significance and practical value.
[0005] The small sample image classification method based on feature decoupling comprises the following steps:
[0006] S1, pre-process the image data and divide it into a support set and a query set, and then input it into a feature decoupling network for processing. The feature decoupling network comprises a semantic convolutional encoder, a label convolutional encoder, a graph neural network and a decoder.
[0007] S2, the semantic convolutional encoder performs feature extraction processing on the support set image to obtain latent semantic information;
[0008] S3, a label convolutional encoder extracts features of the support set image and the query set image, and obtains latent label information after processing the latent semantic information obtained in S2;
[0009] S4, label propagation is performed on the latent label information by a graph neural network to realize prediction of the query set label, and a classification loss is calculated based on the predicted label and the real label of the query set;
[0010] S5, a decoder reconstructs an image based on the latent semantic information and the latent label information, calculates a reconstruction loss based on the reconstructed image and the original image, and integrates the classification loss and the reconstruction loss to obtain a total loss to optimize the feature decoupling network and determine network parameters.
[0011] Further, S1 comprises:
[0012] S101, pre-process the images in the original data set to match the size of the feature decoupling network;
[0013] S102, divide the processed data set into a training set, a validation set and a test set, each data set including multiple small sample tasks, and each small sample task including a support set and a query set. Further, S2 comprises:
[0014] S201, input the support set image into a semantic convolutional encoder to extract features of the image;
[0015] S202, map the features extracted in 201 into a Gaussian distribution, use two fully connected layers to obtain the mean and variance of the Gaussian distribution, and then obtain a Gaussian distribution that can be backpropagated through reparameterization to obtain latent semantic information.
[0016] Further, S3 comprises:
[0017] S301, input the support set and query set images into a label convolutional encoder to extract features;
[0018] S302, combine the features extracted in S301 with the latent semantic information obtained in S2 to fuse into a Gaussian distribution form of label information, obtain the mean and variance of the Gaussian distribution through two fully connected layers, and then obtain a Gaussian distribution that can be backpropagated through reparameterization to obtain latent label information.
[0019] Further, the specific process of S4 is:
[0020] S401, calculate the weight of any two feature vectors in the latent label information obtained in S3 on an undirected graph to obtain an undirected graph composed of the support set and the query set;
[0021] S402, label propagation is performed on the undirected graph, the known labeled support set is taken as a known node of the graph, and label propagation is performed through a label propagation algorithm to predict labels of the query set;
[0022] S403, a cross-entropy loss function is calculated according to the predicted labels and the true labels of the query set, and a classification loss is obtained.
[0023] Further, the S5 comprises:
[0024] S501, the latent semantic information obtained in S2 and the latent label information obtained in S3 are accumulated and input into a decoder for image reconstruction;
[0025] S502, a mean square error is calculated using the reconstructed image and the original image as a reconstruction loss;
[0026] S503, the KL divergence of the latent semantic information of S2 and the standard normal distribution is calculated, and the KL divergence of the latent label information of S3 and the standard normal distribution is calculated as a distribution alignment loss in the variational autoencoder, and the variational autoencoder comprises the semantic convolutional encoder and the decoder;
[0027] S504, the classification loss in S4, the reconstruction loss in S502 and the distribution alignment loss in S503 are added together as a total loss, then the gradient descent of the feature decoupling network is performed, the network parameters are optimized through back propagation, and the network is converged.
[0028] The small sample image classification device based on feature decoupling comprises:
[0029] The memory is used for storing the computer program of the small sample image classification method based on feature decoupling;
[0030] The processor is used for implementing the small sample image classification method based on feature decoupling when the computer program is executed.
[0031] A computer readable storage medium stores a computer program, and the computer program can implement the small sample image classification method based on feature decoupling when the computer program is executed by a processor.
[0032] The beneficial effects of the present application include:
[0033] The feature decoupling network used in the present application can detect in two ways, one way detects label information, and the other way detects semantic information, the two ways of information are transmitted to each other, and finally the label information and the semantic information are decoupled to strengthen the extraction of image label information.
[0034] The label propagation is used to predict the classification result, the semantic information is effectively utilized, the image classification accuracy is improved, and the network convergence speed is also improved. BRIEF DESCRIPTION OF DRAWINGS
[0035] Figure 1 A network diagram related to the embodiments of the present application.
[0036] Figure 2 A flowchart of a small sample image classification method based on feature decoupling related to the embodiments of the present application.
[0037] Figure 3 A detailed flowchart of S1 related to the embodiments of the present application.
[0038] Figure 4 A detailed flowchart of S2 related to the embodiments of the present application.
[0039] Figure 5 A detailed flowchart of S3 related to the embodiments of the present application.
[0040] Figure 6 A detailed flowchart of S4 related to the embodiments of the present application.
[0041] Figure 7 A detailed flowchart of S5 related to the embodiments of the present application. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Therefore, the detailed description of the embodiments of the present application provided in the following drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the present application.
[0043] For existing small sample image classification methods, because the ultimate goal of training the network is to obtain the label information of the predicted picture to achieve the effect of target classification, most methods focus on how to effectively extract the features of the target category under a small amount of training samples, and then use the features for classification. However, images are composed of different features, such as style, design pattern and context information, which are not necessarily relevant discriminative features for classification. In this method, these information is referred to as semantic information. On the other hand, there are some feature attributes (such as the wings of a bird, the nose of an elephant, and the hump on the back of a camel) that are crucial for classification, and these features do not need to consider the context information. In this method, such features are referred to as label information. The semantic information of the picture plays a leading role in the picture, and the label information is mixed in the semantic information and is relatively weak, but the degree of extraction of these weak label information determines the effectiveness of the classification algorithm.
[0044] The small sample image classification method based on feature decoupling, as shown in Figures 1-2 , comprises the following steps:
[0045] S1, pre-process the image data and divide it into a support set and a query set, and then input it into a feature decoupling network for processing, wherein the feature decoupling network comprises a semantic convolutional encoder, a label convolutional encoder, a graph neural network and a decoder;
[0046] S2, the semantic convolutional encoder performs feature extraction processing on the support set image to obtain latent semantic information;
[0047] S3, the label convolutional encoder performs feature extraction on the support set image and the query set image in combination with the latent semantic information obtained in S2 to obtain latent label information;
[0048] S4, based on the latent label information, label propagation is performed through the graph neural network to realize prediction of the label of the query set, and a classification loss is calculated based on the predicted label of the query set and the real label;
[0049] S5, the decoder reconstructs the image based on the latent semantic information and the latent label information, calculates a reconstruction loss based on the reconstructed image and the original image, integrates the classification loss and the reconstruction loss to obtain a total loss, and optimizes the feature decoupling network to determine the network parameters.
[0050] In another embodiment, as shown in Figure 3 , S1 comprises:
[0051] S101, pre-process the images in the original data set to match the size of the feature decoupling network;
[0052] S102, divide the processed data set into a training set, a validation set and a test set, each data set including a plurality of small sample tasks, each small sample task including a support set and a query set. Specifically, a small sample data set is obtained, and the data is preprocessed and divided into a form suitable for small sample learning, into a support set and a query set. In small sample learning, the data set is usually divided into a training task form for training, each training task contains a support set and a query set, the support set contains N categories, each category has K samples, referred to as N-way K-shot, the categories of the query set are the same as the support set, and each category has Q samples; in the field of deep learning, the data set is usually divided into a training set, a validation set and a test set, the three data sets have no intersection, which is also the case in small sample learning. In the training network stage, a plurality of small sample tasks are randomly sampled from the training set. In each task, the original network is first parameter copied, the copied network is first trained in the support set, the obtained loss is used to update the network parameters as an inner loop, and then trained in the query set, the parameters of the query set are used to update the parameters of the original network as an outer loop.
[0053] In another embodiment, as shown in Figure 4 S2 includes:
[0054] S201, input the support set image into the semantic convolutional encoder to extract the features of the picture;
[0055] S202, map the features extracted in 201 to a Gaussian distribution, use two fully connected layers to obtain the mean and variance of the Gaussian distribution, and then use the reparameterization trick to obtain a Gaussian distribution that can be backpropagated to obtain latent semantic information.
[0056] Specifically, the extraction of latent semantic information is mainly realized by a variational autoencoder. The variational autoencoder includes a convolutional encoder for extracting semantic information and a decoder. The image is input into the convolutional encoder for extracting semantic information, i.e., a semantic convolutional encoder. The obtained latent features are mapped to a Gaussian distribution to obtain low-dimensional semantic information. The convolutional encoder is a feature extraction module. In this embodiment, a 4-layer convolutional layer plus a pooling layer feature extraction head is used. Other types of feature extraction heads can also be used. Two fully connected layers are used to map the extracted features to a Gaussian distribution, converting the extracted features to a Gaussian distribution form to obtain two feature vectors σ and μ. Then, the reparameterization trick is used to obtain a Gaussian distribution that can be backpropagated as latent semantic information z s ; wherein the calculation method of the reparameterization is: z s = μ + σ × ε
[0057] wherein z sFor the extracted potential semantic information, μ is the previously obtained mean vector, σ is the previously obtained standard deviation vector, and ε is noise sampled from a standard normal distribution.
[0058] In another embodiment, as shown in Figure 5 S3 includes:
[0059] S301, input the support set and the query set image into the label convolutional encoder together for feature extraction;
[0060] S302, combine the extracted features in S301 with the potential semantic information obtained in S2 to map into a Gaussian distribution form of label information, pass through two fully connected layers to obtain the mean and variance of the Gaussian distribution, and then pass through a reparameterization technique to obtain a Gaussian distribution that can be backpropagated to obtain potential label information.
[0061] Specifically, the extraction of potential label information takes the support set and the query set as input, extracts features, combines the obtained potential features with the potential semantic information obtained in S2, maps into a Gaussian distribution, and obtains low-dimensional label information z l .
[0062] In another embodiment, as shown in Figure 6 The specific process of S4 is:
[0063] S401, for any two feature vectors in the potential label information obtained in S3, use a Gaussian similarity function to calculate their weights on the undirected graph to obtain an undirected graph composed of the support set and the query set;
[0064] S402, label propagation is performed on the undirected graph, the labeled support set is taken as a known node of the graph, label propagation is performed through a label propagation algorithm, and the label of the query set is predicted;
[0065] S403, calculate the cross-entropy loss function according to the predicted label of the query set and the true label to obtain a classification loss.
[0066] Specifically, the classification loss is calculated by performing label propagation on the potential label information obtained in S3 through a graph neural network constructed by feature embedding and an undirected graph to predict the query set and obtain a classification loss. The specific implementation process is as follows: for any two feature vectors in the potential label information obtained in S3, use a Gaussian similarity function to calculate their weights on the undirected graph to obtain an undirected graph composed of the support set and the query set, then use a label propagation algorithm to perform label propagation to obtain a classification result of the query set; calculate the cross-entropy loss function according to the predicted result and the true label to obtain a classification loss L class The calculation method of the classification loss is as follows:
[0067] n is the number of samples, k is the number of categories, y ic is the one-hot encoding of the sample target value, h θ (x i ) c is the observed sample x i belongs to category c.
[0068] It should be noted that the Gaussian similarity function is used as the calculation method of the weight in the embodiment, and other distance algorithms can also be used to calculate the weight between two features.
[0069] In another embodiment, as Figure 7 shown, S5 includes:
[0070] S501, the latent semantic information obtained in S2 and the latent label information obtained in S3 are accumulated and input into the decoder for image reconstruction;
[0071] S502, the mean square error between the reconstructed image and the original image is calculated as the reconstruction loss;
[0072] S503, the relative entropy of the latent semantic information of S2 and the standard normal distribution, also known as Kullback-Leibler divergence, is calculated, and the KL divergence of the latent label information of S3 and the standard normal distribution is calculated. The sum of the two KL divergences is added as the distribution alignment loss in the variational autoencoder, and the variational autoencoder includes the semantic convolutional encoder and the decoder;
[0073] S504, the classification loss in S4, the reconstruction loss in S502, and the distribution alignment loss in S503 are added together as the total loss, and then the gradient descent of the entire network is performed. Back propagation, optimize the parameters of the network, make the network converge.
[0074] Specifically, the semantic features extracted in S2 and the label features extracted in S3 are used for image reconstruction to obtain a reconstructed image and the original image y i The mean square error is calculated as the reconstruction loss L recon , and the calculation method is as follows:
[0075] Where n represents the total number of samples, y i represents the original image, represents the reconstructed image, and L recon represents the reconstruction loss.
[0076] The latent semantic information z sThe relative entropy with respect to the standard normal distribution is also called Kullback-Leibler divergence, hereinafter referred to as KL divergence, denoted as L s-kl ; Calculate the potential label information z in S3 l KL divergence with the standard normal distribution, denoted as L l-kl , put L s-kl With L l-kl The sum is used as the distribution alignment loss in the variational self-encoder. The KL divergence with the standard normal distribution is calculated as follows:
[0077] The role of the variational autoencoder is to effectively extract features, because the encoder of the variational autoencoder actually maps the extracted features into a Gaussian distribution. This Gaussian distribution needs to be as similar as possible to the standard normal distribution to ensure the effectiveness of feature extraction. Therefore, a distribution alignment loss is required to ensure that the distribution is as similar as possible.
[0078] Where μ is the mean vector of the Gaussian distribution, and σ is the variance of the Gaussian distribution.
[0079] The classification loss L in S4 class , reconstruction loss L recon , distribution alignment loss L kl Add them up as the total loss L.
[0080] The calculation method is as follows: L = α c L class +α r L recon +β s L s-kl +β l L l-kl
[0081] Among them L class is the classification loss, L recon To reconstruct the loss, L s-kl is the distribution alignment loss of latent semantic information, L l-kl is the distribution alignment loss of the latent label information, α c For L class The weight, α r For L recon The weight, β s For L s-kl The weight, β l For L l-kl The weight of .
[0082] Then for Figure 1 The feature decoupling classification network shown in the figure performs back propagation and gradient descent to optimize the parameters of the network and make the network converge.
[0083] The optimization target of the whole network is jointly acted on by the multi-class loss, so that the whole network can achieve better performance in classification and reconstruction tasks. It is guaranteed that the label information extraction and semantic information extraction achieve good effects
[0084] The application achieves good classification performance in the small sample image classification task, as shown in Table 1, and achieves 90.77% classification accuracy in the 5-way, 1-shot task on the miniImage dataset, and achieves 96.12% classification accuracy in the 5-way, 5-shot task, reaching the advanced level of the current small sample classification model.
[0085] Table 1: Comparison of classification performance
[0086]
[0087] In another embodiment, a small sample image classification device based on feature decoupling is involved, comprising:
[0088] Memory: for storing the computer program of the cross-domain small sample image classification method based on local correlation reasoning;
[0089] Processor: for implementing the small sample image classification method based on feature decoupling when the computer program is executed.
[0090] In another embodiment, a computer readable storage medium is involved, and the computer readable storage medium stores a computer program, and the computer program can implement the small sample image classification method based on feature decoupling when executed by a processor.
[0091] The above embodiments only express the specific implementation of the application, and the description is more specific and detailed, but it cannot be understood as a limitation on the protection scope of the application. It should be pointed out that for ordinary skilled persons in the art, without departing from the technical scheme concept of the application, a number of modifications and improvements can be made, which all belong to the protection scope of the application.
Claims
1. A small sample image classification method based on feature decoupling, characterized by: The following steps are involved: S1. Preprocess the image data and divide it into a support set and a query set, and then input it into a feature decoupling network for processing. The feature decoupling network includes a semantic convolutional encoder, a label convolutional encoder, a graph neural network, and a decoder. S2, the semantic convolutional encoder extracts features from the support set image to obtain potential semantic information; S3, the label convolution encoder extracts features from the support set image and the query set image and combines them with the potential semantic information obtained in S2 to obtain potential label information; S4. Based on the potential label information, label propagation is performed through the graph neural network to predict the query set label, and the classification loss is calculated based on the predicted label and the true label of the query set; S5. The decoder reconstructs the image based on the potential semantic information and potential label information, and calculates the reconstruction loss based on the reconstructed image and the original image; the classification loss and reconstruction loss are integrated to obtain the total loss to optimize the feature decoupling network and determine the network parameters.
2. The small sample image classification method based on feature decoupling according to claim 1, characterized in that: The S1 includes: S101, preprocessing the images in the original dataset to make their sizes match those of the feature decoupling network; S102: Divide the processed data set into a training set, a validation set, and a test set. Each data set includes multiple small sample tasks, and each small sample task includes a support set and a query set.
3. The small sample image classification method based on feature decoupling according to claim 1, characterized in that: The S2 includes: S201, input the support set image into the semantic convolution encoder to extract features of the image; S202: Map the features extracted in step 201 into a Gaussian distribution, use two fully connected layers to obtain the mean and variance of the Gaussian distribution, and then reparameterize to obtain a Gaussian distribution that can be back-propagated to obtain potential semantic information.
4. The small sample image classification method based on feature decoupling according to claim 1, characterized in that: The S3 includes: S301, input the support set and query set images into the label convolution encoder for feature extraction; S302: Combine the features extracted in S301 with the potential semantic information obtained in S2, and map them into a Gaussian distribution of label information. The mean and variance of the Gaussian distribution are obtained through two fully connected layers, and then a Gaussian distribution that can be back-propagated is obtained through reparameterization to obtain the potential label information.
5. The small sample image classification method based on feature decoupling according to claim 1, characterized in that: The specific process of S4 is as follows: S401, calculating the weights of any two feature vectors in the potential label information obtained in S3 on an undirected graph, and obtaining an undirected graph consisting of a support set and a query set; S402: Perform label propagation on the undirected graph, using the known annotated support set as the known nodes of the graph, and perform label propagation through the label propagation algorithm to predict the label of the query set; S403: Calculate the cross entropy loss function based on the predicted labels and true labels of the query set to obtain the classification loss.
6. The small sample image classification method based on feature decoupling according to claim 1, characterized in that: The S5 includes: S501, accumulating the latent semantic information obtained in S2 and the latent label information obtained in S3 and inputting the accumulated information into the decoder for image reconstruction; S502, using the reconstructed image and the original image to calculate the mean square error as the reconstruction loss; S503, calculating the KL divergence between the potential semantic information of S2 and the standard normal distribution, and calculating the KL divergence between the potential label information of S3 and the standard normal distribution, as the distribution alignment loss in the variational autoencoder, the variational autoencoder including the semantic convolutional encoder and decoder; S504: Add the classification loss in S4, the reconstruction loss in S502, and the distribution alignment loss in S503 as the total loss, then perform gradient descent and back propagation on the feature decoupling network to optimize the network parameters and make the network converge.
7. A small sample image classification device based on feature decoupling, characterized in that: The small sample image classification method based on feature decoupling according to any one of claims 1 to 6 is applied, comprising: Memory: used for storing the computer program of the small sample image classification method based on feature decoupling; Processor: used to implement a small sample image classification method based on feature decoupling when executing the computer program.
8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the small sample image classification method based on feature decoupling according to any one of claims 1 to 6.