Training method of fine-grained image clustering model based on diffusion model and fine-grained image clustering method

Abstract

The present invention discloses a training method for a fine-grained image clustering model based on a diffusion model and a fine-grained image clustering method. This training method first trains the semantic extractor of the fine-grained image clustering model through a pre-trained diffusion model, and forms a background removal mask through an attention mechanism during the training process, which can normalize the diffusion target, help text conditions accurately focus on the object semantics, and achieve rapid pre-optimization of the semantic extractor; after the semantic extractor is first optimized separately, the clustering loss is added to jointly train and optimize the semantic extractor and clustering head of the fine-grained image clustering model, which can align the proxy words obtained by the semantic extractor with the clustering target, and achieve effective image grouping. This training method does not require data augmentation, and at the same time solves the problem of excessive redundant background information, so that the trained fine-grained image clustering model can accurately perform fine-grained image clustering.

Description

Technical Field

[0001] The present invention relates to the field of image classification, and in particular to a training method for a fine-grained image clustering model based on a diffusion model and a fine-grained image clustering method. Background Art

[0002] Currently, traditional image clustering algorithms mainly focus on clustering analysis at the coarse-grained data level, but often ignore the in-depth mining of fine-grained data. Figure 2 The difference between coarse-grained and fine-grained data is intuitively demonstrated. In sharp contrast to traditional coarse-grained clustering methods, fine-grained clustering achieves more refined divisions by capturing subtle differences in target feature attributes, which is undoubtedly a more challenging task. Given the diversity of organisms in the real world, the need for fine-grained clustering is becoming increasingly prominent in various practical applications. For example, in the wild, when infrared cameras can only provide image data, the advantages of fine-grained clustering technology are particularly significant. It can help researchers and ecological conservation workers accurately classify biological subspecies based on image features, even when faced with biological subspecies that are difficult to identify with the naked eye.

[0003] Deep clustering methods have made significant progress in recent years, leveraging the feature extraction capabilities of neural networks. Existing deep clustering research can be roughly divided into two paradigms: discriminative clustering and generative clustering. Specifically, discriminative clustering methods optimize cluster boundaries by pulling instances within clusters together while separating instances between clusters. Contrastive clustering methods, as a representative example, use data augmentation to construct pairs of positive and negative samples. Another line of research is generative clustering, which assumes that instances from different clusters follow different distributions. Leveraging this distribution prior, generative clustering methods learn the underlying data distribution through models such as variational autoencoders and generative adversarial networks, grouping instances with similar distributions to achieve clustering.

[0004] The first existing technology is the representative work of the discriminant clustering method (CC model) such as Figure 3 As shown in Figure 1, the CC model consists of three main components: a pair of identical backbone networks, connected to the instance-level comparison head and the cluster-level comparison head. Specifically, CC first constructs data pairs through data augmentation. The augmented images are fed into the two backbone networks to extract features. Finally, the instance-level comparison head and the cluster-level comparison head perform comparative learning on the rows and columns of the feature matrix, respectively. After training, the cluster distribution is derived based on the soft labels predicted by the cluster comparison head.

[0005] The disadvantage of the existing technology is that it relies heavily on data augmentation strategies. However, fine-grained features such as bird crowns may be destroyed by data augmentation strategies such as color jittering and random cropping, resulting in poor inter-cluster separation, such as Figure 4 shown.

[0006] The second existing technology is the representative work of the generative clustering method (C3-GAN), which is implemented as follows Figure 5 As shown, in general, C3-GAN is based on a generative adversarial network, which has two generators: foreground generator and background generator , and a discriminator C3-GAN inputs random noise to the foreground generator to generate a foreground image containing subject information (such as the outline and details of the bird), and inputs random noise and potential code at the same time. c Generate a picture with only background information (such as blue sky and branches) for the background generator. c is a one-hot vector that represents the unique labels for different categories. Suppose we have a set of items and we want to group them into categories “Shiba Inu”, “Shepherd Dog”, and “Golden Retriever”. For the “Shiba Inu” class, the latent code ; For the "Sheepdog" class, the potential code ; For the "Golden Retriever" class, the potential code The generated foreground image and background image are then synthesized into a whole and compared with the original image in the discriminator, which outputs the comparison result (Boolean value: true or false) and image features. The image features are then projected into the cluster space by a linear network, and the latent code c The image features are projected into the cluster space by a linear network as the cluster center. In the cluster space, the image features are brought closer to the cluster center and different cluster centers are pushed further away to achieve clustering.

[0007] The disadvantage of the second prior art is that the latent features used for clustering contain redundant background information, which drowns out the tiny fine-grained semantic information. Since the latent features are used to reconstruct all image pixels, they are inevitably contaminated by redundant background information, thus obscuring the fine-grained semantics. For example, Figure 6 As shown, due to the large differences in background, birds of the same species may show different underlying traits, resulting in poor intra-cluster compactness.

[0008] In summary, the key to fine-grained image clustering is to capture subtle differences between categories. However, such subtle signals can be destroyed by data augmentation in discriminative methods (represented by prior art 1) or overwhelmed by redundant background information in generative methods (represented by prior art 2). Summary of the Invention

[0009] In response to the above-mentioned deficiencies in the prior art, the present invention provides a training method for a fine-grained image clustering model based on a diffusion model and a fine-grained image clustering method, which solves the problem in the prior art that it is difficult to obtain a model or method that can be effectively used for fine-grained image clustering.

[0010] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0011] A method for training a fine-grained image clustering model based on a diffusion model is provided. The fine-grained image clustering model includes a semantic extractor and a clustering head. The training method includes the following steps:

[0012] S1. Build a pre-trained diffusion model; add noise to the original image sample to obtain a noisy image, and obtain the image features of the noisy image;

[0013] S2, mapping the image features of the noise image into proxy words through a semantic extractor; generating corresponding text prompts based on the proxy words;

[0014] S3. Using the noise image and its corresponding text prompt as input to the pre-trained diffusion model, obtaining the noise image generated by the pre-trained diffusion model in the training phase or the original image generated by the pre-trained diffusion model in the prediction phase;

[0015] S4, based on the image features of the noisy image and the corresponding text prompt, a background removal mask is formed through the attention mechanism;

[0016] S5. Generate mask image:

[0017] Combine the original image sample with the background removal mask to obtain the masked original image; or combine the noise image with the background removal mask to obtain the masked noise image;

[0018] The noise image generated by the pre-trained diffusion model in the training phase is combined with the background-removed mask to obtain a masked noise generated image; or the original image generated by the pre-trained diffusion model in the prediction phase is combined with the background-removed mask to obtain a masked original generated image;

[0019] S6. Calculate the current loss and update the semantic extractor based on the current loss:

[0020] Calculate the current generation loss based on the masked original image and the masked original generated image, or calculate the current diffusion loss based on the masked noise generated image and the masked noise image, and update the semantic extractor based on the current diffusion loss or the current generation loss;

[0021] S7, determine whether the semantic extractor has reached the end training condition, if so, proceed to step S8; otherwise, return to step S2;

[0022] S8, using the same method as steps S2 to S6 to obtain the proxy word and the current loss;

[0023] S9, clustering and assigning the proxy words obtained in step S8 through the clustering head to obtain a clustering result;

[0024] S10. Calculate the clustering loss of the fine-grained image clustering model based on the clustering results, and take the sum of the clustering loss and the current diffusion loss as the total loss, or take the sum of the clustering loss and the current generation loss as the total loss; update the semantic extractor and clustering head based on the current total loss;

[0025] S11, determining whether the fine-grained image clustering model has reached the end training condition, if so, completing the training of the fine-grained image clustering model; otherwise, returning to step S8;

[0026] The fine-grained image clustering model is used for image clustering.

[0027] A fine-grained image clustering method is provided, which includes the following steps:

[0028] A1. Obtain image features of the image to be clustered;

[0029] A2, using the trained semantic extractor to map the image features of the image to be clustered into proxy words;

[0030] A3. Use the trained clustering head to predict the proxy words obtained in step A2 to complete fine-grained image clustering.

[0031] The beneficial effects of the present invention are as follows: This training method first trains the semantic extractor of the fine-grained image clustering model using a pre-trained diffusion model, and during the training process, forms a background removal mask through the attention mechanism. This can normalize the diffusion target, help the text conditions accurately focus on the object semantics, and achieve rapid pre-optimization of the semantic extractor. After the semantic extractor is first optimized separately, a clustering loss is added to jointly train and optimize the semantic extractor and clustering head of the fine-grained image clustering model. This can align the proxy words obtained by the semantic extractor with the clustering target, achieving effective image grouping. This training method does not require data augmentation and solves the problem of excessive redundant background information, allowing the trained fine-grained image clustering model to accurately perform fine-grained image clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flowchart of the training method of the fine-grained image clustering model based on the diffusion model;

[0033] Figure 2 is the difference between coarse-grained and fine-grained data;

[0034] Figure 3 This is the implementation method of the existing technology 1 (CC model);

[0035] Figure 4 Schematic diagram of the shortcomings of the prior art 1;

[0036] Figure 5 This is the implementation method of the second existing technology;

[0037] Figure 6 Schematic diagram of the shortcomings of the second prior art;

[0038] Figure 7 Schematic diagram of data flow for training using diffusion loss;

[0039] Figure 8 Schematic diagram of data flow for training using generative loss. DETAILED DESCRIPTION

[0040] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0041] like Figure 1 As shown, the training method of the fine-grained image clustering model based on the diffusion model includes a semantic extractor and a clustering head. The training method includes the following steps:

[0042] S1. Build a pre-trained diffusion model; add noise to the original image sample to obtain a noisy image, and obtain image features of the noisy image;

[0043] S2, mapping the image features of the noise image into proxy words through a semantic extractor; generating corresponding text prompts based on the proxy words;

[0044] S3. Using the noise image and its corresponding text prompt as input to the pre-trained diffusion model, obtaining the noise image generated by the pre-trained diffusion model in the training phase or the original image generated by the pre-trained diffusion model in the prediction phase;

[0045] S4, based on the image features of the noisy image and the corresponding text prompt, a background removal mask is formed through the attention mechanism;

[0046] S5. Generate mask image:

[0047] Combine the original image sample with the background removal mask to obtain the masked original image; or combine the noise image with the background removal mask to obtain the masked noise image;

[0048] The noise image generated by the pre-trained diffusion model in the training phase is combined with the background-removed mask to obtain a masked noise generated image; or the original image generated by the pre-trained diffusion model in the prediction phase is combined with the background-removed mask to obtain a masked original generated image;

[0049] S6. Calculate the current loss and update the semantic extractor based on the current loss:

[0050] Calculate the current generation loss based on the masked original image and the masked original generated image, or calculate the current diffusion loss based on the masked noise generated image and the masked noise image, and update the semantic extractor based on the current diffusion loss or the current generation loss;

[0051] S7, determine whether the semantic extractor has reached the end training condition, if so, proceed to step S8; otherwise, return to step S2;

[0052] S8, using the same method as steps S2 to S6 to obtain the proxy word and the current loss;

[0053] S9, clustering and assigning the proxy words obtained in step S8 through the clustering head to obtain a clustering result;

[0054] S10. Calculate the clustering loss of the fine-grained image clustering model based on the clustering results, and take the sum of the clustering loss and the current diffusion loss as the total loss, or take the sum of the clustering loss and the current generation loss as the total loss; update the semantic extractor and clustering head based on the current total loss;

[0055] S11, determining whether the fine-grained image clustering model has reached the end training condition, if so, completing the training of the fine-grained image clustering model; otherwise, returning to step S8;

[0056] The fine-grained image clustering model is used for image clustering.

[0057] It should be noted that the clustering in this embodiment is unsupervised clustering, so no label information is needed during the training process.

[0058] In the specific implementation process, the pre-trained diffusion model includes a UNet network and a text encoder, and the conditional generation model Stable Diffusion can be used; the text encoder is used to encode the text prompts as the input of the UNet network, and the encoded text prompts are used to guide the UNet network to generate corresponding images based on the input noise image.

[0059] The diffusion model includes a training phase (forward diffusion process, corresponding to diffusion loss) and a prediction phase (reverse generation process, corresponding to generation loss) in the data processing process. The training phase gradually "noises" the original data until it is completely noisy. That is, the diffusion model gradually adds noise to the input image through a series of time steps (from t=0 to t=T). At each time step t, the input image A standard normal distribution noise is added to become a noise image , the degree of noise increases with time t, which can be expressed by the following recursive formula:

[0060]

[0061] in is the conditional probability distribution of the forward diffusion process, which represents the distribution of the image at the next time step t predicted by the diffusion model given the image at the current time step t-1; is the variance parameter of the noise, which gradually increases as the time step t increases; represents Gaussian distribution; I is the identity matrix; is the image at the t−1th time step; is the image at the t-th time step.

[0062] The goal of the prediction phase is to learn how to reverse the noise process, that is, to generate images from random noise by gradually denoising. This process is achieved by learning a parameterized model to approximate the true back-diffusion process, which can be expressed as:

[0063]

[0064] in is the conditional probability distribution of the reverse generation process, which represents the distribution of the image at the previous time step t-1 predicted by the diffusion model given the image at the current time step t; is the denoised mean of the model predictions; is the variance of the noise. At each time step t in the reverse process, the variance controls the degree of denoising at each step. As the number of time steps increases, the noise gradually decreases, so the variance also gradually decreases.

[0065] like Figure 7 As shown in , if the noise image generated in the training phase is used to calculate the corresponding diffusion loss, the diffusion loss represents the difference between the noise image generated by the diffusion model and the added noise. Figure 8As shown, if the original image generated during the prediction phase is used to calculate the corresponding generation loss, the generation loss represents the difference between the original image generated by the diffusion model and the original image sample. While the two loss calculation methods differ, their goal is to ensure that the proxy words output by the semantic extractor are closer to the actual results. To reduce data computation, this embodiment prefers the diffusion loss for subsequent processing.

[0066] The specific method of adding noise to the original image sample in step S1 is:

[0067] Perform Gaussian noise sampling on the original image sample to generate a noise image, whose expression is:

[0068]

[0069] in for t Noise image when ; for t The noise intensity at is the original image sample; is Gaussian noise, that is, the added noise.

[0070] The objective function of the existing diffusion model is:

[0071]

[0072] in Generate an image for the noise, represents a text encoder, Represents a model for extracting image features. The diffusion model iteratively generates images by predicting and removing noise at each time step, which is different from the objective function usually used for training and Different from this, our method proposes to derive the image generation algorithm from the pre-trained diffusion model. Text prompt c The derived textual hint encapsulates the image semantics and can therefore be used for clustering. To achieve semantic distillation, the most straightforward solution is to derive a textual hint for each image. c However, this implementation has two inherent limitations. On the one hand, it is very difficult to process large datasets because the very time-consuming sampling process must be repeated for each image. On the other hand, such a distillation process is independent for each image, hindering the potential collaboration between different images.

[0073] Therefore, instead of optimizing textual hints directly, this method introduces a shared semantic extractor that can integrate image features into Mapped to text prompts c Agent words in (For example, c =“a photo of a ”) Here the image features The intermediate layer representation corresponding to the b-th block of the UNet network is:

[0074]

[0075] in for When is the feature output by the middle layer of the bth block of the UNet network, that is, the image feature of the noise image; Represents the middle layer of the b-th block of the UNet network; for Noise image when ; Indicates the text prompt content. Represents the output of the text encoder. When the UNet network first obtains the image features of the noise image, Empty, that is, the input text is an empty string; is the time step for calculating image features.

[0076] It should be noted that although this embodiment uses the output of the intermediate layer of the bth block of the UNet network as the image feature of the noise image, other means can also be used to obtain the image features of the noise image, including but not limited to the intermediate layer of the pre-trained CNN and the bottleneck feature of the autoencoder, which can achieve the same effect.

[0077] Pre-trained CNN intermediate layers: Use feature maps from the intermediate layers of a pre-trained convolutional neural network (e.g., ResNet, VGG, or EfficientNet). For example, extract features from a specific convolutional layer or fully connected layer before the classification head.

[0078] Autoencoder bottleneck features: Use the latent space representation of the autoencoder’s output as features.

[0079] In the specific implementation process, The value of is 150. In order to prevent inconsistent Gaussian noise from interfering with image features, we extract image features For all images at time step The same noise is sampled at In all experiments, b The value of is 19. Therefore, in order to perform semantic extraction, the above objective function is rewritten as follows:

[0080]

[0081] In this rewritten objective function, textual cues such as “A photo of a” are omitted for brevity. By minimizing this rewritten objective function, the semantic extractor is able to refine image semantics into proxy words Since the semantic extractor is shared across different images, it takes less time to converge than extracting textual terms for each image independently.

[0082] In addition, during the denoising process, the diffusion model tends to recover high-level semantics first (corresponding to large time steps) and then fill in low-level details (corresponding to small time steps). Since fine-grained clustering aims to distinguish images with subtle differences, we propose the following weighted time step sampling scheme during the semantic distillation process:

[0083]

[0084] in corresponds to the probability of sampling time step t from the total diffusion time step T; π is the circumference of the circle. By emphasizing small diffusion time steps, the extracted proxy It can better capture fine-grained details, thus facilitating the subsequent clustering process.

[0085] Since the diffusion model is trained to restore all pixels in the image, the extracted text conditions inevitably contain redundant background information, which interferes with the clustering based on foreground objects. To alleviate this problem, we introduce an object-centered mask to normalize the diffusion target. Specifically, the specific method of forming the background removal mask through the attention mechanism in step S4 includes the following sub-steps:

[0086] S4-1, generating an image query vector based on the image features of the noise image; generating a text key vector based on the corresponding text prompt;

[0087] S4-2, based on the image query vector, text key vector and feature dimension parameters, the softmax function is used to process and obtain the attention map corresponding to the current noise image;

[0088] S4-3. Model the attention value distribution of the attention map corresponding to the current noise image through a bimodal Gaussian mixture model, and its expression is:

[0089]

[0090] in is the variable in the bimodal Gaussian mixture model The probability density function of and is the Gaussian mixing coefficient; variable is the attention map corresponding to the current noise image; The mean of the Gaussian probability density function is And the variance is The probability density of the components of ; The mean of the Gaussian probability density function is And the variance is The probability density of the components of ;

[0091] S4-4. Parameters based on average values , mean value parameter The attention value at each position in the attention map corresponding to the current noise image is used to calculate the binary mask as the background removal mask; the calculation expression of the binary mask is:

[0092]

[0093] in Indicates that the position in the current noise image is The binary mask value at , , is a binary mask, i.e., a mask to remove the background; when When the value is 1, it means that the position in the current noise image is retained. The original characteristics of When the value is 0, it means removing the current noise image at position The original characteristics of the place; express The middle position is The original characteristics of the place.

[0094] The expressions for generating image query vectors and text key vectors are:

[0095]

[0096] in Indicates the first i The image query vector corresponding to each UNet block; and are all weight matrices; Indicates the i a vector of text keys; Indicates the i UNet blocks in The image features output at is the time step for calculating image features;

[0097] Based on the image query vector, text key vector and feature dimension parameters, the softmax function is used to process the attention map corresponding to the current noise image:

[0098]

[0099] in For the i The attention map corresponding to the image features output by each UNet block; for L The average attention map corresponding to the image features output by the UNet blocks is the attention map corresponding to the current noise image. When the value of L is 1, it means that the attention map corresponding to the image features output by one of the UNet blocks is directly selected as the attention map corresponding to the current noise image. Represents the transpose of a matrix; d is the image feature dimension; Represents the softmax function.

[0100] The calculation expression of the diffusion loss in step S6 is:

[0101]

[0102] in is the diffusion loss; is the added noise; For mask; Indicates term-by-term multiplication; Generate an image for the noise; represents the L2 norm; Expressing expectation.

[0103] In the specific implementation, the proxy words contain rich semantics of the main objects in the image. To further align the refined image semantics with the clustering target, we introduce a clustering head to group the proxy words based on neighborhood similarity. Specifically, given a set of proxy words, the clustering head predicts a soft cluster assignment for each proxy word, encouraging consistent cluster assignment between each proxy word and its neighboring proxy words. The clustering target can be expressed as:

[0104]

[0105]

[0106]

[0107] To prevent the clustering head from assigning most samples to a few clusters, this embodiment introduces an entropy regularization term:

[0108]

[0109]

[0110] In order to accurately reflect the overall cluster assignment distribution, the number of samples N in each batch should be larger than the number of clusters C. However, due to the high memory requirements of the diffusion model, the batch size N is usually much smaller than C. Therefore, the entropy regularization term cannot effectively reflect the distribution of all cluster assignments, resulting in poor performance. As a solution, we add a repository of length U = 512 to store historical cluster assignments. All assignments in the repository will participate in the entropy regularization term. The total clustering loss is in the form of:

[0111]

[0112] in is the clustering loss; is the neighbor similarity item; is the entropy regularization term; Indicates agent The cluster assignment of represents the clustering head; Indicates agent A randomly selected neighbor proxy word Cluster assignment of Indicates that the N proxy words currently input are clustered into The average probability of each category; Indicates the current input n The proxy words are clustered into The probability of each category; Represents the transpose of a matrix; It represents the logarithm with the natural constant e as the base; is the strength of the entropy regularization term, .

[0113] The expression of the total loss as the sum of clustering loss and current diffusion loss is:

[0114]

[0115] The above total loss not only optimizes the clustering head to achieve clustering, but also optimizes the semantic extractor to guide proxy word extraction.

[0116] In one embodiment of the present invention, fine-grained data (This can be specific image data of birds, dogs, plants, etc.) as an example to describe the process of the present invention. We use the pre-trained conditional generation model Stable Diffusion and freeze all the parameters of the model so that it does not participate in training. The network we need to train is the semantic extractor and clustering head The complete training process is divided into two stages: warm-up and training. The clustering head is not trained in the warm-up stage. The purpose of warm-up is to improve the accuracy of neighbors and prevent the clustering head from using incorrect neighbor information to reduce clustering performance during training. The warm-up stage is as follows:

[0117] 1. First, fine-grained data The image is fed into the pre-trained conditional generative model Stable Diffusion to extract image features. The image features are then fed into the semantic extractor to obtain proxy words. Finally, the proxy words and fixed text prompts are concatenated into a sentence as the condition for the conditional generative model.

[0118] 2. According to diffusion loss Calculate the loss and optimize the parameters and weights of the semantic extractor through backpropagation.

[0119] 3. Repeat steps 1 and 2 for a total of 100 rounds.

[0120] The training phase is as follows:

[0121] 1. After 100 rounds of warm-up, introduce the clustering head First, the same process converts the fine-grained data The image is fed into the pre-trained conditional generative model, Stable Diffusion, to extract image features. These features are then fed into a semantic extractor to obtain proxy words. Finally, the proxy words and the fixed text prompt are concatenated into a sentence as the condition. Simultaneously, the proxy words are passed through a clustering head to obtain soft cluster assignments.

[0122] 2. According to the total loss Calculate the loss and optimize the parameters and weights of the semantic extractor and clustering head through backpropagation.

[0123] 3. Repeat steps 1 and 2 for a total of 150 rounds.

[0124] After the training process converges, the fine-grained image clustering method is implemented based on the fine-grained image clustering model, including the following steps:

[0125] A1. Obtain image features of the image to be clustered;

[0126] A2, using the trained semantic extractor to map the image features of the image to be clustered into proxy words;

[0127] A3. Use the trained clustering head to predict the proxy words obtained in step A2 to complete fine-grained image clustering.

[0128] In this embodiment, the network structure adopted by the semantic extractor is as follows:

[0129] The first layer is a convolutional layer with an input of 1280 and an output of 1280, and an activation function of ReLU, followed by a maximum pooling layer.

[0130] The second layer is a convolutional layer with an input of 1280 and an output of 768. The activation function is ReLU, followed by a maximum pooling layer.

[0131] The third layer is an adaptive average pooling layer with an input of 768 and an output of 3072.

[0132] The fourth layer is a fully connected layer with an input shape of 3072 and an output shape of 768.

[0133] The network structure used by the clustering head is as follows:

[0134] First layer: fully connected layer with input shape of 768 and output shape of 768, followed by batch normalization layer with input shape of 768 and output shape of 768. The activation function is ReLU.

[0135] The second layer is a fully connected layer with an input shape of 768, an output shape equal to the number of clusters, and a Softmax activation function.

[0136] Weight initialization: The weights of the first and second fully connected layers are initialized using a truncated normal distribution with a standard deviation of 0.02.

[0137] It should be noted that although this embodiment provides the network structure of the semantic extractor and clustering head, this training method is also applicable to semantic extractors and clustering heads of other network structures, and the training process has no direct relationship with the specific network structure.

[0138] In order to quantify the effect of the present invention, two sets of comparative experiments and visualizations are designed in this embodiment. All experiments are performed on the following four classic fine-grained datasets: CUB, Car, Dog, and Flower. Brief information about these datasets is summarized in Table 1. It is worth noting that we train the clustering model on the experimental training set partition and evaluate it on the experimental test set partition. Due to the small number of training images, the experimental training set partition of CUB and Flower includes the entire dataset. We use two widely used indicators, namely clustering accuracy (ACC) and normalized mutual information (NMI), to evaluate the clustering performance. The higher the values ​​of the two indicators, the better the results.

[0139] Table 1

[0140]

[0141] Experiment 1:

[0142] For discriminant clustering methods requiring data augmentation, as exemplified by prior art 1, our present invention was compared with the discriminant clustering methods IIC, SimCLR, MoCo, SCAN, and SeCu. The experimental results, shown in Table 2, demonstrate that our method surpasses these representative discriminant clustering methods. In particular, despite SeCu's advanced pseudo-labeling strategy, our method still outperforms the discriminant method SeCu. This result can be attributed to the inherent limitations of data augmentation in capturing fine-grained features, which our method cleverly circumvents.

[0143] Table 2

[0144]

[0145] Experiment 2:

[0146] For generative clustering represented by the second prior art, we compare the present invention with InfoGAN, FineGAN, MixNMatch, OneGAN, Stable Diffusion, and C3-GAN. The experimental results are shown in Table 3. These results prove two conclusions: (1) The present method surpasses the above-mentioned generative clustering methods that learn latent image features, demonstrating the superiority of the present method in extracting semantics into text conditions. (2) Since the present method is built on the pre-trained StableDiffusion model, the clustering performance is benchmarked by directly applying the k-means method on the image features extracted by the UNet network encoder. It turns out that the present method significantly outperforms this simple baseline, indicating that the diffusion model itself cannot learn discriminative features, while the present method can effectively capture image semantics.

[0147] Table 3

[0148]

[0149] In summary, this training method first trains the semantic extractor of the fine-grained image clustering model using a pre-trained diffusion model. During the training process, an attention mechanism is used to form a background removal mask, which can normalize the diffusion target, help the text conditional accurately focus on the object semantics, and achieve rapid pre-optimization of the semantic extractor. After the semantic extractor is first optimized separately, a clustering loss is added to jointly train and optimize the semantic extractor and clustering head of the fine-grained image clustering model. This allows the proxy words obtained by the semantic extractor to be aligned with the clustering target, achieving effective image grouping. This training method does not require data augmentation and solves the problem of excessive redundant background information, allowing the trained fine-grained image clustering model to accurately perform fine-grained image clustering.

Claims

1. A training method for a fine-grained image clustering model based on a diffusion model, characterized in that: The fine-grained image clustering model includes a semantic extractor and a clustering head. The training method includes the following steps: S1. Build a pre-trained diffusion model; add noise to the original image sample to obtain a noisy image, and obtain the image features of the noisy image; S2, mapping the image features of the noise image into proxy words through a semantic extractor; generating corresponding text prompts based on the proxy words; S3. Using the noise image and its corresponding text prompt as input to the pre-trained diffusion model, obtaining the noise image generated by the pre-trained diffusion model in the training phase or the original image generated by the pre-trained diffusion model in the prediction phase; S4, based on the image features of the noisy image and the corresponding text prompt, a background removal mask is formed through the attention mechanism; S5. Generate mask image: Combine the original image sample with the background removal mask to obtain the masked original image; or combine the noise image with the background removal mask to obtain the masked noise image; The noise image generated by the pre-trained diffusion model in the training phase is combined with the background-removed mask to obtain a masked noise generated image; or the original image generated by the pre-trained diffusion model in the prediction phase is combined with the background-removed mask to obtain a masked original generated image; S6. Calculate the current loss and update the semantic extractor based on the current loss: Calculate the current generation loss based on the masked original image and the masked original generated image, or calculate the current diffusion loss based on the masked noise generated image and the masked noise image, and update the semantic extractor based on the current diffusion loss or the current generation loss; S7, determine whether the semantic extractor has reached the end training condition, if so, proceed to step S8; otherwise, return to step S2; S8, using the same method as steps S2 to S6 to obtain the proxy word and the current loss; S9, clustering and assigning the proxy words obtained in step S8 through the clustering head to obtain a clustering result; S10. Calculate the clustering loss of the fine-grained image clustering model based on the clustering results, and take the sum of the clustering loss and the current diffusion loss as the total loss, or take the sum of the clustering loss and the current generation loss as the total loss; update the semantic extractor and clustering head based on the current total loss; S11, determining whether the fine-grained image clustering model has reached the end training condition, if so, completing the training of the fine-grained image clustering model; otherwise, returning to step S8; The fine-grained image clustering model is used for image clustering.

2. The method for training a fine-grained image clustering model based on a diffusion model according to claim 1, characterized in that: The pre-trained diffusion model includes a UNet network and a text encoder; the text encoder is used to encode text prompts as input to the UNet network, and the encoded text prompts guide the UNet network to generate corresponding images based on the input noise image.

3. The method for training a fine-grained image clustering model based on a diffusion model according to claim 1, characterized in that: The specific method of adding noise to the original image sample in step S1 is: Perform Gaussian noise sampling on the original image sample to generate a noise image, whose expression is: in for t Noise image when ; for t The noise intensity at is the original image sample; is Gaussian noise, that is, added noise.

4. The method for training a fine-grained image clustering model based on a diffusion model according to claim 2, characterized in that: The specific method for obtaining the image features of a noisy image is: The noise image and the current text prompt are input into the pre-trained diffusion model, and the features output by the middle layer of the UNet network are used as the image features of the noise image, which is expressed as: in for When is the feature output by the middle layer of the bth block of the UNet network, that is, the image feature of the noise image; Represents the middle layer of the b-th block of the UNet network; for Noise image when ; Indicates the text prompt content. Represents the output of the text encoder. When the UNet network first obtains the image features of the noise image, Empty, that is, the input text is an empty string; is the time step for calculating image features.

5. The method for training a fine-grained image clustering model based on a diffusion model according to claim 4, characterized in that: The value of is 150; b The value of is 19.

6. The method for training a fine-grained image clustering model based on a diffusion model according to claim 1, characterized in that: The specific method of forming a background removal mask by the attention mechanism in step S4 includes the following sub-steps: S4-1, generating an image query vector based on the image features of the noise image; generating a text key vector based on the corresponding text prompt; S4-2, based on the image query vector, text key vector and feature dimension parameters, the softmax function is used to process and obtain the attention map corresponding to the current noise image; S4-3. Model the attention value distribution of the attention map corresponding to the current noise image through a bimodal Gaussian mixture model, and its expression is: in is the variable in the bimodal Gaussian mixture model The probability density function of and is the Gaussian mixing coefficient; variable is the attention map corresponding to the current noise image; The mean of the Gaussian probability density function is And the variance is The probability density of the components of ; The mean of the Gaussian probability density function is And the variance is The probability density of the components of ; S4-4. Parameters based on average values , mean value parameter The attention value at each position in the attention map corresponding to the current noise image is used to calculate the binary mask as the background removal mask; the calculation expression of the binary mask is: in Indicates that the position in the current noise image is The binary mask value at , , is a binary mask, i.e., a mask to remove the background; when When the value is 1, it means that the position in the current noise image is retained. The original characteristics of When the value is 0, it means removing the current noise image at position The original characteristics of the place; express The middle position is The original characteristics of the place.

7. The method for training a fine-grained image clustering model based on a diffusion model according to claim 6, characterized in that: The expressions for generating image query vectors and text key vectors are: in Indicates the first i The image query vector corresponding to each UNet block; and are all weight matrices; Indicates the i a vector of text keys; Indicates the i UNet blocks in The image features output at represents the output of the text encoder; is the time step for calculating image features; Based on the image query vector, text key vector and feature dimension parameters, the softmax function is used to process the attention map corresponding to the current noise image: in For the i The attention map corresponding to the image features output by each UNet block; for L The average attention map corresponding to the image features output by the UNet block, that is, the attention map corresponding to the current noise image; Represents the transpose of a matrix; d is the image feature dimension; Represents the softmax function.

8. The method for training a fine-grained image clustering model based on a diffusion model according to claim 1, wherein: The calculation expression of the diffusion loss in step S6 is: in is the diffusion loss; is the added noise; For mask; Indicates term-by-term multiplication; Generate an image for the noise; represents the L2 norm; Expressing expectation.

9. The method for training a fine-grained image clustering model based on a diffusion model according to claim 1, characterized in that: The expression of clustering loss is: in is the clustering loss; is the nearest neighbor similarity item; is the entropy regularization term; N is the batch size of clustering input; , Indicates agent The cluster assignment of represents the clustering head; , Indicates agent A randomly selected neighbor proxy word Cluster assignment of Represents the transpose of a matrix; Indicates that the N proxy words currently input are clustered into The average probability of the categories, , Indicates the current input n The proxy words are clustered into The probability of each category; represents the logarithm with the natural constant e as the base; C is the number of clusters; is the strength of the entropy regularization term.

10. A fine-grained image clustering method based on the training method of the fine-grained image clustering model based on the diffusion model according to any one of claims 1 to 9, characterized in that: The following steps are involved: A1. Obtain image features of the image to be clustered; A2, using the trained semantic extractor to map the image features of the image to be clustered into proxy words; A3. Use the trained clustering head to predict the proxy words obtained in step A2 to complete fine-grained image clustering.

Images