Fine-grained image clustering method and system based on FG-CLIP and language enhancement

Abstract

The invention provides a fine-grained image clustering method and system based on FG-CLIP and language enhancement, and belongs to the field of image clustering. A pre-training text generation model is utilized to generate diversified coarse-grained text descriptions, then a text fine-grained module and a text screening module are utilized to perform fine processing on a text, and two enhanced views are fused to form consistent text representation, so that the comprehensiveness of text information is enhanced. Then, a neighbor set is constructed according to the feature similarity through a random neighbor sample selection module, attention to samples in the same cluster is improved, and the selection range of positive samples is expanded; and finally, inputting the image features, the text features and neighbor samples thereof into each modal cluster-level mapping head for dimension mapping, and realizing alignment and joint training of the two modal features through random neighbor contrast loss. Under the assistance of the text information, the fine-grained clustering targets can be accurately distinguished, so that the accuracy and robustness of fine-grained image clustering are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of image clustering technology, and in particular to a fine-grained image clustering method and system based on FG-CLIP and language enhancement. Background Technology

[0002] Fine-grained image analysis is an important area of ​​computer vision, the core of which is to distinguish fine-grained subclasses with subtle visual differences from coarse-grained categories, such as identifying Siberian Huskies and Alaskan Malamutes among canines. Currently, research in this area mainly focuses on recognition and retrieval, while fine-grained image clustering, a key technology for unsupervised fine-grained classification, is still in its early stages.

[0003] Fine-grained image clustering faces unique challenges: samples within the same category exhibit vastly different visual features due to variations in pose, lighting, and other factors, while the visual differences between different subcategories are minimal. Furthermore, fine-grained image annotation relies on specialized knowledge, is time-consuming and labor-intensive, and the large amount of unannotated data further increases the difficulty. Existing clustering methods are mostly designed for coarse-grained tasks and struggle to capture the subtle differences in features of fine-grained images, thus failing to meet the requirements.

[0004] To address these issues, two main technical solutions have emerged in recent years, but both have limitations: one type extracts component features through parametric localization networks to train the model, but this increases model complexity, redundant regions can easily interfere with learning, and it lacks sufficient semantic guidance; the other type utilizes external knowledge from a vision-language pre-trained model to assist clustering, but this type of model struggles to handle fine-grained visual details, resulting in coarse text descriptions that fail to cover the required semantic information. Current technologies have significant shortcomings in semantic utilization, feature extraction accuracy, and model lightweighting, making the need for efficient and accurate unsupervised fine-grained clustering techniques urgent. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a fine-grained image clustering method and system based on FG-CLIP and language enhancement. With the assistance of textual information, it can accurately distinguish fine-grained clustering targets, thereby effectively improving the accuracy and robustness of fine-grained image clustering.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a fine-grained image clustering method based on FG-CLIP and language enhancement, comprising: By performing data augmentation on image samples in different ways, augmented image pairs are obtained; The enhanced image pairs are input into a multimodal contrastive clustering network, where the enhanced image pairs are respectively input into a pre-trained FG-CLIP visual encoder and a BLIP module to extract visual embedding features and coarse-grained text; By using an external semantic knowledge base, based on the similarity with the visual embedded features, the target objects in the coarse-grained text are lexical replaced and expanded to obtain a fine-grained text description. A pre-trained FG-CLIP text encoder is used to encode fine-grained text descriptions to obtain text embedding vectors, and consistent text representations are generated based on visual embedding features. A random nearest neighbor set is constructed based on the random nearest neighbor strategy, which consists of visual embedded features and consistent text representation. These are then input into a multimodal contrastive clustering head to obtain the clustering distribution. The multimodal contrastive clustering network is then jointly optimized based on the loss function. The images to be clustered are input into the trained FG-CLIP visual encoder and image clustering head to obtain fine-grained clustering results.

[0007] Secondly, the present invention provides a fine-grained image clustering system based on FG-CLIP and language enhancement, comprising: The image enhancement unit is configured to perform data enhancement on image samples in different ways to obtain enhanced image pairs; The network training unit is configured to input augmented image pairs into a multimodal contrastive clustering network, wherein the augmented image pairs are respectively input into a pre-trained FG-CLIP visual encoder and a BLIP module to extract visual embedding features and coarse-grained text; By using an external semantic knowledge base, based on the similarity with the visual embedded features, the target objects in the coarse-grained text are lexical replaced and expanded to obtain a fine-grained text description. A pre-trained FG-CLIP text encoder is used to encode fine-grained text descriptions to obtain text embedding vectors, and consistent text representations are generated based on visual embedding features. A random nearest neighbor set is constructed based on the random nearest neighbor strategy, which consists of visual embedded features and consistent text representation. These are then input into a multimodal contrastive clustering head to obtain the clustering distribution. The multimodal contrastive clustering network is then jointly optimized based on the loss function. The clustering unit is configured to input the image to be clustered into the trained FG-CLIP visual encoder and image clustering head to obtain fine-grained clustering results.

[0008] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the fine-grained image clustering method based on FG-CLIP and language enhancement described in the first aspect.

[0009] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the fine-grained image clustering method based on FG-CLIP and language enhancement described in the first aspect.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention utilizes a pre-trained FG-CLIP visual encoder and BLIP module to efficiently extract visual and coarse-grained text features, and uses these two to initially associate image and text semantics. The text fine-grained module combines an external semantic knowledge base to optimize the text based on image feature similarity, generating a more accurate fine-grained text description. After encoding and text filtering by the FG-CLIP text encoder, a unique and crucial text semantic embedding is obtained. Based on the random nearest neighbor selection strategy, a set is constructed and input into a multimodal contrastive clustering head, which can fully explore the correlation and local structure of visual and text features, realize accurate fine-grained image clustering, and effectively improve the fine-grained discrimination ability of image clustering and the clustering effect under multimodal fusion.

[0011] (2) This invention uses a large-scale vision-language pre-trained model to perform semantic enhancement modeling on fine-grained images. During the training phase, the generated fine-grained text description is used to pre-optimize the semantic extractor, enabling the model to better capture local feature differences. In this process, cross-modal semantic constraints are introduced through an improved random nearest neighbor sample comparison clustering mechanism, which effectively improves the distinguishability of fine-grained samples.

[0012] (3) This invention utilizes text descriptions to explicitly enhance target attributes and local details, reducing the interference of redundant backgrounds on latent space learning. This results in the semantic representation extracted by the semantic extractor being highly aligned with the clustering target, achieving more accurate image grouping. This invention can fully leverage the complementary advantages of visual and linguistic modalities under unsupervised conditions, significantly improving the stability and accuracy of fine-grained image clustering.

[0013] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0014] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0015] Figure 1 The main flowchart of a fine-grained image clustering method based on FG-CLIP and language enhancement provided for embodiments of the present invention; Figure 2A network framework diagram for implementing a fine-grained image clustering method based on FG-CLIP and language enhancement, provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the implementation of the text fine-grained module in an embodiment of the present invention. Figure 4 This is a schematic diagram of the text filtering module provided in an embodiment of the present invention. Detailed Implementation

[0016] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0017] Explanation of technical terms 1. FG-CLIP (Fine-Grained Visual and Textual Alignment) is a cross-modal image-text alignment model proposed by the 360 ​​AI Research Institute, with an overall structure similar to CLIP. This model comprises two core modules: a visual encoder and a text encoder. The visual encoder, based on a convolutional neural network (CNN) and visual transformer (ViT) architecture, includes an input processing unit, several visual feature extraction layers, and a feature projection layer, mapping the input image to a uniform-dimensional image embedding vector. The text encoder, employing a transformer-based model, consists of a text embedding layer, multiple self-attention processing layers, and a feature projection layer, mapping the input text to a fixed-dimensional semantic embedding vector, achieving fine-grained cross-modal alignment between images and text.

[0018] 2. BLIP (Bootstrapping Language-Image Pre-training) is a unified pre-trained model for visual-language understanding and generation, used to generate or understand image-text pairs. The BLIP module includes a visual encoding submodule, a text decoding submodule, and a cross-modal attention interaction unit, achieving image-text alignment and text generation through multimodal feature fusion.

[0019] Example 1 like Figure 1 As shown, this embodiment discloses a fine-grained image clustering method based on FG-CLIP and language enhancement, including the following steps: S1: Perform data augmentation on image samples using different methods to obtain augmented image pairs; S2: The enhanced image pairs are input into the multimodal contrastive clustering network, where the enhanced image pairs are respectively input into the pre-trained FG-CLIP visual encoder and BLIP module to extract visual embedding features and coarse-grained text; S3: Using an external semantic knowledge base, based on the similarity with the visual embedded features, perform word replacement and expansion on the target object in the coarse-grained text to obtain a fine-grained text description; S4: A pre-trained FG-CLIP text encoder is used to encode fine-grained text descriptions to obtain text embedding vectors, and consistent text representations are generated based on visual embedding features. S5: Construct a random nearest neighbor set based on the random nearest neighbor strategy for visual embedding features and consistent text representation, and input them together into the multimodal contrastive clustering head to obtain the clustering distribution. Perform joint optimization on the multimodal contrastive clustering network based on the loss function. S6: Input the image to be clustered into the trained FG-CLIP visual encoder and image clustering head to obtain fine-grained clustering results.

[0020] Next, combined Figure 2 This embodiment provides a detailed description of a fine-grained image clustering method based on FG-CLIP and language enhancement.

[0021] In S1, the original image samples are obtained, and two different data augmentations are performed to obtain augmented image pairs.

[0022] Regarding the type of the original image, those skilled in the art can choose according to the actual situation. It can be an animal such as a dog or penguin, a vehicle such as a car or airplane, or a subcategory under a major category. For example, the original image sample in this embodiment is a dog, such as an Alaskan Malamute or a Beagle.

[0023] Two different enhancement methods are used and For example, where, By simply cropping and center-aligning, the global image structure is preserved; It employs random scale adjustment and cropping operations, combined with scale jitter, random horizontal flipping, and random grayscale transformation, to introduce greater variations in scale, perspective, and appearance, thereby promoting the generation of text descriptions with richer semantics and stronger robustness.

[0024] In S2, the frozen FG-CLIP visual encoder is used to extract features from the enhanced image pairs to obtain the corresponding image features.

[0025] Specifically, for any given pair of enhanced images, i.e., two enhanced samples... and ,in , Representing different enhanced views, Indicates the first Images. A pre-trained FG-CLIP visual encoder is used to obtain visual embedding features. The specific process is as follows: ; in, , These are the enhanced images, , These are the features of the corresponding enhanced image, i.e., visual embedding features; This is a frozen FG-CLIP visual encoder. Image embedding is obtained from this. and ; Meanwhile, coarse-grained text descriptions are generated for the augmented image pairs using the pre-trained large visual language model BLIP.

[0026] The pre-trained BLIP model generates the required text descriptions for all images, for any augmented image. and The specific method for generating coarse-grained text is as follows: ; in, , These are the corresponding enhanced coarse-grained texts.

[0027] It should be understood that both the frozen FG-CLIP visual encoder and the pre-trained BLIP model are pre-trained models that can be selected and set by those skilled in the art according to actual needs, which facilitates the subsequent actual training work.

[0028] In S3, coarse-grained text input is as follows: Figure 3 The text fine-grained module shown performs lexical replacement and expansion on target objects in the text based on the external semantic knowledge base WordNet, resulting in multiple fine-grained text descriptions. Specifically: S301. Use Spacy to perform syntactic parsing on coarse-grained text and extract the main target objects. .

[0029] It should be understood that Spacy is a high-performance natural language processing library that performs part-of-speech analysis, named entity recognition, dependency characterization, and extraction of main target objects from natural language text. .

[0030] S302. Based on WordNet, automatically extract target objects according to their semantic relationships. The candidate word set consists of synonyms, hypernyms, and hyponyms. .

[0031] It should be understood that WordNet is a large English vocabulary database. Nouns, verbs, adjectives, and adverbs are stored in this database as sets of synonyms.

[0032] S303, Calculation The similarity between each candidate word and the visual embedding feature is used to select the top words whose similarity reaches a preset threshold. The candidate words replace the target object and are combined with the original coarse-grained text to construct a fine-grained text set.

[0033] In this embodiment, the target objects are first extracted from the text, and then the target objects in the text are replaced and expanded by relying on an external semantic knowledge base. Subsequently, the similarity between the replaced and expanded text features and the visual embedded features is compared to select suitable replacement words. This process realizes the effective integration of text modality and visual modality, which enables the text description to more accurately match the visual content while retaining the core semantics, thereby improving the processing effect and performance under the synergistic effect of multimodal data.

[0034] In S4, the FG-CLIP text encoder is used to encode fine-grained text, obtaining two types of text embedding vectors: ; in, This is the fine-grained text processed by the text fine-graining module. It is an FG-CLIP text encoder.

[0035] Furthermore, such as Figure 4 As shown, the multiple text embedding vectors are filtered through a text filtering module. This module generates a semantic center based on visual embedding features, calculates the similarity between the text embedding and the semantic center, selects the optimal candidate embedding, and uses an attention mechanism to generate a consistent text representation. Specifically: S401. Set a large number of categories and use K-means to cluster the image features to obtain semantic centers. The specific process is as follows: ; in, This represents the set of image embeddings assigned to cluster l in view m. It is the number of samples in the cluster. These are visual embedding features.

[0036] S402. Calculate the cosine similarity between the text embedding and the semantic center for each enhanced view: ; in, This indicates that different enhancements are consistent with step J. For fine-grained text embedding vectors; S403. To select the most suitable text for each image sample, all text embeddings in each cluster are sorted, and the top ones that match the highest similarity score are selected. 100 texts are used to form a local candidate set. The selection process is as follows: ; in, Indicates the first A collection of candidate text embeddings for an enhanced view. Return to When all similarities are sorted in descending order, the embedding Ranking; S404. In order to set the most accurate text for each image sample, all candidate texts for each enhanced view are... Combining together to form And generate corresponding text representations for each image through an attention mechanism: ; in, and The values ​​are all The values ​​are respectively , , Then, using image embedding, it can be represented as: , Indicates the number of candidate texts.

[0037] Furthermore, the text semantic embeddings from different enhanced views are weighted and fused to obtain a consistent fine-grained text representation.

[0038] Specifically, to explore the complementarity between different enhanced text embeddings and obtain comprehensive clustering results, the two enhanced view texts are fused to obtain a consistent text representation. The specific method is as follows: ; in, Constrained to the range (0,1) and adaptively updated via gradient descent; each enhanced image has a corresponding text description. ,in Corresponding to enhanced attempt a, The corresponding enhancement attempt b; the text embeddings generated by the two enhancement methods will be merged to form a consistent text representation.

[0039] In this embodiment, visual features guide text embedding and selection, and the text representations from different augmented views are then fused to achieve deep interaction between visual and text modalities. This approach utilizes visual semantic centers to accurately select matching text, and through multi-augmented view text fusion, fully leverages complementary information, making the text representation more closely aligned with the visual content. This enhances the accuracy and consistency of text semantic expression under multimodal data collaboration.

[0040] In S5, random nearest neighbor sets are constructed for both visual embedded features and consistent text representations based on a random nearest neighbor selection strategy. These sets are used to constrain the selection of positive sample pairs. Specifically: S501. Visual embedding features and consistent text representation for each enhanced image. Calculate the similarity matrix, and for each enhanced image embedding or consistent text embedding, select the top-ranked embeddings from the remaining embeddings of the same modality based on similarity. One sample.

[0041] Specifically, using image embedding For example, first calculate the similarity matrix. Each row of the similarity matrix represents the similarity between a sample and other samples; then, the top-ranked samples are selected from the similarity scores. One sample.

[0042] S502, from A nearest neighbor set is randomly selected from each sample and defined as a positive sample pair to avoid mistakenly treating similar samples as negative samples.

[0043] Furthermore, visual embedding features, fine-grained consistent text representations, and their random nearest neighbor samples are input into a multimodal contrastive clustering head, which includes an image clustering head. Image clustering head Text clustering head The clustering distributions of the image features and text representations are output respectively, specifically including: S511. Considering that the two enhanced images share the same textual semantics, two image clustering heads and one text clustering head are set up respectively to achieve cluster-level feature alignment. Image clustering head The method used to enhance the image features of 'a' is as follows: ; in, , These are the image embeddings and random nearest neighbor samples of enhancement a, respectively. Represented as the actual number of clusters; S512, Image Clustering Header Used for processing enhancements The image features, specifically the method is as follows: ; in, , These are the image embedding of enhanced b and its random nearest neighbor samples, respectively. S513, Text Clustering Header The method for processing the consistent text representation after fusion is as follows: ; in, , These are the consistent text embedding and its random nearest neighbor samples, respectively.

[0044] Ultimately, two image embeddings and nearest neighbor samples were obtained. , , , Consistent text embeddings and their nearest neighbor samples , .

[0045] In this embodiment, a random nearest neighbor strategy is used to construct a nearest neighbor set for visual and textual images to ensure accurate selection of positive sample pairs. Then, a multimodal contrastive clustering head is used to align the feature distributions of different enhanced images and fused text, thereby achieving efficient interaction and accurate clustering of multimodal features.

[0046] Specifically, the multimodal contrastive clustering network is jointly optimized using a random nearest neighbor sample contrastive loss function and a regularized loss function to obtain an optimized training model. S521. The multimodal contrastive clustering network is jointly optimized using random nearest neighbor comparison loss and regularization loss. , , , For example, the random nearest neighbor comparison loss is: ; ; ; in, , , and These represent the nth element in the distribution matrix output by each clustering head. List.

[0047] S522. To avoid the imbalance caused by most samples being assigned to a few classes, a regularization loss is set to... and For example, the specific process is as follows: ; in, , These represent the mean of each cluster in the image and text modalities, respectively. S523, Total Loss Function: ; in, These are the weighting coefficients.

[0048] In S6, after training is complete, the image to be clustered is input into the visual branch of the trained clustering model (i.e., the FG-CLIP visual encoder and the image clustering head) to output the corresponding fine-grained image clustering results.

[0049] Specifically, the test images are sequentially input into the visual encoder. Image clustering head The corresponding clustering prediction results are obtained by using the same structure as the test image. For example, the prediction process is as follows: ; in, For the predicted category results, Indicates in The category corresponding to the maximum value in the output probability distribution is selected.

[0050] As one implementation method, using fine-grained data This example uses specific aircraft or dog image data as an example. This embodiment first uses a pre-trained BLIP model to generate text descriptions corresponding to different augmented views, and then uses a pre-trained FG-CLIP model to extract image and text features. Throughout the process, the parameters of the pre-trained model remain frozen and do not participate in training; the model only needs to optimize the image clustering head. and Text clustering head and text fusion parameters This training method is convenient and lightweight. By adopting the technical solution of this embodiment, the accuracy and efficiency of fine-grained image clustering can be effectively improved, and the accuracy and stability of the clustering results can be significantly enhanced.

[0051] This specific embodiment leverages the FG-CLIP visual encoder to extract visual features from images, taking advantage of its strengths in fine-grained feature extraction while avoiding the increased complexity and redundant interference issues associated with parametric localization networks. It uses the BLIP module to generate coarse-grained text, which is then processed by a text fine-grained module. Combining an external semantic knowledge base with image feature similarity, the target text is expanded through word substitution to obtain a fine-grained text description, compensating for the shortcomings of the visual-language pre-trained model in generating coarse text and insufficient semantic coverage. Through FG-CLIP text encoder encoding, text filtering module filtering, and the construction of a set based on a random nearest neighbor strategy and input into a multimodal comparative clustering head, it fully utilizes semantic information while ensuring feature extraction accuracy, without requiring additional complex structures, thus achieving model lightweighting. This provides an effective technical approach for efficient and accurate unsupervised fine-grained clustering.

[0052] Example 1 To verify the effectiveness of the present invention, the following simulation will further illustrate the effects of the present invention.

[0053] Simulation experimental conditions: Simulation experimental conditions for this invention: Server GPU: NVIDIA GeForce RTX 3090, 24GB video memory.

[0054] The software platform for the simulation experiment of this invention is: Windows 11 Professional Edition, Python 3.9, and Torch+Cu1212.3.1.

[0055] Simulation content and analysis of experimental results: This invention simulates the image clustering task by introducing fine-grained text descriptions as guidance, based on the existing pre-trained contrastive learning large model FG-CLIP. The main fine-grained task datasets used in the simulation are: DTD, Animal10N, Stanford Dogs, and FGVC-Aircraft.

[0056] DTD is a public dataset focused on texture recognition, containing 47 describable texture categories such as "stripes," "spots," and "waves," with approximately 120 images per category. This dataset emphasizes fine-grained differences in surface textures and is suitable for evaluating a model's ability to capture local image patterns.

[0057] Animal10N is a dataset for animal image classification and recognition, covering 10 common animal categories. It is characterized by a large number of images and includes a certain proportion of noisy labels, enabling it to effectively simulate fine-grained recognition and clustering challenges in real-world environments.

[0058] Stanford Dogs is a fine-grained canine recognition dataset containing over 20,000 images from 120 different canine species. This dataset not only covers significant differences in canine appearance but also includes a large number of pose, background, and lighting variations, making it a classic benchmark for fine-grained image clustering.

[0059] FGVC-Aircraft is a fine-grained aircraft recognition dataset containing approximately 10,000 images of 100 different aircraft models. This dataset is challenging in capturing subtle differences between aircraft categories, such as wing shape, engine location, and fuselage structure, making it an important dataset for evaluating a model's ability to capture detail.

[0060] This embodiment is based on CLIP and FG-CLIP, respectively, and combines pseudo-labels with fixed sentence structures to construct comparison methods, namely CLIP (Pseudo-Label) and FG-CLIP (Pseudo-Label). The clustering experimental results on the above four datasets are given below.

[0061] Table 1. Comparison of simulation results of the present invention on four datasets;

[0062] This invention uses three commonly used metrics—Clustering Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI)—to evaluate clustering performance. Higher values ​​for all three indicate better clustering results. As shown in Table 1, compared to various existing benchmark methods, this invention achieves optimal results on all fine-grained datasets and evaluation metrics, fully demonstrating that the proposed method can effectively improve the accuracy and stability of fine-grained image clustering, exhibiting significant technical advantages.

[0063] Example 2 This embodiment provides a fine-grained image clustering system based on FG-CLIP and language enhancement, including: The image enhancement unit is configured to perform data enhancement on image samples in different ways to obtain enhanced image pairs; The network training unit is configured to input augmented image pairs into a multimodal contrastive clustering network, wherein the augmented image pairs are respectively input into a pre-trained FG-CLIP visual encoder and a BLIP module to extract visual embedding features and coarse-grained text; By using an external semantic knowledge base, based on the similarity with the visual embedded features, the target objects in the coarse-grained text are lexical replaced and expanded to obtain a fine-grained text description. A pre-trained FG-CLIP text encoder is used to encode fine-grained text descriptions to obtain text embedding vectors, and consistent text representations are generated based on visual embedding features. A random nearest neighbor set is constructed based on the random nearest neighbor strategy, which consists of visual embedded features and consistent text representation. These are then input into a multimodal contrastive clustering head to obtain the clustering distribution. The multimodal contrastive clustering network is then jointly optimized based on the loss function. The clustering unit is configured to input the image to be clustered into the trained FG-CLIP visual encoder and image clustering head to obtain fine-grained clustering results.

[0064] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a fine-grained image clustering method based on FG-CLIP and language enhancement as described in Embodiment 1 above.

[0065] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the fine-grained image clustering method based on FG-CLIP and language enhancement as described in Embodiment 1 above.

[0066] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0067] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A fine-grained image clustering method based on FG-CLIP and language enhancement, characterized in that, The method comprises the following steps: different ways of data enhancement are performed on the image samples to obtain enhanced image pairs; the enhanced image pairs are input into a multi-modal contrast clustering network, wherein the enhanced image pairs are respectively input into a pre-trained FG-CLIP visual encoder and a BLIP module to extract visual embedding features and coarse-grained text; based on the similarity with the visual embedding features, target objects in the coarse-grained text are replaced and expanded based on an external semantic knowledge base to obtain fine-grained text descriptions; a pre-trained FG-CLIP text encoder is used to encode the fine-grained text descriptions to obtain text embedding vectors, and consistent text representations are screened based on the visual embedding features; a random neighbor set of the visual embedding features and the consistent text representations is constructed based on a random neighbor strategy, and is input into a multi-modal contrast clustering head to obtain a clustering distribution, and the multi-modal contrast clustering network is jointly optimized based on a loss function; a to-be-clustered image is input into the trained FG-CLIP visual encoder and an image clustering head to obtain a fine-grained clustering result.

2. The FG-CLIP and language enhancement based fine-grained image clustering method of claim 1, wherein, The step of replacing and expanding the target objects in the coarse-grained text based on the similarity with the visual embedding features based on the external semantic knowledge base to obtain the fine-grained text descriptions comprises the following steps: the coarse-grained text is parsed to extract main target objects; synonyms, hypernyms and hyponyms of the target objects are obtained based on the external semantic knowledge base to form a candidate word set; the similarity of each candidate word in the candidate word set with the visual embedding features is calculated, and the candidate word with a similarity reaching a preset threshold is selected to replace the target object, and the original coarse-grained text is used to construct a fine-grained text set.

3. The FG-CLIP and language enhancement based fine-grained image clustering method of claim 1, wherein, The step of screening the consistent text representations based on the visual embedding features comprises the following steps: the visual embedding features are clustered to obtain a semantic center; the cosine similarity of the text embedding vectors with the semantic center is calculated, and the optimal candidate embedding is screened based on the similarity; the optimal candidate embedding of each image is aggregated through an attention mechanism to generate a corresponding text representation; the corresponding text representations are weighted and fused based on different enhancement methods to obtain consistent text representations.

4. The FG-CLIP and language enhancement based fine-grained image clustering method of claim 1, wherein, The step of constructing the random neighbor set of the visual embedding features and the consistent text representations based on the random neighbor strategy comprises the following steps: part of the samples corresponding to the visual embedding features are screened based on the similarity to form a random neighbor set of the visual embedding features; part of the samples corresponding to the consistent text representations are screened based on the similarity to form a random neighbor set of the consistent text representations.

5. The FG-CLIP and language enhancement based fine-grained image clustering method of claim 1, wherein, The step of inputting the visual embedding features and the consistent text representations into the multi-modal contrast clustering head to obtain a clustering distribution comprises the following steps: the multi-modal contrast clustering head comprises two image clustering heads and one text clustering head; the visual embedding features and their random neighbor set are respectively input into the two image clustering heads according to different enhancement methods, and the consistent text representations and their random neighbor set are input into the text clustering head; through multi-modal contrast learning, the image features and the text representations are matched with each other in the clustering space to obtain the clustering distribution of the image features and the text representations.

6. The FG-CLIP and language enhancement based fine-grained image clustering method of claim 1, wherein, The loss function comprises a random neighbor contrast loss function and a regularization loss function.

7. A fine-grained image clustering system based on FG-CLIP and language augmentation, characterized in that, The method comprises the following steps: An image enhancement unit is configured to perform different data augmentations on the image samples to obtain an enhanced image pair; A network training unit is configured to input the enhanced image pair into a multi-modal contrast clustering network, wherein the enhanced image pair is input into a pre-trained FG-CLIP visual encoder and a BLIP module respectively to extract visual embedding features and coarse-grained text; Through an external semantic knowledge base, target objects in the coarse-grained text are replaced and expanded based on the similarity with the visual embedding features to obtain fine-grained text descriptions; A pre-trained FG-CLIP text encoder is used to encode the fine-grained text descriptions to obtain text embedding vectors, and consistent text representations are screened based on the visual embedding features; A random neighbor set of the visual embedding features and the consistent text representations is constructed based on a random neighbor strategy, and is input into a multi-modal contrast clustering head to obtain a clustering distribution, and the multi-modal contrast clustering network is jointly optimized based on a loss function; A clustering unit is configured to input an image to be clustered into a trained FG-CLIP visual encoder and an image clustering head to obtain a fine-grained clustering result.

8. A fine-grained image clustering system based on FG-CLIP and language augmentation as claimed in claim 7, wherein, The method further comprises the following steps: The coarse-grained text is parsed to extract main target objects; Synonyms, hypernyms and hyponyms of the target objects are obtained based on an external semantic knowledge base to form a candidate word set; The similarity of each candidate word in the candidate word set with the visual embedding features is calculated, and a candidate word with a similarity reaching a preset threshold is selected to replace the target object to form a fine-grained text set together with the original coarse-grained text.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the fine-grained image clustering method based on FG-CLIP and language enhancement.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the fine-grained image clustering method based on FG-CLIP and language enhancement.

Images