Image depth clustering method and system based on multi-modal large model guidance

Abstract

The invention discloses an image deep clustering method and system based on multi-modal large model guidance, and the method comprises the steps: generating a text description for each to-be-clustered image, and carrying out the feature extraction of the text description, and obtaining a text feature set; inputting all images to be clustered into an image encoder of the CLIP model to obtain an image feature set; for each image feature, calculating a cosine distance between the image feature and each text feature in the text feature set, selecting the text features corresponding to the first N maximum cosine distances, and constructing a semantic feature space; and inputting the fused image features into a clustering algorithm to obtain a clustering result and calculate a clustering index. By using the multi-modal large model to generate the fine-grained description of the image, the method can accurately distinguish similar images belonging to different categories, thereby remarkably improving the image clustering performance. Through the fusion of the multi-modal features, the method effectively enhances the discrimination of similar images among different categories, thereby improving the accuracy of image clustering.

Description

Technical Field

[0001] The present application relates to the technical field of image processing technology, and specifically to a method and system for deep image clustering guided by a multimodal large model. Background Art

[0002] In recent years, multimodal large models (such as BLIP, BLIP2, and LLaVA) have demonstrated outstanding capabilities in semantic understanding. These models can accurately describe the content of an image, conduct image question-answering, and even generate relevant captions for images by accepting text prompts. These advanced multimodal large models have achieved a deep understanding of image content to a certain extent and promoted the semantic association between images and text. However, when faced with large-scale image clustering tasks, existing multimodal large models still face certain limitations and it is difficult to achieve ideal results.

[0003] In the study of image semantic understanding, many works have attempted to capture the semantic information of images through existing vocabularies (such as WordNet). These methods rely on hierarchical relationships and synonym networks in the vocabulary, hoping to better represent the image content through rich vocabulary resources. However, there are some problems in dictionaries such as WordNet that affect their performance in image semantic understanding. For example, many words have multiple meanings in different contexts, resulting in the phenomenon of "polysemy", which makes it difficult for a single word to accurately describe the complex semantics of an image. In addition, a single word often cannot fully capture all the information contained in an image, because the content of the image usually involves multiple levels of semantics and details.

[0004] These problems severely restrict the performance of image clustering tasks based on existing vocabularies, especially when dealing with images with complex and diverse content. Therefore, how to accurately capture the semantic information in images, especially how to overcome the influence of polysemy in vocabulary, has become a key issue to be solved in the field of deep image clustering.

[0005] In order to improve the effect of image clustering, future research needs to explore more flexible and precise semantic expression methods that can better adapt to the diversity and complexity of images while reducing the negative impact of vocabulary limitations. By improving the modeling method of image semantic representation, it is expected to further promote the development of large-scale image clustering technology and improve its performance in practical applications. Summary of the invention

[0006] In this embodiment, a method, system, electronic device and storage medium for deep image clustering guided by a multimodal large model are provided, which utilize the fine-grained description of images by multimodal multi-models to enhance the capture of image semantic information, thereby improving the accuracy of small sample image classification.

[0007] In a first aspect, an embodiment of the present invention provides an image deep clustering method guided by a multimodal large model, and the image deep clustering method guided by a multimodal large model includes: (1) Generate a text description for each image to be clustered, and extract features from the text description to obtain a text feature set; (2) Input all images to be clustered into the image encoder of the CLIP model to obtain an image feature set; for each image feature, calculate its cosine distance with each text feature in the text feature set, select the text features corresponding to the first N largest cosine distances, and construct a semantic feature space; (3) calculating the cosine similarity between each image feature and the text feature in the semantic feature space, weighting and normalizing the text features based on the cosine similarity, adding the normalized text features to the corresponding image features to obtain fused image features; (4) Inputting the fused image features into a clustering algorithm to obtain clustering results and calculate clustering indicators.

[0008] In an optional embodiment, step (1) includes: Generate a text prompt for each image to be clustered, input the text prompt and the corresponding image into the multimodal large model to generate a text description; The text description is input into the text encoder of the CLIP model to obtain text features and form a text feature set.

[0009] In an optional embodiment, in step (2), the value of N is 5.

[0010] In an optional embodiment, the multimodal large model is one of BLIP, BLIP2 or LLaVA models. In an optional embodiment, in step (3), the modulus normalization process uses the L2 norm.

[0011] In an optional embodiment, in step (4), the clustering algorithm is one of k-means, spectral clustering or hierarchical clustering.

[0012] In an optional embodiment, in step (4), the clustering index includes accuracy, adjusted Rand index, normalized mutual information or silhouette coefficient.

[0013] Compared with the prior art, the image deep clustering method guided by a multimodal large model of the present invention has the following beneficial effects: The present invention can effectively mine the semantic information contained in the image. By using a large multimodal model to generate a fine-grained description of the image, the present invention can accurately distinguish similar images belonging to different categories, thereby significantly improving the image clustering performance. Compared with the traditional word-guided clustering method, the present invention is more accurate in capturing image semantic information, avoiding the interference of the "polysemy" phenomenon on semantic understanding, and is an efficient and accurate image deep clustering method.

[0014] The present invention uses a multimodal large model to generate text descriptions of images through carefully designed text prompts without introducing additional training parameters. Through the fusion of multimodal features, the present invention effectively enhances the distinguishability of similar images between different categories, thereby improving the accuracy of image clustering. This method is concise and efficient, and is a powerful image semantic enhancement method that can significantly improve the performance of image clustering tasks.

[0015] In a second aspect, an embodiment of the present invention provides an image deep clustering system guided by a multimodal large model, comprising: An image description generation module is used to generate a text description for each image to be clustered, and extract features from the text description to obtain a text feature set; A semantic feature space construction module is used to input all images to be clustered into the image encoder of the CLIP model to obtain an image feature set; for each image feature, the cosine distance between the image feature and each text feature in the text feature set is calculated, and the text features corresponding to the first N largest cosine distances are selected to construct a semantic feature space; A multimodal feature fusion module, used for calculating the cosine similarity between each image feature and the text feature in the semantic feature space, performing weighted and modulus normalization processing on the text features based on the cosine similarity, and adding the normalized text features to the corresponding image features to obtain fused image features; The image clustering module is used to input the fused image features into a clustering algorithm to obtain clustering results and calculate clustering indicators.

[0016] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory communicate with each other through the bus, and the processor can call logic instructions in the memory to execute the steps of the method provided in the first aspect.

[0017] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image deep clustering method guided by a multimodal large model as described in the first aspect.

[0018] Compared with the prior art, the beneficial effects of the image deep clustering system, electronic device and storage medium guided by a multimodal large model of the present invention are the same as those of the image deep clustering method guided by a multimodal large model described in the first aspect, so they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flowchart of an image deep clustering method guided by a multimodal large model in an embodiment of the present invention; Figure 2 It is a principle diagram of an image deep clustering method guided by a multimodal large model in an embodiment of the present invention; Figure 3 It is a structural block diagram of an image deep clustering system guided by a multimodal large model in an embodiment of the present invention; Figure 4 4 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0022] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0023] In an embodiment of the present invention, a method for deep image clustering based on guidance of a multimodal large model is provided. Figure 1 is a flow chart of the image deep clustering method guided by a multimodal large model of the present invention, such as Figure 1 As shown, the process includes the following steps: S1, generate a text description for each image to be clustered, and extract features from the text description to obtain a text feature set; Specifically, in an optional embodiment, step S1 includes: A text prompt is generated for each image to be clustered, and the text prompt and the corresponding image are input into a multimodal large model to generate a text description; in this embodiment, the multimodal large model is one of the BLIP, BLIP2 or LLaVA models.

[0024] The text description is input into the text encoder of the CLIP model to obtain text features and form a text feature set.

[0025] In this embodiment, for each picture, a text prompt "Please help me describe this picture, reflecting the main features of the object in the picture" is constructed, and the text prompt and each picture are sequentially input into the multimodal large model to obtain a text description of each picture, and the text description is input into the text encoder of the CLIP model to obtain the corresponding text features. Exemplarily, image description generation includes the following specific steps: Constructing query hint text for semantic understanding capabilities of multimodal large models “Please help me describe this picture and reflect the main features of the object in the picture”; Then, the query prompt text query and the image to be clustered (image) are input into the multimodal large model to obtain the text description of the image caption = MLLM (query, image), where MLLM is the multimodal large model and caption is the text description corresponding to the image image; Input the image caption into the text encoder TextEncoder of the CLIP model to obtain text features , all text features constitute a text feature set .

[0026] S2. Input all the images to be clustered into the image encoder of the CLIP model to obtain an image feature set; for each image feature, calculate its cosine distance with each text feature in the text feature set, select the text features corresponding to the first N largest cosine distances, and construct a semantic feature space; It should be noted that in step S2, the value of N is 5. In this embodiment, all images are input into the image encoder of the CLIP model to obtain corresponding image features, and then the cosine distance is calculated for each image feature and all text features, and they are arranged in descending order according to the cosine distance. For each image, the text features corresponding to the first five cosine distances are selected to form a set to construct a semantic feature space.

[0027] For example, all images are input into the image encoder ImageEncoder of the CLIP model to obtain the feature set of the image ,in ; Then calculate the cosine distance between each image feature and the text feature set , arrange the cosine distances from large to small, and select the text features corresponding to the first 5 large cosine distances for each image to form a semantic feature space .

[0028] S3, calculating the cosine similarity between each image feature and the text feature in the semantic feature space, weighting and normalizing the text features based on the cosine similarity, adding the normalized text features to the corresponding image features to obtain the fused image features; It should be noted that, in step S3, the modulus length normalization process adopts the L2 norm.

[0029] According to the semantic feature space constructed in S2, the image features are fused. The cosine similarity is calculated for each image feature and the text feature in the semantic feature space. According to the cosine similarity, the text features are weighted and normalized. Then, the normalized text features are added to the image features to obtain the fused image features.

[0030] For example, the cosine similarity is calculated between the image features and the text in the semantic feature space. According to weight Weighting text features Get the weighted text features, is the text feature vector; Then the weighted text features are normalized by modulus length , and then add and fuse the corresponding image features ; is the image feature vector.

[0031] S4. Input the fused image features into a clustering algorithm to obtain clustering results and calculate clustering indicators. It should be noted that the clustering algorithm is one of k-means, spectral clustering or hierarchical clustering. Clustering indicators include accuracy, adjusted Rand index, normalized mutual information or silhouette coefficient.

[0032] In this embodiment, the fused image features are passed through the kmeans algorithm to obtain the clustering results and calculate three indicators: accuracy, normalized mutual information and adjusted Rand index. The larger the values ​​of these three indicators, the better the clustering effect.

[0033] In this embodiment, the text description generated by the multimodal large model is used to capture the semantic information of the image, avoiding the semantic ambiguity problem caused by using a single word. The text features are then weighted by cosine similarity and fused with the image features to guide image clustering.

[0034] The semantic information in the image is captured by the text description generated by the multimodal large model, avoiding the semantic ambiguity that may be caused by using a single word. Subsequently, the text features are fused with the image features by calculating the weighted cosine similarity to guide the image clustering process.

[0035] The present invention can effectively mine the semantic information contained in the image. By using a large multimodal model to generate a fine-grained description of the image, the present invention can accurately distinguish similar images belonging to different categories, thereby significantly improving the image clustering performance. Compared with the traditional word-guided clustering method, the present invention is more accurate in capturing image semantic information, avoiding the interference of the "polysemy" phenomenon on semantic understanding, and is an efficient and accurate image deep clustering method.

[0036] The present invention uses a multimodal large model to generate text descriptions of images through carefully designed text prompts without introducing additional training parameters. Through the fusion of multimodal features, the present invention effectively enhances the distinguishability of similar images between different categories, thereby improving the accuracy of image clustering. This method is concise and efficient, and is a powerful image semantic enhancement method that can significantly improve the performance of image clustering tasks.

[0037] The embodiment of the present invention also provides an image deep clustering system guided by a multimodal large model, which is used to implement the above method embodiment, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware or a combination of software and hardware is also possible and conceived.

[0038] like Figure 3 As shown, Figure 3 It is a structural block diagram of the image deep clustering system guided by a multimodal large model in the present invention, and the system includes: An image description generating module 101 is used to generate a text description for each image to be clustered, and extract features from the text description to obtain a text feature set; The semantic feature space construction module 102 is used to input all the images to be clustered into the image encoder of the CLIP model to obtain an image feature set; for each image feature, the cosine distance between the image feature and each text feature in the text feature set is calculated, and the text features corresponding to the first N largest cosine distances are selected to construct the semantic feature space; A multimodal feature fusion module 103 is used to calculate the cosine similarity between each image feature and the text feature in the semantic feature space, perform weighted and modulus normalization processing on the text features based on the cosine similarity, and add the normalized text features to the corresponding image features to obtain fused image features; The image clustering module 104 is used to input the fused image features into a clustering algorithm to obtain clustering results and calculate clustering indicators.

[0039] Figure 4A structural block diagram of an electronic device provided by an embodiment of the present invention, such as Figure 4 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method: S1, generate a text description for each image to be clustered, and extract features from the text description to obtain a text feature set; S2, input all the images to be clustered into the image encoder of the CLIP model to obtain an image feature set; for each image feature, calculate its cosine distance with each text feature in the text feature set, select the text features corresponding to the first N largest cosine distances, and construct a semantic feature space; S3, calculating the cosine similarity between each image feature and the text feature in the semantic feature space, weighting and normalizing the text features based on the cosine similarity, adding the normalized text features to the corresponding image features to obtain fused image features; S4. Input the fused image features into a clustering algorithm to obtain clustering results and calculate clustering indicators.

[0040] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0041] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in the above embodiments is implemented.

[0042] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep image clustering method based on multimodal large model guidance, characterized in that: The image deep clustering method comprises: (1) Generate a text description for each image to be clustered, and extract features from the text description to obtain a text feature set; (2) Input all images to be clustered into the image encoder of the CLIP model to obtain an image feature set; for each image feature, calculate its cosine distance with each text feature in the text feature set, select the text features corresponding to the first N largest cosine distances, and construct a semantic feature space; (3) calculating the cosine similarity between each image feature and the text feature in the semantic feature space, weighting and normalizing the text features based on the cosine similarity, adding the normalized text features to the corresponding image features to obtain fused image features; (4) Inputting the fused image features into a clustering algorithm to obtain clustering results and calculate clustering indicators.

2. The image deep clustering method based on multimodal large model guidance according to claim 1 is characterized in that: Step (1) comprises: Generate a text prompt for each image to be clustered, input the text prompt and the corresponding image into the multimodal large model to generate a text description; The text description is input into the text encoder of the CLIP model to obtain text features and form a text feature set.

3. The image deep clustering method based on multimodal large model guidance according to claim 1 is characterized in that: In step (2), the value of N is 5.

4. The image deep clustering method based on multimodal large model guidance according to claim 2 is characterized in that: The multimodal large model is one of BLIP, BLIP2 or LLaVA models.

5. The image deep clustering method based on multimodal large model guidance according to claim 1, characterized in that: In step (3), the modulus normalization process uses the L2 norm.

6. The image deep clustering method based on multimodal large model guidance according to claim 1 is characterized in that: In step (4), the clustering algorithm is one of k-means, spectral clustering or hierarchical clustering.

7. The image deep clustering method based on multimodal large model guidance according to claim 1 is characterized in that: In step (4), the clustering index includes accuracy, adjusted Rand index, normalized mutual information or silhouette coefficient.

8. A deep image clustering system guided by a multimodal large model, characterized in that: include: An image description generation module is used to generate a text description for each image to be clustered, and extract features from the text description to obtain a text feature set; A semantic feature space construction module is used to input all images to be clustered into the image encoder of the CLIP model to obtain an image feature set; for each image feature, the cosine distance between the image feature and each text feature in the text feature set is calculated, and the text features corresponding to the first N largest cosine distances are selected to construct a semantic feature space; A multimodal feature fusion module, used for calculating the cosine similarity between each image feature and the text feature in the semantic feature space, performing weighted and modulus normalization processing on the text features based on the cosine similarity, and adding the normalized text features to the corresponding image features to obtain fused image features; The image clustering module is used to input the fused image features into a clustering algorithm to obtain clustering results and calculate clustering indicators.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the image deep clustering method based on multimodal large model guidance as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image deep clustering method based on multimodal large model guidance as described in any one of claims 1 to 7 are implemented.

Images