Image clustering method and system and readable storage medium

Abstract

The invention relates to an image clustering method and system and a readable storage medium, and the method comprises the steps: constructing a positive sample pair according to images and texts of the same defect type, and constructing a negative sample pair according to images and texts of different defect types, training based on the consistency loss of the positive sample pair and the inconsistency loss of the negative sample pair to obtain an image encoder and a text encoder; obtaining a plurality of images to be clustered, obtaining an embedded vector of each image to be clustered through the image encoder, and obtaining a first similarity between every two embedded vectors; and calculating the offset similarity of any two to-be-clustered images to obtain a second similarity, and performing clustering based on the first similarity and the second similarity to obtain an image clustering result. An adversarial training strategy is combined with a clustering target, and a novel loss function is designed, so that the model pursues improvement of clustering performance and enhancement of adversarial robustness in the optimization process at the same time.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of integrated circuits, and particularly relates to an image clustering method, system and readable storage medium. BACKGROUND

[0002] Traditional clustering methods usually rely on internal supervision signals to achieve clustering goals by optimizing data distribution assumptions or extracting discriminative features. However, these methods often perform poorly when faced with complex, high-dimensional data, especially in the absence of explicit supervision signals. In recent years, deep clustering methods have significantly improved clustering performance by combining the powerful feature extraction capabilities of neural networks. However, these methods still mainly rely on the inherent characteristics of the data itself and fail to fully utilize external knowledge to enhance clustering results.

[0003] In the image clustering task, image data usually has complex semantic structures, and traditional clustering methods often struggle to fully utilize these semantic information. For example, visually similar images may be confused during the clustering process, despite significant semantic differences. In this case, relying solely on feature extraction and clustering strategies from the images themselves may not achieve the desired clustering results. SUMMARY

[0004] To solve all or part of the problems of the prior art, the present application provides an image clustering method based on adversarial learning, aiming to combine image and external text knowledge to construct positive and negative sample pairs, and optimize the feature representation capability of the model through adversarial training. This method not only fully utilizes external knowledge to enhance clustering results, but also improves the generalization ability of the model through adversarial learning, thereby achieving higher quality clustering results in complex scenarios.

[0005] In a first aspect, an image clustering method is provided in the present embodiment, comprising: constructing positive sample pairs according to images and texts of the same defect type and negative sample pairs according to images and texts of different defect types, and training an image encoder and a text encoder based on consistency loss of the positive sample pairs and inconsistency loss of the negative sample pairs; obtaining a plurality of images to be clustered, obtaining an embedding vector of each of the images to be clustered through the image encoder, and obtaining a first similarity between each pair of embedding vectors; calculating an offset similarity between any two images to be clustered to obtain a second similarity, and performing clustering based on the first similarity and the second similarity to obtain an image clustering result.

[0006] In some embodiments, the positive sample pair includes first positive samples of images and texts matched under the same defect type, each first positive sample obtains a first image vector and a first text vector through the image encoder and the text encoder, and the consistency loss of the positive sample includes a loss L1 obtained according to a first similarity value of the first image vector and the first text vector.

[0007] In some embodiments, the method further includes a first classifier, the positive sample pair includes two second positive samples formed by interchanging images and texts of the two first positive samples, and the consistency loss of the positive sample includes: a loss L2 obtained according to a classification result of the image vector and the text vector of the two second positive samples and the classification by the classifier.

[0008] In some embodiments, the loss L2 obtained according to the classification result of the image vector and the text vector of the two second positive samples and the classification by the classifier includes: splicing the image vector and the text vector of each second positive sample to obtain two first splicing vectors; inputting the two first splicing vectors into the classifier to obtain two prediction results, calculating a cross-entropy loss according to the two prediction results, and obtaining the loss L2 of the two second positive samples.

[0009] In some embodiments, the negative sample pair includes first negative samples formed by freely combining the image and the text of the first positive sample of different defect types, each first negative sample obtains a second image vector and a second text vector through the image encoder and the text encoder, and the consistency loss of the negative sample includes a loss L3 obtained according to a second similarity value of the second image vector and the second text vector.

[0010] In some embodiments, the negative sample pair includes two second negative samples formed by interchanging images and texts of the two first negative samples, and the consistency loss of the negative sample includes: a loss L4 obtained according to a classification result of the image vector and the text vector of the two second negative samples and the classification by the classifier.

[0011] In some embodiments, the loss L4 obtained according to the classification result of the image vector and the text vector of the two second negative samples and the classification by the classifier includes: splicing the image vector and the text vector of each second negative sample to obtain two second splicing vectors; inputting the two second splicing vectors into the classifier to obtain two prediction results, calculating a KL divergence loss according to the two prediction results, and obtaining the loss L4 of the two second negative samples.

[0012] In some of these embodiments, the two prediction results are preset as follows: and Calculate the KL divergence value: and / or ; The loss L4 includes: and / or L4=−[KL_div(PAC|PCA+KL_div(PCA|PAC].

[0013] In some embodiments, calculating the offset similarity between any two images to be clustered to obtain a second similarity includes: The image to be clustered is converted into a three-dimensional matrix to obtain the pixel value at each coordinate position in each channel; The DTW similarity of two images to be clustered is calculated using pixel values ​​in each channel, and a second similarity is obtained based on the DTW similarity in different channels.

[0014] In a second aspect, the present invention provides an image clustering system, comprising: The training module constructs positive sample pairs based on images and text with the same defect type and negative sample pairs based on images and text with different defect types. Based on the consistency loss of the positive samples and the inconsistency loss of the negative samples, it trains an image encoder and a text encoder. The clustering module acquires multiple images to be clustered, obtains the embedding vector of each image to be clustered through the image encoder, and obtains the first similarity between any two embedding vectors; calculates the similarity between any two images to be clustered to obtain the second similarity, and performs clustering based on the first and second similarities to obtain the image clustering result.

[0015] In a third aspect, the present invention provides a readable storage medium storing a program that, when executed, implements the image clustering method of the first aspect described above.

[0016] Compared with the prior art, the embodiments disclosed herein have at least one of the following beneficial effects: By combining adversarial training strategies with clustering objectives through the above technical solution, and designing a novel loss function, the model simultaneously pursues improved clustering performance and enhanced adversarial robustness during the optimization process. Specifically, the loss for positive sample pairs is minimized to enhance the model's clustering ability, while the loss for negative sample pairs is maximized to improve the model's generalization ability. This joint optimization strategy can effectively avoid the problem of model overfitting to a specific data distribution. By introducing the concept of adversarial learning, positive and negative sample pairs are dynamically constructed, enabling the model to more flexibly capture the distribution characteristics of the data. Specifically, positive sample pairs consist of highly similar image-text pairs, while negative sample pairs consist of significantly different image-text pairs. This dynamic construction method can effectively improve the model's adaptability to complex data distributions. Secondly, this solution applies DTW to the image domain and designs a recursive formula for dynamic DTW of images, which further narrows the similarity between images with offset patterns. Finally, this similarity calculation is integrated into clustering, which can effectively improve the clustering effect between images with the same offset. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the method flow in one embodiment; Figure 2 This is a schematic diagram of model training in one embodiment; Figure 3 This is a schematic diagram of the clustering process in one embodiment. Detailed Implementation

[0019] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0021] It should also be understood that the terms "comprising / including" or "having," etc., specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. Meanwhile, in this specification, the term "and / or" includes any and all combinations of the associated listed items.

[0022] In existing technologies, classic clustering methods (such as k-means, hierarchical clustering, spectral clustering, etc.) are based on assumptions about data distribution and achieve clustering by optimizing the compactness, connectivity, or sparsity of the data. These methods perform well when dealing with low-dimensional, well-structured data, but they often struggle to extract effective features when faced with complex, high-dimensional image data, resulting in limited clustering performance.

[0023] Deep clustering methods extract discriminative features through neural networks and optimize network parameters by combining clustering objectives, significantly improving the ability to process complex data. However, these methods mainly rely on internal supervision signals (such as neighborhood consistency or pseudo-labels) and have difficulty fully utilizing external knowledge (such as semantic information), which limits their potential in practical applications.

[0024] Self-supervised clustering methods improve the robustness and generalization ability of models by constructing supervision signals through data augmentation or momentum strategies. However, these methods are still limited to mining internal supervision signals from data, ignoring the richness and availability of external knowledge, and are difficult to achieve optimal performance in complex scenarios.

[0025] Based on this, this embodiment provides a method such as Figure 1 The image clustering models or methods shown include: Positive sample pairs are constructed based on images and text with the same defect type, and negative sample pairs are constructed based on images and text with different defect types. An image encoder and a text encoder are trained based on the consistency loss of the positive sample pairs and the inconsistency loss of the negative sample pairs. Multiple images to be clustered are obtained, and the embedding vector of each image to be clustered is obtained through the image encoder. The first similarity between any two of the embedding vectors is obtained. Calculate the offset similarity between any two images to be clustered to obtain the second similarity. Perform clustering based on the first and second similarities to obtain the image clustering results.

[0026] This application improves image clustering performance by constructing positive and negative sample pairs by combining image and external textual knowledge and optimizing the model's feature representation capabilities through adversarial training. This method, based on existing text-assisted clustering methods, incorporates an adversarial learning strategy to design a novel clustering fine-tuning framework to enhance the generalization and discriminative power of image embeddings. This approach not only fully utilizes external knowledge to enhance clustering results but also improves the model's generalization ability through adversarial learning, thereby achieving higher-quality clustering results in complex scenarios.

[0027] Based on the optimized image encoder, image embedding vectors with good discriminative power can be obtained. The first similarity is used for clustering based on the image embedding vectors. At the same time, there may be some offset for images of the same shape (defect type), which will cause similarity offset. Therefore, the second similarity is obtained by calculating the offset similarity of two images. Clustering is performed based on the weighted combination of the first and second similarities, which can effectively classify the same type of defect and distinguish different defect types.

[0028] Reference Figure 2 As shown, the clustering model includes an image encoder and a text encoder. By fine-tuning these two encoders, matching image-text pairs are brought closer together and mismatched image-text pairs are moved further apart in the embedding space. Specifically, multiple positive and negative samples are constructed. Based on the introduction of contrastive learning and adversarial learning, the intra-sample contrastive loss is controlled under the same defect type, and the inter-sample adversarial loss is controlled under different defect types. The image encoder and text encoder are trained and optimized by combining the two losses, so that even images with low similarity can obtain discriminative embedding vectors.

[0029] In the process of constructing positive and negative sample pairs, the following can be included: a first positive sample that matches the image and text under the same defect type; two second positive samples formed by swapping the image and text of two first positive samples; a first negative sample formed by freely combining the image and text of first positive samples with different defect types; and two second negative samples formed by swapping the image and text of two first negative samples. The required positive and negative samples can be freely selected according to the characteristics of the image.

[0030] In some embodiments, the positive sample pair includes a first positive sample that matches the image and text under the same defect type. Each first positive sample is processed by the image encoder and the text encoder to obtain a first image vector and a first text vector. The consistency loss of the positive sample includes a loss L1 obtained based on a first similarity value of the first image vector and the first text vector.

[0031] The first positive sample uses image-text pairs matched under the same defect type. The first image vector (image embedding vector) and the first text vector (text embedding vector) can be obtained through the image encoder and text encoder. The maximum cosine similarity (i.e., the first similarity value) of the two embedding vectors is calculated. The formula for calculating the loss L1 is as follows: L1=1−cos_sim(lembed, Tembed) Here, Iembed is the first image vector, Tembed is the first text vector, and cos_sim is the cosine similarity. In practice, for example, 100 first positive samples can be used to calculate 100 maximum cosine similarities, requiring all cosine similarity values ​​to be as close to 1 as possible.

[0032] In some embodiments, the method further includes a first classifier, wherein the positive sample pairs comprise two second positive samples formed by swapping the images and text of the two first positive samples; the consistency loss of the positive samples includes: Classification is performed based on the image vectors and text vectors of two second positive samples, as well as the classifier, and the loss L2 is obtained based on the classification results.

[0033] Loss L1 can be used to obtain the contrast loss between the image and the text in the first positive sample of image-text matching. Loss L2 is used to calculate the contrast loss between different positive samples under the same defect type. The second positive sample can be constructed from the first positive sample of image-text matching.

[0034] The images and text in the samples are arranged in the order of image first, followed by text. Input samples AC and BD consist of two images (A, B) and two texts (C, D). The L1 loss enables the model to correctly classify input samples AC and BD, while the L2 loss teaches the model that the two vectors output by samples AD and BC also belong to the same class. The difference between L2 and L1 losses is that L2 loss selects two positive samples from the same class each time. For example, for images of a Golden Retriever and a Shiba Inu; for instance, L1 inputs a Golden Retriever and its text description, while L2 inputs an image of a Golden Retriever and its description, or vice versa. Since both Golden Retrievers and Shiba Inus belong to the canine family, they are considered to be in the same class, and images and text can be cross-combined.

[0035] Regarding the classifier, the second positive sample and the second negative sample can share the same classifier, which can be a fully connected layer. The input is the concatenation of the image embedding vector and the text embedding vector (so the input dimension is the image embedding dimension + the text embedding dimension), and the output is the prediction results for several categories. This classifier is used in both the positive sample pair consistency loss and the negative sample pair inconsistency loss, so it needs to be trained. During training, the classifier is trained together with two encoders.

[0036] In some embodiments, the classification based on the image vectors and text vectors of two second positive samples, and the classifier, and the loss L2 obtained from the classification result, includes: The image vector of each second positive sample is concatenated with the text vector to obtain two first concatenated vectors; The two first concatenated vectors are input into the classifier to obtain two prediction results. The cross-entropy loss is calculated based on the two prediction results to obtain the loss L2 of the two second positive samples.

[0037] Take two first positive samples (e.g., sample A and sample B), concatenate their images and text respectively (e.g., concatenate the image embedding of sample A and the text embedding of sample B, and vice versa), and then classify them using a classifier (e.g., a linear layer). The classification results of the two concatenated vectors (referring to the first concatenated vectors) should be consistent (i.e., the classification labels should be the same): The concatenation vector concat_AB = [I_A; T_B] — concatenates the image vector I_A of sample A and the text vector T_B of sample B; The concatenation vector concat_BA = [I_B; T_A] — concatenates the image vector I_B of sample B and the text vector T_A of sample A; Then, we use a shared classifier (a fully connected layer) to classify the two first concatenated vectors to obtain the prediction results logits_AB and logits_BA. We calculate the loss L2 using cross-entropy loss, specifically requiring that the defect categories predicted by logits_AB and logits_BA are consistent.

[0038] In some embodiments, the negative samples are formed by freely combining images and text from the first positive samples, which include different defect types; each first negative sample is processed by an image encoder and a text encoder to obtain a second image vector and a second text vector, and the consistency loss of the negative samples includes a loss L3 obtained based on a second similarity value of the second image vector and the second text vector.

[0039] The first negative sample uses mismatched image-text pairs from different defect types, i.e., the image and text come from different defect types. A second image vector (image embedding vector) and a second text vector (text embedding vector) are obtained through an image encoder and a text encoder. The cosine similarity (i.e., the second similarity value) between the two embedding vectors is calculated, aiming to minimize the cosine similarity between them. The formula for calculating the L3 loss is as follows: Where Iembed is the second image vector, Tembed is the second text vector, and cos_sim is the cosine similarity.

[0040] In some embodiments, the negative sample pair comprises two second negative samples formed by swapping the images and text of two first negative samples, and the consistency loss of the negative samples includes: Classification is performed based on the image vectors and text vectors of the two second negative samples, as well as the classifier, and the loss L4 is obtained based on the classification results.

[0041] The L4 loss requires that the classification results of two second negative sample pairs be inconsistent. We use two different samples (such as sample B and sample D, both of which are first negative samples) to concatenate them, and then require the classifier to classify the concatenated vector inconsistently.

[0042] In some embodiments, the classification based on the image vectors and text vectors of the two second negative samples, and the classifier, and the loss L4 obtained from the classification result, includes: The image vector of each second negative sample is concatenated with the text vector to obtain two second concatenated vectors; The two concatenated vectors are input into the classifier to obtain two prediction results. The KL divergence loss is calculated based on the two prediction results to obtain the loss L4 of the two second negative samples.

[0043] In some of these embodiments, the two prediction results are preset as follows: and Calculate the KL divergence value: and / or ; The loss L4 includes: and / or L4=−[KL_div(PAC|PCA+KL_div(PCA|PAC].

[0044] Specifically, we construct: The concatenation vector concat_BD = [I_B; T_D] concatenates the image vector of sample B and the text vector of sample D (the text of sample D does not match the text of sample B). The concatenation vector concat_DB = [I_D; T_B] — concatenates the image vector of sample D and the text vector of sample B (the image of sample D and the text of sample B do not match); For a concatenated vector, we have a real class label (e.g., the class label y_B of sample B). For the concatenated vector concat_BD, we expect the classifier to classify it as y_B. Therefore, we require that the embedding generated by the encoder can prevent the classifier from classifying it correctly (i.e., adversarial learning).

[0045] We compute the predicted values ​​logits_BD and logits_DB from the concatenated vectors concat_BD and concat_DB, and then require that the distributions of these two predicted values ​​differ as much as possible (i.e., are inconsistent). The loss function (L4) can be designed as follows: ; We want the KL divergence to be as large as possible (i.e., the difference to be large), so we add a negative sign to make the loss negative. However, we usually don't minimize negative values, so we can instead minimize the negative KL divergence, i.e., maximize the KL divergence. But the KL divergence is asymmetric, so we use the symmetric form: L4=−[KL_div(PAC|PCA+KL_div(PCA|PAC] Minimizing the loss L4 is equivalent to maximizing the KL divergence (i.e., the difference) between the two distributions.

[0046] In summary, the loss for positive samples includes L1 + L2, and the loss for negative samples includes L3 + L4. Therefore, the total loss can be written as: L_total =α* (L1) +β* L2 +γ* L3 +δ* (L4); where α, β, γ, δ are weight coefficients.

[0047] The training process for image encoders, text encoders, and classifiers may include the following: Input: A batch contains N matching image-text pairs (positive samples) and M non-matching image-text pairs (negative samples); note that negative samples can be obtained by randomly shuffling the text of positive samples (i.e., matching an image with a text that is not its corresponding text). Step a: Obtain the embedding vectors of all images and text using the image encoder and text encoder respectively; Step b, positive sample pair loss For each positive sample pair, calculate the loss L1 (cosine similarity loss). Randomly select two pairs of positive samples from the positive samples, swap the images and text of the two pairs of positive samples, and calculate the loss L2 (two pairs of positive samples are required, so each batch must have at least two pairs of positive samples). Step c, negative sample pair loss For each negative sample pair, calculate the loss L3; Select symmetrical negative sample pairs of images and text, and calculate the loss L4, which is the KL divergence difference loss; Step d: Calculate the total loss and backpropagate to update the parameters of the image encoder, text encoder, and classifier (the fully connected layer used for concatenating vectors for classification).

[0048] Reference Figure 3As shown, after the image encoder is fine-tuned, the image we need to cluster is input, and the image encoder obtains the embedding vector. Because the model has been fine-tuned for both text and images, it can obtain discriminative embedding vectors even for images with low similarity. Clustering can be performed based on the first similarity of pairwise embedding vectors. Therefore, conventional clustering algorithms, such as spectral clustering, k-means, and Dbscan, can achieve good results.

[0049] In some embodiments, calculating the offset similarity between any two images to be clustered to obtain a second similarity includes: The image to be clustered is converted into a three-dimensional matrix to obtain the pixel value at each coordinate position in each channel; The DTW similarity of two images to be clustered is calculated using pixel values ​​in each channel, and a second similarity is obtained based on the DTW similarity in different channels.

[0050] In practice, we've found that images with the same shape may exhibit some shifts. While the vectors generated by the model for images with the same type of shift are similar, they still possess a certain degree of distinguishability. To further reduce the similarity between shifted images and improve model performance, we've added a DTW-based image similarity calculation method to our similarity calculation approach.

[0051] Taking the calculation of similarity between two images as an example, for images A / B, the images can be transformed into a three-dimensional matrix, such as A[i][j][k] representing the positions of pixels i and j in the k channel of image A. Based on the DTW algorithm, our goal is to find a regular path. Each of them This represents the correspondence between image A and image B at pixel positions (i, j). Since the different channels of an image are independent of each other, the similarity between image A and image B can be calculated separately, and then the average of the different channels can be taken as the DTW similarity between image A and image B.

[0052] For any channel, the recursive formula for dynamic programming is: in, This refers to the pixel value of image A at the (1,1) pixel position. This refers to the pixel value of image B at the (1,1) pixel position. Since the DTW principle of conventional time series data is an existing technology, the recursive formula of the above dynamic programming will not be introduced in detail here.

[0053] Finally, the minimum path distance between images A and B is obtained. After obtaining the DTW distance of the images, the distance values ​​are normalized to obtain the DTW similarity, ensuring the consistency of the similarity measurement units. In the clustering process, the similarity calculation can use a weighted average of vector similarity (first similarity) and DTW similarity (second similarity) in the clustering calculation.

[0054] In one embodiment, an image clustering system is provided, comprising: The training module constructs positive sample pairs based on images and text with the same defect type and negative sample pairs based on images and text with different defect types. Based on the consistency loss of the positive samples and the inconsistency loss of the negative samples, it trains an image encoder and a text encoder. The clustering module acquires multiple images to be clustered, obtains the embedding vector of each image to be clustered through the image encoder, and obtains the first similarity between any two embedding vectors; calculates the similarity between any two images to be clustered to obtain the second similarity, and performs clustering based on the first and second similarities to obtain the image clustering result.

[0055] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described image clustering method embodiments.

[0056] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0057] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0058] In the description of this specification, the references to terms such as "some embodiments," "other embodiments," "ideal embodiments," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example that are included in at least one embodiment or example of this application. In this specification, the illustrative descriptions of the above terms do not necessarily refer to the same embodiments or examples.

[0059] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0060] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image clustering method, characterized in that, include: Positive sample pairs are constructed based on images and text with the same defect type, and negative sample pairs are constructed based on images and text with different defect types. An image encoder and a text encoder are trained based on the consistency loss of the positive sample pairs and the inconsistency loss of the negative sample pairs. Multiple images to be clustered are obtained, and the embedding vector of each image to be clustered is obtained through the image encoder. The first similarity between any two of the embedding vectors is obtained. Calculate the offset similarity between any two images to be clustered to obtain the second similarity. Perform clustering based on the first and second similarities to obtain the image clustering results.

2. The image clustering method according to claim 1, characterized in that, The positive sample pair includes a first positive sample that matches the image and text under the same defect type. Each first positive sample obtains a first image vector and a first text vector through the image encoder and the text encoder. The consistency loss of the positive sample includes a loss L1 obtained based on the first similarity value of the first image vector and the first text vector.

3. The image clustering method according to claim 2, characterized in that, The method further includes a first classifier, wherein the positive sample pairs consist of two second positive samples formed by swapping images and text between the two first positive samples; the consistency loss of the positive samples includes: Classification is performed based on the image vectors and text vectors of two second positive samples, as well as the classifier, and the loss L2 is obtained based on the classification results.

4. The image clustering method according to claim 3, characterized in that, The classification based on the image vectors and text vectors of two second positive samples, and the classifier, and the loss L2 obtained from the classification result, includes: The image vector of each second positive sample is concatenated with the text vector to obtain two first concatenated vectors; The two first concatenated vectors are input into the classifier to obtain two prediction results. The cross-entropy loss is calculated based on the two prediction results to obtain the loss L2 of the two second positive samples.

5. The image clustering method according to claim 2, characterized in that, The negative sample is formed by freely combining the image and text of the first positive sample, which includes different defect types; each first negative sample is processed by an image encoder and a text encoder to obtain a second image vector and a second text vector, and the consistency loss of the negative sample includes a loss L3 obtained based on the second similarity value of the second image vector and the second text vector.

6. The image clustering method according to claim 5, characterized in that, The negative sample pair comprises two second negative samples formed by swapping the images and text of two first negative samples. The consistency loss of the negative samples includes: Classification is performed based on the image vectors and text vectors of the two second negative samples, as well as the classifier, and the loss L4 is obtained based on the classification results.

7. The image clustering method according to claim 6, characterized in that, The classification based on the image vectors and text vectors of the two second negative samples, and the classifier, and the loss L4 obtained from the classification result, includes: The image vector of each second negative sample is concatenated with the text vector to obtain two second concatenated vectors; The two concatenated vectors are input into the classifier to obtain two prediction results. The KL divergence loss is calculated based on the two prediction results to obtain the loss L4 of the two second negative samples.

8. The image clustering method according to claim 7, characterized in that, The two predicted results are assumed to be: and Calculate the KL divergence value: and / or ; The loss L4 includes: and / or L4=−[KL_div(PAC|PCA+KL_div(PCA|PAC].

9. The image clustering method according to claim 1, characterized in that, The calculation of the offset similarity between any two images to be clustered, to obtain the second similarity, includes: The image to be clustered is converted into a three-dimensional matrix to obtain the pixel value at each coordinate position in each channel; The DTW similarity of two images to be clustered is calculated using pixel values ​​in each channel, and a second similarity is obtained based on the DTW similarity in different channels.

10. An image clustering model, characterized in that, include: The training module constructs positive sample pairs based on images and text with the same defect type and negative sample pairs based on images and text with different defect types. Based on the consistency loss of the positive samples and the inconsistency loss of the negative samples, it trains an image encoder and a text encoder. The clustering module acquires multiple images to be clustered, obtains the embedding vector of each image to be clustered through the image encoder, and obtains the first similarity between any two embedding vectors; calculates the similarity between any two images to be clustered to obtain the second similarity, and performs clustering based on the first and second similarities to obtain the image clustering result.

11. A readable storage medium storing a program, characterized in that, When the program is executed, it implements the image clustering method according to any one of claims 1 to 9.

Images