Untrained referential image segmentation method based on diffusion model

Abstract

This invention presents a training-free referential image segmentation method based on a diffusion model, belonging to the field of referential image segmentation in computer vision. It addresses the problem of difficulty in achieving referential image segmentation under conditions of limited existing data. The method includes: acquiring the original image and text to be segmented; adding Gaussian noise to the original image and generating a matrix of all calculated results from the cross-attention function; identifying all target matrices corresponding to the core word positions from the matrix of all calculated results from the cross-attention function; expanding the target matrices using bilinear interpolation and then averaging them to obtain a referential relation matrix; segmenting the original image using a SAM model to obtain multiple candidate segmentation result matrices; calculating the similarity between each candidate segmentation result matrix and the referential relation matrix; and selecting the candidate segmentation result matrix with the highest similarity as the referential image segmentation result. This invention achieves training-free referential image segmentation.

Description

Technical Field

[0001] This invention relates to a training-free referential image segmentation method based on a diffusion model, belonging to the field of referential image segmentation in computer vision. Background Technology

[0002] The goal of training-free referential image segmentation is to segment the regions described by the text in an image, given an image and text, without any prior data. Training a usable referential image segmentation system often requires a large amount of data, but collecting enough referential image segmentation data in real-world scenarios is extremely difficult. Therefore, traditional methods struggle with referential image segmentation in situations where data is extremely scarce. Summary of the Invention

[0003] To address the problem of difficulty in achieving referential image segmentation due to the scarcity of existing data, this invention provides a training-free referential image segmentation method based on a diffusion model.

[0004] The present invention provides a training-free referential image segmentation method based on a diffusion model, comprising:

[0005] Step 1: Obtain the original image and text to be segmented. Add Gaussian noise to the original image and input it together with the text into the text-to-image generator to obtain the full calculation result matrix of the cross-attention function.

[0006] Step 2: Use syntactic analysis tools to determine the positions of the core words in the text, and determine all target matrices corresponding to the positions of the core words in the total calculation result matrix of the cross-attention function; use bilinear interpolation algorithm to expand each target matrix to the same size as the original image, and then perform mean normalization to obtain the reference relation matrix;

[0007] Step 3: Use the SAM model to segment the original image to obtain multiple candidate segmentation result matrices; calculate the similarity between each candidate segmentation result matrix and the referential relationship matrix, and select the candidate segmentation result matrix with the highest similarity as the referential image segmentation result.

[0008] According to the diffusion model-based training-free referential image segmentation method of the present invention, in step one, the method for obtaining the text-to-image generator is to download an open-source text-to-image diffusion model-based generator, or a pre-trained diffusion model-based generator.

[0009] In the training-free referential image segmentation method based on the diffusion model according to the present invention, in step two, the syntactic analysis tool is SpaCy.

[0010] According to the diffusion model-based training-free referential image segmentation method of the present invention, the method for obtaining the core word position in step two is as follows:

[0011] Syntactic information for each word in the text is obtained using syntactic analysis tools; words whose syntactic information is the root word are selected as core words, and the positions of the core words are determined.

[0012] According to the diffusion model-based training-free referential image segmentation method of the present invention, in step two, the target matrix is ​​represented as A′:

[0013] A′=R k ,

[0014] In the formula R k This is the matrix corresponding to the core word position k in the total calculation result matrix of the cross-attention function.

[0015] According to the diffusion model-based training-free referential image segmentation method of the present invention, in step two, the target matrix is ​​expanded to the same size as the original image and represented as matrix A:

[0016] A = φ(A′),

[0017] In the formula, φ is a bilinear interpolation function.

[0018] According to the diffusion model-based training-free referential image segmentation method of the present invention, in step three, the method for calculating the similarity between the candidate segmentation result matrix and the referential relationship matrix is ​​as follows:

[0019]

[0020] In the formula q i Let P be the segmentation result matrix of the i-th candidate. i The similarity to the referential relation matrix S, where f is the element summation function.

[0021] The beneficial effects of the present invention are as follows: The method of the present invention uses a text-to-image generation model to obtain the image's referential segmentation, and designs a method for obtaining the referential relationship matrix and similarity calculation from the text-to-image generation model, so as to obtain the relationship between the text and the image region without training, and thus achieve referential segmentation without the need for referential segmentation training data. Attached Figure Description

[0022] Figure 1 This is a flowchart of an implementation of the training-free referential image segmentation method based on the diffusion model described in this invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0025] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0026] Specific Implementation Method 1: Combination Figure 1 As shown, this invention provides a training-free referential image segmentation method based on a diffusion model, comprising:

[0027] Step 1: Obtain the original image and text to be segmented. Add Gaussian noise to the original image, and then input the noisy original image and text together into the text-to-image generator to obtain the complete calculation result matrix of the cross-attention function. Since the image processed by the text-to-image generator needs to be in a noisy environment, Gaussian noise is added to the original image.

[0028] Step 2: Use syntactic analysis tools to determine the positions of the core words in the text, and determine all target matrices corresponding to the positions of the core words in the total calculation result matrix of the cross-attention function; use bilinear interpolation algorithm to expand each target matrix to the same size as the original image, and then perform mean normalization to obtain the reference relation matrix;

[0029] Step 3: Use the SAM model to segment the original image to obtain multiple candidate segmentation result matrices; calculate the similarity between each candidate segmentation result matrix and the referential relationship matrix, and select the candidate segmentation result matrix with the highest similarity as the referential image segmentation result.

[0030] This implementation considers the role of the DiffusionModel-based text-to-image generation model in understanding the relationship between text and images. It designs a method to obtain the referential relationship matrix and similarity calculation from the text-to-image generation model, thus solving the problem of referential segmentation in the absence of training data. Therefore, given an image and text, the corresponding region described by the text in the image can be directly segmented.

[0031] As an example, in step one, the method to obtain the text-to-image generator is to download an open-source text-to-image generator based on a diffusion model, or to obtain a pre-trained generator based on a diffusion model.

[0032] In this implementation, if a pre-trained generator is available, it can be used directly; otherwise, the generator needs to be trained on image and text data until convergence. The pre-trained generator based on a diffusion model is obtained by collecting a sufficient number of images for training.

[0033] As an example, in step two, the syntactic analysis tool is SpaCy.

[0034] Furthermore, the method for obtaining the location of core words in step two is as follows:

[0035] Syntactic information for each word in the text is obtained using syntactic analysis tools; words whose syntactic information is the root word are selected as core words, and the positions of the core words are determined.

[0036] Furthermore, in step two, the dimensions corresponding to the core word positions in all calculation results of the cross-attention function are selected, and the target matrix is ​​represented as A′:

[0037] A′=R k ,

[0038] In the formula R k This is the matrix corresponding to the core word position k in the total calculation result matrix of the cross-attention function.

[0039] In step two, the target matrix is ​​expanded to the same size as the original image, and is represented as matrix A:

[0040] A = φ(A′),

[0041] In the formula, φ is a bilinear interpolation function.

[0042] The mean of the bilinear interpolation results of all target matrices is used to obtain the referential matrix.

[0043] Furthermore, in step three, it is assumed that the SAM model segments the original image, obtaining n candidate segmentation result matrices P1, P2, ..., P n The similarity between the candidate segmentation result matrix and the referential relation matrix is ​​calculated as follows:

[0044]

[0045] In the formula q i Let P be the segmentation result matrix of the i-th candidate. iThe similarity to the referential relation matrix S, where f is the element-sum function. The sign for multiplying all elements of the matrix is ​​used. The candidate segmentation matrix corresponding to the highest similarity value is selected as the representative image segmentation result.

[0046] Compared with existing methods, the method of this invention does not require manually labeled data references for image segmentation data, which greatly reduces the application cost.

[0047] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A training-free referential image segmentation method based on a diffusion model, characterized in that... include, Step 1: Obtain the original image and text to be segmented. Add Gaussian noise to the original image and input it together with the text into the text-to-image generator to obtain the full calculation result matrix of the cross-attention function. Step 2: Use syntactic analysis tools to determine the positions of the core words in the text, and determine all target matrices corresponding to the positions of the core words in the total calculation result matrix of the cross-attention function; use bilinear interpolation algorithm to expand each target matrix to the same size as the original image, and then perform mean normalization to obtain the reference relation matrix; Step 3: Use the SAM model to segment the original image to obtain multiple candidate segmentation result matrices; Calculate the similarity between each candidate segmentation result matrix and the referential relationship matrix, and select the candidate segmentation result matrix with the highest similarity as the referential image segmentation result; The method for obtaining the location of core words in step two is as follows: Syntactic information for each word in the text is obtained using syntactic analysis tools; words whose root syntactic information is selected as core words are then identified, and their positions are determined. In step three, the similarity between the candidate segmentation result matrix and the referential relation matrix is ​​calculated as follows: ， In the formula The matrix of the i-th candidate segmentation results Similarity to the referential relation matrix S, A function to sum the elements.

2. The training-free referential image segmentation method based on a diffusion model according to claim 1, characterized in that, In step one, the text-to-image generator is obtained by downloading an open-source text-to-image generator based on a diffusion model, or by obtaining a pre-trained generator based on a diffusion model.

3. The training-free referential image segmentation method based on a diffusion model according to claim 2, characterized in that, In step two, the syntactic analysis tool is SpaCy.

4. The training-free referential image segmentation method based on a diffusion model according to claim 3, characterized in that, In step two, the target matrix is ​​represented as : ， In the formula This is the matrix corresponding to the core word position k in the total calculation result matrix of the cross-attention function.

5. The training-free referential image segmentation method based on a diffusion model according to claim 4, characterized in that, In step two, the target matrix is ​​expanded to the same size as the original image and then represented as a matrix. : ， In the formula It is a bilinear interpolation function.

Images