Image segmentation method, model, training method of the model, and image segmentation system

Abstract

The present invention relates to the field of image processing technology, and in particular to an image segmentation method, model, model training method, and image segmentation system. The method comprises: obtaining an image to be segmented containing a target object and corresponding text information; encoding the text information through a text encoder of the image segmentation model to generate text embedding features, and encoding the image through an image encoder of the image segmentation model to generate image latent features; processing the image latent features through an attention module and a splicing module of the image segmentation model based on the text embedding features according to a preset number of fusions until the noisy masked latent features generated at the last fusion are obtained, and decoding the latent features through a latent space decoder of the image segmentation model after denoising to generate a predicted target mask for segmenting the target object; and segmenting the target object in the image based on the predicted target mask. The present invention improves the accuracy of image segmentation.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image segmentation method, a model, a model training method and an image segmentation system. Background Art

[0002] Open-Vocabulary Semantic Segmentation (OVSS) is an important computer vision task that aims to semantically annotate each pixel in an image based on a textual description. Unlike traditional semantic segmentation methods, OVSS requires the ability to recognize unseen categories to adapt to diverse targets in open environments. While multimodal models based on text-image matching perform well for single-target classification, they struggle to accurately understand the relationships between objects in complex scenes, limiting their application to pixel-level segmentation. Furthermore, while recent text-to-image diffusion models have demonstrated strong capabilities in generating high-quality images, most OVSS research has used these diffusion models solely as fixed feature extractors, failing to fully leverage their rich visual-linguistic understanding capabilities. This results in limited performance for pixel-level semantic alignment and open-category recognition, hindering their segmentation capabilities in zero-shot environments and complex scenes. Therefore, there is a need for an image segmentation method, model, model training method, and image segmentation system. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide an image segmentation method, model, model training method and image segmentation system, which improve the problem of low recognition accuracy of targets and unseen categories in complex scenes when performing open vocabulary semantic segmentation tasks in the prior art.

[0004] To achieve the above-mentioned and other related purposes, the present invention provides an image segmentation method, comprising: obtaining an image to be segmented containing a target object and text information corresponding thereto; wherein the text information is used to characterize the category of the target object in the image; encoding the text information through a text encoder of an image segmentation model to generate text embedding features, and encoding the image through an image encoder of the image segmentation model to generate image latent features; according to a preset number of fusions, based on the text embedding features, processing the image latent features through an attention module and a splicing module of the image segmentation model within a preset number of fusions until a noisy mask generated at the last fusion is obtained. The image latent features are decoded and, after denoising, are decoded by the latent space decoder of the image segmentation model to generate a predicted target mask for segmenting the target object; wherein, at each fusion: the image latent features and the noisy mask latent features are fused by the splicing module of the image segmentation model, and the visual features are extracted from the fused features obtained after the fusion by the attention module, and the visual features are aligned and fused with the text embedding features to update the noisy mask latent features at the time of the next fusion; wherein, the noisy mask latent features at the time of the first fusion are random noise generated by the noise generator of the image segmentation model; and the target object in the image is segmented based on the predicted target mask.

[0005] In one embodiment of the present invention, the latent space decoder generates K predicted target masks, where K is a positive integer greater than 1. The processed mask latent features are decoded to generate a predicted target mask for segmenting the target object, including: decoding the processed mask latent features to obtain K initial predicted target masks; and calculating the average of the K initial predicted target masks as the final predicted target mask.

[0006] In one embodiment of the present invention, the image potential features and the noisy mask potential features are fused through the splicing module of the image segmentation model, and the visual features are extracted from the fused features obtained after the fusion through the attention module, and the visual features are aligned and fused with the text embedding features to update the noisy mask potential features at the time of the next fusion, including: fusing the image potential features and the noisy mask potential features through the splicing module of the image segmentation model to obtain fused features; extracting visual features of corresponding scales from the fused features based on downsampling networks of different scales through the diffusion coding unit of the attention module; wherein the encoder module includes a cascade of multiple downsampling networks with decreasing scales in sequence; and the attention unit of the attention module uses the content attention mechanism to convert the minimum scale into the minimum scale. The visual features of the text embedding features are cross-modally fused and aligned to obtain fused aligned features; the intermediate features and the visual features of the corresponding scale are fused in each upsampling unit through the diffusion decoding unit of the attention module to obtain the intermediate features input to the next upsampling unit, and the prediction noise is generated based on the intermediate features obtained from the last upsampling unit; wherein, the initial intermediate features are the fused aligned features, and the decoder module includes a plurality of cascaded upsampling units with increasing scales, and the scale of the upsampling unit corresponds to the scale of the downsampling network; the denoising unit of the attention module is used to denoise the noisy masked latent features at the time of current fusion based on the predicted noise to obtain the denoised masked latent features, and use them as the noisy masked latent features for the next fusion.

[0007] In one embodiment of the present invention, the fusing of the image latent features and the noisy mask latent features includes: superimposing the image latent features and the noisy mask latent features to generate a joint feature; and splicing the joint feature and the noisy mask latent features to generate a fused feature.

[0008] In one embodiment of the present invention, fusing the image latent features and the noisy mask latent features includes: performing a splicing process on the image latent features and the noisy mask latent features to generate a fused feature.

[0009] In one embodiment of the present invention, the random noise is Gaussian noise.

[0010] In one embodiment of the present invention, the attention unit of the attention module performs cross-modal feature fusion and alignment on the minimum-scale visual features and the text embedding features based on the content attention mechanism to obtain fused and aligned features, including: generating a query matrix, a key matrix and a first value matrix based on the minimum-scale visual features, and generating an intermediate matrix and a second value matrix based on the text embedding features; calculating the self-attention weight of the visual features based on the query matrix and the key matrix according to the scaled dot product attention mechanism; calculating the interactive attention weight of the text features to the visual features based on the key matrix and the intermediate matrix according to the scaled dot product attention mechanism; and performing weighted summation on the first value matrix and the second value matrix based on the self-attention weight and the interactive attention weight to obtain a fused and aligned feature.

[0011] In one embodiment of the present invention, an image segmentation model is also provided, which includes: an image encoder, a noise generator, a text encoder, a splicing module, an attention module and a latent space decoder, wherein: the output end of the image encoder is connected to the first input end of the splicing module; the output end of the noise generator is connected to the second input end of the splicing module; the output end of the text encoder is connected to the first input end of the attention module; the output end of the splicing module is connected to the second input end of the attention module; wherein the attention module is used to fuse the image latent features obtained after the image encoder encodes the image and the random noise generated by the noise generator to extract visual features, and align and fuse the visual features and the text embedding features obtained by the text encoder from encoding the text information, and obtain denoised mask latent features after denoising; the output end of the attention module is connected to the input end of the latent space decoder; wherein the latent space decoder is used to decode the denoised mask latent features to generate a predicted target mask for segmenting the target object in the image.

[0012] In one embodiment of the present invention, a training method for an image segmentation model is also provided, the training method comprising: obtaining an image containing a target object and text information corresponding thereto, and a target mask of the target object in the image; wherein the text information is used to characterize the category of the target object in the image; inputting the text information into a text encoder of a semantic segmentation model for encoding to generate text embedding features, inputting the image into an image encoder of the semantic segmentation model for encoding to generate image latent features, and inputting the target mask into a mask encoder of the semantic segmentation model for encoding to generate mask latent features; according to a preset number of fusions, The attention module of the semantic segmentation model is fine-tuned and trained at each fusion to obtain a trained attention module of the semantic segmentation model; wherein, at each fusion: the mask latent features are noised and fused with the image latent features through the noise addition module and the splicing module of the semantic segmentation model, the visual features are extracted from the fused features obtained after the fusion through the attention module of the semantic segmentation model, the visual features and the text embedding features are aligned and fused based on the attention mechanism to generate noisy masked latent features, and the attention module of the semantic segmentation model is trained through the noisy masked latent features.

[0013] In one embodiment of the present invention, an image segmentation system is further provided, comprising: an image encoder, a mask encoder, a noise generator, a text encoder, a noise summation module, a splicing module, and an attention module, wherein: an output end of the mask encoder is connected to a first input end of the noise summation module; wherein the mask encoder is used to encode a target mask of a target object in the image into a masked latent feature; an output end of the noise generator is connected to a second input end of the noise summation module; wherein the noise generator is used to generate random noise; and an output end of the image encoder is connected to a first input end of the splicing module; Wherein, the image encoder is used to encode the image to generate image latent features; the output end of the noise addition module is connected to the second input end of the splicing module; the output end of the text encoder is connected to the first input end of the attention module; wherein, the text encoder is used to encode text information to generate text embedding features; the output end of the splicing module is connected to the second input end of the attention module; wherein, the attention module is used to extract visual features from the fused features generated by the splicing module, and align and fuse the visual features and the text embedding features through the attention mechanism to generate noisy masked latent features.

[0014] As described above, the present invention provides an image segmentation method, model, model training method, and image segmentation system, each having the following beneficial effects: Textual information of a target object is input into a text encoder, which encodes the textual information to generate text embedding features, enabling the model to understand image content and adapt to unseen categories, thereby improving zero-shot segmentation capabilities. Furthermore, the image encoder encodes the image into latent features. After encoding, the image latent features are fused with noisy mask latent features according to a preset number of fusions, and visual information is extracted. This information is then cross-modally aligned with the text embedding features, and the mask features are continuously optimized to enhance the model's ability to capture object boundaries and semantic information. During the initial fusion, the noisy mask latent features are initialized with random noise. After a multi-step iterative diffusion denoising process, clear target mask features are gradually extracted. The latent space decoder decodes the mask to generate a high-quality predicted target mask, which is then used to accurately segment the original image. Compared to traditional segmentation methods, the present invention can effectively remove irrelevant interference in complex background environments. Furthermore, by combining cross-modal information fusion, it improves generalization and segmentation accuracy for unseen categories, achieving more efficient open-lexicon semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic flow chart of an image segmentation method provided by an embodiment of the present invention;

[0016] Figure 2 A schematic diagram of the overall architecture of an image segmentation model provided by an embodiment of the present invention;

[0017] Figure 3 A schematic flow chart of a method for training an image segmentation model according to an embodiment of the present invention;

[0018] Figure 4 Shown is a schematic diagram of the overall architecture of an image segmentation system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0020] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0021] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0022] Open Vocabulary Semantic Segmentation (OVSS) is a task that aims to semantically label each pixel in an image based on a natural language description. Its goal is to break through the reliance of traditional semantic segmentation methods on a fixed set of categories and effectively recognize unseen categories. Currently, methods based on large-scale pre-trained image-text models such as CLIP demonstrate good generalization in open-category recognition, but still suffer from misjudgments and insufficient understanding when handling spatial relationships between objects in complex scenes. Furthermore, text-to-image diffusion models have become a focus of multimodal research due to their excellent image generation quality, strong semantic guidance capabilities, and excellent cross-modal alignment properties. However, existing methods typically use diffusion models as static feature extractors and fail to effectively tune their internal visual language modeling modules (such as U-Net), thus failing to fully unleash the potential of diffusion models. Consequently, the current OVSS field still faces the challenge of insufficient generalization in zero-shot scenarios.

[0023] To address the above situation, the present invention provides an image segmentation method that inputs the text information of the target object into a text encoder, encodes the text information to generate text embedding features, and enables the model to understand the image content and adapt to unseen categories, thereby improving zero-shot segmentation capabilities. Furthermore, the image encoder encodes the image into image latent features. After encoding, the image latent features are fused with noisy mask latent features according to a preset number of fusions, and visual information is extracted. This is then cross-modally aligned with the text embedding features, and the mask features are continuously optimized to enhance the model's ability to capture target boundaries and semantic information. During the initial fusion, the noisy mask latent features are initialized with random noise. After a multi-step iterative diffusion denoising process, clear target mask features are gradually extracted. Finally, the latent space decoder decodes the mask to generate a high-quality predicted target mask, which is then used to accurately segment the original image. Compared to traditional segmentation methods, the present invention can effectively remove irrelevant interference in complex background environments. At the same time, combined with cross-modal information fusion, it improves the generalization ability and segmentation accuracy of unseen categories, achieving more efficient open vocabulary semantic segmentation.

[0024] like Figure 1 As shown, the image segmentation method provided by the present invention includes the following steps:

[0025] S11. Acquire an image to be segmented containing a target object and corresponding text information; wherein the text information is used to represent the category of the target object in the image.

[0026] Textual information is used to characterize the categories of target objects in an image, thus supporting the subsequent alignment of visual and semantic information. For example, the textual information might be "a photo of {*}, a photo of multiple {*}, ...," where "{*}" is a coarse category descriptor (e.g., "sky," "trees," "buildings," etc.). To ensure that the acquired image meets the input requirements of the subsequent image segmentation model, the image can be preprocessed. Preprocessing methods include, but are not limited to, normalization and resolution adjustment.

[0027] S12. Encode the text information through the text encoder 1 of the image segmentation model to generate text embedding features, and encode the image through the image encoder 2 of the image segmentation model to generate image potential features.

[0028] like Figure 2 As shown, for RGB images, given an image x and its text information ，The text information is input into the text encoder 1 for encoding to convert the text information into a high-dimensional embedding feature to obtain a text embedding feature, which is used as a guiding factor to denoise the generated noisy latent mask feature to update the noisy latent mask feature for subsequent image segmentation. The text encoder 1 can be any Transformer-based encoder, including but not limited to CLIP, BERT, etc. Preferably, in order to better capture cross-modal semantic relationships, the text encoder 1 is CLIP. The image is input into the image encoder 2 to extract image features and map the image features to a low-dimensional latent space to reduce computational complexity and generate image latent features. The image encoder 2 includes, but is not limited to, a variational autoencoder (VAE), a ResNet, or a convolutional neural network, as long as the image can be mapped to a low-dimensional latent space. Preferably, in order to effectively learn the latent representation of the image and be applicable to generative tasks, the image encoder 2 is an encoder of a variational autoencoder.

[0029] S13, according to a preset number of fusions, based on the text embedding features, the image latent features are processed by the attention module 5 and the splicing module 4 of the image segmentation model within the preset number of fusions until the noisy mask latent features generated at the last fusion are obtained, and after denoising, the latent space decoder 6 of the image segmentation model is used to decode the latent features to generate a predicted target mask for segmenting the target object; wherein, at each fusion:

[0030] The image latent features and the noisy mask latent features are fused through the splicing module 4 of the image segmentation model, and the visual features are extracted from the fused features obtained after fusion through the attention module 5, and are aligned and fused with the text embedding features to update the noisy mask latent features during the next fusion; wherein the noisy mask latent features during the first fusion are the random noise generated by the noise generator 3 of the image segmentation model.

[0031] Continue as Figure 2 As shown, in each fusion process, the splicing module 4 of the image segmentation model converts the image potential features Noisy masked latent features corresponding to the current fusion number t The fusion is performed, and the fused visual features are extracted through the attention module 5, which are aligned with the text embedding features to update the noisy masked latent features required for the next fusion time t-1 When fused for the first time, the noisy mask potential feature is the random noise generated by noise generator 3 In order to achieve more efficient and accurate image segmentation, the random noise is Gaussian noise. The above fusion process continues until the preset fusion times are reached, and the noisy mask potential features after the last fusion are obtained. . The mask latent features After denoising, it is input into the latent space decoder 6 of the image segmentation model for decoding to generate the predicted target mask , in order to achieve accurate segmentation of the target object. It should be noted that in order to speed up the inference process, this application performs non-Markov sampling by recalibrating the number of steps to obtain random noise.

[0032] Continue as Figure 2 As shown, in one embodiment of the present invention, S13 includes S131 to S135 (not shown in the figure):

[0033] S131 , fusing the image latent features and the noisy mask latent features through the splicing module 4 of the image segmentation model to obtain fused features.

[0034] At each fusion, the image latent features and the noisy mask latent features are fused by the splicing module 4 of the image segmentation model to obtain the fused features. At each fusion (i.e. at the current time step t): Figure 2 As shown, the image potential features and the noisy masked latent features at the current time step t Through the splicing module 4, the fusion feature of the current time step is obtained. The fusion features obtained after splicing It contains both the visual information of the original image and the noise information of the mask feature during the current fusion.

[0035] Optionally, the present invention can obtain fusion features in the following two different ways. The following takes the t-th fusion (i.e., the t-th time step) as an example: Method 1: The image potential features are converted into and noisy masked latent features Perform superposition processing to generate joint features; combine joint features and noisy masked latent features Perform splicing processing to generate fusion features ,Right now =cat( , ). Specifically, the image latent features and the noisy masked latent features at the current time step t Add element by element and generate joint features through this superposition process. Combine the joint features with the noisy mask latent features Splicing along the channel dimension to obtain the final fusion feature This fusion method not only retains the original mask information, but also enhances its interaction with image features, so that the text information can better guide the subsequent denoising process, thereby providing a richer feature expression for the subsequent attention mechanism to improve the accuracy of target object segmentation. Method 2: Splice the image latent features and the noisy mask latent features to generate fused features. Specifically, the image latent features and the noisy masked latent features at the current time step t Splicing along the channel dimension to generate fusion features ,Right now =cat( , ). This adjustment helps to speed up the convergence of the model and significantly accelerates the training process without prediction errors. The fused features obtained in this way not only contain the local and global visual features of the image, but also retain the noise characteristics of the mask, so that the subsequent attention mechanism can optimize the mask more accurately. After obtaining the fused features in the above way, the number of input channels is doubled to fit the expanded input .

[0036] S132. Extract visual features of corresponding scales from fused features through the diffusion coding unit of the attention module 5 based on downsampling networks of different scales; wherein the diffusion coding unit includes a cascade of multiple downsampling networks with decreasing scales.

[0037] After the fused features are input into the diffusion coding unit, the features are propagated downward layer by layer through multiple cascaded downsampling networks. The feature map output by each downsampling network serves as the input of the next downsampling network. This layer-by-layer propagation method can gradually reduce the size of the feature map to extract higher-level semantic information. Specifically, for the initial downsampling network, basic information such as edges and textures of the feature map can be extracted. As the feature map propagates downward layer by layer, the intermediate-scale downsampling network pays more attention to the overall shape of the target object and the semantic information of the local area. When the feature enters the deeper downsampling network, the scale gradually decreases. The model focuses on extracting high-level semantic information, strengthening the ability to distinguish target categories, and focusing on key areas through the attention mechanism. These visual features at different scales provide richer feature representations for the subsequent semantic alignment and decoding processes, thereby improving the segmentation accuracy and generalization ability of the model.

[0038] S133. Through the attention unit of the attention module 5, the minimum-scale visual features and text embedding features are cross-modally fused and aligned based on the content attention mechanism to obtain fused and aligned features.

[0039] Through the attention unit, the smallest-scale visual features are cross-modally aligned with the text embedding features. Since the smallest-scale visual features provide global semantic information, and the text embedding features provide a semantic description of the target object, the attention mechanism aligns the text information and the semantic information, so that the model can more accurately understand the correspondence between text information in the visual space.

[0040] In an optional embodiment, S133 includes: generating a query matrix, a key matrix and a first value matrix based on the minimum-scale visual features, and generating an intermediate matrix and a second value matrix based on the text embedding features; based on the query matrix and the key matrix, according to the scaled dot product attention mechanism, calculating the self-attention weight of the visual features; based on the key matrix and the intermediate matrix, according to the scaled dot product attention mechanism, calculating the interactive attention weight of the text features to the visual features; based on the self-attention weight and the interactive attention weight, performing weighted summation on the first value matrix and the second value matrix to obtain a fused alignment feature.

[0041] Specifically, the query matrix Q, key matrix K and first value matrix are generated based on the minimum scale visual features. , where the query matrix represents the most representative part of the minimum-scale visual features, the key matrix is used to provide a reference for the query matrix to calculate the attention distribution, and the first value matrix is used for weighted aggregation during attention calculation. The text embedding features are generated by linear projection into the intermediate matrix I and the second value matrix , where the middle matrix is used to match visual features so that the text and visual features are mapped and associated, and the second value matrix can store text embedding features to provide basic information for subsequent fusion. Using the scaled dot product attention mechanism, as shown in formula (1), the self-attention weight of the visual feature is calculated :

[0042] (1)

[0043] Among them, d is the feature dimension, Q is the query matrix, and K is the key matrix. The self-attention weight can help the model determine which areas in the image are more important, so as to better understand the internal structure and content of the image and realize subsequent image segmentation. Similarly, according to the key matrix and the intermediate matrix, the interactive attention weight is calculated using formula (2) :

[0044] (2)

[0045] Where I is the intermediate matrix, combined with the calculated attention weight, the visual features and text features are weighted as shown in formula (3) to obtain the fused alignment feature :

[0046] (3)

[0047] in, and The first value matrix and the second value matrix, respectively. This query-key-value mechanism allows visual features to better align with text semantics, thereby improving the accuracy of subsequent semantic segmentation. This process narrows the gap between text embeddings and visual features, leveraging the unique strengths of both text and image modalities to improve image segmentation accuracy.

[0048] S134. Through the diffusion decoding unit of the attention module 5, the intermediate features and the visual features of the corresponding scale are fused in each upsampling unit to obtain the intermediate features input to the next upsampling unit, and the prediction noise is generated based on the intermediate features obtained by the last upsampling unit; wherein, the initial intermediate features are fused alignment features, and the decoder module includes a plurality of cascaded upsampling units with increasing scales, and the scale of the upsampling unit corresponds to the scale of the downsampling network.

[0049] In each upsampling unit, the intermediate features are upsampled to gradually increase their resolution. The upsampled features are then fused with the visual features of the encoder at the corresponding scale to retain key information about the target area and enhance the model's ability to express local structures. This process is performed sequentially in multiple upsampling units to restore image features layer by layer. The last upsampling unit generates the final intermediate features and calculates the predicted noise through a diffusion process for further denoising or generating a target mask. By gradually increasing the resolution and decoding layer by layer, the image segmentation model can gradually restore the mask features from the overall outline to the edge details, keeping the semantic information consistent during the restoration process. This allows the image segmentation model to more accurately restore the shape and boundaries of the target object in high-resolution images.

[0050] S135. Denoising the noisy masked latent features during the current fusion based on the predicted noise is performed by the denoising unit of the attention module 5 to obtain the denoised masked latent features, which are used as the noisy masked latent features during the next fusion.

[0051] like Figure 2 As shown in Figure 5, the denoising unit of attention module 5 uses the predicted noise at the current time step to denoise the noisy mask latent features, removing unnecessary noise interference and extracting clearer target mask information. The above process follows the reverse diffusion process in the diffusion model: the denoised mask latent features generated at each time step are used as the noisy mask input for the next time step, iterating sequentially to gradually approximate the final true target mask, thereby improving the accuracy of image segmentation.

[0052] like Figure 2 As shown, in one embodiment of the present invention, the latent space decoder 6 generates K predicted target masks, where K is a positive integer greater than 1. The processed mask latent features are decoded to generate a predicted target mask for segmenting the target object, including:

[0053] First, the processed mask latent features are decoded to obtain K initial predicted target masks.

[0054] The masked latent features are gradually upsampled and reconstructed through the latent space decoder 6 to restore them to the resolution of the original input image, and K initial predicted target masks are obtained.

[0055] Then the mean of the K initial predicted target masks is calculated as the final predicted target mask.

[0056] Since the original image is copied into K channels during encoding, after decoding, the K initial predicted target masks need to be averaged to obtain the final predicted target mask. For example, K is 3, which means that we can simulate an RGB image and copy the original image into three channels. After decoding, the three initial predicted target masks are averaged to obtain the final predicted target mask.

[0057] S14. Segment the target object in the image based on the predicted target mask.

[0058] The predicted target mask provides a probability distribution of the category or target region to which each pixel belongs. The model then performs pixel-level classification on the image based on this mask. The target mask accurately determines the boundary region of the target object, thereby completing the segmentation of the target object in the image. It is understandable that since the categories of target objects in an image may not be unique, the predicted target masks of target objects of different categories can be mapped to different depth values according to a preset mapping rule. These depth values are then combined to form a single depth map for segmenting the target object.

[0059] like Figure 2 As shown, the present invention also provides an image segmentation model, including: an image encoder 2, a noise generator 3, a text encoder 1, a splicing module 4, an attention module 5 and a latent space decoder 6, wherein:

[0060] The output terminal of the image encoder 2 is connected to the first input terminal of the splicing module 4;

[0061] The output end of the noise generator 3 is connected to the second input end of the splicing module 4;

[0062] The output terminal of the text encoder 1 is connected to the first input terminal of the attention module 5;

[0063] The output end of the splicing module 4 is connected to the second input end of the attention module 5; wherein the attention module 5 is used to extract visual features from the fused features generated by the splicing module 4, align and fuse the visual features with the text embedding features obtained by encoding the text information by the text encoder 1, and obtain the denoised masked latent features after denoising;

[0064] The output of the attention module 5 is connected to the input of the latent space decoder 6; wherein the latent space decoder 6 is used to decode the denoising mask latent features to generate a predicted target mask for segmenting the target object in the image.

[0065] like Figure 2 As shown, the output end of the image encoder 2 is connected to the second input end of the splicing module 4, which is used to convert the image potential features into The output end of the noise generator 3 is connected to the second input end of the splicing module 4 to generate the noise. The two input ends of the attention module 5 are connected to the splicing module 4 and the text encoder 1 respectively, and are used to receive the text embedding features and the fusion features after splicing. , and after content attention processing, the noisy masked latent features are obtained , after denoising, we get the noisy masked potential features of the next fusion times t-1 Repeat the above process until the last fusion number of noisy mask potential features are obtained, and after denoising, the predicted target mask is obtained through the latent space decoder 6 .

[0066] like Figure 3 and Figure 4 As shown, the present invention also provides a training method for an image segmentation model, the training method comprising:

[0067] S31. Acquire an image containing a target object and corresponding text information, as well as a target mask of the target object in the image; wherein the text information is used to characterize the category of the target object in the image.

[0068] The text information is used to characterize the target object category in the image, providing guidance for subsequent semantic alignment. The object mask is a pixel-level annotation of the target object in the image, which is used to supervise the image segmentation model to learn accurate segmentation boundaries.

[0069] S32. Input the text information into the text encoder 1 of the semantic segmentation model for encoding to generate text embedding features, input the image into the image encoder 2 of the semantic segmentation model for encoding to generate image latent features, and input the target mask into the mask encoder 7 of the semantic segmentation model for encoding to generate mask latent features.

[0070] The text encoder 1 of the semantic segmentation model is used to encode text information (such as a picture of the sky), and the text embedding features of the text information are obtained through steps such as word segmentation, embedding mapping, and attention calculation. The image encoder 2 is used to extract features from the image x and generate low-dimensional image potential features. Similarly, the target mask y is encoded by the mask encoder 7 to generate a low-dimensional mask potential feature . Among them, the image encoder 2 and the mask encoder 7 can be encoders of variational autoencoders.

[0071] S33. Fine-tune the attention module 5 of the semantic segmentation model at each fusion according to a preset number of fusions to obtain a trained attention module 5 of the semantic segmentation model; wherein, at each fusion:

[0072] The masked latent features are denoised by the noise generator 3 of the semantic segmentation model, and the noisy masked latent features and the image latent features are fused by the noise addition module 8 and the splicing module 4 of the semantic segmentation model. The visual features are extracted from the fused features obtained after fusion by the attention module 5 of the semantic segmentation model, and the visual features and the text embedding features are aligned and fused based on the attention mechanism to generate noisy masked latent features, and the attention module 5 of the semantic segmentation model is trained by the noisy masked latent features.

[0073] In each fusion, the mask latent features are input into the noise generator 3 of the semantic segmentation model for noise processing to simulate the noisy mask features in the diffusion process. The noise summation module 8 and the splicing module 4 are used to fuse the noisy mask latent features with the image latent features to obtain richer visual information. The key visual features are extracted and fused with the text embedding features through the attention mechanism to learn a more accurate representation of the target area. Based on this process, the noisy masked latent features are generated. This involves training an image containing prediction noise and continuously optimizing the performance of the attention module 5 of the image segmentation model at different time steps. Ultimately, an efficient attention module 5 is trained, improving the accuracy and generalization of the segmentation task. It is understood that the attention module 5 of the present invention is built on top of a pre-trained text-to-image LDM, effectively leveraging the image prior knowledge in the pre-trained dataset. With minimal modifications to the model components, the model is converted into an open-vocabulary semantic segmentation model by adding the splicing module 4 and modifying the attention mechanism. It should be further noted that the parameters of the text encoder 1, image encoder 2, mask encoder 7, and latent space decoder 6 of the present invention remain unchanged during the aforementioned training process. Furthermore, to simulate RGB images, the target mask is copied to all three channels. This step is crucial because the encoder is initially configured to process three-channel (RGB) input, while the target mask only has a single channel. The present invention reconstructs the target mask from the latent code without modifying the latent space or VAE. During inference, the predicted target mask is obtained by averaging the values across the three channels after decoding the latent code of the mask at the end of the inverse process. In addition, in the training stage of the present invention, the random noise is multi-resolution noise.

[0074] The present invention also provides an image segmentation system, comprising: an image encoder 2, a mask encoder, a noise generator 3, a text encoder 1, a noise summation module, a splicing module 4 and an attention module 5, wherein: the output end of the mask encoder is connected to the first input end of the noise summation module; wherein the mask encoder is used to encode the target mask of the target object in the image into a mask potential feature; the output end of the noise generator 3 is connected to the second input end of the noise summation module; wherein the noise generator 3 is used to generate random noise; the output end of the image encoder 2 is connected to the first input end of the splicing module 4; wherein the image The encoder 2 is used to encode the image to generate image latent features; the output end of the noise addition module is connected to the second input end of the splicing module 4; the output end of the text encoder 1 is connected to the first input end of the attention module 5; wherein, the text encoder 1 is used to encode text information to generate text embedding features; the output end of the splicing module 4 is connected to the second input end of the attention module 5; wherein, the attention module 5 is used to extract visual features from the fused features generated by the splicing module 4, and align and fuse the visual features and the text embedding features through the attention mechanism to generate noisy masked latent features.

[0075] like Figure 4As shown, the output end of the mask encoder and the output end of the noise generator 3 are connected to the first input end and the second input end of the noise summation module respectively. The mask encoder encodes the real target mask into the mask potential feature , the image encoder 2 converts the image x Encoded as image latent features , the noise summation module 8 will mask the potential features and random noise Fusion is performed. At the tth fusion (i.e., the tth time step), the splicing module 4 combines the fused features and image latent features Splicing. The spliced features And the text embedding feature is input to the attention module 5 to obtain the noisy masked potential feature .

[0076] In summary, the present invention discloses an image segmentation method, model, model training method and image segmentation system. The text information of the target object is input into a text encoder, and the text information is encoded to generate text embedding features, so that the model can understand the image content and adapt to unseen categories to improve the zero-sample segmentation ability. In addition, the image is encoded into image latent features by the image encoder. After encoding, the image latent features and the noisy mask latent features are fused according to the preset number of fusions, and the visual information is extracted and cross-modally aligned with the text embedding features, and the mask features are continuously optimized to enhance the model's ability to capture target boundaries and semantic information. During the initial fusion, the noisy mask latent features are initialized by random noise. After a multi-step iterative diffusion denoising process, clear target mask features are gradually extracted, and finally decoded by the latent space decoder to generate a high-quality predicted target mask, and the original image is accurately segmented based on this.

[0077] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. An image segmentation method, characterized in that: The segmentation method includes: Acquire an image to be segmented containing a target object and corresponding text information; wherein the text information is used to characterize the category of the target object in the image; Encoding the text information through a text encoder of an image segmentation model to generate text embedding features, and encoding the image through an image encoder of the image segmentation model to generate image latent features; According to a preset number of fusions, based on the text embedding features, the image latent features are processed by the attention module and the splicing module of the image segmentation model within the preset number of fusions until the noisy masked latent features generated at the last fusion are obtained, and after denoising, they are decoded by the latent space decoder of the image segmentation model to generate a predicted target mask for segmenting the target object; wherein, at each fusion: The image latent features and the noisy mask latent features are fused through the splicing module of the image segmentation model, and visual features are extracted from the fused features obtained after fusion through the attention module, and the visual features are aligned and fused with the text embedding features to update the noisy mask latent features for the next fusion; wherein the noisy mask latent features for the first fusion are random noise generated by the noise generator of the image segmentation model; Segmenting the target object in the image based on the predicted target mask; Among them, the image encoder extracts image features and maps the image features to a low-dimensional latent space.

2. The image segmentation method according to claim 1, wherein: The latent space decoder generates K predicted target masks, where K is a positive integer greater than 1, and decodes the processed mask latent features to generate a predicted target mask for segmenting the target object, including: Decode the processed mask potential features to obtain K initial predicted target masks; Calculate the mean of the K initial predicted target masks as the final predicted target mask.

3. The image segmentation method according to claim 1, wherein: The image latent features and the noisy mask latent features are fused by the splicing module of the image segmentation model, and visual features are extracted from the fused features obtained by the attention module, and the visual features are aligned and fused with the text embedding features to update the noisy mask latent features in the next fusion, including: fusing the image potential features and the noisy mask potential features through a splicing module of an image segmentation model to obtain a fused feature; Extracting visual features of corresponding scales from the fused features through a diffusion coding unit of the attention module based on downsampling networks of different scales; wherein the diffusion coding unit includes a cascade of multiple downsampling networks of decreasing scales; By using the attention unit of the attention module, cross-modal feature fusion and alignment are performed on the minimum-scale visual features and the text embedding features based on the content attention mechanism to obtain fused and aligned features; Through the diffusion decoding unit of the attention module, the intermediate features and the visual features of the corresponding scale are fused in each upsampling unit to obtain the intermediate features input to the next upsampling unit, and prediction noise is generated based on the intermediate features obtained by the last upsampling unit; wherein the initial intermediate features are the fused alignment features, and the diffusion decoding unit includes a cascade of multiple upsampling units with increasing scales, and the scale of the upsampling unit corresponds to the scale of the downsampling network; The denoising unit of the attention module performs denoising on the noisy masked latent features during the current fusion based on the predicted noise to obtain the denoised masked latent features, which are used as the noisy masked latent features during the next fusion.

4. The image segmentation method according to claim 1, wherein: The image latent features are fused with the noisy mask latent features, including: Superimposing the image latent feature and the noisy mask latent feature to generate a joint feature; The joint feature and the noisy masked latent feature are concatenated to generate a fusion feature.

5. The image segmentation method according to claim 1, wherein: The image potential features and the noisy mask potential features are fused, comprising: splicing the image potential features and the noisy mask potential features to generate fused features.

6. The image segmentation method according to claim 1, wherein: The random noise is Gaussian noise.

7. The image segmentation method according to claim 3, characterized in that: The attention unit of the attention module performs cross-modal feature fusion and alignment on the minimum-scale visual features and the text embedding features based on the content attention mechanism to obtain fused and aligned features, including: Generate a query matrix, a key matrix and a first value matrix based on the minimum-scale visual features, and generate an intermediate matrix and a second value matrix based on the text embedding features; Calculating the self-attention weights of the visual features based on the query matrix and the key matrix according to a scaled dot product attention mechanism; Based on the key matrix and the intermediate matrix, calculating the interactive attention weight of the text embedding feature to the visual feature according to the scaled dot product attention mechanism; According to the self-attention weight and the interactive attention weight, the first value matrix and the second value matrix are weightedly summed to obtain a fused alignment feature.

8. An image segmentation model, characterized in that The image segmentation model includes: an image encoder, a noise generator, a text encoder, a splicing module, an attention module and a latent space decoder, wherein: The output end of the image encoder is connected to the first input end of the splicing module; The output terminal of the noise generator is connected to the second input terminal of the splicing module; The output of the text encoder is connected to the first input of the attention module; The output end of the splicing module is connected to the second input end of the attention module; wherein the attention module is used to extract visual features from the fused features generated by the splicing module, align and fuse the visual features with the text embedding features obtained by encoding the text information by the text encoder, and obtain the denoised masked latent features after denoising; The output of the attention module is connected to the input of the latent space decoder; wherein the latent space decoder is used to decode the latent features of the denoising mask to generate a predicted target mask for segmenting the target object in the image; Among them, the image encoder extracts image features and maps the image features to a low-dimensional latent space.

9. A training method for an image segmentation model, characterized in that: The training method comprises: Acquire an image containing a target object and text information corresponding thereto, as well as a target mask of the target object in the image; wherein the text information is used to characterize the category of the target object in the image; Inputting the text information into the text encoder of the semantic segmentation model for encoding to generate text embedding features, inputting the image into the image encoder of the semantic segmentation model for encoding to generate image latent features, and inputting the target mask into the mask encoder of the semantic segmentation model for encoding to generate mask latent features; According to the preset number of fusions, the attention module of the semantic segmentation model is fine-tuned and trained at each fusion to obtain a trained attention module of the semantic segmentation model; wherein, at each fusion: The masked latent features are denoised and fused with the image latent features by the noise summation module and the splicing module of the semantic segmentation model; visual features are extracted from the fused features obtained by the fusion by the attention module of the semantic segmentation model; the visual features and the text embedding features are aligned and fused based on the attention mechanism to generate noisy masked latent features; and the attention module of the semantic segmentation model is trained by the noisy masked latent features; Among them, the image encoder extracts image features and maps the image features to a low-dimensional latent space.

10. An image segmentation system, characterized in that: The image segmentation system includes: an image encoder, a mask encoder, a noise generator, a text encoder, a noise summation module, a splicing module and an attention module, wherein: The output end of the mask encoder is connected to the first input end of the noise summation module; wherein the mask encoder is used to encode the target mask of the target object in the image into a mask potential feature; The output terminal of the noise generator is connected to the second input terminal of the noise summing module; wherein the noise generator is used to generate random noise; The output end of the image encoder is connected to the first input end of the splicing module; wherein the image encoder is used to encode the image to generate image potential features; The output end of the noise summing module is connected to the second input end of the splicing module; The output end of the text encoder is connected to the first input end of the attention module; wherein the text encoder is used to encode text information to generate text embedding features; The output end of the splicing module is connected to the second input end of the attention module; wherein the attention module is used to extract visual features from the fused features generated by the splicing module, and align and fuse the visual features with the text embedding features through the attention mechanism to generate noisy masked latent features; Among them, the image encoder extracts image features and maps the image features to a low-dimensional latent space.

Images