A semantic segmentation method, a method for training a model, a model, and a system

Abstract

The present invention relates to a semantic segmentation method, model training method, model, and system. The semantic segmentation model includes: an image encoder, a noise generator, a text encoder, a splicing module, an attention module, and a latent space decoder, wherein: the output of the image encoder is connected to the first input of the splicing module; the output of the noise generator is connected to the second input of the splicing module; the output of the text encoder is connected to the first input of the attention module; the output of the splicing module is connected to the second input of the attention module; the output of the attention module is connected to the input of the latent space decoder; wherein the latent space decoder is used to decode the latent features of the denoising mask to generate a predicted target mask for segmenting the target object in the image. The present invention improves the accuracy of image segmentation.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a semantic segmentation method, a model training method, a model and a system. Background Art

[0002] Open-Vocabulary Semantic Segmentation (OVSS) is an important computer vision task that aims to semantically annotate each pixel in an image based on a textual description. Unlike traditional semantic segmentation methods, OVSS requires the ability to recognize unseen categories to adapt to diverse targets in open environments. While multimodal models based on text-image matching perform well for single-target classification, they struggle to accurately understand the relationships between objects in complex scenes, limiting their application to pixel-level segmentation tasks. Furthermore, while recent text-to-image diffusion models have demonstrated strong capabilities in generating high-quality images, most OVSS research uses these diffusion models solely as fixed feature extractors and lacks deep integration between modules. For example, they lack dynamic information interaction between the image encoder, mask modeling module, and attention mechanism. Consequently, they fail to fully utilize the model's rich visual-language understanding capabilities, resulting in performance bottlenecks in pixel-level semantic alignment and unseen category recognition. Therefore, a semantic segmentation method, model training method, model, and system are needed. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a semantic segmentation method, model training method, model and system, which improve the problem of low target recognition accuracy in complex scenarios when performing open vocabulary semantic segmentation tasks in the prior art.

[0004] To achieve the above-mentioned objectives and other related objectives, the present invention provides a semantic segmentation model, which includes: an image encoder, a noise generator, a text encoder, a splicing module, an attention module and a latent space decoder, wherein: the output end of the image encoder is connected to the first input end of the splicing module; the output end of the noise generator is connected to the second input end of the splicing module; the output end of the text encoder is connected to the first input end of the attention module; the output end of the splicing module is connected to the second input end of the attention module; wherein the attention module is used to extract visual features from the fusion features generated by the splicing module, and align and fuse the visual features with the text embedding features obtained by encoding the text information by the text encoder, and obtain denoising mask latent features after denoising; the output end of the attention module is connected to the input end of the latent space decoder; wherein the latent space decoder is used to decode the denoising mask latent features to generate a predicted target mask for segmenting the target object in the image.

[0005] In one embodiment of the present invention, the splicing module includes: a superposition unit, a first input end of which is connected to the output end of the image encoder, and a second input end of which is connected to the output end of the noise generator; wherein the superposition unit is used to weightedly fuse the image potential features and the noisy mask potential features generated by the noise generator to generate a joint feature; a splicing unit, a first input end of which is connected to the output end of the superposition unit, and a second input end of which is connected to the output end of the noise generator; wherein the splicing unit is used to splice the joint feature and the noisy mask potential features to generate a fused feature.

[0006] In one embodiment of the present invention, the splicing module includes: a splicing unit, a first input end of which is connected to the output end of the image encoder, and a second input end of which is connected to the output end of the noise generator; wherein the splicing unit is used to splice the image latent features and the noisy mask latent features generated by the noise generator to generate a fusion feature.

[0007] In one embodiment of the present invention, the attention module includes: a diffusion coding unit, whose input end is connected to the output end of the splicing module, and is used to extract visual features of corresponding scales from the fusion features generated by the splicing module based on downsampling networks of different scales; wherein the diffusion coding unit includes a cascade of multiple downsampling networks with successively decreasing scales; a content attention unit, whose first input end is connected to the output end of the text encoder, and whose second input end is connected to the output end of the diffusion coding unit, and is used to perform cross-modal feature fusion and alignment of the minimum scale visual features and the text embedding features based on the content attention mechanism , obtaining fused alignment features; a diffusion decoding unit, a first input end of which is connected to the output end of the content attention unit, and a second input end of which is connected to the output end of the diffusion coding unit, for fusing the intermediate features with the visual features of the corresponding scale in each upsampling network to obtain intermediate features input to the next upsampling network, and generating an image with prediction noise based on the intermediate features obtained by the last upsampling network; wherein the initial intermediate features are the fused alignment features, and the diffusion decoding unit includes a plurality of cascaded upsampling networks with increasing scales, and the scale of the upsampling network corresponds to the scale of the downsampling network.

[0008] In one embodiment of the present invention, the content attention unit includes: a matrix generation subunit, a first input end of which is connected to the output end of the text encoder, and a second input end of which is connected to the output end of the diffusion coding unit, for generating a query matrix, a key matrix and a first value matrix based on the minimum scale visual feature, and correspondingly generating an intermediate matrix and a second value matrix based on the text embedding feature; a self-attention weight subunit, an input end of which is connected to the first output end of the matrix generation subunit, for calculating the self-attention weight of the visual feature based on the query matrix and the key matrix according to the scaled dot product attention mechanism; an interactive attention weight subunit , its input end is connected to the second output end of the matrix generation subunit, and is used to calculate the interactive attention weight of the text feature to the visual feature based on the key matrix and the intermediate matrix according to the scaled dot product attention mechanism; a feature fusion subunit, whose first input end is connected to the output end of the self-attention weight subunit, its second input end is connected to the output end of the interactive attention weight subunit, and its third input end is connected to the third output end of the matrix generation subunit, and is used to perform weighted summation on the first value matrix and the second value matrix according to the self-attention weight and the interactive attention weight to obtain a fused alignment feature.

[0009] In one embodiment of the present invention, the attention module further includes: a denoising unit, whose input end is connected to the output end of the diffusion decoding unit, for denoising the image containing predicted noise generated each time within a preset denoising number of times, and using the denoised image as the noisy masked latent feature generated by the noise generator for re-extracting visual features.

[0010] In one embodiment of the present invention, the image is a K-channel image, where K is a positive integer greater than 1, and the latent space decoder includes: a decoding unit, whose input end is connected to the output end of the attention module, and is used to obtain K initial predicted target masks based on the image containing prediction noise; a mask generation unit, whose input end is connected to the output end of the decoding unit, and is used to perform weighted averaging on the K initial predicted target masks and use the weighted average as the final predicted target mask.

[0011] In one embodiment of the present invention, a semantic segmentation method is further provided, the semantic segmentation method comprising: obtaining an image to be segmented containing a target object and text information corresponding thereto; wherein the text information is used to represent the category of the target object in the image;

[0012] The text information is encoded by the text encoder of the image segmentation model to generate text embedding features, and the image is encoded by the image encoder of the image segmentation model to generate image latent features; according to a preset diffusion time step, based on the text embedding features, the image latent features are processed by the attention module and the splicing module of the image segmentation model at the preset diffusion time step until the noisy mask latent features generated at the last fusion are obtained, and after denoising, they are decoded by the latent space decoder of the image segmentation model to generate a predicted target mask for segmenting the target object; wherein, at each fusion: the image latent features and the noisy mask latent features are fused by the splicing module of the image segmentation model, and the visual features are extracted from the fused features obtained after fusion by the attention module, and aligned and fused with the text embedding features to update the noisy mask latent features at the next fusion; wherein the noisy mask latent features at the first fusion are random noise generated by the noise generator of the image segmentation model; the target object in the image is segmented based on the predicted target mask.

[0013] In one embodiment of the present invention, a semantic segmentation system is further provided, comprising: an image encoder, a mask encoder, a noise generator, a text encoder, a noise summation module, a splicing module, and an attention module, wherein: an output end of the mask encoder is connected to a first input end of the noise summation module; wherein the mask encoder is used to encode a target mask of a target object in the image into a masked latent feature; an output end of the noise generator is connected to a second input end of the noise summation module; wherein the noise generator is used to generate random noise; and an output end of the image encoder is connected to a first input end of the splicing module; Wherein, the image encoder is used to encode the image to generate image latent features; the output end of the noise addition module is connected to the second input end of the splicing module; the output end of the text encoder is connected to the first input end of the attention module; wherein, the text encoder is used to encode text information to generate text embedding features; the output end of the splicing module is connected to the second input end of the attention module; wherein, the attention module is used to extract visual features from the fused features generated by the splicing module, and align and fuse the visual features and the text embedding features through the attention mechanism to generate noisy masked latent features.

[0014] In one embodiment of the present invention, a training method for a semantic segmentation model is also provided, the training method comprising: obtaining an image containing a target object and text information corresponding thereto, and a target mask of the target object in the image; wherein the text information is used to characterize the category of the target object in the image; inputting the text information into a text encoder of a semantic segmentation model for encoding to generate text embedding features, inputting the image into an image encoder of the semantic segmentation model for encoding to generate image latent features, and inputting the target mask into a mask encoder of the semantic segmentation model for encoding to generate mask latent features; and performing the multi-step training according to a preset diffusion time step. , fine-tuning the attention module of the semantic segmentation model at each fusion to obtain a trained attention module of the semantic segmentation model; wherein, at each fusion: the mask latent features are noised and fused with the image latent features through the noise addition module and the splicing module of the semantic segmentation model, visual features are extracted from the fused features obtained after fusion through the attention module of the semantic segmentation model, the visual features and the text embedding features are aligned and fused based on the attention mechanism to generate noisy mask latent features, and the attention module of the semantic segmentation model is trained through the noisy mask latent features.

[0015] As described above, a semantic segmentation method, model training method, model and system of the present invention have the following beneficial effects: the image latent features generated by the image encoder and the noisy mask latent features generated by the noise generator are fused through the splicing module to generate fused features. The attention module takes the fused features as input, and combines them with the text embedding features generated by the text encoder to achieve semantic alignment of cross-modal information between image and text through the content attention mechanism, thereby helping to improve the semantic segmentation model's ability to detect and recognize target semantic positions. The denoised mask latent features finally obtained are decoded by the latent space decoder, thereby restoring the predicted target mask consistent with the target object structure to improve segmentation accuracy. The semantic segmentation model can achieve accurate and stable image segmentation in open categories and zero-sample scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the overall structure of the semantic segmentation model provided by an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of the overall structure of a splicing module provided in one embodiment of the present invention;

[0018] Figure 3 A schematic diagram of the overall structure of a splicing module provided by another embodiment of the present invention;

[0019] Figure 4A schematic diagram of the overall structure of the content attention module provided by an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of the overall structure of the content attention unit provided by an embodiment of the present invention;

[0021] Figure 6 A schematic diagram of the overall structure of a latent space decoder provided by an embodiment of the present invention;

[0022] Figure 7 A schematic diagram of the overall architecture of the semantic segmentation model provided by an embodiment of the present invention;

[0023] Figure 8 A flowchart of a semantic segmentation method provided by an embodiment of the present invention;

[0024] Figure 9 A schematic diagram of the overall architecture of an image segmentation system provided by an embodiment of the present invention;

[0025] Figure 10 A flowchart of a method for training a semantic segmentation model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0027] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0028] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0029] Open vocabulary semantic segmentation (OVSS) is a challenging computer vision task whose goal is to label every pixel in an image based on a textual description. Traditional semantic segmentation methods rely on predefined categories, which makes it difficult to recognize unseen categories during inference, limiting their applicability in open settings. In contrast, humans possess powerful visual generalization abilities, capable of recognizing and understanding a large number of unseen categories. To emulate this ability, much research has been devoted to exploring OVSS in recent years to achieve pixel-level segmentation of a wide range of semantic categories, including those not present in the training data.

[0030] Many open-vocabulary semantic segmentation approaches leverage the strong generalization capabilities of text-image discriminative models trained on large internet datasets. While these pre-trained models perform well at classifying individual objects or pixels, they may not be fully suited for scene-level analysis. Recent research has shown that models like CLIP often misjudge the spatial relationships between objects. Therefore, the shortcomings of text-image discriminative models in spatial and relational understanding have become a barrier to open-vocabulary semantic segmentation. In recent years, diffusion models trained on internet-scale datasets for text-to-image generation have revolutionized the field of image synthesis. These models offer superior image quality, adaptability, composability, and semantic guidance via text input. Notably, diffusion models facilitate image generation by creating cross-attention between text embeddings and their internal visual representations. This suggests that diffusion model representations are potentially unique and connected to higher-level semantic concepts expressible in language. While there have been some attempts to use diffusion models for open-vocabulary semantic segmentation, these efforts typically freeze-stabilize the diffusion model and use it solely as a means of feature extraction. They did not fine-tune and train the U-Net network in the model, which resulted in failure to fully utilize the rich vision-language capabilities of stable diffusion.

[0031] In addition, the current progress in the field of OVSS mainly relies on the improvement of model capacity, but existing methods still have limitations when processing unfamiliar images or new categories, and their visual language understanding is usually limited by training data. This has prompted researchers to explore more powerful prior knowledge to improve the zero-shot generalization ability of the model. In recent years, text-to-image diffusion models have shown strong capabilities in generating images with diverse open vocabulary descriptions. Therefore, a question worthy of attention is whether these diffusion models can provide more comprehensive visual-linguistic prior knowledge for OVSS tasks, thereby enhancing the model's generalization ability on unseen categories. However, there is still a lack of a complete framework that can simultaneously parse object instances and scene semantics and fully utilize the capabilities of diffusion models to improve the performance of OVSS in open environments.

[0032] In response to the above situation, the present invention provides a semantic segmentation model, which fuses the image latent features generated by the image encoder and the noisy mask latent features generated by the noise generator through a splicing module to generate fused features. The attention module takes the fused features as input, and combines them with the text embedding features generated by the text encoder to achieve semantic alignment of cross-modal information between image and text through the content attention mechanism, thereby helping to improve the semantic segmentation model's ability to detect and recognize the target semantic position. The final denoised mask latent features are decoded by a latent space decoder, thereby restoring a predicted target mask consistent with the target object structure to improve segmentation accuracy. This semantic segmentation model can achieve accurate and stable image segmentation in open categories and zero-sample scenarios.

[0033] The semantic segmentation model is used in the inference phase of the model. Based on the diffusion model, it gradually optimizes the masked latent features through diffusion modeling to achieve semantically guided image segmentation. This semantic segmentation model is used to input an image to be segmented into the semantic segmentation model in the inference phase, segment the image, and combine the image's text information to perform semantic understanding and processing on the image, thereby accurately segmenting the target object from the image.

[0034] like Figure 1 As shown in Figure 1, the semantic segmentation model includes an image encoder, a noise generator, a text encoder, a splicing module, an attention module, and a latent space decoder. Each module is connected in an orderly manner according to its function, forming an image segmentation framework based on a diffusion mechanism and guided by text semantics. The following is a detailed introduction to each module:

[0035] Specifically, the text information is input into a text encoder. The encoded text embedding features are then fed into the attention module via its output and the first input of the attention module. The text information is used to characterize the category of the target object in the image, thereby supporting the subsequent alignment of visual and semantic information. For example, the text information may be "a photo of {*}, a photo of more {*}, ...," where "{*}" is a coarse category descriptor (e.g., "sky," "tree," "building," etc.).

[0036] An image containing the target object is input into an image encoder, which encodes the image, extracts image features, and maps them into a low-dimensional latent space. This generates latent features with rich semantic information, reducing the complexity of subsequent computations. The output of the image encoder is connected to the first input of the splicing module, sending the image latent features to the splicing module for fusion with mask-related features in the subsequent diffusion modeling process.

[0037] The output of the noise generator is connected to the second input of the splicing module to provide mask-related features at each diffusion time step as input to the attention module. Specifically, in the first diffusion time step, the noise generator generates and outputs random noise as the initial noisy masked latent features; and in subsequent diffusion time steps, the noisy masked latent features after denoising by the attention module in the previous diffusion time step are received, and the noisy masked latent features of the current time step are obtained according to the preset diffusion scheduling rules as input to the splicing module. Optionally, the random noise is multi-resolution noise to adapt to the multi-scale structure of the image potential features and improve the image segmentation model's ability to process different semantic information.

[0038] The splicing module fuses the image latent features with the corresponding mask-related features according to the different diffusion time steps. Specifically, at the first diffusion time step, the splicing module fuses the image latent features generated by the image encoder with the random noise output by the noise generator to obtain the fused features of the first diffusion time step; at non-first diffusion time steps, the splicing module fuses the image latent features with the noisy mask latent features of the current diffusion time step to generate the fused features of the corresponding diffusion time step. After obtaining the fused features, the splicing module sends the fused features of the current diffusion time step to the attention module through the second input terminal of the attention module, so that the attention module further extracts semantic features to achieve cross-modal alignment with the text features.

[0039] like Figure 2 As shown, in one embodiment of the present invention, the splicing module includes: an overlay unit and a splicing unit, which work together to achieve feature fusion processing between the image latent features and the noisy mask latent features. The overlay unit has two inputs and one output. The first input of the overlay unit is connected to the output of the image encoder to receive the image latent features obtained after the image encoder encodes the input image. The second input of the overlay unit is connected to the output of the noise generator. In the first diffusion time step, it is used to receive the random noise generated by the noise generator. In subsequent diffusion time steps, it receives the noisy mask latent features generated by the noise generator based on the denoising results of the previous time step and the diffusion scheduling rule. The overlay unit performs a weighted fusion of the image latent features with the random noise or noisy mask latent features to generate a joint feature. The joint feature contains both the contextual features of the image and the gradual change of the mask from noise to the target shape during the diffusion process. The weighted fusion method includes but is not limited to element-by-element weighted fusion or linear overlay processing.

[0040] Continue as Figure 2As shown, the splicing module also includes a splicing unit. The first input of the splicing unit is connected to the output of the superposition unit for receiving the joint feature, and the second input of the splicing unit is connected to the output of the noise generator for synchronously receiving the noisy masked latent feature generated by the current diffusion time step. The splicing unit uses channel-dimensional splicing to splice the joint feature with the mask feature of the current diffusion time step to generate a fused feature with rich semantic information.

[0041] Furthermore, if Figure 3 As shown, the splicing module may also only include a splicing unit, which is used to realize the fusion of the image latent features and the noisy mask latent features. The splicing unit has two input ends, the first input end is connected to the output end of the image encoder, and is used to receive the image latent features obtained after the image encoder encodes the original image. The second input end is connected to the output end of the noise generator, and is used to receive the random noise generated by the noise generator in the first diffusion time step or the noisy mask latent features iteratively obtained during the diffusion process. Specifically, the splicing unit splices the image latent features and the noisy mask latent features according to the channel dimension to generate a fusion feature with richer semantic information. The fusion feature not only contains the visual semantic information of the image, but also integrates the noise representation of the mask, which helps the subsequent attention module to perform more effective semantic alignment and feature extraction, thereby improving the accuracy and semantic consistency of mask reconstruction.

[0042] Through the above process, the fused features output by the splicing module are input into the attention module to participate in the cross-modal alignment and feature optimization process, thereby ensuring that the model can gradually improve the accuracy and boundary integrity of the target mask in each diffusion time step, thereby improving the mask reconstruction performance of the diffusion model under semantic guidance.

[0043] The attention module, a key component of the diffusion model, is responsible for feature extraction and semantic guidance of the input fused features within each diffusion time step. Specifically, the attention module, based on the attention mechanism, extracts multi-scale visual features from the fused features generated by the splicing module and aggregates the visual features at each scale to generate a fused, aligned feature for the current diffusion time step. The resulting fused, aligned feature incorporates both the spatial semantics of the image and the category information of the target object provided by the text embedding features, thereby integrating image and text information. During the aggregation process, the text embedding features serve as semantic guidance, guiding the attention module to focus on regions of the visual features that are highly relevant to the text description, thereby achieving cross-modal feature alignment and fusion of text and image. After aggregation, the attention module generates prediction noise for that diffusion time step based on the fused, aligned features. The prediction noise is then inversely denoised based on the diffusion mechanism to obtain a denoised masked latent feature for the current diffusion time step. This denoised masked latent feature is then passed as input to the splicing module for fusion with the image latent feature at the next diffusion time step, completing the next round of diffusion denoising. The above iterative denoising process is repeated until all diffusion time steps are completed. The denoising mask potential features obtained in the last diffusion time step are input into the latent space decoder to generate the predicted mask of the target object to achieve the segmentation of the target object in the image.

[0044] like Figure 4As shown, in one embodiment of the present invention, the attention module includes: a diffusion coding unit, a content attention unit, and a diffusion decoding unit. These modules work together to achieve text-guided image mask prediction and target object segmentation. The input of the diffusion coding unit is connected to the output of the splicing module to receive fused features formed by fusing image latent features with noisy mask latent features. The diffusion coding unit includes a cascade of multiple downsampling networks with successively decreasing scales. Through layer-by-layer downsampling, it extracts multi-scale semantic information of the image from the fused features, thereby obtaining a multi-level visual feature representation from low-level local details to high-level global semantics, providing semantic support for the subsequent attention mechanism and decoding process. The content attention unit is used to achieve cross-modal semantic alignment. Its first input is connected to the output of the text encoder to receive text embedding features, and its second input is connected to the output of the diffusion coding unit to receive multi-scale visual features generated by downsampling the fused features. The content attention unit utilizes a content attention mechanism to weight the text embedding features based on the correspondence between the minimum-scale visual features and the text embedding features. This weighted attention weight guides the text embedding features, generating fused aligned features that combine text semantic information with visual information, improving the semantic segmentation model's ability to identify objects in the image. The diffusion decoding unit has two inputs: the first input is connected to the output of the content attention unit and is used as the initial intermediate features. The second input is connected to the output of the diffusion coding unit and is used to synchronously acquire visual features at the corresponding scale during downsampling by the diffusion coding unit. During the decoding process, the diffusion coding unit progressively restores the feature map to its pre-downsampling size through multiple upsampling networks of increasing scale. During each upsampling process, the intermediate features corresponding to the upsampling network are fused with the visual features at the corresponding scale, and the fused features are used as the intermediate features for the next upsampling network. This upsampling process is repeated until the intermediate features output by the final upsampling network are obtained. This intermediate feature is then used as an image containing prediction noise, which implicitly contains structural information about the object mask, allowing it to be restored to the final segmentation mask after subsequent decoding. It should be noted that the scale of the upsampling network in the content attention module is consistent with the corresponding downsampling network, so that the features can be effectively aligned between encoding and decoding to ensure the coherence of semantic information transmission and improve the accuracy and robustness of subsequent target mask generation.

[0045] Furthermore, the content attention unit is used to achieve deep alignment between image visual features and text semantic features, such as Figure 5As shown, the content attention unit includes a matrix generation subunit, a self-attention weight subunit, an interactive attention weight subunit, and a feature fusion subunit. The subunits work together to complete the cross-modal feature fusion process to improve the accuracy of subsequent target mask generation. Specifically, the matrix generation subunit has two input terminals, the first input terminal of which is connected to the output terminal of the text encoder for receiving the text embedding features generated by the text encoder, and the second input terminal is connected to the output terminal of the diffusion coding unit for receiving multi-scale visual features. The matrix generation subunit generates the query matrix Q, the key matrix K, and the first value matrix based on the minimum scale visual features. , where the query matrix Q represents the most representative part of the visual features of the minimum scale, the key matrix K is used to provide a reference for calculating the attention distribution of the query matrix Q, and the first value matrix Used for weighted aggregation during attention calculation. The text embedding features are generated by linear projection into the intermediate matrix I and the second value matrix , where the middle matrix I is used to match visual features to associate text with visual features, and the second value matrix Text embedding features can be stored to provide text information for subsequent fusion processing.

[0046] The self-attention weight subunit is used to calculate the attention relationship within the visual feature, and its input is connected to the first output of the matrix generation subunit. Thus, according to the scaled dot product attention mechanism, the self-attention weight of the visual feature is calculated using the generated query matrix Q and key matrix K. , where d is the feature dimension. The self-attention weight can help the model determine which areas in the image are more important, thereby better understanding the internal structure and content of the image and achieving subsequent image segmentation.

[0047] The input end of the interaction attention weight subunit is connected to the second output end of the matrix generation subunit, which calculates the interaction attention weight using the key matrix K and the intermediate matrix I according to the scaled dot product attention mechanism. .

[0048] The feature fusion subunit is connected to the third output terminal of the self-attention weight subunit, the interactive attention weight subunit and the matrix generation subunit respectively to receive 、 、 and The first value matrix of the feature fusion subunit and the second value matrix Perform double attention weighted fusion to obtain fused alignment features This query-key-value mechanism allows visual features to better align with text semantics, thereby improving the accuracy of subsequent semantic segmentation. This process narrows the gap between text embeddings and visual features, fully leveraging the advantages of both text and image modalities to improve image segmentation accuracy.

[0049] In addition, the attention module also includes a denoising unit (not shown) for iteratively optimizing the mask latent features over multiple diffusion time steps. The input of this denoising unit is connected to the output of the diffusion decoding unit to receive the image containing prediction noise generated by the diffusion decoding unit at each diffusion time step. The denoising unit performs inverse denoising on each image containing prediction noise within a preset number of denoising steps (e.g., T steps). Specifically, the denoising unit utilizes the inverse process of the diffusion model to gradually restore the image containing prediction noise into a denoised mask latent feature that more closely resembles the actual mask structure. The denoised data is used as a new image containing prediction noise for subsequent decoding.

[0050] Furthermore, the image is a K-channel image, where K is a positive integer greater than 1. In order to achieve accurate semantic segmentation of the multi-channel image, as shown in FIG. Figure 6 As shown, the latent space decoder in this application includes a decoding unit and a mask generation unit, which work together to decode the denoised image containing prediction noise into a predicted target mask. The input end of the decoding unit is connected to the output end of the attention module to receive the denoised image that integrates visual and textual semantics (i.e., the image containing prediction noise denoised by the denoising unit). The decoding unit restores the input image to the resolution of the original input image through upsampling and feature reconstruction, obtaining K initial predicted target masks.

[0051] The input of the mask generation unit is connected to the output of the decoding unit to receive these K initial prediction masks. Since the original image is copied into K channels during encoding, after decoding, it is necessary to average the K initial prediction target masks to obtain the final prediction target mask. For example, K is 3, which means that we can simulate an RGB image and copy the original image into three channels. After decoding, the three initial prediction target masks are averaged to obtain the final prediction target mask.

[0052] like Figure 7 The output of the image encoder is connected to the second input of the splicing module to convert the image potential features into The output of the noise generator is connected to the second input of the splicing module to generate the noise. The two input ends of the attention module are connected to the splicing module and the text encoder respectively to receive the text embedding features and the spliced fusion features. , and after content attention processing, the noisy masked latent features are obtained , after denoising, we get the noisy masked potential features of the next diffusion time step t-1 , repeat the above process until the noisy mask potential feature of the last diffusion time step is obtained, and after denoising, the predicted target mask is obtained through the latent space decoder .

[0053] like Figure 8 As shown, the present invention also provides a semantic segmentation method, the method comprising:

[0054] S81. Acquire an image to be segmented containing a target object and corresponding text information; wherein the text information is used to represent the category of the target object in the image.

[0055] S82. Encode the text information through a text encoder of the image segmentation model to generate text embedding features, and encode the image through an image encoder of the image segmentation model to generate image latent features.

[0056] S83. According to a preset diffusion time step, based on the text embedding feature, the image latent feature is processed at a preset diffusion time step by the attention module and the stitching module of the image segmentation model until the noisy mask latent feature generated at the last fusion is obtained, and after denoising, it is decoded by the latent space decoder of the image segmentation model to generate a predicted target mask for segmenting the target object; wherein, at each fusion: the image latent feature and the noisy mask latent feature are fused by the stitching module of the image segmentation model, and the visual feature is extracted from the fused feature obtained after the fusion by the attention module, and it is aligned and fused with the text embedding feature to update the noisy mask latent feature at the next fusion; wherein, the noisy mask latent feature at the first fusion is the random noise generated by the noise generator of the image segmentation model.

[0057] S84: Segment the target object in the image based on the predicted target mask.

[0058] The textual information of the target object is input into a text encoder, which encodes the textual information to generate text embedding features, enabling the model to understand the image content and adapt to unseen categories, thereby improving zero-shot segmentation capabilities. Furthermore, the image is encoded into image latent features via an image encoder. After encoding, the image latent features are fused with the noisy mask latent features according to a preset diffusion time step, and visual information is extracted. This is then cross-modally aligned with the text embedding features, and the mask features are continuously optimized to enhance the model's ability to capture target boundaries and semantic information. During the initial fusion, the noisy mask latent features are initialized with random noise generated by the noise generator. After a multi-step iterative diffusion denoising process, target mask features with clear and accurate boundaries are gradually extracted. These features are decoded by the latent space decoder to generate a predicted target mask, which is then used to accurately segment the target object in the original image.

[0059] The present application also provides a semantic segmentation system, which includes: an image encoder, a mask encoder, a noise generator, a text encoder, a noise addition module, a splicing module and an attention module, wherein: the output end of the mask encoder is connected to the first input end of the noise addition module; wherein the compression encoder is used to encode the target mask of the target object in the image into a mask potential feature; the output end of the noise generator is connected to the second input end of the noise addition module; wherein the noise generator is used to generate random noise; the output end of the image encoder is connected to the first input end of the splicing module; wherein the image encoder is used to encode the image to generate image potential features; the output end of the noise addition module is connected to the second input end of the splicing module; the output end of the text encoder is connected to the first input end of the attention module; wherein the text encoder is used to encode text information to generate text embedding features; the output end of the splicing module is connected to the second input end of the attention module; wherein the attention module is used to extract visual features from the fused features generated by the splicing module, and align and fuse the visual features and the text embedding features through the attention mechanism to generate noisy mask potential features.

[0060] like Figure 9 As shown, the output of the mask encoder and the output of the noise generator are connected to the first input and the second input of the noise summation module respectively. The mask encoder encodes the real target mask into the mask potential feature , the image encoder encodes the image x into image latent features , the noise summation module will mask the latent features and random noise Fusion is performed. At the tth fusion (i.e., the tth diffusion time step), the splicing module combines the fused features and image latent features Splicing. The spliced features And the text embedding feature is input into the attention module to obtain the noisy masked potential feature .

[0061] like Figure 10 As shown, the present application also provides a training method for a semantic segmentation model, the training method comprising:

[0062] S101, acquiring an image containing a target object and corresponding text information, as well as a target mask of the target object in the image; wherein the text information is used to characterize the category of the target object in the image;

[0063] S102, inputting the text information into the text encoder of the semantic segmentation model for encoding to generate text embedding features, inputting the image into the image encoder of the semantic segmentation model for encoding to generate image latent features, and inputting the target mask into the mask encoder of the semantic segmentation model for encoding to generate mask latent features;

[0064] S103. According to the preset diffusion time step, the attention module of the semantic segmentation model is fine-tuned and trained at each fusion to obtain the trained attention module of the semantic segmentation model; wherein, at each fusion: the mask latent features are noised and fused with the image latent features through the noise addition module and the splicing module of the semantic segmentation model, the visual features are extracted from the fused features obtained after the fusion through the attention module of the semantic segmentation model, the visual features and the text embedding features are aligned and fused based on the attention mechanism to generate noisy masked latent features, and the attention module of the semantic segmentation model is trained through the noisy masked latent features.

[0065] like Figure 9 and Figure 10 As shown in the figure, during the training phase, an image containing a target object is obtained, and the text information and target mask corresponding to the image are obtained. The text information is used to describe the category of the target object in the image, which is used to provide a basis for subsequent semantic guidance. The target mask is used to mark the actual position of the target object in the image as a label for model training. The text information is input into the text encoder of the semantic segmentation model to extract its semantic features and obtain text embedding features; the image x is input into the image encoder to extract image features and map the image features to the latent space to generate image latent features. Similarly, the target mask is fed into the mask encoder to generate the mask latent features The semantic segmentation model is iteratively trained according to the preset diffusion time step T. In each diffusion time step, the noise summation module is used to mask the potential features. Add noise to get the feature corresponding to the diffusion time step t . and image latent features Input them together into the splicing module for fusion processing to obtain fusion features The fusion features Input to the attention module, extract fusion features through the attention unit The multi-scale visual features are then cross-modally aligned with the text embedding features based on the content attention mechanism to generate an intermediate feature with semantic guidance capability. This intermediate feature is used as the noisy masked latent feature at the current time step. , which is used to supervise the training of the parameters of the attention module at this diffusion time step. Among them, the image encoder and the mask encoder can be variational autoencoders. It should be noted that the text encoder, image encoder, mask encoder and latent space decoder of the present invention do not change the parameters during the above training process. In addition, in order to simulate the RGB image, the target mask is copied to the three channels. This step is crucial because the encoder is originally configured to process 3-channel (RGB) input, while the target mask has only a single channel. The present invention can reconstruct the target mask from the latent code without modifying the latent space or VAE. During the inference process, the predicted target mask is obtained by averaging the values on the three channels after decoding the latent code of the mask at the end of the inverse process. In addition, during the training phase of the present invention, the random noise generated by the noise generator in the first diffusion time step is multi-resolution noise.

[0066] For the specific definition of the training system of the semantic segmentation model, please refer to the definition of the training method of the semantic segmentation model above, which will not be repeated here. Each module in the above-mentioned training system of the semantic segmentation model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware format, or can be stored in the memory of the computer device in software format, so that the processor can call the operations corresponding to the above modules.

[0067] It should be noted that, in order to highlight the innovative part of the present invention, this embodiment does not introduce modules that are not closely related to solving the technical problems raised by the present invention, but this does not mean that there are no other modules in this embodiment.

[0068] In summary, the present invention discloses a semantic segmentation method, a model training method, a model and a system, which fuse the image latent features generated by the image encoder and the noisy mask latent features generated by the noise generator through a splicing module to generate fused features. The attention module takes the fused features as input, and combines them with the text embedding features generated by the text encoder to achieve semantic alignment of cross-modal information between the image and the text through the content attention mechanism, thereby helping to improve the semantic segmentation model's ability to detect and recognize the target semantic position. The denoised mask latent features finally obtained are decoded by the latent space decoder, so that the predicted target mask consistent with the target object structure can be restored to improve the segmentation accuracy. The semantic segmentation model can achieve accurate and stable image segmentation in open categories and zero-sample scenarios. Therefore, the present invention effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0069] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A semantic segmentation model, characterized in that The semantic segmentation model includes: an image encoder, a noise generator, a text encoder, a splicing module, an attention module and a latent space decoder, wherein: The output end of the image encoder is connected to the first input end of the splicing module; The output terminal of the noise generator is connected to the second input terminal of the splicing module; The output of the text encoder is connected to the first input of the attention module; The output end of the splicing module is connected to the second input end of the attention module; wherein the attention module is used to extract visual features from the fused features generated by the splicing module, align and fuse the visual features with the text embedding features obtained by encoding the text information by the text encoder, and obtain the denoised masked latent features after denoising; The output of the attention module is connected to the input of the latent space decoder; wherein the latent space decoder is used to decode the latent features of the denoising mask to generate a predicted target mask for segmenting the target object in the image; The image encoder is used to extract image features and map them to a low-dimensional latent space to generate image latent features; At the first diffusion time step, the splicing module fuses the image potential features with the random noise output by the noise generator to obtain the fused features of the first diffusion time step; at the non-first diffusion time step, the splicing module fuses the image potential features with the noisy mask potential features of the current diffusion time step to generate the fused features of the corresponding diffusion time step; the noisy mask potential features are generated by the noise generator.

2. The semantic segmentation model according to claim 1, characterized in that The splicing module includes: a superposition unit, wherein a first input end thereof is connected to the output end of the image encoder, and a second input end thereof is connected to the output end of the noise generator; wherein the superposition unit is configured to perform weighted fusion of the image latent features and the noisy mask latent features generated by the noise generator to generate a joint feature; A splicing unit, whose first input end is connected to the output end of the superposition unit, and whose second input end is connected to the output end of the noise generator; wherein the splicing unit is used to splice the joint feature and the noisy masked latent feature to generate a fusion feature.

3. The semantic segmentation model according to claim 1, characterized in that The splicing module includes: A splicing unit, wherein a first input end is connected to the output end of the image encoder, and a second input end is connected to the output end of the noise generator; wherein the splicing unit is used to splice the image potential features and the noisy mask potential features generated by the noise generator to generate a fusion feature.

4. The semantic segmentation model according to claim 1, characterized in that The attention module includes: a diffusion coding unit, whose input end is connected to the output end of the splicing module, and is used to extract visual features of corresponding scales from the fused features generated by the splicing module based on downsampling networks of different scales; wherein the diffusion coding unit includes a cascade of multiple downsampling networks of decreasing scales; a content attention unit, having a first input connected to the output of the text encoder and a second input connected to the output of the diffusion coding unit, for performing cross-modal feature fusion and alignment on the minimum-scale visual features and the text embedding features based on a content attention mechanism to obtain a fused and aligned feature; A diffusion decoding unit, a first input end of which is connected to the output end of the content attention unit, and a second input end of which is connected to the output end of the diffusion coding unit, is used to fuse the intermediate features and the visual features of the corresponding scale in each upsampling network to obtain the intermediate features input to the next upsampling network, and generate an image with prediction noise based on the intermediate features obtained by the last upsampling network; wherein the initial intermediate features are the fused alignment features, and the diffusion decoding unit includes a plurality of cascaded upsampling networks with increasing scales, and the scale of the upsampling network corresponds to the scale of the downsampling network.

5. The semantic segmentation model according to claim 4, characterized in that The content attention unit includes: a matrix generation subunit, a first input end of which is connected to the output end of the text encoder, a second input end of which is connected to the output end of the diffusion coding unit, and is used to generate a query matrix, a key matrix, and a first value matrix based on the minimum-scale visual features, and to generate an intermediate matrix and a second value matrix based on the text embedding features; a self-attention weight subunit, an input end of which is connected to the first output end of the matrix generation subunit, and is used to calculate the self-attention weight of the visual feature based on the query matrix and the key matrix according to the scaled dot product attention mechanism; an interaction attention weight subunit, whose input end is connected to the second output end of the matrix generation subunit, and is used to calculate the interaction attention weight of the text embedding feature to the visual feature based on the key matrix and the intermediate matrix according to the scaled dot product attention mechanism; A feature fusion subunit, whose first input end is connected to the output end of the self-attention weight subunit, whose second input end is connected to the output end of the interaction attention weight subunit, and whose third input end is connected to the third output end of the matrix generation subunit, is used to perform weighted summation on the first value matrix and the second value matrix according to the self-attention weight and the interaction attention weight to obtain a fused alignment feature.

6. The semantic segmentation model according to claim 4, characterized in that The attention module also includes: A denoising unit, whose input end is connected to the output end of the diffusion decoding unit, is used to denoise the image containing predicted noise generated each time within a preset denoising number of times, and use the denoised image as the noisy masked latent feature generated by the noise generator to be used for re-extracting visual features.

7. The semantic segmentation model according to claim 1, characterized in that The image is a K-channel image, where K is a positive integer greater than 1, and the latent space decoder includes: A decoding unit, whose input is connected to the output of the attention module, and is used to obtain K initial predicted target masks based on the image containing prediction noise; The mask generation unit has an input end connected to the output end of the decoding unit and is used to perform weighted averaging on the K initial prediction target masks and use the weighted average as the final prediction target mask.

8. A semantic segmentation method, characterized in that: The semantic segmentation method comprises: Acquire an image to be segmented containing a target object and corresponding text information; wherein the text information is used to characterize the category of the target object in the image; Encoding the text information through a text encoder of an image segmentation model to generate text embedding features, and encoding the image through an image encoder of the image segmentation model to generate image latent features; According to a preset diffusion time step, based on the text embedding features, the image latent features are processed by the attention module and the splicing module of the image segmentation model at the preset diffusion time step until the noisy mask latent features generated at the last fusion are obtained, and after denoising, the latent features are decoded by the latent space decoder of the image segmentation model to generate a predicted target mask for segmenting the target object; wherein, at each fusion: The image latent features and the noisy mask latent features are fused through the splicing module of the image segmentation model, and visual features are extracted from the fused features obtained after fusion through the attention module, and the visual features are aligned and fused with the text embedding features to update the noisy mask latent features for the next fusion; wherein the noisy mask latent features for the first fusion are random noise generated by the noise generator of the image segmentation model; Segmenting the target object in the image based on the predicted target mask; The image encoder is used to extract image features and map them to a low-dimensional latent space to generate image latent features; At the first diffusion time step, the splicing module fuses the image potential features with the random noise output by the noise generator to obtain the fused features of the first diffusion time step; at the non-first diffusion time step, the splicing module fuses the image potential features with the noisy mask potential features of the current diffusion time step to generate the fused features of the corresponding diffusion time step; the noisy mask potential features are generated by the noise generator.

9. A semantic segmentation system, characterized in that The semantic segmentation system includes: an image encoder, a mask encoder, a noise generator, a text encoder, a noise summation module, a splicing module and an attention module, wherein: The output end of the mask encoder is connected to the first input end of the noise summation module; wherein the mask encoder is used to encode the target mask of the target object in the image into a mask potential feature; The output terminal of the noise generator is connected to the second input terminal of the noise summing module; wherein the noise generator is used to generate random noise; The output end of the image encoder is connected to the first input end of the splicing module; wherein the image encoder is used to encode the image to generate image potential features; The output end of the noise summing module is connected to the second input end of the splicing module; The output end of the text encoder is connected to the first input end of the attention module; wherein the text encoder is used to encode text information to generate text embedding features; The output end of the splicing module is connected to the second input end of the attention module; wherein the attention module is used to extract visual features from the fused features generated by the splicing module, and align and fuse the visual features with the text embedding features through the attention mechanism to generate noisy masked latent features; The image encoder is used to extract image features and map them to a low-dimensional latent space to generate image latent features; At the first diffusion time step, the splicing module fuses the image potential features with the random noise output by the noise generator to obtain the fused features of the first diffusion time step; at the non-first diffusion time step, the splicing module fuses the image potential features with the noisy mask potential features of the current diffusion time step to generate the fused features of the corresponding diffusion time step; the noisy mask potential features are generated by the noise generator.

10. A method for training a semantic segmentation model, characterized in that: The training method comprises: Acquire an image containing a target object and text information corresponding thereto, as well as a target mask of the target object in the image; wherein the text information is used to characterize the category of the target object in the image; Inputting the text information into the text encoder of the semantic segmentation model for encoding to generate text embedding features, inputting the image into the image encoder of the semantic segmentation model for encoding to generate image latent features, and inputting the target mask into the mask encoder of the semantic segmentation model for encoding to generate mask latent features; According to the preset diffusion time step, the attention module of the semantic segmentation model is fine-tuned and trained at each fusion to obtain a trained attention module of the semantic segmentation model; wherein, at each fusion: The masked latent features are denoised and fused with the image latent features by the noise summation module and the splicing module of the semantic segmentation model; visual features are extracted from the fused features obtained by the fusion by the attention module of the semantic segmentation model; the visual features and the text embedding features are aligned and fused based on the attention mechanism to generate noisy masked latent features; and the attention module of the semantic segmentation model is trained by the noisy masked latent features; The image encoder is used to extract image features and map them to a low-dimensional latent space to generate image latent features; At the first diffusion time step, the splicing module fuses the image potential features with the random noise output by the noise generator to obtain the fused features of the first diffusion time step; at the non-first diffusion time step, the splicing module fuses the image potential features with the noisy mask potential features of the current diffusion time step to generate the fused features of the corresponding diffusion time step; the noisy mask potential features are generated by the noise generator.

Images