Zero-shot semantic segmentation method for text image pre-training model based on regional clues

Abstract

The application provides a zero-shot semantic segmentation method for improving a text image pre-training model based on a region clue, and comprises the following steps: constructing input data for a preset text image pre-training model; performing deep feature extraction on an image in the input data through a preset region level bridge to distinguish the category features of each region; adding an additional constraint condition to a preset semantic segmentation decoder through a preset recovery decoder with a recovery loss based on the category features of each region to reduce overfitting; and performing zero-shot semantic segmentation based on the image features and text features after the overfitting is reduced. The application solves the problems that the existing model has poor generalization and the pixel-level classification cannot meet the actual needs in zero-shot semantic segmentation.

Description

Technical Field

[0001] This invention relates to the field of semantic segmentation technology, and in particular to a zero-shot semantic segmentation method based on region cues to improve a pre-trained text image model. Background Technology

[0002] With the rapid development of deep learning technology, deep neural networks have played a central role in image understanding and analysis, particularly in the research area of ​​zero-shot semantic segmentation. Zero-shot semantic segmentation aims to enable models to recognize and segment categories they have never seen during training, which places extremely high demands on the model's generalization ability and depth of understanding. Traditional semantic segmentation methods often only effectively handle categories present in the training set, exhibiting significant limitations for novel or unlabeled categories.

[0003] With the rise of pre-trained visual-language models, such as CLIP (a text-image pre-trained model employing contrastive learning), researchers have begun to explore applying the cross-modal understanding capabilities of these models for images and text to zero-shot semantic segmentation tasks. CLIP models, through large-scale image-text pair pre-training, have demonstrated a deep understanding of image content and powerful zero-shot classification capabilities. However, directly applying CLIP to pixel-level semantic segmentation tasks still faces the challenge of translating image-level understanding into accurate pixel-level predictions.

[0004] Furthermore, existing methods for applying CLIP models to zero-shot semantic segmentation tasks mostly employ a two-stage process, including using a class-agnostic mask generator to produce preliminary segmentation results, and then using CLIP for zero-shot classification. This approach is not only computationally expensive but also difficult to implement end-to-end training, limiting further improvements in model performance. Therefore, there is an urgent need to develop a new method that can fully utilize the powerful cross-modal understanding capabilities of pre-trained models such as CLIP while effectively addressing the pixel-level classification problem in zero-shot semantic segmentation tasks. Summary of the Invention

[0005] This invention provides a zero-shot semantic segmentation method based on region cues to improve the text image pre-training model, in order to solve the problems of poor generalization of existing models in zero-shot semantic segmentation and the inability of pixel-level classification to meet practical needs.

[0006] This invention provides a zero-shot semantic segmentation method based on region cues to improve a text image pre-trained model, comprising:

[0007] Construct input data for a pre-trained text-image model;

[0008] By using pre-defined region-level bridges, deep features are extracted from the images in the input data to distinguish the category features of each region;

[0009] Based on the category features of each region, an additional constraint is added to the preset semantic segmentation decoder using a preset recovery decoder with recovery loss to reduce overfitting.

[0010] Zero-shot semantic segmentation is performed based on image and text features that have been reduced from overfitting.

[0011] According to the present invention, a zero-shot semantic segmentation method for improving a pre-trained text image model based on region cues is provided, wherein constructing input data for a pre-trained text image model specifically includes:

[0012] Construct the first layer of input for the visual encoder of the pre-trained text image model;

[0013] The first layer input includes: global image features, depth cue tuning markers, and image features with added positional encoding.

[0014] According to the present invention, a zero-shot semantic segmentation method for improving a text image pre-training model based on region cues is provided. The method involves extracting deep features from the input image data using pre-defined region-level bridges to distinguish the category features of each region. Specifically, this includes:

[0015] Based on the regional level bridge, the feature extraction direction is determined by a pre-set mask attention;

[0016] Image regions are segmented and labeled based on feature extraction directions to generate labeling information, and deep features are extracted based on the labeling information.

[0017] Based on the extracted deep features, the segmented regions are distinguished by category features, resulting in global image category features with classification labels, image features extracted by the visual encoder, and region category features including region-level bridges.

[0018] According to the present invention, a zero-shot semantic segmentation method for improving a pre-trained text image model based on region cues is provided. During the training phase, the method adds additional constraints to a pre-set semantic segmentation decoder based on the category features of each region using a pre-set recovery decoder with recovery loss to reduce overfitting. Specifically, this includes:

[0019] Based on the classification results of each region, the collected global category features of the image and the category features of the region containing the region-level bridges are aligned to generate an alignment result.

[0020] Based on the alignment result, the global category features and regional category features of the image are upsampled to the size of the image features and concatenated with the image features to obtain the features to be decoded;

[0021] During the training phase, the features to be decoded are decoded by a preset recovery decoder with recovery loss, which adds additional constraints to the semantic segmentation decoder and reduces overfitting.

[0022] According to the present invention, a zero-shot semantic segmentation method for improving a text image pre-trained model based on region cues is provided. The method aligns the collected global image category features and the region category features extracted including region-level bridges based on the category feature differentiation results of each region to generate an alignment result. Specifically, this includes:

[0023] Define the dimensions of the regional category features extracted from regional-level bridges;

[0024] Alignment results are generated by aligning the global category features of the image with the category features of the region containing the bridge at the region level, based on the set dimensions.

[0025] According to the present invention, a zero-shot semantic segmentation method for improving a text image pre-training model based on region cues is provided, wherein the text image pre-training model is pre-trained using a large amount of paired image and text data;

[0026] During training, images and text are mapped to a shared vector space, enabling the model to understand the semantic relationships between images and text;

[0027] After training, the text-image pre-trained model can determine the alignment relationship between the image and the text;

[0028] Based on the alignment relationship between images and text, a text-image pre-trained model is used to determine the text aligned to the image in the input data.

[0029] This invention also provides a zero-shot semantic segmentation system based on improved text image pre-training models using region cues, the system comprising:

[0030] The data acquisition module is used to construct input data for a pre-trained text image model.

[0031] The deep feature extraction module is used to extract deep features from the input data image through preset region-level bridges, and to distinguish the category features of each region.

[0032] The recovery decoding module is used to add additional constraints to the preset semantic segmentation decoder based on the category features of each region through a preset recovery decoder with recovery loss to reduce overfitting.

[0033] The semantic segmentation module is used for zero-shot semantic segmentation based on image and text features that have been reduced from overfitting.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the zero-shot semantic segmentation method for improving a text image pre-training model based on region cues as described above.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the zero-shot semantic segmentation method for improving a text image pre-training model based on region cues as described above.

[0036] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the zero-shot semantic segmentation method for improving a text image pre-trained model based on region cues as described above.

[0037] This invention provides a zero-shot semantic segmentation method based on improved text-image pre-trained models using region cues. By extending the powerful image-text cross-modal understanding capabilities of pre-trained text-image models to zero-shot semantic segmentation tasks, it leverages the robust capabilities of pre-trained visual-language models to connect image-level and pixel-level semantic understanding through region-level bridges, thereby achieving finer-grained semantic segmentation. This significantly improves the model's accuracy and generalization ability when handling unseen categories. This not only provides a new technical path for the field of zero-shot semantic segmentation but also offers valuable experience and reference for future related research. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 This is one of the flowcharts of the zero-shot semantic segmentation method based on the improved text image pre-training model with regional cues provided by the present invention.

[0040] Figure 2 This is the second flowchart of the zero-shot semantic segmentation method based on the improved text image pre-training model using region cues provided by this invention.

[0041] Figure 3This is the third flowchart of the zero-shot semantic segmentation method based on the improved text image pre-training model with regional cues provided by this invention.

[0042] Figure 4 This is the fourth flowchart of the zero-shot semantic segmentation method based on the improved text image pre-training model with regional cues provided by this invention.

[0043] Figure 5 This is the fifth flowchart of the zero-shot semantic segmentation method based on improved text image pre-training models using region cues provided by this invention.

[0044] Figure 6 This is a schematic diagram of a region-level bridge in the image coding layer provided by the present invention.

[0045] Figure 7 This is a visual schematic diagram of the attention mask in CLIP provided by the present invention.

[0046] Figure 8 This is a schematic diagram of the module connections of the zero-shot semantic segmentation system based on the improved text image pre-training model using region cues provided by the present invention.

[0047] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] First, let's explain the relevant concepts of CLIP. CLIP stands for Constrastive Language-Image Pre-training, which is a text-image pre-training model that uses contrastive learning. CLIP has a very simple architecture and good performance, performing well in zero-shot text-image retrieval, zero-shot image classification, the text-image generation task quidance, and open-domain detection and segmentation tasks.

[0050] The core of the CLIP model is pre-training with a large amount of paired image and text data to learn the alignment relationships between images and text. The CLP model has two modalities: a text modality and a visual modality, comprising two main parts.

[0051] 1. Text Encoder: Used to convert text into a low-dimensional vector representation - Embedding.

[0052] 2. Image Encoder: Used to convert images into a similar vector representation - Embedding.

[0053] In the prediction phase, the CLP model generates predictions by calculating the cosine similarity between text and image vectors. This model is particularly suitable for zero-shot learning tasks, meaning the model can make predictions without seeing new training examples of images or text. The CLIP model performs well in multiple domains, such as image text retrieval and image-text generation.

[0054] CLIP's innovation lies in its ability to map images and text into a shared vector space, enabling the model to understand the semantic relationships between them. This shared vector space allows CLIP to achieve unsupervised joint learning between images and text, making it applicable to various visual and language tasks. CLIP's design is inspired by a simple idea: enabling the model to understand the relationships between images and text not just through supervised training, but through self-supervised learning. CLIP is trained using a large number of image-text pairs, allowing the model to embed corresponding images and text close to each other in the vector space.

[0055] CLIP first inputs the image and text into an image encoder and a text encoder, respectively, to obtain the vector representations I_f and T_f of the image and text. Then, it maps these vector representations to a joint multimodal space, resulting in new, directly comparable vector representations I_e and T_e (this is a common method in multimodal learning; gaps may exist between data representations of different modalities, making direct comparison impossible. Therefore, mapping data from different modalities to the same multimodal space facilitates subsequent similarity calculations). Next, it calculates the cosine similarity between the image and text vectors. Finally, the objective function of contrastive learning Q is to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs.

[0056] CLIP features a unified vector space: A key innovation of CLIP is that it maps both images and text to the same vector space. This allows the model to directly compute the similarity between images and text in the vector space without requiring additional intermediate representations.

[0057] Contrastive Learning: CLIP uses contrastive learning for pre-training. The model is instructed to map image and text embeddings from the same sample to nearby locations, while mapping embeddings from different samples to more distant locations. This allows the model to learn common features between images and text.

[0058] Unsupervised learning: CLIP's pre-training is unsupervised, meaning it doesn't require large amounts of labeled data to guide training. It learns from text and image data on the internet, enabling it to excel in tasks across various domains.

[0059] The following is combined Figures 1-5 The present invention describes a zero-shot semantic segmentation method for improving a pre-trained text image model based on region cues, comprising: S100, constructing input data for a preset pre-trained text image model.

[0060] In this invention, based on the input data, the powerful capabilities of pre-trained visual-language models (such as CLIP) are utilized to connect image-level and pixel-level semantic understanding through region-level bridges, thereby achieving finer-grained semantic segmentation, narrowing the gap between image-level recognition and pixel-level semantic segmentation, and improving the model's generalization ability to unseen categories.

[0061] Specifically, S101, constructs the first layer input for the visual encoder of the pre-trained text image model.

[0062] S102, The first layer input includes: global image features, depth cue tuning markers, and image features with added position encoding.

[0063] In this invention, the first layer of the CLIP visual encoder is constructed as input.

[0064] X 0 =[G 0 P 0 I 0 R 0 ].

[0065] in, The [CLS] tag represents a visual encoder designed to capture global image features, where D represents the dimension of the features. This represents the K depth cue tuning markers for the first layer, from which a marker set is selected. These are task-specific learnable parameters in the input space. This indicates that location-encoded image features have been added. Finally, This indicates an initial area-level bridge marked with an M. R 0 Each tag in the G array uses G. 0The weights are used as their initial weights, acting as a bridge between image-level features and pixel-level features.

[0066] The [CLS] token is a classification token used to indicate the beginning of the input sequence. During training, BERT learns to use the [CLS] token for sentence classification tasks, such as sentiment analysis. In the input sequence, [CLS] should be placed at the beginning of the sentence. For example: [CLS] This is an example.

[0067] S200: Extract deep features from the images in the input data using a preset region-level bridge to distinguish the category features of each region.

[0068] Specifically: S201, Based on the region-level bridge, the feature extraction direction is determined by the preset mask attention.

[0069] S202. Segment and label the image region based on the feature extraction direction, generate label information, and perform deep feature extraction at the region level based on the label information.

[0070] S203. Based on the extracted deep features, the category features of the segmented regions are distinguished to obtain the global category features of the image with classification labels, the image features extracted by the visual encoder, and the category features of the regions containing the region-level bridges.

[0071] In this invention, such as Figure 6 As shown, we continue to extract image features. The input consists of [CLS] markers, depth cue markers, image features, and region-level bridges (from top to bottom). Each marker in the region-level bridges is responsible for a different image region, resulting in more refined image region classification features. After the inference process is complete, we obtain the image features and the [CLS] markers containing both global image features and region category features.

[0072] For an image feature sequence of length N, the original size is... Mask attention is used to extract region category features of image regions through region-level bridging. For example... Figure 7 As shown, (a) each marker in the region-level bridge is responsible for a 2×2 region of a 4×4-sized image feature. (b) For the [CLS] marker and the depth cue marker, each marker interacts with all image features, including interactions between them. (c) Visualization of the attention mask during self-attention, where white blocks represent the mask and no interaction.

[0073] Attention mask The calculation direction for region-level bridges is defined, where E = 1 + K + N + M. Each marker in the region-level bridge is responsible for... Features are extracted from each patch. In this way, each marker in the region-level bridge is responsible for a region in the original whole feature set, representing an intermediate granularity between the image and pixel levels. This feature extraction method is almost identical to the feature extraction used by CLIP for image classification.

[0074] After obtaining the attention mask, the mask attention operation in the visual encoder can be described as follows.

[0075]

[0076] Among them, X l and X l+1 These are the converter layers l+1. The input and output of the visual encoder are such that, to ensure the generality of region-level bridges across unseen categories, their gradients are not updated. The final output of the visual encoder (in X...) L The expression is given by the following formula.

[0077] X L = [G, -, I, R],.

[0078] Where G represents the global category feature of the image marked with [CLS], I represents the image feature extracted by the visual encoder, and R contains the region category feature extracted by the region-level bridge.

[0079] In this invention, a region-level bridge is used to extract region-level category features from an image, serving as a bridge connecting image-level recognition and pixel-level segmentation. By effectively capturing region-level category features, the region-level bridge can provide more refined semantic information for zero-shot semantic segmentation tasks.

[0080] S300: Based on the category features of each region, additional constraints are added to the preset semantic segmentation decoder through a preset recovery decoder with recovery loss to reduce overfitting.

[0081] Specifically, S301, based on the classification results of each region, the collected global category features of the image and the category features of the region containing the bridge at the region level are aligned to generate an alignment result.

[0082] S302. Based on the alignment result, the global category features and regional category features of the image are upsampled to the size of the image features and concatenated with the image features to obtain the features to be decoded.

[0083] S303. During the training phase, the features to be decoded are decoded by a preset recovery decoder with recovery loss, which adds additional constraints to the semantic segmentation decoder and reduces overfitting.

[0084] In this invention, to mitigate potential overfitting during model training, a recovery decoder with recovery loss is introduced. The recovery decoder with recovery loss aims to add additional constraints to the semantic segmentation decoder, ensuring that the features learned by the model are both specific and capable of generalization.

[0085] Based on the classification results of each region, the collected global category features of the image and the category features of the region containing the bridge at the region level are aligned to generate alignment results, specifically including S401 and S402.

[0086] S401. Set the dimension of the regional category features extracted from regional-level bridges to include regional-level bridges.

[0087] S402. Align the global category features of the image with the region category features extracted from the region-level bridges based on the set dimensions to generate an alignment result.

[0088] In this invention, based on the output of the CLIP visual encoder, the collected global image category features G are aligned with region- and image-specific features R. To align the feature sets, this invention reshapes R as follows: The dimension of I represents the class features of each region in the image. Then, we upsample the G and R, which contain global image class features, to the size of the image features and concatenate them with I.

[0089]

[0090] In addition, RLB is merged with the global [CLS] token.

[0091] R a =(Upsample(G)+R),.

[0092] R a The shape is M×D, then.

[0093]

[0094] This represents the raw text embedding extracted by the CLIP text encoder, where C is the number of classes. It is a region relation descriptor.

[0095] For the obtained features to be decoded, the present invention uses a decoder to perform decoding.

[0096]

[0097] in, and This represents a decoder for multi-head cross-attention semantic segmentation. and These are image features and region-specific text queries, resulting in the final segmentation image. It is obtained by averaging the output.

[0098]

[0099] During training, the recovery decoder restores the features extracted by the decoder into features with strong generalization ability. The network structure of the recovery decoder is exactly the same as that of the semantic segmentation decoder. Their recovery process is as follows.

[0100]

[0101] and This indicates the characteristics after restoration.

[0102] The method of restoring the features extracted by the decoder into features with strong generalization ability by using a preset recovery decoder with recovery loss based on the decoding result also includes setting a specific recovery loss for the recovery decoder.

[0103] The recovery loss is used to ensure that the recovery decoder maintains equal weight between learning the specific content of the current task and maintaining a broad general knowledge base.

[0104] To ensure that the output of the recovery decoder remains consistent with the original features of the backbone network, this invention proposes a recovery loss for use by the recovery decoder. This recovery loss aims to help the semantic segmentation decoder strike a balance between learning the specific content of the task at hand and maintaining a broad general knowledge base, thereby mitigating the problem of overfitting to unseen categories.

[0105]

[0106] S400 performs zero-shot semantic segmentation based on image and text features that have been reduced from overfitting.

[0107] The text-image pre-training model is pre-trained using a large amount of paired image and text data.

[0108] During training, images and text are mapped to a shared vector space, enabling the model to understand the semantic relationships between images and text.

[0109] Once trained, the text-image pre-trained model can determine the alignment relationship between the image and the text.

[0110] Based on the alignment relationship between images and text, a text-image pre-trained model is used to determine the text aligned to the image in the input data.

[0111] This invention successfully extends the powerful image-text cross-modal understanding capabilities of the CLIP model to zero-shot semantic segmentation tasks, significantly improving the model's accuracy and generalization ability when handling unseen categories. This improvement not only provides a new technical path for the field of zero-shot semantic segmentation but also offers valuable experience and reference for future related research.

[0112] refer to Figure 8 The present invention also discloses a zero-shot semantic segmentation system based on an improved text image pre-training model using region cues, the system comprising:

[0113] The data acquisition module 110 is used to construct input data for a pre-trained text image model;

[0114] The deep feature extraction module 120 is used to extract deep features from the image in the input data through a preset region-level bridge, and to distinguish the category features of each region.

[0115] The recovery decoding module 130 is used to add additional constraints to the preset semantic segmentation decoder based on the category features of each region through a preset recovery decoder with recovery loss to reduce overfitting.

[0116] Semantic segmentation module 140 is used for zero-shot semantic segmentation based on image and text features that have been reduced from overfitting.

[0117] The data acquisition module constructs the first layer of input for the visual encoder of the pre-trained text image model.

[0118] The first layer input includes: global image features, depth cue tuning markers, and image features with added positional encoding.

[0119] The deep feature extraction module determines the feature extraction direction based on the region-level bridge using a preset mask attention.

[0120] Image regions are segmented and labeled based on feature extraction directions to generate label information, and deep feature extraction is performed based on the label information.

[0121] Based on the extracted depth features, the segmented regions are classified, resulting in global image category features with classification labels, image features extracted by the visual encoder, and region category features including region-level bridges.

[0122] The recovery decoding module, based on the classification results of each region, aligns the collected global category features of the image with the category features of the regions containing the bridges at the region level, and generates an alignment result.

[0123] Based on the alignment result, the global category features and regional category features of the image are upsampled to the size of the image features and concatenated with the image features to obtain the features to be decoded.

[0124] During the training phase, the features to be decoded are decoded by a preset recovery decoder with recovery loss, which adds additional constraints to the semantic segmentation decoder and reduces overfitting.

[0125] Specifically, based on the category feature differentiation results of each region, the collected global category features of the image and the category features of the regions containing the regional bridges are aligned to generate the alignment result, which includes:

[0126] The dimensions of the regional category features extracted from regional-level bridges are set.

[0127] Alignment results are generated by aligning the global category features of the image with the category features of the region containing the bridge at the region level, based on the set dimensions.

[0128] The text-image pre-training model is pre-trained using a large amount of paired image and text data.

[0129] During training, images and text are mapped to a shared vector space, enabling the model to understand the semantic relationships between images and text.

[0130] Once trained, the text-image pre-trained model can determine the alignment relationship between the image and the text.

[0131] Based on the alignment relationship between images and text, a text-image pre-trained model is used to determine the text aligned to the image in the input data.

[0132] This invention provides a zero-shot semantic segmentation system based on region cues to improve the pre-trained text-image model. This extends the powerful image-text cross-modal understanding capabilities of the pre-trained model to zero-shot semantic segmentation tasks. Leveraging the robust capabilities of the pre-trained visual-language model, it connects image-level and pixel-level semantic understanding through region-level bridges, thereby achieving finer-grained semantic segmentation and significantly improving the model's accuracy and generalization ability when handling unseen categories. This not only provides a new technical path for the field of zero-shot semantic segmentation but also offers valuable experience and reference for future research.

[0133] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a zero-shot semantic segmentation method based on region cues to improve a pre-trained text image model. This method includes: constructing input data for a pre-trained text image model; extracting deep features from the image in the input data through a pre-defined region-level bridge to distinguish the category features of each region; adding additional constraints to a pre-defined semantic segmentation decoder based on the category features of each region through a pre-defined recovery decoder with recovery loss to reduce overfitting; and performing zero-shot semantic segmentation based on the image features and text features that have been reduced from overfitting.

[0134] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the zero-shot semantic segmentation method based on the region-based improved text image pre-training model provided by the above methods. The method includes: constructing input data for a preset text image pre-training model; extracting deep features from the image in the input data through a preset region-level bridge to distinguish the category features of each region; adding additional constraints to the preset semantic segmentation decoder based on the category features of each region through a preset recovery decoder with recovery loss to reduce overfitting; and performing zero-shot semantic segmentation based on the image features and text features that have been reduced from overfitting.

[0136] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a zero-shot semantic segmentation method for improving a pre-trained text image model based on region cues, as provided by the methods described above. This method includes: constructing input data for a pre-trained text image model; extracting deep features from the image in the input data through pre-defined region-level bridges to distinguish the category features of each region; adding additional constraints to a pre-defined semantic segmentation decoder based on the category features of each region using a pre-defined recovery decoder with recovery loss to reduce overfitting; and performing zero-shot semantic segmentation based on the overfitted image features and text features.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A zero-shot semantic segmentation method based on improved text image pre-training models using region cues, characterized in that, include: Construct input data for a pre-trained text-image model; The input data for constructing the preset text image pre-trained model includes: Construct the first layer of input for the visual encoder of the pre-trained text image model; The first layer input includes: global image features, depth cue tuning markers, and image features with added positional encoding; The image in the input data is subjected to depth feature extraction through a preset region-level bridge to distinguish the category features of each region; the process of extracting depth features from the image in the input data through a preset region-level bridge to distinguish the category features of each region includes: Based on the regional level bridge, the feature extraction direction is determined by a pre-set mask attention; Image regions are segmented and labeled based on feature extraction directions to generate labeling information, and deep features are extracted based on the labeling information. Based on the extracted deep features, the category features of the segmented regions are distinguished, resulting in global image category features with classification labels, image features extracted by the visual encoder, and region category features including region-level bridges. Based on the category features of each region, an additional constraint is added to the preset semantic segmentation decoder using a preset recovery decoder with recovery loss to reduce overfitting. Zero-shot semantic segmentation is performed based on image and text features that have been reduced from overfitting.

2. The zero-shot semantic segmentation method based on region cues to improve the text image pre-training model according to claim 1, characterized in that, During the training phase, the pre-defined semantic segmentation decoder is supplemented with additional constraints based on the category features of each region using a pre-defined recovery decoder with recovery loss to reduce overfitting. Specifically, this includes: Based on the classification results of each region, the collected global category features of the image and the category features of the region containing the region-level bridges are aligned to generate an alignment result. Based on the alignment result, the global category features and regional category features of the image are upsampled to the size of the image features and concatenated with the image features to obtain the features to be decoded; During the training phase, the features to be decoded are decoded by a preset recovery decoder with recovery loss, which adds additional constraints to the semantic segmentation decoder and reduces overfitting.

3. The zero-shot semantic segmentation method based on improved text image pre-training models using region cues according to claim 2, characterized in that, The classification results based on the category features of each region are used to align the collected global category features of the image with the category features of the regions containing the bridges at the region level, generating an alignment result, specifically including: Define the dimensions of the regional category features extracted from regional-level bridges; Alignment results are generated by aligning the global category features of the image with the category features of the region containing the bridge at the region level, based on the set dimensions.

4. The zero-shot semantic segmentation method based on improved text image pre-training models using region cues as described in claim 2, characterized in that, The text-image pre-training model is pre-trained using a large amount of paired image and text data; During training, images and text are mapped to a shared vector space, enabling the model to understand the semantic relationships between images and text; After training, the text-image pre-trained model can determine the alignment relationship between the image and the text; Based on the alignment relationship between images and text, a text-image pre-trained model is used to determine the text aligned to the image in the input data.

5. A zero-shot semantic segmentation system based on improved text image pre-training models using region cues, characterized in that, The system includes: The data acquisition module is used to construct input data for a preset text image pre-trained model; constructing input data for the preset text image pre-trained model includes: constructing a first layer of input for the visual encoder of the preset text image pre-trained model; the first layer of input includes: global image features, depth cue tuning markers, and image features with added position encoding; The deep feature extraction module is used to extract deep features from the image in the input data through a preset region-level bridge, and distinguish the category features of each region. The process of extracting deep features from the image in the input data through the preset region-level bridge and distinguishing the category features of each region includes: determining the feature extraction direction based on the region-level bridge using a preset mask attention; segmenting and labeling the image regions based on the feature extraction direction to generate label information; extracting deep features based on the label information; and distinguishing the category features of the segmented regions based on the extracted deep features, resulting in global image category features with classification labels, image features extracted by the visual encoder, and region category features including those extracted by the region-level bridge. The recovery decoding module is used to add additional constraints to the preset semantic segmentation decoder based on the category features of each region through a preset recovery decoder with recovery loss to reduce overfitting. The semantic segmentation module is used for zero-shot semantic segmentation based on image and text features that have been reduced from overfitting.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the zero-shot semantic segmentation method for improving the text image pre-training model based on region cues as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the zero-shot semantic segmentation method for improving the text image pre-training model based on region cues as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the zero-shot semantic segmentation method for improving the text image pre-training model based on region cues as described in any one of claims 1 to 4.

Images