A narrow zero-shot semantic segmentation system and method based on multimodal information alignment

Abstract

This invention relates to the field of image processing technology, specifically to a narrow-sense zero-shot semantic segmentation system and method based on multimodal information alignment. The system includes a pre-trained feature extraction module, a semantic-visual feature alignment module, and a prototype metric segmentation module. The pre-trained feature extraction module is used to extract semantic and visual features from a specially processed image; wherein the pre-trained feature extraction module includes a feature extractor and a pre-trained language model; the semantic-visual feature alignment module is used to align the extracted semantic and visual features to obtain aligned semantic features; the prototype metric segmentation module uses the aligned semantic features to perform metric segmentation on the visual features to be segmented, obtaining a narrow-sense semantic segmentation result.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a narrow zero-shot semantic segmentation system and method based on multimodal information alignment. Background Technology

[0002] Image semantic segmentation: Image semantic segmentation is a key task in the field of computer vision. Its goal is to perform pixel-level analysis and labeling of images, thereby achieving fine differentiation and annotation of different objects or regions within an image. This technology not only segments an image into individual parts, but also strives to understand the image and divide it into semantically meaningful regions. Through semantic segmentation, computers can gain a deeper understanding of image content, providing crucial foundational support for fields such as autonomous driving and medical image analysis.

[0003] Zero-shot image semantic segmentation: Zero-shot image semantic segmentation is a challenging task in the field of image segmentation, characterized by the lack of similar samples or category information during the training phase compared to the testing phase. In this situation, traditional learning methods cannot be directly applied due to the lack of prior training data. This segmentation method attempts to achieve semantic segmentation of images by utilizing category descriptors, attribute information, or other prior knowledge, as well as techniques such as transfer learning, achieving satisfactory segmentation results even in the absence of training data.

[0004] Narrow semantic segmentation: Narrow semantic segmentation is a specific form of image semantic segmentation that focuses on segmenting only specific categories of objects or regions. Unlike full-image semantic segmentation, narrow semantic segmentation focuses on accurately segmenting specific categories of targets or regions in an image. Because narrow semantic segmentation focuses on specific categories of targets or regions, it can provide more accurate segmentation results. Compared to full-image semantic segmentation, narrow segmentation can more accurately identify and segment specific targets, which is helpful for refined image analysis.

[0005] Training and test classes, seen and unseen classes: Zero-shot semantic segmentation is a semantic segmentation task performed on test classes without labeled samples. Current methods are based on transfer learning. The model is first trained on some training classes (training and test classes are disjoint), and the trained model is expected to be directly transferred to the test class for testing. Therefore, in zero-shot semantic segmentation, the training class refers to the seen classes used to train the model during the training phase, while the test class refers to the unseen classes used to test the segmentation performance of the zero-shot model during the testing phase.

[0006] Image semantic segmentation enables pixel-level labeling and recognition of different objects or regions within an image. It provides a crucial foundation for computer vision, allowing computers to accurately understand image content. Through semantic segmentation, computers can assign each pixel in an image to a corresponding semantic category, providing vital support for applications such as autonomous driving, medical image analysis, and security monitoring.

[0007] Zero-shot segmentation arose because in many real-world scenarios, it's difficult to collect enough labeled samples for training. This can be due to high data acquisition costs, labeling difficulties, or insufficient data volume. The advantage of zero-shot segmentation is that, even without any samples or category information similar to those used in the testing phase, it attempts to achieve effective semantic segmentation of images by utilizing category descriptors, attribute information, or other prior knowledge. This expands the model's applicability, enabling it to achieve satisfactory segmentation results even when faced with unknown categories or samples.

[0008] However, zero-shot semantic segmentation faces challenges such as low accuracy in practical applications. Due to the lack of a large amount of prior data, the model may fail to accurately capture the features of new categories or samples, leading to a decrease in the accuracy of the segmentation results. This limits the applicability of zero-shot segmentation methods in certain complex scenarios, making them somewhat limited when handling unknown categories or samples. To address this technical problem, a narrow zero-shot semantic segmentation system and method based on multimodal information alignment is proposed. Summary of the Invention

[0009] To address the technical problems existing in the prior art, the present invention provides a narrow zero-sample semantic segmentation method and system based on multimodal information alignment.

[0010] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0011] In a first aspect, in one embodiment of the present invention, a narrow zero-shot semantic segmentation system based on multimodal information alignment is provided, the system comprising: a pre-trained feature extraction module, a semantic-visual feature alignment module, and a prototype metric segmentation module;

[0012] The pre-trained feature extraction module is used to extract semantic and visual features from the image to be processed; wherein the pre-trained feature extraction module includes a feature extractor and a pre-trained language model, and the pre-trained language model is based on the bag-of-words model and is trained using text semantic information.

[0013] The semantic-visual feature alignment module is used to align the extracted semantic features and visual features to obtain aligned semantic features.

[0014] The prototype measurement and segmentation module uses the aligned semantic features to measure and segment the visual features to be segmented, thereby obtaining the narrow semantic segmentation result.

[0015] As a further aspect of the present invention, the feature extractor obtaining step includes:

[0016] A generalized semantic segmentation network is directly trained using labeled data from the training set.

[0017] The parameters of the feature extractor in the generalized semantic segmentation network are frozen, and the feature extractor with frozen parameters is then transferred to the pre-trained feature extraction module.

[0018] As a further aspect of the present invention, the generalized semantic segmentation network includes a feature extractor and a segmenter; the feature extractor is used to extract three-dimensional depth visual features of the image; the segmenter is used to perform pixel-level classification of the three-dimensional depth visual features.

[0019] As a further embodiment of the present invention, the generalized semantic segmentation network may further include a spatial information capture module, which is used to analyze the three-dimensional depth visual features obtained by the feature extractor to obtain more refined features.

[0020] As a further aspect of the present invention, the semantic-visual feature alignment module includes a semantic feature mapper and a similarity calculator;

[0021] The semantic feature mapper is used to map semantic features to an embedded feature space that is similar to visual features to achieve alignment. The alignment loss calculation process is used to calculate the cross-entropy loss between the similarity map and the visual label to achieve alignment between semantic features and visual features.

[0022] The similarity calculator is used to calculate the similarity relationship between semantic features and visual features, providing alignment support.

[0023] As a further aspect of the present invention, the semantic-visual feature alignment module aligns the extracted semantic features and visual features to obtain aligned semantic features, including the following steps:

[0024] Alignment is achieved by using a semantic feature mapper to map the set of word vectors into an embedding feature space that is similar to the visual features;

[0025] The semantic features and the visual features of the training images obtained by the pre-trained feature extraction module are input into the similarity calculator to obtain a similarity map. The segmentation result map of image X is obtained using the similarity map. The loss is calculated using the segmentation result map and the label to supervise the alignment process and achieve alignment.

[0026] As a further aspect of the present invention, the semantic feature F wThe visual features of the training image X obtained from the pre-trained feature extraction module are input into a similarity calculator to obtain a similarity map. The segmentation result map of image X is obtained using the similarity map. The loss is calculated using the segmentation result map and the labels to supervise the alignment process and achieve alignment, including:

[0027] Visual features are transformed and their dimensions are calculated.

[0028] Semantic features and visual features for transformation dimension calculation are input into a similarity calculator to obtain a similarity map. The loss is calculated using the similarity map to align the semantic features and visual features.

[0029] As a further aspect of the present invention, the prototype metric segmentation module is used to segment the visual features to be segmented using aligned semantic features to obtain a narrow semantic segmentation result, and the steps are as follows:

[0030] Extract features corresponding to the foreground category from the aligned semantic features to obtain a foreground prototype set;

[0031] After expanding the resolution of the foreground prototypes in the foreground prototype set to the size of the semantic features to be segmented, they are concatenated one by one with the semantic features to be segmented. Then, a convolutional structure is used as a metric to obtain the metric result. The metric result is upsampled to the size of the label Y to obtain the narrow semantic segmentation result.

[0032] Secondly, in another embodiment provided by the present invention, a narrow zero-sample semantic segmentation method based on multimodal information alignment is provided, the method comprising:

[0033] Semantic and visual features are extracted from the image to be processed;

[0034] The extracted semantic features and visual features are aligned to obtain aligned semantic features;

[0035] This is used to segment the visual features to be segmented using aligned semantic features, thereby obtaining narrow semantic segmentation results.

[0036] The technical solution provided by this invention has the following beneficial effects:

[0037] This invention includes a semantic-visual feature alignment module that aligns semantic and visual spaces by calculating the similarity between word vector features and visual features. The prototype measurement and segmentation module extracts multi-category prototypes and finally uses the prototypes to segment specific categories, achieving more accurate segmentation results.

[0038] These or other aspects of the invention will become more apparent from the following description of embodiments. It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of a narrow zero-sample semantic segmentation method based on multimodal information alignment according to an embodiment of the present invention.

[0041] Figure 2 This is a detailed flowchart illustrating the structural block diagram of a narrow zero-sample semantic segmentation system based on multimodal information alignment, according to an embodiment of the present invention.

[0042] Figure 3 This is a training flowchart of a narrow zero-shot semantic segmentation system based on multimodal information alignment, according to an embodiment of the present invention.

[0043] Figure 4 This is an example result diagram of a narrow zero-sample semantic segmentation system based on multimodal information alignment according to an embodiment of the present invention.

[0044] In the figure: Pre-trained feature extraction module-100, semantic-visual feature alignment module-200, prototype metric segmentation module-300. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0047] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0048] Specifically, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0049] Please see Figure 1 , Figure 1 This is a flowchart of a narrow zero-shot semantic segmentation method based on multimodal information alignment provided by an embodiment of the present invention, as shown below. Figure 1 As shown, the narrow zero-sample semantic segmentation method based on multimodal information alignment includes steps S10 to S30.

[0050] S10. Extract semantic and visual features from the image to be processed;

[0051] In an embodiment of the present invention, S10 involves extracting semantic and visual features from the image to be processed, including obtaining visual features using a feature extractor.

[0052] The feature extractor acquisition steps include:

[0053] A generalized semantic segmentation network is directly trained using labeled data from the training set.

[0054] The feature extractor parameters in the generalized semantic segmentation network are frozen to obtain the feature extractor.

[0055] In an embodiment of the present invention, the generalized semantic segmentation network includes a feature extractor and a segmenter; the feature extractor is used to extract three-dimensional depth visual features of the image; the segmenter is used to perform pixel-level classification of the three-dimensional depth visual features.

[0056] In embodiments of the present invention, the generalized semantic segmentation network may further include a spatial information capture module, which is used to analyze the three-dimensional depth visual features obtained by the feature extractor to obtain more refined features, thereby improving segmentation performance.

[0057] S10, extracting semantic and visual features from the image to be processed, includes:

[0058] The pre-trained language model W obtained through open-source pre-training maps all categories of text labels to their corresponding semantic features.

[0059] The formula for calculating semantic features is as follows:

[0060] w = W(c word )

[0061] Where w is the word vector, w∈R 1×d d is the dimension of the word vector, c word This represents the word corresponding to category c. For example, to obtain the word vector for the category "cow", the input c is... wordThe value is 'cow'. Since there are similarity calculation relationships between all word vectors, word vectors from all categories are extracted for calculation during the training of the semantic-visual feature alignment module to increase the amount of usable information. The final set of word vectors obtained is w. all ∈R n×d , where n represents the total number of all categories.

[0062] S20. Align the extracted semantic features and visual features to obtain aligned semantic features.

[0063] In an embodiment of the present invention, S20, aligning the extracted semantic features and visual features to obtain aligned semantic features, includes the following steps:

[0064] S21. Use a semantic feature mapper to map the set of semantic features to an embedded feature space that is similar to the visual features to achieve alignment;

[0065] in,

[0066] In the formula, F w ∈R n×c This represents the semantic features after mapping, where c represents the feature dimension. This represents a semantic feature mapper, which consists of fully connected layers.

[0067] S22, semantic feature F w and the visual features F of the training image X obtained by the pre-trained feature extraction module x ∈R c ×h×w Input the similarity calculator to obtain a similarity map; use the similarity map to obtain the segmentation result map of image X; use the segmentation result map and labels to calculate the loss, supervise the alignment process, and achieve alignment.

[0068] In this embodiment of the invention, the semantic feature F w and the visual features F of the training image X obtained by the pre-trained feature extraction module x ∈R c×h×w Input the similarity calculator to obtain a similarity map; use the similarity map to obtain a segmentation result map of image X; use the segmentation result map and labels to calculate the loss, supervise the alignment process, and achieve alignment, including:

[0069] S221, Visual Feature F x Perform transformation dimension calculation;

[0070] The visual feature F x The formula for calculating the transformation dimension is as follows:

[0071]

[0072] In the formula, This means transforming the input to dimension R. c×hw .

[0073] S222, Semantic Feature F w And the visual features F for dimensional transformation calculation x Input the similarity calculator to obtain a similarity map, use the similarity map to calculate the loss, and align semantic features and visual features.

[0074] The formula for calculating the similarity map is as follows:

[0075]

[0076] In the formula, Sim represents the similarity graph.

[0077] Similarity maps can be viewed as activations of similar visual features by aligned semantic features. Therefore, we can directly use the similarity map Sim to extract the similarity maps of the corresponding foreground and background categories based on the category index. For example, during training, we can train category C. train Foreground category, Test category C test For the background category, the opposite applies during testing. Therefore, the Sim dimension is adjusted to R. c×h×w Then, the foreground category similarity map set SimSet fg SimSet, a set of similarity graphs to background categories bg They are respectively:

[0078]

[0079] in, This indicates that categories are converted into corresponding numbers according to a predefined category order to index the corresponding similarity graph. train and n test This represents the number of corresponding training and testing categories, where n = n train +n test Then, the background similarity map is reweighted using a convolution operation:

[0080] SimSet bg =F conv (SimSet bg )∈R 1×h×w

[0081] Among them, F conv This represents the convolution operation. Then, it is concatenated with the foreground category similarity map to obtain the segmentation result image X. The concatenated result, Logits, is:

[0082]

[0083] in, For the concatenation function, use SimSet bg and SimSet fg splicing along the channel dimension; This is an upsampling function based on bilinear interpolation, used to align the size of Logits with the label Y in order to calculate the alignment loss. The alignment loss is:

[0084]

[0085] S30 is used to segment the visual features to be segmented using the aligned semantic features, and obtain the narrow semantic segmentation result.

[0086] S30, which is used to segment the visual features to be segmented using the aligned semantic features to obtain a narrow semantic segmentation result, includes the following steps:

[0087] S31. Extract the features corresponding to the foreground category from the aligned semantic features to obtain the foreground prototype set;

[0088] S32. After expanding the resolution of the foreground prototypes in the foreground prototype set to the size of the semantic features to be segmented, concatenate them one by one with the semantic features to be segmented, and then use a convolutional structure as a metric to obtain the metric result. Upsample the metric result to the size of the label Y to obtain the narrow semantic segmentation result.

[0089]

[0090] In the formula, output is the final segmentation result. It is a convolution metric tool.

[0091] Calculate the cross-entropy loss function to supervise the model's optimization process:

[0092]

[0093] In the formula, Loss seg This is the segmentation loss function.

[0094] Finally, the model's loss is obtained by weighting the alignment loss and the segmentation loss:

[0095] Loss = α * Loss ali +β*Loss seg

[0096] Where α and β are the loss weights.

[0097] It should be understood that although the above description follows a certain order, these steps are not necessarily executed in that order. Unless otherwise expressly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, some steps in this embodiment may include multiple steps or multiple stages, which are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least a portion of the steps or stages in other steps.

[0098] In one embodiment, see Figure 2 As shown, an embodiment of the present invention also provides a narrow zero-shot semantic segmentation system based on multimodal information alignment, which includes a pre-trained feature extraction module 100, a semantic-visual feature alignment module 200, and a prototype metric segmentation module 300.

[0099] The pre-trained feature extraction module 100 is used to extract semantic and visual features from the image to be processed; wherein the pre-trained feature extraction module 100 includes a feature extractor and a pre-trained language model W, the pre-trained language model W is based on the bag-of-words model and is trained using text semantic information.

[0100] The feature extractor is used to extract visual features, and the pre-trained language model W is used to extract semantic features.

[0101] It should be noted that the pre-trained language model W extracts semantic features, including:

[0102] The open-source pre-trained language model W maps all categories of text labels to corresponding semantic features; the formula for calculating the semantic features is as follows:

[0103] w = W(c word )

[0104] Where w is a semantic feature, w∈R 1×d d is the dimension of the word vector, c word This represents the word corresponding to category c. For example, to obtain the word vector for the category "cow", the input c is... word The value is 'cow'. Since there are similarity calculation relationships between all word vectors, word vectors from all categories are extracted for calculation during the training of the semantic-visual feature alignment module to increase the amount of usable information. The final set of word vectors obtained is w. all ∈R n×d , where n represents the total number of all categories.

[0105] It should be noted that the dataset contains data in categories such as "cat" and "dog" for segmentation. The text labels of the categories, "cat" and "dog", are input into the language model W to extract word vectors w.

[0106] It's important to note that because zero-shot segmentation tasks cannot use labeled samples of specific classes from the test dataset to train the feature extraction network, traditional deep learning-based methods often directly use the feature extractor of a classification network pre-trained on the ImageNet dataset as the feature extractor for the zero-shot segmentation network. This approach presents two problems: First, the absence of a test class during pre-training on the ImageNet dataset can cause the feature extractor's parameters to be biased towards the training class, leading to overfitting. Second, training on the ImageNet dataset often involves using a feature extractor like ResNet or VGG combined with a classifier, treating it as a classification task. However, the features required for classification tasks differ from those for segmentation tasks. Semantic segmentation is a pixel-level classification task, requiring more refined and comprehensive 3D features as input to the segmenter. Directly using a feature extractor from a classification task will degrade segmentation performance.

[0107] Given the aforementioned issues, this system does not directly use the feature extractor of a classification network based on classification tasks. Instead, it first trains a generalized semantic segmentation network using samples and labels of the training class, and then directly transfers the feature extractor to the narrow zero-shot segmentation system of this system. First, the feature extractor trained for semantic segmentation tasks is clearly more suitable for downstream semantic segmentation tasks. Second, the feature extractor transferred to the narrow zero-shot segmentation system has its parameters frozen to alleviate overfitting.

[0108] Specifically, the feature extractor acquisition steps include:

[0109] A generalized semantic segmentation network is directly trained using labeled data from the training set.

[0110] The parameters of the feature extractor in the generalized semantic segmentation network are frozen, and the feature extractor with frozen parameters is transferred to the pre-trained feature extraction module 100.

[0111] In an embodiment of the present invention, the generalized semantic segmentation network includes a feature extractor and a segmenter; the feature extractor is used to extract three-dimensional depth visual features of the image; the segmenter is used to perform pixel-level classification of the three-dimensional depth visual features.

[0112] In embodiments of the present invention, the generalized semantic segmentation network may further include a spatial information capture module, which is used to analyze the three-dimensional depth visual features obtained by the feature extractor to obtain more refined features, thereby improving segmentation performance.

[0113] The semantic-visual feature alignment module 200 is used to align the extracted semantic features and visual features to obtain aligned semantic features.

[0114] In embodiments of the present invention, the semantic-visual feature alignment module 200 includes a semantic feature mapper and a similarity calculator. The semantic-visual feature alignment module 200 aims to construct a relationship between the training class and the test class through word vectors, enabling the model to directly utilize the embedded word vectors of the test class as segmentation information for the test class in the zero-shot semantic segmentation process.

[0115] The semantic feature mapper is used to map semantic features to an embedded feature space that is similar to visual features to achieve alignment. The alignment loss calculation process is used to calculate the cross-entropy loss between the similarity map and the visual label to achieve alignment between semantic features and visual features.

[0116] The similarity calculator is used to calculate the similarity relationship between semantic features and visual features, providing alignment support.

[0117] The semantic-visual feature alignment module 200 of this invention maps word vectors to an embedding feature space that is similar to visual features. Then, it uses a similarity calculation tool to calculate and align the embedded word vectors and visual features. Thus, an alignment relationship is established between the visual features of the training category and the word vectors of the training category during the training phase, and the segmenter can use this alignment relationship to support the segmentation process.

[0118] The semantic-visual feature alignment module 200 aligns the extracted semantic features and visual features to obtain aligned semantic features, including the following steps:

[0119] S21. Use a semantic feature mapper to map the set of semantic features to an embedded feature space that is similar to the visual features to achieve alignment;

[0120] in,

[0121] In the formula, F w ∈R n×c This represents the semantic features after mapping, where c represents the feature dimension. This represents a semantic feature mapper, which consists of fully connected layers.

[0122] S22, semantic feature F w and the visual features F of the training image X obtained by the pre-trained feature extraction module x ∈R c ×h×wInput the similarity calculator to obtain a similarity map; use the similarity map to obtain the segmentation result map of image X; use the segmentation result map and labels to calculate the loss, supervise the alignment process, and achieve alignment.

[0123] In this embodiment of the invention, the semantic feature F w and the visual features F of the training image X obtained by the pre-trained feature extraction module x ∈R c×h×w Input the similarity calculator to obtain a similarity map; use the similarity map to obtain a segmentation result map of image X; use the segmentation result map and labels to calculate the loss, supervise the alignment process, and achieve alignment, including:

[0124] S221, Visual Feature F x Perform transformation dimension calculation;

[0125] The visual feature F x The formula for calculating the transformation dimension is as follows:

[0126]

[0127] In the formula, This means transforming the input to dimension R. c×hw .

[0128] S222, Semantic Feature F w And the visual features F for dimensional transformation calculation x Input the similarity calculator to obtain a similarity map, use the similarity map to calculate the loss, and align semantic features and visual features.

[0129] The formula for calculating the similarity map is as follows:

[0130]

[0131] In the formula, Sim represents the similarity graph.

[0132] Similarity maps can be viewed as activations of similar visual features by aligned semantic features. Therefore, we can directly use the similarity map Sim to extract the similarity maps of the corresponding foreground and background categories based on the category index. For example, during training, we can train category C. train Foreground category, Test category C test For the background category, the opposite applies during testing. Therefore, the Sim dimension is adjusted to R. c×h×w Then, the foreground category similarity map set SimSet fg SimSet, a set of similarity graphs to background categories bg They are respectively:

[0133]

[0134] in, This indicates that categories are converted into corresponding numbers according to a predefined category order to index the corresponding similarity graph. train and n test This represents the number of corresponding training and testing categories, where n = n train +n test Then, the background similarity map is reweighted using a convolution operation:

[0135] SimSet bg =F conv (SimSet bg )∈R 1×h×w

[0136] Among them, F conv This represents the convolution operation. Then, it is concatenated with the foreground category similarity map to obtain the segmentation result image X. The concatenated result, Logits, is:

[0137]

[0138] in, For the concatenation function, use SimSet bg and SimSet fg splicing along the channel dimension; This is an upsampling function based on bilinear interpolation, used to align the size of Logits with the label Y in order to calculate the alignment loss. The alignment loss is:

[0139]

[0140] The prototype measurement and segmentation module 300 uses the aligned semantic features to measure and segment the visual features to be segmented, thereby obtaining a narrow semantic segmentation result.

[0141] In an embodiment of the present invention, the prototype metric segmentation module 300 is used to segment the visual features to be segmented using aligned semantic features to obtain a narrow semantic segmentation result. The steps are as follows:

[0142] S31. Extract the features corresponding to the foreground category from the aligned semantic features to obtain the foreground prototype set;

[0143] The formula for extracting the prototype of the foreground category is as follows:

[0144]

[0145] In the formula, Props is the set of foreground prototypes; F w ∈R n×c Let C represent the prototype of n categories, and define the set of foreground categories as C. trainThen use C train Go to F w Index (indexing process) The corresponding foreground prototype can be obtained if it is known.

[0146] S32. After expanding the resolution of the foreground prototypes in the foreground prototype set to the size of the semantic features to be segmented, concatenate them one by one with the semantic features to be segmented, and then use a convolutional structure as a metric to obtain the metric result. Upsample the metric result to the size of the label Y to obtain the final segmentation result.

[0147]

[0148] In the formula, output is the final segmentation result. It is a convolution metric tool.

[0149] Calculate the cross-entropy loss function to supervise the model's optimization process:

[0150]

[0151] In the formula, Loss seg This is the segmentation loss function.

[0152] Finally, the model's loss is obtained by weighting the alignment loss and the segmentation loss:

[0153] Loss = α * Loss ali +β*Loss seg

[0154] Where α and β are the loss weights.

[0155] The training set D needs to be prepared in advance during the training of the system of this invention. train and test set D test The two contain disjoint segmentation categories C, that is... C train and C test The number of categories included is n train and n test The model in D train After training, it will directly transfer to D. test The test is then conducted. Both the training and test sets contain labeled data. The labels in the training set are used to train the model, while the labels in the test set are only used to evaluate the test results. Furthermore, for each segmentation category c∈C... train ∪C test The pre-trained language model W will be mapped to a word vector W[c]∈R. d Where d represents the dimension. Therefore, when sampling for training on the training and test sets, the data for each sampling batch is... Where X∈R 3×H×W Represents the sampled image, Y∈R H×W W[C] represents the category label of X. train ∩C test This means that all categories defined in the dataset are converted into a set of word vectors using a language model.

[0156] This invention's pre-trained feature extraction module trains a generalized image semantic segmentation network using training categories, then transfers the feature extractor from this generalized image semantic segmentation network to a narrow semantic segmentation system. It also utilizes a pre-trained language model W to obtain semantic features. To mitigate overfitting, the transferred feature extractor parameters are frozen. The semantic-visual feature alignment module uses multi-modal information to support the segmentation task. An additional alignment loss is added using a similarity calculation method to achieve multi-modal alignment of semantic and visual features. The prototype measurement segmentation module directly uses the aligned semantic features as prototypes for visual features in the segmentation process.

[0157] The testing procedure for this system is as follows:

[0158] Step 1: Input test set C test The pre-trained feature extraction module is used to extract features from the input image;

[0159] Step 2: Obtain the aligned semantic features using the semantic-visual feature alignment module;

[0160] Step 3: Use the prototype metric segmentation module to concatenate the category semantic features with the input image features and then perform metric segmentation.

[0161] Step 4: Use the measurement results to conduct accuracy tests and visualization tests.

[0162] For example, an implementation instance based on a real dataset is provided, using two public datasets: PASCAL VOC 2012 and SBD (dataset SBD is used to provide additional segmentation labels for PASCAL VOC 2012). Since this method is used for narrow semantic segmentation, the experimental n is [not specified in the original text]. train For 19, n test The value is 1. Furthermore, PyTorch is used as the artificial intelligence model framework.

[0163] First, a generalized semantic segmentation model is defined (as shown in Table 1). The feature extractor of this model is based on ResNet (the structure is Layer in Table 1). 0 -Layer 5 The spatial information capture module is based on the scene parsing module (Layer).6 Finally, convolution is used to complete the segmentation (Layer). 7 The model is optimized using the cross-entropy loss function on the training classes. After training, layers are extracted. 0 -Layer 6 The parameters are frozen and used as the pre-trained feature extraction module of this system.

[0164] Table 1. Structure of the generalized semantic segmentation model in this implementation example.

[0165]

[0166]

[0167] Secondly, the open-source pre-trained language model used in this example is word2vec, which is trained on the Google News dataset and can output d(300)-dimensional vectors.

[0168] Next, the semantic feature mapper in the semantic-visual feature alignment module This structure is used to map the semantic features contained in word vectors to an embedded feature space that is similar to the visual features to achieve alignment, as shown in Table 2.

[0169] Table 2. Semantic Feature Mapper Structure in this Implementation Example

[0170]

[0171] Finally, in the invention introduction Both use convolutions with a kernel of 1, a stride of 1, and no padding.

[0172] During the training and testing phases, the input image resolution is 473×473 pixels (i.e., H and W are both 473), and the size of the feature map output by the feature extraction module is 1 / 16 of the input image size. During training, the training images are randomly cropped and flipped. The SGD optimizer is used to train the entire system for 2000 iterations, with a learning rate of 0.001, a batch size of 8, a momentum of 0.99, and a weight decay rate of 0.0005. In Equation 6-14, α is 1 and β is 1.

[0173] After training, the system was tested on a randomly selected narrow zero-shot segmentation task. The test results are as follows: Figure 4 As shown, the segmentation results in the image have been labeled with different colors. Experimental results demonstrate that the system trained in this example can achieve accurate semantic segmentation of narrow-sense zero-shot images.

[0174] It should be understood that, as used herein, the singular form "a" is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations of one or more of the associatedly listed items. The embodiment numbers disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0175] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A narrow-sense zero-shot semantic segmentation system based on multimodal information alignment, characterized in that, The system includes: a pre-trained feature extraction module, a semantic-visual feature alignment module, and a prototype metric segmentation module; The pre-trained feature extraction module is used to extract semantic and visual features from the image to be processed; wherein the pre-trained feature extraction module includes a feature extractor and a pre-trained language model, and the pre-trained language model is based on the bag-of-words model and is trained using text semantic information. The semantic-visual feature alignment module is used to align the extracted semantic features and visual features to obtain aligned semantic features. The process of aligning the extracted semantic and visual features to obtain aligned semantic features includes the following steps: A semantic feature mapper is used to map the set of word vectors into an embedding feature space that is similar to the visual features to achieve alignment. The alignment loss calculation process is used to calculate the cross-entropy loss between the similarity map and the visual label to achieve alignment between semantic features and visual features. The semantic features and the visual features of the training images obtained by the pre-trained feature extraction module are input into the similarity calculator to obtain a similarity map. The segmentation result map of image X is obtained using the similarity map. The loss is calculated using the segmentation result map and the label to supervise the alignment process and achieve alignment. The prototype measurement and segmentation module is used to measure and segment the visual features to be segmented using the aligned semantic features, so as to obtain the narrow semantic segmentation result. The process involves segmenting the visual features to be segmented using aligned semantic features to obtain a narrow semantic segmentation result. The steps are as follows: Extract features corresponding to the foreground category from the aligned semantic features to obtain a foreground prototype set; After expanding the resolution of the foreground prototypes in the foreground prototype set to the size of the semantic features to be segmented, they are concatenated one by one with the semantic features to be segmented. Then, a convolutional structure is used as a metric to obtain the metric result. The metric result is upsampled to the size of the label Y to obtain the narrow semantic segmentation result.

2. The narrow zero-shot semantic segmentation system based on multimodal information alignment as described in claim 1, characterized in that, The feature extractor acquisition steps include: A generalized semantic segmentation network is directly trained using labeled data from the training set. The parameters of the feature extractor in the generalized semantic segmentation network are frozen, and the feature extractor with frozen parameters is then transferred to the pre-trained feature extraction module.

3. The narrow zero-shot semantic segmentation system based on multimodal information alignment as described in claim 2, characterized in that, The generalized semantic segmentation network includes a feature extractor and a segmenter; the feature extractor is used to extract the three-dimensional depth visual features of the image; The segmenter is used to perform pixel-level classification of three-dimensional depth visual features.

4. The narrow zero-shot semantic segmentation system based on multimodal information alignment as described in claim 3, characterized in that, The generalized semantic segmentation network also includes a spatial information capture module, which is used to analyze the 3D depth visual features obtained by the feature extractor to obtain more refined features.

5. The narrow zero-shot semantic segmentation system based on multimodal information alignment as described in claim 1, characterized in that, The semantic-visual feature alignment module includes a semantic feature mapper and a similarity calculator; The semantic feature mapper is used to map semantic features to an embedded feature space that is similar to visual features to achieve alignment. The alignment loss calculation process is used to calculate the cross-entropy loss between the similarity map and the visual label to achieve alignment between semantic features and visual features. The similarity calculator is used to calculate the similarity relationship between semantic features and visual features.

6. The narrow zero-shot semantic segmentation system based on multimodal information alignment as described in claim 1, characterized in that, Input the semantic features and the visual features of the training image X obtained from the pre-trained feature extraction module into the similarity calculator to obtain a similarity map, including: Visual features are transformed and their dimensions are calculated. Semantic features and visual features for transformation dimension calculation are input into a similarity calculator to obtain a similarity map. The loss is calculated using the similarity map to align the semantic features and visual features.

7. A narrow zero-sample semantic segmentation method based on multimodal information alignment, characterized in that, The method includes: Semantic and visual features are extracted from the image to be processed; The extracted semantic features and visual features are aligned to obtain aligned semantic features; The process of aligning the extracted semantic and visual features to obtain aligned semantic features includes the following steps: A semantic feature mapper is used to map the set of word vectors into an embedding feature space that is similar to the visual features to achieve alignment. The alignment loss calculation process is used to calculate the cross-entropy loss between the similarity map and the visual label to achieve alignment between semantic features and visual features. The semantic features and the visual features of the training images obtained by the pre-trained feature extraction module are input into the similarity calculator to obtain a similarity map. The segmentation result map of image X is obtained using the similarity map. The loss is calculated using the segmentation result map and the label to supervise the alignment process and achieve alignment. The aligned semantic features are used to segment the visual features to be segmented, thus obtaining the narrow semantic segmentation result. The steps for segmenting the visual features to be segmented using aligned semantic features to obtain narrow semantic segmentation results are as follows: Extract features corresponding to the foreground category from the aligned semantic features to obtain a foreground prototype set; After expanding the resolution of the foreground prototypes in the foreground prototype set to the size of the semantic features to be segmented, they are concatenated one by one with the semantic features to be segmented. Then, a convolutional structure is used as a metric to obtain the metric result. The metric result is upsampled to the size of the label Y to obtain the narrow semantic segmentation result.

Images