Narrow zero sample semantic segmentation system and method based on multi-modal information alignment

Abstract

The invention relates to the technical field of image processing, in particular to a narrow zero sample semantic segmentation system and method based on multi-modal information alignment. The system comprises a pre-training feature extraction module, a semantic-visual feature alignment module and a prototype measurement segmentation module. The pre-training feature extraction module is used for carrying out semantic feature and visual feature extraction on the specially processed picture; wherein the pre-training feature extraction module comprises a feature extractor and a pre-training language model; the semantic-visual feature alignment module is used for carrying out alignment processing on the extracted semantic features and visual features so as to obtain aligned semantic features; and the prototype measurement segmentation module performs measurement segmentation on the visual features to be segmented by using the aligned semantic features to obtain a narrow semantic segmentation result.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a narrow zero-sample semantic segmentation system and method based on multimodal information alignment. Background Art

[0002] Image semantic segmentation: Image semantic segmentation is a key task in the field of computer vision. Its goal is to analyze and label images at the pixel level, so as to achieve fine distinction and labeling of different objects or regions in the image. This technology not only divides the image into parts, but also aims to understand the image and divide it into semantically meaningful regions. Through semantic segmentation, computers can have a deep understanding of the content of the image, providing important basic support for fields such as autonomous driving and medical image analysis.

[0003] Zero-shot image semantic segmentation: Zero-shot image semantic segmentation is a challenging task in the field of image segmentation, which is characterized by the lack of samples or category information similar to those in the test phase during the training phase. In this case, traditional learning methods cannot be directly applied due to the lack of prior training data. This segmentation method attempts to achieve semantic segmentation of images by utilizing category descriptors, attribute information or other prior knowledge, as well as techniques such as transfer learning, and can achieve satisfactory segmentation results even in the absence of training data.

[0004] Narrow semantic segmentation: Narrow semantic segmentation is a specific form of image semantic segmentation, which focuses on segmenting only specific categories of objects or regions. Unlike full-image semantic segmentation, narrow semantic segmentation focuses on accurately segmenting specific categories of objects or regions in the image. Since narrow semantic segmentation focuses on specific categories of objects or regions, it can provide more accurate segmentation results. Compared with full-image semantic segmentation, narrow segmentation can more accurately identify and segment specific targets, which helps in refined image analysis.

[0005] Training class and test class, seen class and unseen class: Zero-shot semantic segmentation is a semantic segmentation task for test classes without labeled samples. Current methods are all based on transfer learning. The model is first trained on some training classes (training classes and test classes are disjoint), and the trained model is expected to be directly transferred to the test class for testing. Therefore, the training class that appears in zero-shot semantic segmentation is the seen class used to train the model in the training phase, and the test class refers to the unseen class used to test the segmentation performance of the zero-shot model in the test phase.

[0006] The role of image semantic segmentation is to achieve pixel-level labeling and identification of different objects or regions in an image. It provides a key foundation for the field of computer vision, enabling computers to accurately understand the content of an image. Through semantic segmentation, computers can classify each pixel in an image into corresponding semantic categories, which provides important support for applications such as autonomous driving, medical image analysis, and security monitoring.

[0007] The emergence of zero-shot segmentation stems from the fact that in many practical scenarios, it is difficult to collect enough labeled samples for training. This may be due to reasons such as high data collection costs, difficulty in labeling, or insufficient data. The advantage of zero-shot segmentation is that it attempts to achieve effective semantic segmentation of images by utilizing category descriptors, attribute information, or other prior knowledge when there are no samples or category information similar to those in the test phase. This can expand the scope of application of the model so that it can still achieve satisfactory segmentation results when faced with unknown categories or samples.

[0008] However, zero-shot semantic segmentation faces problems such as low accuracy in practical applications. Due to the lack of a large amount of prior data, the model may not be able to accurately capture the characteristics of new categories or samples, resulting in a decrease in the accuracy of the segmentation results. This limits the applicability of the zero-shot segmentation method in some complex scenarios, and makes it have certain limitations when dealing with unknown categories or samples. In order to solve this technical problem, a narrow zero-shot semantic segmentation system and method based on multimodal information alignment is proposed. Summary of the invention

[0009] In order to solve the technical problems existing in the above-mentioned prior art, the present invention provides a narrow zero-sample semantic segmentation method and system based on multimodal information alignment.

[0010] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0011] In a first aspect, in an embodiment provided by the present invention, a narrow zero-shot semantic segmentation system based on multimodal information alignment is provided, the system comprising: a pre-trained feature extraction module, a semantic-visual feature alignment module and a prototype metric segmentation module;

[0012] The pre-trained feature extraction module is used to extract semantic features and visual features from the special processed pictures; wherein the pre-trained feature extraction module includes a feature extractor and a pre-trained language model, and the pre-trained language model is based on a bag-of-words model and is obtained by training using text semantic information;

[0013] The semantic-visual feature alignment module is used to align the extracted semantic features and visual features to obtain aligned semantic features;

[0014] The prototype metric segmentation module uses the aligned semantic features to perform metric segmentation on the visual features to be segmented to obtain a narrow semantic segmentation result.

[0015] As a further solution of the present invention, the feature extractor obtaining step includes:

[0016] Directly train the generalized semantic segmentation network using labeled data in the training set;

[0017] The feature extractor parameters in the generalized semantic segmentation network are frozen, and the feature extractor with frozen parameters is migrated to the pre-trained feature extraction module.

[0018] As a further solution of the present invention, the generalized semantic segmentation network includes a feature extractor and a segmenter; the feature extractor is used to extract three-dimensional deep visual features of the image; and the segmenter is used to perform pixel-level classification of the three-dimensional deep visual features.

[0019] As a further solution of the present invention, the generalized semantic segmentation network may also include a spatial information capture module, which is used to analyze the three-dimensional deep visual features obtained by the feature extractor to obtain more refined features.

[0020] As a further solution of the present invention, the semantic-visual feature alignment module includes a semantic feature mapper and a similarity calculator;

[0021] The semantic feature mapper is used to map the semantic features to an embedded feature space similar to the visual features to achieve alignment, and the alignment loss calculation process is used to calculate the cross entropy loss between the similarity map and the visual label to achieve alignment between the semantic features and the visual features;

[0022] The similarity calculator is used to calculate the similarity relationship between the semantic features and the visual features, and provide alignment support.

[0023] As a further solution of the present invention, the semantic-visual feature alignment module performs alignment processing on the extracted semantic features and visual features to obtain aligned semantic features, comprising the following steps:

[0024] Use the semantic feature mapper to map the word vector set into an embedding feature space close to the visual features to achieve alignment;

[0025] The semantic features and the visual features of the training images obtained by the pre-trained feature extraction module are input into the similarity calculator to obtain a similarity graph; the segmentation result graph of the image X is obtained by using the similarity graph, and the loss is calculated by using the segmentation result graph and the label to supervise the alignment process and achieve alignment.

[0026] As a further solution of the present invention, the semantic feature F wThe visual features of the training image X obtained by the pre-trained feature extraction module are input into the similarity calculator to obtain a similarity graph; the segmentation result graph of the image X is obtained by using the similarity graph, and the loss is calculated by using the segmentation result graph and the label to supervise the alignment process and realize alignment, including:

[0027] Visual features are converted into dimension calculations;

[0028] The semantic features and the visual features for which the transformed dimension is calculated are input into a similarity calculator to obtain a similarity graph, and the loss is calculated using the similarity graph to align the semantic features with the visual features.

[0029] As a further solution of the present invention, the prototype metric segmentation module is used to segment the visual features to be segmented using the aligned semantic features to obtain a narrow semantic segmentation result, and the steps are as follows:

[0030] Extract features corresponding to the foreground category from the aligned semantic features to obtain a foreground prototype set;

[0031] After the resolution of the foreground prototypes in the foreground prototype set is expanded to the size of the semantic features to be segmented, they are concatenated with the semantic features to be segmented one by one, and then the convolution structure is used as a measurer to obtain the measurement result, which is upsampled to the size of the label Y to obtain the narrow semantic segmentation result.

[0032] In a second aspect, in another embodiment provided by the present invention, a narrow zero-sample semantic segmentation method based on multimodal information alignment is provided, the method comprising:

[0033] Extract semantic features and visual features from specially processed images;

[0034] Aligning the extracted semantic features and visual features to obtain aligned semantic features;

[0035] It is used to segment the visual features to be segmented using the aligned semantic features to obtain narrow semantic segmentation results.

[0036] The technical solution provided by the present invention has the following beneficial effects:

[0037] The semantic-visual feature alignment module of the present invention completes the alignment of the semantic space and the visual space by calculating the similarity relationship between the word vector features and the visual features. The prototype metric segmentation module extracts multi-category prototypes, and finally uses the prototypes to segment specific categories to achieve a more accurate segmentation effect.

[0038] These and other aspects of the present invention will be more clearly understood in the following description of the embodiments. It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0040] Figure 1 The present invention is a flowchart of a narrow zero-shot semantic segmentation method based on multimodal information alignment according to an embodiment of the present invention.

[0041] Figure 2 This is a specific flow chart showing an example of a structural block diagram of a narrow zero-shot semantic segmentation system based on multimodal information alignment according to an embodiment of the present invention.

[0042] Figure 3 This is a training flowchart of a narrow zero-shot semantic segmentation system based on multimodal information alignment according to an embodiment of the present invention.

[0043] Figure 4 This is an example result diagram of a narrow zero-shot semantic segmentation system based on multimodal information alignment according to an embodiment of the present invention.

[0044] In the figure: pre-trained feature extraction module-100, semantic-visual feature alignment module-200, prototype metric segmentation module-300. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0047] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0048] Specifically, the embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0049] See also Figure 1 , Figure 1 is a flowchart of a narrow zero-shot semantic segmentation method based on multimodal information alignment provided by an embodiment of the present invention, such as Figure 1 As shown, the narrow zero-shot semantic segmentation method based on multimodal information alignment includes steps S10 to S30.

[0050] S10, extracting semantic features and visual features from the specially processed images;

[0051] In the embodiment of the present invention, S10, extracting semantic features and visual features from the specially processed image, includes obtaining visual features using a feature extractor.

[0052] The feature extractor obtaining step comprises:

[0053] Directly train the generalized semantic segmentation network using labeled data in the training set;

[0054] Freeze the feature extractor parameters in the generalized semantic segmentation network to obtain the feature extractor.

[0055] In an embodiment of the present invention, the generalized semantic segmentation network includes a feature extractor and a segmenter; the feature extractor is used to extract three-dimensional deep visual features of the image; and the segmenter is used to perform pixel-level classification on the three-dimensional deep visual features.

[0056] In an embodiment of the present invention, the generalized semantic segmentation network may further include a spatial information capture module, which is used to analyze the three-dimensional deep visual features obtained by the feature extractor to obtain more refined features to improve the segmentation performance.

[0057] The step S10, extracting semantic features and visual features from the specially processed image, comprises:

[0058] The pre-trained language model W obtained through open source pre-training maps all categories of text labels to corresponding semantic features;

[0059] The calculation formula of semantic features is as follows:

[0060] w=W(c word )

[0061] Among them, w is the word vector, w∈R 1×d , d is the dimension of the word vector, c word Represents the word corresponding to category c. For example, to obtain the word vector of the cow category, enter c wordSince there is a similarity calculation relationship between all word vectors, in the process of training the semantic-visual feature alignment module, all categories of word vectors are extracted to participate in the calculation to increase the amount of available information. The final word vector set is w all ∈R n×d , where n represents the total number of all categories.

[0062] S20, aligning the extracted semantic features and visual features to obtain aligned semantic features.

[0063] In an embodiment of the present invention, S20, aligning the extracted semantic features and visual features to obtain aligned semantic features, comprises the following steps:

[0064] S21, using a semantic feature mapper to map the semantic feature set into an embedded feature space similar to the visual feature to achieve alignment;

[0065] in,

[0066] In the formula, F w ∈R n×c represents the semantic features after mapping, where c represents the feature dimension; Represents the semantic feature mapper, which consists of fully connected layers.

[0067] S22, the semantic feature F w and the visual features F of the training image X obtained by the pre-trained feature extraction module x ∈R c ×h×w Input the similarity calculator to obtain the similarity graph; use the similarity graph to obtain the segmentation result graph of image X, use the segmentation result graph and the label to calculate the loss, supervise the alignment process, and achieve alignment.

[0068] In the embodiment of the present invention, the semantic feature F w and the visual features F of the training image X obtained by the pre-trained feature extraction module x ∈R c×h×w Input the similarity calculator to obtain a similarity graph; use the similarity graph to obtain the segmentation result graph of image X, use the segmentation result graph and the label to calculate the loss, supervise the alignment process, and achieve alignment, including:

[0069] S221, Visual Features F x Perform conversion dimension calculation;

[0070] The visual feature F x The conversion dimension calculation formula is as follows:

[0071]

[0072] In the formula, Indicates that the input dimension is converted to R c×h w .

[0073] S222, semantic feature F w and the visual features F for transformation dimension calculation x Input the similarity calculator to obtain the similarity graph, use the similarity graph to calculate the loss, and align the semantic features and visual features.

[0074] The formula for calculating the similarity graph is as follows:

[0075]

[0076] Where Sim represents the similarity graph.

[0077] The similarity graph can be regarded as the activation of the aligned semantic features to the same visual features. Therefore, the similarity graph Sim is directly used to extract the similarity graphs of the corresponding foreground category and background category according to the category index. For example, during training, the training category C train is the foreground category, the test category C test is the background category; during testing, it is the opposite. Therefore, the Sim dimension is adjusted to R c×h×w After that, the foreground category similarity graph set SimSet fg and background category similarity graph set SimSet bg They are:

[0078]

[0079]

[0080] in, Indicates that according to the predefined category order, the categories are converted into corresponding serial numbers to index the corresponding similarity graph. train and n test Indicates the number of corresponding training categories and test categories, n = n train +n test . Then re-weight the background similarity map using the convolution operation:

[0081] SimSet bg =F conv (SimSet bg )∈R 1×h×w

[0082] Among them, F convRepresents the convolution operation. Then, it is concatenated with the foreground category similarity map as the segmentation result map of image X. The concatenated result Logits is:

[0083]

[0084] in, For the splicing function, SimSet bg and SimSet fg Splicing in the channel dimension; It is an upsampling function based on bilinear interpolation, which is used to align the size of Logits with the label Y in order to calculate the alignment loss. The alignment loss is:

[0085]

[0086] S30, segmenting the visual features to be segmented by using the aligned semantic features to obtain a narrow semantic segmentation result.

[0087] The S30, for segmenting the visual features to be segmented using the aligned semantic features to obtain a narrow semantic segmentation result, comprises the following steps:

[0088] S31, extracting features corresponding to the foreground category from the aligned semantic features to obtain a foreground prototype set;

[0089] S32. After expanding the resolution of the foreground prototypes in the foreground prototype set to the size of the semantic features to be segmented, they are concatenated with the semantic features to be segmented one by one, and then the convolution structure is used as a measurer to obtain the measurement result, and the measurement result is upsampled to the size of the label Y to obtain the narrow semantic segmentation result.

[0090]

[0091] Where output is the final segmentation result, It is a convolution metric tool.

[0092] Calculate the cross entropy loss function to supervise the optimization process of the model:

[0093]

[0094] In the formula, Loss seg is the segmentation loss function.

[0095] Finally, the model loss is obtained by weighting the alignment loss and the segmentation loss:

[0096] Loss = α * Loss ali +β*Loss seg

[0097] Among them, α and β are loss weights.

[0098] It should be understood that, although described in a certain order, these steps are not necessarily performed in sequence in the above order. Unless there is clear explanation in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other orders. Moreover, a part of the steps of the present embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a part of the steps or stages in other steps or other steps.

[0099] In one embodiment, see Figure 2 As shown, in an embodiment of the present invention, a narrow zero-sample semantic segmentation system based on multimodal information alignment is also provided, which includes a pre-trained feature extraction module 100, a semantic-visual feature alignment module 200 and a prototype metric segmentation module 300.

[0100] The pre-trained feature extraction module 100 is used to extract semantic features and visual features of the specially processed image; wherein the pre-trained feature extraction module 100 includes a feature extractor and a pre-trained language model W, and the pre-trained language model W is based on a bag-of-words model and is trained using text semantic information.

[0101] The feature extractor is used to extract visual features, and the pre-trained language model W is used to extract semantic features.

[0102] It should be noted that the pre-trained language model W extracts semantic features, including:

[0103] All categories of text labels are mapped to corresponding semantic features through the open source pre-trained language model W; the calculation formula of the semantic feature is as follows:

[0104] w=W(c word )

[0105] Among them, w is the semantic feature, w∈R 1×d , d is the dimension of the word vector, c word Represents the word corresponding to category c. For example, to obtain the word vector of the cow category, enter c word Since there is a similarity calculation relationship between all word vectors, in the process of training the semantic-visual feature alignment module, all categories of word vectors are extracted to participate in the calculation to increase the amount of available information. The final word vector set is w all ∈R n×d , where n represents the total number of all categories.

[0106] It should be noted that there are categories of data such as "cat, dog" in the dataset for segmentation. The text labels of the categories are "cat, dog" and are input into the language model W to extract the word vector w.

[0107] It should be noted that since the zero-shot segmentation task cannot use labeled samples of a specific class during testing to train the feature extraction network, traditional deep learning-based methods often directly use the feature extractor of the classification network pre-trained on the ImageNet dataset as the feature extractor of the zero-shot segmentation network. There are two problems with the above approach. First, during the pre-training process on the ImageNet dataset, there is no participation of the test class, which will make the parameters of the feature extractor biased towards the training class, causing the model to overfit to the training class; second, when training on the ImageNet dataset, the feature extractor ResNet or VGG plus the classifier method is often used to perform it as a classification task. However, the features that the classification task focuses on are inconsistent with the segmentation task, because the semantic segmentation task is a pixel-level classification task, which requires more refined and sufficient three-dimensional features as the input of the segmentor. Directly using the feature extractor of the classification task will reduce the segmentation performance.

[0108] In view of the above problems, this system does not directly use the feature extractor of the classification network based on the classification task, but first uses the samples of the training class and their labels to train a generalized semantic segmentation network, and then directly migrates the feature extractor to the narrow zero-shot segmentation system of this system for use. First, the feature extractor trained based on the semantic segmentation task is obviously more suitable for the downstream semantic segmentation task. Second, the feature extractor migrated to the narrow zero-shot segmentation system will have its parameters frozen to alleviate overfitting.

[0109] Specifically, the feature extractor obtaining step includes:

[0110] Directly train the generalized semantic segmentation network using labeled data in the training set;

[0111] The feature extractor parameters in the generalized semantic segmentation network are frozen, and the feature extractor with frozen parameters is migrated to the pre-trained feature extraction module 100.

[0112] In an embodiment of the present invention, the generalized semantic segmentation network includes a feature extractor and a segmenter; the feature extractor is used to extract three-dimensional deep visual features of the image; and the segmenter is used to perform pixel-level classification on the three-dimensional deep visual features.

[0113] In an embodiment of the present invention, the generalized semantic segmentation network may further include a spatial information capture module, which is used to analyze the three-dimensional deep visual features obtained by the feature extractor to obtain more refined features to improve the segmentation performance.

[0114] The semantic-visual feature alignment module 200 is used to align the extracted semantic features and visual features to obtain aligned semantic features.

[0115] In an embodiment of the present invention, the semantic-visual feature alignment module 200 includes a semantic feature mapper and a similarity calculator. The semantic-visual feature alignment module 200 in the present invention aims to construct the relationship between the training class and the test class through word vectors, so that the model can directly use the embedded word vectors of the test class as the segmentation information of the test class to participate in the zero-sample semantic segmentation process.

[0116] The semantic feature mapper is used to map the semantic features to an embedded feature space close to the visual features to achieve alignment, and the alignment loss calculation process is used to calculate the cross entropy loss between the similarity map and the visual label to achieve alignment between the semantic features and the visual features.

[0117] The similarity calculator is used to calculate the similarity relationship between the semantic features and the visual features, and provide alignment support.

[0118] The semantic-visual feature alignment module 200 of the present invention maps the word vector to an embedded feature space close to the visual feature, and then calculates and aligns the embedded word vector and the visual feature using a similarity calculation tool. Thus, through the training phase, an alignment relationship between the training category visual features and the training category word vector is established, and the segmenter can use this alignment relationship to support the segmentation process.

[0119] The semantic-visual feature alignment module 200 performs alignment processing on the extracted semantic features and visual features to obtain aligned semantic features, including the following steps:

[0120] S21, using a semantic feature mapper to map the semantic feature set into an embedded feature space similar to the visual feature to achieve alignment;

[0121] in,

[0122] In the formula, F w ∈R n×c represents the semantic features after mapping, where c represents the feature dimension; Represents the semantic feature mapper, which consists of fully connected layers.

[0123] S22, the semantic feature F wand the visual features F of the training image X obtained by the pre-trained feature extraction module x ∈R c ×h×w Input the similarity calculator to obtain the similarity graph; use the similarity graph to obtain the segmentation result graph of image X, use the segmentation result graph and the label to calculate the loss, supervise the alignment process, and achieve alignment.

[0124] In the embodiment of the present invention, the semantic feature F w and the visual features F of the training image X obtained by the pre-trained feature extraction module x ∈R c×h×w Input the similarity calculator to obtain a similarity graph; use the similarity graph to obtain the segmentation result graph of image X, use the segmentation result graph and the label to calculate the loss, supervise the alignment process, and achieve alignment, including:

[0125] S221, Visual Features F x Perform conversion dimension calculation;

[0126] The visual feature F x The conversion dimension calculation formula is as follows:

[0127]

[0128] In the formula, Indicates that the input dimension is converted to R c×h w .

[0129] S222, semantic feature F w and the visual features F for transformation dimension calculation x Input the similarity calculator to obtain the similarity graph, use the similarity graph to calculate the loss, and align the semantic features and visual features.

[0130] The formula for calculating the similarity graph is as follows:

[0131]

[0132] Where Sim represents the similarity graph.

[0133] The similarity graph can be regarded as the activation of the aligned semantic features to the same visual features. Therefore, the similarity graph Sim is directly used to extract the similarity graphs of the corresponding foreground category and background category according to the category index. For example, during training, the training category C train is the foreground category, the test category C test is the background category; during testing, it is the opposite. Therefore, the Sim dimension is adjusted to R c×h×w After that, the foreground category similarity graph set SimSet fg and background category similarity graph set SimSetbg They are:

[0134]

[0135]

[0136] in, Indicates that according to the predefined category order, the categories are converted into corresponding serial numbers to index the corresponding similarity graph. train and n test Indicates the number of corresponding training categories and test categories, n = n train +n test . Then re-weight the background similarity map using the convolution operation:

[0137] SimSet bg =F conv (SimSet bg )∈R 1×h×w

[0138] Among them, F conv Represents the convolution operation. Then, it is concatenated with the foreground category similarity map as the segmentation result map of image X. The concatenated result Logits is:

[0139]

[0140] in, For the splicing function, SimSet bg and SimSet fg Splicing in the channel dimension; It is an upsampling function based on bilinear interpolation, which is used to align the size of Logits with the label Y in order to calculate the alignment loss. The alignment loss is:

[0141]

[0142] The prototype metric segmentation module 300 uses the aligned semantic features to perform metric segmentation on the visual features to be segmented to obtain a narrow semantic segmentation result.

[0143] In an embodiment of the present invention, the prototype metric segmentation module 300 is used to segment the visual features to be segmented using the aligned semantic features to obtain a narrow semantic segmentation result, and the steps are as follows:

[0144] S31, extracting features corresponding to the foreground category from the aligned semantic features to obtain a foreground prototype set;

[0145] Among them, the formula for extracting the prototype of the foreground category is:

[0146]

[0147] Where Props is the foreground prototype set; F w ∈R n×c , represents the prototype of n categories, and defines the foreground category set as C train , then use C train Go to F w Indexing (indexing process is known) the corresponding foreground prototype can be obtained

[0148] S32. After expanding the resolution of the foreground prototypes in the foreground prototype set to the size of the semantic features to be segmented, they are concatenated with the semantic features to be segmented one by one, and then the convolution structure is used as a measurer to obtain the measurement result, and the measurement result is upsampled to the size of the label Y to obtain the final segmentation result.

[0149]

[0150] Where output is the final segmentation result, It is a convolution metric tool.

[0151] Calculate the cross entropy loss function to supervise the optimization process of the model:

[0152]

[0153] In the formula, Loss seg is the segmentation loss function.

[0154] Finally, the model loss is obtained by weighting the alignment loss and the segmentation loss:

[0155] Loss = α * Loss ali +β*Loss seg

[0156] Among them, α and β are loss weights.

[0157] When training the system of the present invention, it is necessary to prepare the training set D in advance. train and the test set D test , the segmentation categories C contained in the two do not intersect, that is, C train and C test The number of categories included is n train and n test The model is in D train After training, it will be directly transferred to D test The data in both the training set and the test set are labeled. The labels in the training set can be used to train the model, but the labels in the test set can only be used to test the test results. In addition, each segmentation category c∈C train∪C test It will be mapped to a word vector W[c]∈R through the pre-trained language model W d , where d represents the dimension. Therefore, when sampling training on the training set and test set, the data of each sampling batch is where X∈R 3×H×W Represents the sampled image, Y∈R H×W represents the category label of X, W[C train ∩C test ] means that all categories defined in the dataset are converted into a set of word vectors through the language model.

[0158] The pre-trained feature extraction module of the present invention uses training categories to train a generalized image semantic segmentation network, and then migrates the feature extractor of the generalized image semantic segmentation network to the narrow semantic segmentation system for use, and also uses the pre-trained language model W to obtain semantic features. In order to alleviate the overfitting phenomenon, the migrated feature extractor parameters are frozen. The semantic-visual feature alignment module uses information from multiple modalities to support the segmentation task. Using the method of calculating similarity, an additional alignment loss is added to achieve multi-modal information alignment of semantic features and visual features; the prototype metric segmentation module uses the aligned semantic features directly as visual feature prototypes to participate in segmentation.

[0159] The testing process of this system is as follows:

[0160] Step 1: Input test set C test , extract the features of the input image using the pre-trained feature extraction module;

[0161] Step 2: Using the semantic-visual feature alignment module to obtain aligned semantic features;

[0162] Step 3: Use the prototype metric segmentation module to take the category semantic features as prototypes, and then perform metric segmentation after splicing with the input image features;

[0163] Step 4: Use the measurement results to perform accuracy tests and visualization tests.

[0164] For example, an implementation example based on an actual dataset is provided. The example uses two public datasets: PASCAL VOC 2012 and SBD (the SBD dataset is used to provide additional segmentation labels for PASCAL VOC 2012). Since this method is used for narrow semantic segmentation, when the total number of categories in PASCAL VOC 2012 is 20, the number of n in the experiment is train is 19, n test is 1. In addition, pytorch is used as the artificial intelligence model framework.

[0165] First, a generalized semantic segmentation model is defined (as shown in Table 1). The feature extractor of this model is based on ResNet (the structure is the Layer 0 -Layer 5 ), the spatial information capture module is based on the scene analysis module (Layer 6 ), and finally use convolution to complete the segmentation (Layer 7 ). Use the cross entropy loss function on the training category to optimize the model. After training, extract the Layer 0 -Layer 6 And freeze the parameters as the pre-trained feature extraction module of this system.

[0166] Table 1 Generalized semantic segmentation model structure in this implementation example

[0167]

[0168]

[0169] Secondly, the open source pre-trained language model used in this example is word2vec, which is trained on the Google News dataset and can output a d(300)-dimensional vector.

[0170] Next, the semantic feature mapper in the semantic-visual feature alignment module It is used to map the semantic features contained in the word vector to an embedded feature space similar to the visual features to achieve alignment. The structure is shown in Table 2.

[0171] Table 2 Semantic feature mapper structure in this implementation example

[0172]

[0173] Finally, the invention introduction Both use convolution with kernel size 1, stride 1, and no padding.

[0174] During the training and testing phases, the input image resolution is 473×473 pixels (i.e., H and W are both 473), and the size of the feature map output by the feature extraction module is 1 / 16 of the input image size. During training, the training images are randomly cropped and flipped. The SGD optimizer is used to train the entire system for 2000 iterations, with a learning rate of 0.001, a batch size of 8, a momentum of 0.99, and a weight decay rate of 0.0005. α and β in Formula 6-14 are 1 and 1, respectively.

[0175] After training, the system is tested on a randomly selected narrow zero-shot segmentation task. The test results are as follows: Figure 4As shown in the figure, the segmentation results have been marked with different colors in the picture. The experimental results show that the system trained in this example can achieve accurate narrow zero-shot image semantic segmentation.

[0176] It should be understood that, as used herein, the singular form "a" or "an" is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations of one or more of the items listed in association. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments.

[0177] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the embodiments of the present invention as above, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A narrow zero-shot semantic segmentation system based on multimodal information alignment, characterized in that: The system includes: a pre-trained feature extraction module, a semantic-visual feature alignment module and a prototype metric segmentation module; The pre-trained feature extraction module is used to extract semantic features and visual features from the special processed images; wherein the pre-trained feature extraction module includes a feature extractor and a pre-trained language model, and the pre-trained language model is based on a bag-of-words model and is obtained by training using text semantic information; The semantic-visual feature alignment module is used to align the extracted semantic features and visual features to obtain aligned semantic features; The prototype metric segmentation module uses the aligned semantic features to perform metric segmentation on the visual features to be segmented to obtain a narrow semantic segmentation result.

2. The narrow zero-shot semantic segmentation system based on multimodal information alignment according to claim 1, characterized in that: The feature extractor obtaining step comprises: Directly train the generalized semantic segmentation network using labeled data in the training set; The feature extractor parameters in the generalized semantic segmentation network are frozen, and the feature extractor with frozen parameters is migrated to the pre-trained feature extraction module.

3. The narrow zero-shot semantic segmentation system based on multimodal information alignment as claimed in claim 2, characterized in that: The generalized semantic segmentation network includes a feature extractor and a segmenter; the feature extractor is used to extract three-dimensional deep visual features of the image; The segmenter is used to classify three-dimensional deep visual features at the pixel level.

4. The narrow zero-shot semantic segmentation system based on multimodal information alignment according to claim 3, characterized in that: The generalized semantic segmentation network may also include a spatial information capture module, which is used to analyze the three-dimensional deep visual features obtained by the feature extractor to obtain more refined features.

5. The narrow zero-shot semantic segmentation system based on multimodal information alignment according to claim 1, characterized in that: The semantic-visual feature alignment module includes a semantic feature mapper and a similarity calculator; The semantic feature mapper is used to map the semantic features to an embedded feature space similar to the visual features to achieve alignment, and the alignment loss calculation process is used to calculate the cross entropy loss between the similarity map and the visual label to achieve alignment between the semantic features and the visual features; The similarity calculator is used to calculate the similarity relationship between the semantic features and the visual features.

6. The narrow zero-shot semantic segmentation system based on multimodal information alignment according to claim 5, characterized in that: The semantic-visual feature alignment module performs alignment processing on the extracted semantic features and visual features to obtain aligned semantic features, including the following steps: The semantic feature mapper is used to map the word vector set into an embedding feature space close to the visual features to achieve alignment. The alignment loss calculation process is used to calculate the cross entropy loss between the similarity map and the visual label to achieve alignment between the semantic features and the visual features. The semantic features and the visual features of the training images obtained by the pre-trained feature extraction module are input into the similarity calculator to obtain a similarity graph, and the segmentation result graph of the image X is obtained by using the similarity graph. The loss is calculated using the segmentation result graph and the label, and the alignment process is supervised to achieve alignment.

7. The narrow zero-shot semantic segmentation system based on multimodal information alignment according to claim 1, characterized in that: The semantic features and the visual features of the training image X obtained by the pre-trained feature extraction module are input into a similarity calculator to obtain a similarity graph, including: Visual features are converted into dimension calculations; The semantic features and the visual features for which the transformed dimension is calculated are input into a similarity calculator to obtain a similarity graph, and the loss is calculated using the similarity graph to align the semantic features with the visual features.

8. The narrow zero-shot semantic segmentation system based on multimodal information alignment according to claim 7, characterized in that: The prototype metric segmentation module is used to segment the visual features to be segmented using the aligned semantic features to obtain the narrow semantic segmentation results, and the steps are as follows: Extract features corresponding to the foreground category from the aligned semantic features to obtain a foreground prototype set; After the resolution of the foreground prototypes in the foreground prototype set is expanded to the size of the semantic features to be segmented, they are concatenated with the semantic features to be segmented one by one, and then the convolution structure is used as a measurer to obtain the measurement result, which is upsampled to the size of the label Y to obtain the narrow semantic segmentation result.

9. A narrow zero-shot semantic segmentation method based on multimodal information alignment, characterized in that: The method includes: Extract semantic features and visual features from specially processed images; Aligning the extracted semantic features and visual features to obtain aligned semantic features; It is used to segment the visual features to be segmented using the aligned semantic features to obtain narrow semantic segmentation results.

Images