Transfomer-based mapping type zero sample image semantic segmentation method and system

Abstract

The invention provides a mapping type zero sample image semantic segmentation method and system based on Transfomer, and relates to the field of image processing. The method comprises the following steps: acquiring to-be-segmented image data and a to-be-segmented image data label; based on the to-be-segmented image data, extracting visual features through an image feature extractor; based on the to-be-segmented image data labels, text features are extracted through a text feature extractor; adjusting the number of text feature channels based on a semantic mapper to obtain adjusted text features; performing mapping and alignment on the adjusted text features and the adjusted visual features through a transform model to obtain alignment features; and segmenting the alignment features through a lightweight divider, and identifying segmentation categories. The zero-sample image semantic segmentation method and device are used in the mapping type zero-sample image semantic segmentation process, and the technical problem that an existing zero-sample image semantic segmentation technology is not high in precision in practical application is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a mapping type zero-shot image semantic segmentation method and system based on Transfomer. BACKGROUND

[0002] Traditional image semantic segmentation methods are mainly based on manually designed features and pixel-level classification techniques. Although these methods can achieve image semantic segmentation to some extent, they have the shortcomings of large amount of labeled data requirement and limited model generalization ability. To break through this dilemma, zero-shot image semantic segmentation methods have emerged and received widespread attention. The goal of zero-shot image semantic segmentation is to complete accurate semantic segmentation of unknown classes without relying on additional class information. The core challenge of zero-shot image semantic segmentation is how to accurately segment without any unseen class samples or label information, which requires the model to have strong generalization ability and effective representation ability for unknown classes. Mapping method is one of the mainstream methods to solve this problem. The core idea of mapping method is to learn the mapping relationship between feature space and label space, and then realize the segmentation of unknown classes. However, this method has insufficient mapping processing ability for complex text information and visual information, resulting in low accuracy in practical application. Therefore, there is an urgent need for a method to solve the technical problem of low accuracy of existing zero-shot image semantic segmentation technology in practical application. SUMMARY

[0003] The present application provides a mapping type zero-shot image semantic segmentation method and system based on Transfomer, which solves the technical problem of low accuracy of existing zero-shot image semantic segmentation technology in practical application.

[0004] To achieve the above purpose, the technical scheme adopted by the present application is as follows: In a first aspect, a mapping type zero-shot image semantic segmentation method based on Transfomer is provided, comprising: obtaining image data to be segmented and label of the image data to be segmented; extracting visual features based on the image data to be segmented through an image feature extractor; extracting text features based on the label of the image data to be segmented through a text feature extractor; adjusting the number of text feature channels based on a semantic mapper to obtain adjusted text features; mapping and aligning the adjusted text features and the visual features through a transformer model to obtain aligned features; segmenting the aligned features through a lightweight segmenter and identifying the segmentation class.

[0005] In a possible implementation manner of the first aspect, before the obtaining of the image data to be segmented and the label of the image data to be segmented, the method further includes: constructing a training set and a test set based on the segmented image data and the label of the segmented image data; training the image feature extractor based on the training set to obtain the trained image feature extractor; and training the trained image feature extractor, the text feature extractor, the semantic mapper, and the transformer model based on the training set and the test set.

[0006] In a possible implementation manner of the first aspect, the training of the image feature extractor based on the training set to obtain the trained image feature extractor includes: constructing the training set based on the segmented image data and the label of the segmented image data; training a neural network based on the training set to obtain a trained neural network; removing an output layer related to a prediction task of the trained neural network, and only retaining a network main body part used for feature representation learning as the trained image feature extractor.

[0007] In a possible implementation manner of the first aspect, the extracting of the text feature based on the label of the image data to be segmented by the text feature extractor includes: extracting the text feature based on the label of the image data to be segmented by a text feature extractor constructed by a word2vec model.

[0008] In a possible implementation manner of the first aspect, the text feature satisfies the following formula:

[0009] wherein, denotes all category label names, denotes the text feature extractor, denotes a number of categories in the test set, and d denotes a number of text feature channels.

[0010] In a possible implementation manner of the first aspect, the mapping and alignment of the adjusted text feature and the visual feature by the transformer model to obtain the aligned feature includes: mapping the adjusted text feature and the visual feature to the same space to obtain a first text feature and a first visual feature; constructing a cross-modal attention input by taking the first visual feature as a query vector, and taking the first text feature as a key vector and a value vector; calculating a similarity between the query vector and the key vector in a plurality of subspaces in parallel based on a multi-head attention mechanism to obtain corresponding attention weights; generating a multi-head alignment feature by weighting and fusing the value vector according to the attention weights; concatenating and linearly mapping the multi-head alignment feature to make a number of channels of the multi-head alignment feature consistent with the first visual feature to obtain a fusion feature; performing residual connection on the fusion feature and the visual feature, and performing normalization processing to obtain the aligned feature.

[0011] With reference to the first aspect, in a possible implementation manner, the first visual feature satisfies the following formula: The following formula is satisfied:

[0012] wherein, denotes a dimension transformation operation, is a transformer attention head number, is a product of a feature map height and a feature map width, is a feature channel number, is a visual feature; With reference to the first aspect, in a possible implementation manner, the first text feature satisfies the following formula: The following formula is satisfied:

[0013] wherein, denotes a dimension transformation operation, is a transformer attention head number, is a test set category number, is a feature channel number, is an adjusted text feature; With reference to the first aspect, in a possible implementation manner, the multi-head alignment feature satisfies the following formula:

[0014] wherein, is a normalization operation, Q is a query vector, K is a key vector, and V is a value vector, is a transformer attention head number, is a product of a feature map height and a feature map width, is a feature channel number, is a matrix multiplication operation.

[0015] With reference to the first aspect, in a possible implementation manner, before the image data to be segmented and the image data to be segmented label are acquired, the method further comprises training the lightweight segmenter, including: taking the alignment feature as input, predicting a pixel-level segmentation category through the lightweight segmenter; performing up-sampling processing on the pixel-level segmentation category to obtain a prediction result consistent with the size of the original label map; using a cross-entropy loss function, calculating a segmentation loss based on the prediction result consistent with the size of the original label map and the data label; based on the segmentation loss, optimizing the parameters of the lightweight segmenter through a back propagation manner, and continuously iterating until the model converges to complete the training of the segmenter.

[0016] In combination with the first aspect, in a possible implementation, during the training of the lightweight segmenter, the image feature extractor parameters, the text feature extractor parameters, the semantic mapper parameters, and the transformer model parameters are frozen.

[0017] In combination with the first aspect, in a possible implementation, the segmentation loss The following formula is satisfied:

[0018] wherein, The sum of the number of training set categories and the number of test set categories is represented by The prediction result consistent with the original label map size is represented by The alignment feature is represented by i, which represents the index.

[0019] The second aspect provides a Transformer-based mapping zero-shot image semantic segmentation system, comprising a data acquisition device and an electronic device. The data acquisition device is configured to acquire image data to be segmented and a label of the image data to be segmented. The electronic device is configured to extract visual features from the image data to be segmented by an image feature extractor, extract text features from the label of the image data to be segmented by a text feature extractor, adjust the number of text feature channels based on a semantic mapper to obtain adjusted text features, map and align the adjusted text features and the visual features by a transformer model to obtain alignment features, and segment the alignment features by a lightweight segmenter and identify the segmentation categories.

[0020] The present application provides a Transformer-based mapping zero-shot image semantic segmentation method and system. By acquiring image data to be segmented and its category label, visual features and text semantic features are extracted respectively, and the number of text feature channels is adjusted by a semantic mapper to make the visual features and the text features more matched in structure. Then, the mapping and alignment between the text features and the visual features are realized by a Transformer model, so that the text semantic information can directly participate in the pixel-level segmentation decision. Finally, the segmentation category recognition is completed by combining a lightweight segmenter. Without generating unseen category samples, the error caused by unstable sample quality in the generative method is effectively avoided, the modeling ability for complex semantic relationships is enhanced, the difficulty of aligning text and visual features is reduced, and the overfitting problem of seen categories in zero-shot segmentation is alleviated, thereby improving the accuracy, stability, and generalization performance of zero-shot image semantic segmentation, and solving the technical problem of low precision in the actual application of existing zero-shot image semantic segmentation technology.

[0021] It should be understood that the description of technical features, technical solutions, advantages or similar language in this application does not imply that all features and advantages can be achieved in any single embodiment. On the contrary, it can be understood that the description of a feature or advantage means that the specific technical feature, technical solution or advantage is included in at least one embodiment. Therefore, the description of technical features, technical solutions or advantages in this specification does not necessarily refer to the same embodiment. Further, the technical features, technical solutions and advantages described in this embodiment can be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or advantages of a particular embodiment. In other embodiments, additional technical features and advantages can be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A system architecture diagram of a mapping-based zero-shot image semantic segmentation system based on a Transfomer is provided for embodiments of the application. Figure 2 A flowchart of a mapping-based zero-shot image semantic segmentation method based on a Transfomer is provided for embodiments of the application. Figure 3 A segmentation result diagram of another mapping-based zero-shot image semantic segmentation method based on a Transfomer is provided for embodiments of the application. Figure 4 A flowchart of another mapping-based zero-shot image semantic segmentation method based on a Transfomer is provided for embodiments of the application. Figure 5 A flowchart of another mapping-based zero-shot image semantic segmentation method based on a Transfomer is provided for embodiments of the application. DETAILED DESCRIPTION

[0023] In the description of the present application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this document is only a description of the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In addition, "at least one" means one or more, and "multiple" means two or more. "First", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. do not necessarily mean different.

[0024] It should be noted that in the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design solution described as "exemplary" or "for example" in the present application should not be construed as being more advantageous or superior to other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present concepts in a particular manner. The present application is directed to the following technical solutions.

[0025] The mapping-based zero-shot image semantic segmentation method based on the transformer provided by the embodiments of the present application can be applied to a mapping-based zero-shot image semantic segmentation system based on the transformer, as shown in Figure 1 The system includes a data acquisition device 101 and an electronic device 102, as shown in Figure 1

[0026] The data acquisition device 101 is configured to acquire image data to be segmented and a label of the image data to be segmented. The electronic device 102 is configured to extract visual features from the image data to be segmented by an image feature extractor; extract text features from the label of the image data to be segmented by a text feature extractor; adjust the number of text feature channels based on a semantic mapper to obtain adjusted text features; map and align the adjusted text features and the visual features by a transformer model to obtain aligned features; and segment the aligned features by a lightweight segmenter and identify a segmentation category.

[0027] To solve the technical problem of low precision in the actual application of the existing zero-shot image semantic segmentation technology in the prior art, the embodiments of the present application provide a mapping-based zero-shot image semantic segmentation method based on the transformer, which includes: acquiring image data to be segmented and a label of the image data to be segmented; extracting visual features from the image data to be segmented by an image feature extractor; extracting text features from the label of the image data to be segmented by a text feature extractor; adjusting the number of text feature channels based on a semantic mapper to obtain adjusted text features; mapping and aligning the adjusted text features and the visual features by a transformer model to obtain aligned features; and segmenting the aligned features by a lightweight segmenter and identifying a segmentation category.

[0028] Figure 2 A flowchart of the mapping-based zero-shot image semantic segmentation method based on the transformer provided by the embodiments of the present application is shown in Figure 2 The method includes the following steps. S201, acquiring image data to be segmented and a label of the image data to be segmented.

[0029] ​The to-be-segmented image data is input image data that needs to be subjected to semantic segmentation processing, and the to-be-segmented image data label is a set of category names corresponding to the image data, used to indicate semantic category information that needs to participate in segmentation. The label can include a seen category label and an unseen category label.

[0030] In a possible implementation, one or more to-be-segmented images are acquired from an image acquisition device or an image storage medium, and a set of category label names corresponding to the images is acquired or preset at the same time. The category labels are input into the system in the form of text, for subsequent text feature extraction operations.

[0031] This step ensures the consistency of the format of the input data of the subsequent modules and the consistency of the labels, thereby providing a reliable data basis for stable training of a visual feature extractor and evaluation of zero-shot performance, and reducing the influence of data preprocessing differences on model convergence and generalization.

[0032] S202, extracting visual features based on the to-be-segmented image data through an image feature extractor.

[0033] The image feature extractor is a deep neural network module that has been trained and has fixed parameters, and is used to extract visual feature representations with semantic discrimination ability from input images.

[0034] In a possible implementation, the electronic device inputs the to-be-segmented image into the image feature extractor whose parameters have been pre-trained and fixed, performs multi-layer convolution and feature transformation processing on the image, and outputs a set of visual feature maps containing spatial position information and semantic information.

[0035] It should be noted that, in the inference stage, only forward inference calculation is performed, and no backward propagation or parameter updating operation is involved, thereby ensuring the stability and calculation efficiency of the system in an actual deployment environment.

[0036] As an example, in the embodiment of the present application, the image feature The following formula is satisfied:

[0037] X is input image data, is an image feature extractor, is a feature map height, is a feature map width, is a feature channel.

[0038] Based on the above steps, this step can provide a visual representation rich in semantic and spatial position information, so that the subsequent text and visual alignment have sufficient visual context support, thereby improving the accuracy of cross-modal mapping and the effect of pixel-level segmentation.

[0039] S203, extracting text features based on the image data to be segmented label through the text feature extractor.

[0040] The text feature extractor is a pre-trained language model, which is used to convert the category label name into a vectorized semantic representation.

[0041] In a possible implementation, the text features are extracted based on the image data to be segmented label through a text feature extractor constructed by a word2vec model.

[0042] As an example, in the embodiment of the present application, the text features satisfy the following formula:

[0043] wherein, represents all category label names, represents the text feature extractor, represents the number of categories in the test set, and d represents the number of text feature channels.

[0044] It should be noted that the text feature extractor only performs semantic encoding function in the inference stage, and does not involve retraining or parameter fine-tuning operation of the language model, so as to ensure the consistency of the semantic representation.

[0045] Based on the above steps, this step enables the semantic category information to participate in subsequent processing in the form of a vector, so that the unseen categories can participate in segmentation inference without visual samples.

[0046] S204, adjusting the number of text feature channels based on the semantic mapper to obtain adjusted text features.

[0047] The semantic mapper is a pre-trained feature mapping network module, which is used to adjust the channel structure of the text features to a consistent or matched number of channels with the visual features.

[0048] In a possible implementation, the text features are input into the semantic mapper, and after linear transformation and nonlinear activation processing, the adjusted text features with a consistent number of channels with the visual features are output.

[0049] It should be noted that the parameters of the semantic mapper remain fixed in the use stage, and its role is to reduce the difference between the text features and the visual features in numerical distribution and dimension structure, so as to reduce the difficulty of cross-modal alignment.

[0050] As an example, in the embodiment of the present application, the adjusted text features satisfy the following formula:

[0051] wherein, Here, w is the text feature adjuster. c represents the number of categories in the test set, and c represents the number of channels.

[0052] Based on the above steps, this step provides a structurally compatible multimodal input for subsequent Transformer-based feature mapping and alignment operations.

[0053] S205. The adjusted text features and visual features are mapped and aligned using the transformer model to obtain aligned features.

[0054] In one possible implementation, the adjusted text features and visual features are mapped to the same space to obtain first text features and first visual features. The first visual features are used as query vectors, and the first text features are used as key vectors and value vectors to construct cross-modal attention inputs. Based on a multi-head attention mechanism, the similarity between the query vector and the key vector is calculated in parallel in multiple subspaces to obtain the corresponding attention weights. The value vectors are weighted and fused according to the attention weights to generate multi-head aligned features. The multi-head aligned features are concatenated and linearly mapped to make their number of channels consistent with the first visual features to obtain fused features. The fused features are residually connected with the visual features and normalized to obtain aligned features.

[0055] It should be noted that this step only performs forward attention calculation during the inference phase and does not involve training or updating the Transformer model parameters, thereby ensuring the repeatability and determinism of the inference process.

[0056] Based on the above steps, this step realizes spatial-level guidance of visual features by textual semantic information, enabling semantic priors of unseen categories to directly participate in pixel-level semantic segmentation decisions.

[0057] S206. Segment the aligned features using a lightweight segmenter and identify the segmentation category.

[0058] The lightweight segmenter is a pre-trained pixel-level classification module used to map aligned features to the category prediction result corresponding to each pixel.

[0059] In one possible implementation, aligned features are input into a lightweight segmenter, which obtains pixel-level prediction results for each category through convolution and probability normalization operations, and outputs the final semantic segmentation map.

[0060] It should be noted that lightweight segmenters have a simple structure and a small number of parameters, which helps to reduce computational overhead during the inference phase and avoid bias of the model towards known categories.

[0061] As an example, in an embodiment of this application, Figure 3The segmentation prediction results provided for the application of actual data in the embodiments of this application, such as Figure 3 As shown, for the four groups of images that need to be segmented, the baseline model and the method of this embodiment are used for prediction respectively. The prediction results are compared with the label results, and the prediction results of this embodiment are more accurate.

[0062] Based on the above steps, this step completes the semantic segmentation of seen and unseen categories during the usage phase, realizing the practical application effect of zero-shot image semantic segmentation.

[0063] This application's embodiments acquire the image data to be segmented and its corresponding semantic category labels, extract stable visual features and text semantic features respectively, and keep the parameters of each module fixed during the usage phase, achieving zero-shot semantic segmentation without introducing pixel-level annotation or retraining the model. A semantic mapper adjusts the number of text feature channels to make the text features more structurally and numerically matched with the visual features, reducing the difficulty of cross-modal feature alignment. Then, a Transformer-based mapping alignment mechanism is used to establish a spatial relationship between text semantics and visual features, allowing semantic information of unseen categories to directly participate in pixel-level segmentation decisions. Finally, a lightweight segmenter completes the segmentation and category recognition of aligned features. While ensuring inference efficiency, this approach effectively avoids the problem of unstable sample quality in generative methods, alleviating the bias towards seen categories in zero-shot scenarios. Therefore, it improves the accuracy, stability, and generalization ability of zero-shot image semantic segmentation without involving training, solving the technical problem of low accuracy in practical applications of existing zero-shot image semantic segmentation techniques.

[0064] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, the above S205 can be implemented through the following S401 to S406, which are explained in detail below: S401. Map the adjusted text features and visual features to the same space to obtain the first text feature and the first visual feature.

[0065] In one possible implementation, the electronic device performs dimensional rearrangement on visual features to form a visual representation suitable for serialization. Then, it performs expansion or projection processing on the adjusted text features to form a text representation that matches the visual representation in dimensional structure. Next, a linear mapping module projects the visual and text representations respectively, and the projection results are output as the first visual feature and the first text feature to a cross-modal attention module, thereby completing the mapping process of text and visual features to a unified representation space.

[0066] It should be noted that mapping to the same space means that the text features with the number of channels adjusted by the semantic mapper and the visual features obtained by the image feature extractor are made compatible in terms of the number of channels, scale or representation by size transformation, linear transformation or projection transformation, so that they can be used as Query, Key and Value in subsequent attention calculation; the first text feature and the first visual feature represent the text and visual representations after the alignment processing of this space, respectively.

[0067] As an example, in an embodiment of this application, the first visual feature Satisfy the following formula:

[0068] in, This indicates a dimensional transformation operation. For the number of attention heads of the transformer, It is the product of the feature map's height and width. The number of feature channels, Visual features; As an example, in an embodiment of this application, the first text feature Satisfy the following formula:

[0069] in, This indicates a dimensional transformation operation. For the number of attention heads of the transformer, The number of categories in the test set. The number of feature channels, These are the adjusted text features.

[0070] Based on the above steps, this step ensures that the text and visual representations achieve initial consistency in terms of numerical scale, channel structure, and spatial interoperability, thereby providing correct and efficient input for subsequent attention-based cross-modal similarity calculations and reducing alignment errors caused by differences in dimensionality or distribution.

[0071] S402. Construct a cross-modal attention input by using the first visual feature as the query vector and the first text feature as the key vector and value vector.

[0072] In one possible implementation, the electronic device uses the first visual feature as the query vector and the first text feature as the key vector and value vector. The query vector, key vector, and value vector are combined according to a preset input format to form a cross-modal attention input, which is then passed to a multi-head attention computation module, thereby constructing the foundation for cross-modal attention computation of visual features to query text semantics.

[0073] It should be noted that the query vector refers to the set of vectors used in the attention mechanism to initiate similarity matching with all key vectors. The key vector and value vector are used to participate in similarity calculation and subsequent weighted fusion, respectively. The cross-modal attention input is a ternary input composed of visual and textual elements, used to calculate the response strength of visual position to semantics of each category.

[0074] Based on the above steps, this step forms an input framework for similarity matching between visual location and category semantics, enabling subsequent attention calculations to effectively aggregate textual semantic information with visual space as the query center, thereby achieving pixel-level semantic guidance.

[0075] S403. Based on the multi-head attention mechanism, the similarity between the query vector and the key vector is calculated in parallel in multiple subspaces to obtain the corresponding attention weights.

[0076] In one possible implementation, the query vector and key vector are channel-segmented according to a preset number of heads to form multiple subspace vectors. The similarity between the query vector and the key vector is calculated in each subspace. Then, the similarity results are normalized to generate corresponding attention weights. Finally, the attention weights obtained in each subspace are output to the weighted fusion module, thereby completing the cross-modal similarity calculation process under the multi-head attention mechanism.

[0077] Based on the above steps, this step enhances the model's ability to capture multi-scale and multi-semantic relationships by computing similarity in parallel across multiple subspaces. This enables the attention mechanism to not only focus on correspondence from a single perspective, but also to mine complementary information in parallel across different representation subspaces, thereby improving the expressive richness of cross-modal alignment.

[0078] S404. Weighted fusion of value vectors based on attention weights to generate multi-head aligned features.

[0079] In one possible implementation, the value vectors are weighted and fused according to the corresponding attention weights in each subspace to generate head-level aligned features. Then, the head-level aligned features obtained from each subspace are summarized and output as multi-head aligned features.

[0080] As an example, in an embodiment of this application, the multi-head alignment feature satisfies the following formula:

[0081] in, For the normalization operation, Q is the query vector, K is the key vector, and V is the value vector. For the number of attention heads of the transformer, It is the product of the feature map's height and width. The number of feature channels, This is a matrix multiplication operation. As the primary visual feature, As the first text feature, For dimensional transformation operations, This represents the number of categories in the test set.

[0082] In addition to the steps described above, this step efficiently combines attention weights with semantic value vectors to generate an aligned representation that is both semantically guided and spatially aware. This provides a semantically enriched feature base for subsequent stitching, mapping back to the image structure, and participation in pixel-level classification, thereby improving the segmentation and recognition capabilities for unseen categories.

[0083] S405. The multi-head aligned features are spliced ​​and linearly mapped to make their channel number consistent with the first visual feature, thus obtaining the fused feature.

[0084] In one possible implementation, the electronic device concatenates the multi-head alignment features according to the channel dimension to form a high-dimensional feature representation, performs channel mapping processing on the high-dimensional feature representation to adjust its channel number to be consistent with the first visual feature, and outputs the mapped result as a fusion feature to the residual connection and normalization module, thereby realizing the unified mapping of multi-head semantic alignment information to the visual feature structure.

[0085] It should be noted that concatenation and linear mapping refer to concatenating the outputs of each attention point on the channel dimension, and then integrating and mapping the concatenated features through linear transformation or convolution to obtain fused features with the same number of channels as the first visual feature, which facilitates subsequent element-wise or channel-wise fusion operations.

[0086] Based on the above steps, this step normalizes and aligns the semantic fusion results of the multi-head system to the channel structure of the visual features, so that the semantic enhancement information can be seamlessly incorporated into the visual representation. This lays the foundation for channel and semantic compatibility for subsequent residual connection and normalization processing, thereby improving the effectiveness of fused features in pixel-level classification tasks.

[0087] S406. Perform residual connection between the fused features and the visual features, and perform normalization to obtain the aligned features.

[0088] In one possible implementation, the fused features are joined with the first visual features element-wise to generate a residual signal. The residual signal is then normalized to obtain aligned features with stable numerical distribution. The aligned features are then output to a subsequent lightweight segmenter as segmentation input, thereby completing cross-modal feature alignment and providing effective feature representation for pixel-level semantic segmentation.

[0089] As an example, in an embodiment of this application, the alignment feature Satisfy the following formula:

[0090] in, For dimensional transformation operations, The product of the feature map height and width, where h is the feature map height and w is the feature map width. The number of feature channels, For normalization operations, For residuals, For fully connected layer operations, , It is an intermediate feature.

[0091] As an example, in an embodiment of this application, It consists of a linear layer, a ReLU function layer, a Dropout layer, and a linear layer.

[0092] Based on the above steps, this step introduces semantic enhancement while preserving visual structural information through residual fusion and normalization, achieving a balanced alignment between semantics and vision. This makes the aligned features have both spatial details and category discrimination capabilities, improving the accuracy of subsequent pixel-level segmentation in unseen categories and complex scenes.

[0093] This application's embodiments map adjusted text features and visual features to the same feature space, and construct a cross-modal attention mechanism using visual features as queries and text features as keys and values. This allows text semantic information to participate in the modeling of visual features at the pixel level. Under the action of the multi-head attention mechanism, the relationship between text semantics and visual structure is characterized in parallel from multiple subspaces, thereby improving the expressive power of cross-modal feature alignment. Furthermore, by splicing and linearly mapping the multi-head aligned features, the fused features are kept consistent with the visual features in the channel dimension. Combined with residual connections and normalization processing, the semantic guidance effect is enhanced while preserving the original visual information, effectively mitigating the information bias and gradient decay problems that may be introduced during cross-modal mapping. This results in structurally stable and semantically consistent aligned features, providing a feature foundation with stronger discriminativeness and higher generalization ability for subsequent zero-shot semantic segmentation.

[0094] In one possible implementation, combining Figure 2 ,like Figure 5 As shown, prior to S201, the Transformer-based mapping zero-shot image semantic segmentation method provided in this application embodiment further includes training the model, as specifically described in S501 to S503 below: S501. Construct training and testing sets based on segmented image data and segmented image data labels.

[0095] In one possible implementation, a public dataset or a self-built dataset is collected and partitioned, and the training and test sets are split by category to ensure that there are unseen categories in both the training and test categories. Two datasets are defined: the training set... and test set The two contain segmentation categories The following relationship exists: This means that the test set contains some categories that were not seen during training. and The number of categories included are respectively and The model in After training, it will be directly transferred to Conduct tests. Both the training and test sets contain labeled data. The labels in the training set can be used to train the model, but the labels in the test set can only be used to evaluate the test results.

[0096] It should be noted that data augmentation strategies can be considered during the data acquisition phase to improve generalization ability. At the same time, label quality must be ensured. If the labels are noisy, simple post-processing or manual correction can be used to avoid performance degradation due to labeling errors during the training phase.

[0097] As an example, in this embodiment of the application, two publicly available datasets, PASCAL VOC 2012 and SBD, are used to construct the training and test sets, where the SBD dataset is used to provide additional segmentation labels for PASCAL VOC 2012. With a total of 20 categories in PASCAL VOC 2012, the experiment... The value is 19 (including the background class). The value is 21 (including the background class).

[0098] Based on the above steps, this step ensures that training and testing are strictly separated in terms of class distribution and have a unified data format, thereby providing a compliant, reproducible data foundation that can reflect zero-shot capability for subsequent model training, which is conducive to the evaluation of generalization ability and robust adjustment of hyperparameters during the subsequent training process.

[0099] S502. Train the image feature extractor based on the training set to obtain a trained image feature extractor.

[0100] In one possible implementation, a training set is constructed based on segmented image data and segmented image data labels; the neural network is trained based on the training set to obtain a trained neural network; the output layer related to the prediction task is removed from the trained neural network, and only the main part of the network used for feature representation learning is retained as a trained image feature extractor.

[0101] As an example, in this embodiment of the application, Table 1 shows the image feature extractor structure provided in this embodiment. As shown in Table 1, an image feature extractor is defined, and the table lists the structure, input feature size, and output feature size of each layer to meet the requirements of the image feature extraction module. The image feature extractor uses ResNet50 as the network structure for the feature extractor, including the structure shown in Table 1. to Using the scene analysis module ( ) captures spatial information and completes the segmentation task through convolutional layers. On known category data, we use the cross-entropy loss function to optimize and train the model. After training, we extract... to The features are identified and the parameters are frozen, serving as the image feature extractor trained by this system.

[0102] Table 1

[0103] Based on the above steps, this step trains a robust visual representation on the seen categories, ensuring that the subsequent mapping and alignment modules have reliable and discriminative visual feature inputs, thereby improving the stability of cross-modal alignment and providing high-quality visual priors for the zero-shot inference stage.

[0104] S503. Based on the training set and test set, train the trained image feature extractor, text feature extractor, semantic mapper and transformer model.

[0105] In one possible implementation, the electronic device fixes or fine-tunes the trained image feature extractor with a very low learning rate. Images from the training set are input into the image feature extractor to obtain visual features, and the corresponding category names from the training set are processed by a text feature extractor and a semantic mapper to obtain text feature representations. The visual features and the mapped text features are fed into a Transformer model to calculate alignment features, and the alignment features are input into a segmenter to obtain pixel-level predictions. An overall loss function is constructed based on pixel-level cross-entropy loss. An optimizer is used to iteratively update the trainable parameters of the text feature extractor, semantic mapper, Transformer, and segmenter on the training set. At the same time, the model is evaluated on test and validation sets in a preset validation period, and hyperparameters are adjusted or early stopping is performed based on the validation results.

[0106] It should be noted that in this example, during the training and testing phases, the input image resolution is 473×473 pixels (i.e., H and W are both 473), and the size of the feature map output by the image feature extractor is 1 / 16 of the input image size. During training, the training images are randomly cropped and flipped. The SGD optimizer is used to train the entire system for 2000 iterations, with a learning rate of 0.001, a batch size of 8, a momentum of 0.99, and a weight decay rate of 0.0005. The multi-head Transformer has 4 heads.

[0107] Based on the above steps, this step achieves collaborative learning of semantic and visual features by jointly optimizing cross-modal mapping and pixel-level segmentation targets on the training set and monitoring the performance of unseen categories on the validation set. This improves the ability of image feature extractors, text feature extractors, semantic mappers, and transformer models to model complex semantic relationships, enhances the visual adaptability of semantic mappers to text vectors, and ultimately improves segmentation accuracy and generalization robustness in zero-shot scenarios.

[0108] This application's embodiments standardize the segmented image data and its labels, constructing training and test sets that are mutually distinguishable at the category level. This strictly limits the model training process to known categories, conforming to the application constraints of zero-shot semantic segmentation from the outset. Based on this, the image feature extractor is first trained independently to obtain visual feature representations with good discriminative power and stability. Then, it is further combined with a text feature extractor, a semantic mapper, and a Transformer model for joint training. This allows visual features and text semantic features to gradually align and merge within a unified framework, thereby enhancing the model's ability to model cross-modal semantic relationships while ensuring the reliability of visual features. By using the test set for validation and generalization evaluation during the training process, the overfitting tendency of the model to known categories is effectively suppressed, ensuring that the obtained model still has good segmentation performance in scenarios without known categories. This comprehensively improves the accuracy, stability, and generalization ability of the zero-shot image semantic segmentation method.

[0109] In one possible implementation, combining Figure 2 ,like Figure 5 As shown, prior to S201, the Transformer-based mapping zero-shot image semantic segmentation method provided in this application embodiment further includes the following S501 to S504: S501. Using alignment features as input, a lightweight segmenter is used to predict pixel-level segmentation categories.

[0110] In one possible implementation, the alignment features are directly input into a lightweight segmenter. The lightweight segmenter performs convolutional mapping and channel mapping operations on the input features in sequence, converting the alignment features that incorporate textual semantic information into response values ​​for each category corresponding to each pixel, thereby generating low-resolution pixel-level segmentation prediction results.

[0111] It should be noted that during the training of the lightweight segmenter, the parameters of the image feature extractor, text feature extractor, semantic mapper, and transformer model are frozen.

[0112] As an example, in this embodiment of the application, the lightweight segmenter consists of a convolutional layer with a kernel size of 1 and a stride of 21, takes alignment features as input, and outputs pixel-level segmentation categories.

[0113] S502. Upsample the pixel-level segmentation categories to obtain prediction results with the same size as the original label image.

[0114] In one possible implementation, based on the spatial size parameters of the original label map, the prediction results are scaled up using bilinear interpolation or nearest neighbor interpolation, and a predicted segmentation result with the same size as the original label map is generated for subsequent pixel-by-pixel loss calculation.

[0115] Based on the above steps, this step ensures the consistency between the model's prediction results and the true labels in the spatial dimension, enabling the subsequent loss function to make accurate pixel-by-pixel comparisons, which is beneficial to improving the stability of the training process and the accuracy of segmentation error measurement.

[0116] S503. Using the cross-entropy loss function, the segmentation loss is calculated based on the prediction results that are consistent with the original label map size and the data labels.

[0117] In one possible implementation, based on the predicted segmentation results with the same size as the original label map and the corresponding pixel-level ground truth labels, the cross-entropy loss value between the predicted probability and the ground truth class is calculated for each pixel location, and the loss values ​​of all pixels are summed or averaged to obtain the segmentation loss for the current batch.

[0118] As an example, in an embodiment of this application, the segmentation loss Satisfy the following formula:

[0119] in, This represents the sum of the number of categories in the training set and the number of categories in the test set. This indicates a prediction result with the same dimensions as the original label image. This indicates the alignment feature, and 'i' represents the index.

[0120] S504. Based on the segmentation loss, the parameters of the lightweight segmenter are optimized through backpropagation, and the training is iterated until the model converges, thus completing the training of the segmenter.

[0121] In one possible implementation, after obtaining the segmentation loss, the segmentation loss is input into the backpropagation module, and the error signal is backpropagated to each trainable parameter of the lightweight segmenter according to the computation graph. The parameters are then updated using a preset optimizer. The implementing entity repeats the above forward prediction, loss calculation and parameter update process to form a complete training loop. The model performance is evaluated after each or several training rounds to determine whether the convergence condition has been met.

[0122] This application's embodiments use cross-modal aligned features as input and a lightweight segmenter with a simple structure to complete pixel-level category prediction. This reduces the risk of overfitting to seen categories in zero-shot scenarios while ensuring computational efficiency and controllable model size. By performing parameterless upsampling on the low-resolution segmentation results, the predicted results are made consistent with the original label map in spatial size, thereby achieving accurate pixel-by-pixel supervision. Furthermore, the cross-entropy loss function is used to quantify the difference between the predicted results and the true labels, and the parameters of the lightweight segmenter are continuously optimized through a backpropagation mechanism based on the segmentation loss. This allows the segmenter to gradually adapt to the distribution characteristics of cross-modal aligned features, and finally, after training convergence, a pixel-level segmentation capability with stable performance in both seen and unseen category scenarios is obtained.

[0123] The foregoing mainly describes the solutions of the embodiments of this application from the perspective of device implementation. It is understood that each device, such as the Transformer-based mapping zero-shot image semantic segmentation system, includes at least one of the hardware structures and software modules corresponding to the execution of each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0124] This application embodiment can divide the Transformer-based mapping zero-shot image semantic segmentation system into functional units according to the above method example. For example, each function can be divided into separate functional units, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0125] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0126] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0127] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A Transfomer-based mapping type zero-shot image semantic segmentation method, characterized in that, The method comprises the following steps: acquiring image data to be segmented and image data to be segmented labels; extracting visual features by an image feature extractor based on the image data to be segmented; extracting text features by a text feature extractor based on the image data to be segmented labels; adjusting the number of text feature channels based on a semantic mapper to obtain adjusted text features; mapping and aligning the adjusted text features and the visual features by a transformer model to obtain aligned features; segmenting the aligned features by a lightweight segmenter and identifying the segmentation categories.

2. The Transformer-based mapping type zero-shot image semantic segmentation method according to claim 1, characterized in that, Before acquiring the image data to be segmented and the image data to be segmented labels, the method further comprises the following steps: constructing a training set and a test set based on segmented image data and segmented image data labels; training the image feature extractor based on the training set to obtain a trained image feature extractor; training the trained image feature extractor, the text feature extractor, the semantic mapper and the transformer model based on the training set and the test set.

3. The Transformer-based mapping type zero-shot image semantic segmentation method according to claim 2, characterized in that, The training of the image feature extractor based on the training set to obtain the trained image feature extractor comprises the following steps: constructing a training set based on segmented image data and segmented image data labels; training a neural network based on the training set to obtain a trained neural network; removing the output layer related to the prediction task of the trained neural network and only keeping the network main body part for feature representation learning as the trained image feature extractor.

4. The Transformer-based mapping type zero-shot image semantic segmentation method according to claim 1, characterized in that, The extraction of text features by the text feature extractor based on the image data to be segmented labels comprises the following steps: extracting text features by a text feature extractor constructed by a word2vec model based on the image data to be segmented labels; The text features satisfy the following formula: wherein, represents all category label names, represents a text feature extractor, represents the number of test set categories, d represents the number of text feature channels.

5. The Transformer-based mapping type zero-shot image semantic segmentation method according to claim 1, characterized in that, The mapping and alignment of the adjusted text features and the visual features by the transformer model to obtain the aligned features comprises the following steps: mapping the adjusted text features and the visual features to the same space to obtain first text features and first visual features; constructing a cross-modal attention input by taking the first visual features as query vectors, the first text features as key vectors and value vectors; calculating the similarity between the query vectors and the key vectors in multiple subspaces in parallel based on a multi-head attention mechanism to obtain corresponding attention weights; generating multi-head aligned features by weighting and fusing the value vectors according to the attention weights; concatenating and linearly mapping the multi-head aligned features to make their channel numbers consistent with those of the first visual features to obtain fused features; performing residual connection and normalization processing on the fused features and the visual features to obtain aligned features.

6. The Transformer-based mapping type zero-shot image semantic segmentation method according to claim 5, characterized in that, the first visual feature satisfies the following equation: wherein, denotes a dimension transformation operation, is the number of transformer attention heads, is the product of the feature map height and width, is the number of feature channels, is a visual feature; the first text feature satisfies the following equation: wherein, denotes a dimension transformation operation, is the number of transformer attention heads, is the number of test set classes, is the number of feature channels, is the adjusted text feature; The multi-head aligned features satisfy the following formula: wherein, is a normalization operation, Q is a query vector, K is a key vector, V is a value vector, is a number of transformer attention heads, is a product of the height and width of a feature map, is a number of feature channels, is a matrix multiplication operation.

7. The Transformer-based mapping type zero-shot image semantic segmentation method of claim 6, wherein, Before acquiring the image data to be segmented and the image data to be segmented labels, the method further comprises training the lightweight segmenter, comprising the following steps: taking the aligned features as input to predict pixel-level segmentation categories by the lightweight segmenter; performing up-sampling processing on the pixel-level segmentation categories to obtain prediction results consistent with the size of the original label map; The segmentation loss is calculated based on the cross-entropy loss function, the prediction result consistent with the size of the original label map, and the data label. Based on the segmentation loss, the parameters of the lightweight segmenter are optimized by back propagation, and the training is continuously iterated until the model converges, completing the training of the segmenter.

8. The method of claim 7, wherein, During the training of the lightweight segmenter, the image feature extractor parameters, the text feature extractor parameters, the semantic mapper parameters, and the transformer model parameters are frozen.

9. The Transformer-based mapping type zero-shot image semantic segmentation method according to claim 8, characterized in that, The segmentation loss satisfies the following formula: wherein, denotes the sum of the number of classes of the training set and the number of classes of the test set, denotes the prediction result consistent with the original label map size, denotes the aligned feature, i denotes the index.

10. A Transfomer-based mapping-based zero-shot image semantic segmentation system, characterized in that, The Transformer-based mapping zero-shot image semantic segmentation system comprises a data acquisition device and an electronic device. The data acquisition device is configured to acquire image data to be segmented and image data label to be segmented. The electronic device is configured to extract visual features from the image data to be segmented by an image feature extractor, extract text features from the image data label to be segmented by a text feature extractor, adjust the number of text feature channels based on a semantic mapper to obtain adjusted text features, map and align the adjusted text features and the visual features by a transformer model to obtain aligned features, and segment the aligned features by a lightweight segmenter to identify the segmentation category.

Images