Semantic segmentation model and method based on aggregation Transform and feature refinement

Abstract

The invention discloses a semantic segmentation model and method based on aggregation Transform and feature refinement, and belongs to the technical field of computer vision. Aiming at the problems of poor segmentation effect, low efficiency and low precision of the existing Transform type model, the model is composed of two parts, namely an aggregation Transform module and a feature refining module. Due to the importance of contextual information, the aggregation Transform module randomly aggregates the pixels around the pixels of the feature image at first, and then establishes a long-distance remote relationship between each pixel in the feature image and other pixels, so that the feature expression capability is improved. Besides, a refined feature module performs self-attention modeling on a feature map output by an encoder by using an effective self-attention layer, so that feature image pixels in each stage of a backbone network have a long-distance remote dependency relationship, and then, FHL fuses a high-level feature map and a low-level feature map to realize the refinement of the feature map. And finally, guiding feature learning of fusion in each stage by utilizing global context semantic information of the output feature map.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a semantic segmentation model and method based on aggregated Transformer and feature refinement. Background Art

[0002] Semantic segmentation is one of the most basic and challenging tasks in the field of computer vision. It aims to classify each pixel in an image into a specific category, thereby achieving a detailed understanding of the image. Semantic segmentation plays a vital role in many practical applications, such as autonomous driving, medical image analysis, remote sensing image processing, and augmented reality. Through fine-grained analysis of images, semantic segmentation can provide precise object boundaries and rich semantic information, which plays an important role in promoting further visual tasks such as object detection and image understanding.

[0003] With the rapid development of deep learning technology, especially the introduction of convolutional neural networks (CNN), semantic segmentation technology has made significant progress. Traditional image segmentation methods rely on manually designed feature extractors and classifiers, which are usually difficult to handle complex scenes and diverse objects. Deep learning methods can achieve higher accuracy and robustness on large-scale data sets by automatically learning the hierarchical features of images. Since the introduction of FCNet, various advanced models such as UNet, SegNet, DeepLab, PSPNet, etc. have emerged, promoting the continuous progress of semantic segmentation technology. Although the above methods have made great progress in the direction of semantic segmentation, there are still problems, such as limited processing capabilities for complex scenes, poor processing of small objects and fine-grained features leading to unsatisfactory segmentation. The emergence of these problems has inspired researchers to study Transformer. The Transformer model was originally proposed to solve some key problems in machine translation. It effectively solves the problems of traditional sequence models in long-term dependence, parallel processing, sequence information utilization, representation ability and interpretability, so that it has achieved significant performance improvement in natural language processing tasks. Inspired by this, Dosovitskiy et al. applied the Transformer model to visual tasks in 2020 and launched Vision Transformer (ViT), which pioneered the application of Transformer in computer vision. ViT divides the image into fixed-size image blocks and flattens each image block as the input of Transformer, demonstrating the effectiveness of Transformer on large datasets, performing well in image classification tasks, and having stronger global feature capture capabilities than traditional convolutional neural networks (CNNs).

[0004] The success of ViT has inspired a series of subsequent Transformer-based visual models. For example, DETR (Detection Transformer) formulates the object detection task as a set prediction problem, using the Transformer model to directly predict the bounding boxes and category labels of a set of objects. DETR solves the label assignment problem in target detection by introducing a matching algorithm, implements an end-to-end training method, and simplifies the complex post-processing steps in traditional detectors.

[0005] Subsequently, Segmenter used Transformer for image segmentation tasks, and captured global features in the image through the self-attention mechanism, achieving high-quality image segmentation and performing well on multiple image segmentation benchmark datasets. Furthermore, MaskFormer unified different problems in image segmentation tasks (such as semantic segmentation, instance segmentation, and panoptic segmentation) into one framework, and proposed a general segmentation framework. By learning a set of predictable mask embeddings, it simplified the model design and implementation of segmentation tasks, making training and reasoning more consistent.

[0006] As an improved version of MaskFormer, Mask2Former further improves the performance of segmentation tasks by introducing new architectures and training strategies. Mask2Former improves the representation method and training process of mask embedding, achieves new best performance on multiple image segmentation tasks, and verifies the great potential of Transformer in visual tasks. Recently, Shim et al. proposed FeedFormer in 2023, a new Transformer variant that focuses on improving feature extraction and representation capabilities in visual tasks. FeedFormer introduces a new attention mechanism and feature fusion method, which enhances the model's ability to capture details and global information in images, and achieves significant performance improvements on multiple visual tasks (such as image classification, detection, and segmentation), demonstrating its wide applicability and prospects in various visual tasks. Although the above models have made great progress, the computational cost is very large, especially when processing high-resolution images, which will generate a lot of computation. To solve this problem, Swin proposed a layered transformer to increase computational efficiency. SegFormer proposed a lightweight codec, but it relies on the encoder to improve the overall performance, which leads to low efficiency in segmentation. Summary of the invention

[0007] In view of the problems of poor segmentation effect, low segmentation efficiency and low segmentation accuracy of current Transformer-type models, the present invention provides a semantic segmentation model and method based on aggregated Transformer and feature refinement. By making full use of context information, local feature information and codec, a semantic segmentation model based on aggregated Transformer and feature refinement is designed, which integrates high-level features and low-level features with a U-shaped structure to enhance the feature representation capability, and improves the efficiency of model segmentation while improving the accuracy.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A semantic segmentation model based on an aggregated transformer and feature refinement, the model comprising an aggregated transformer module ATM and a feature refinement module FRM;

[0010] The Aggregate Transformer Module ATM includes a convolutional layer and a Transformer mechanism, wherein the convolutional layer is responsible for extracting local features, and the Transformer mechanism processes global and long-distance dependencies through an attention mechanism;

[0011] The feature refinement module FRM includes several self-attention modules ESA and several high-level feature fusion and low-level feature modules FHL;

[0012] The Aggregate Transformer Module ATM is used to aggregate information around pixels and model the complex relationship between pixels and their neighborhood, so that the model can more accurately understand the semantic content of each part of the image;

[0013] The self-attention module ESA uses a self-attention mechanism to refine the generated feature map, so that the model can adaptively strengthen or weaken the connection weights between specific pixels at different levels, making it easier to understand the complex semantic structures and associations in the image; wherein the self-attention mechanism allows the model to dynamically allocate attention when processing features, which is conducive to capturing long-distance dependencies between pixels;

[0014] The high-level feature and low-level feature fusion module FHL uses the semantics of high-level features to guide the learning process of low-level features, so that the model understands and expresses the content in the image at different levels; it is used to enhance the model's ability to understand image semantics and improve the accuracy and stability of semantic segmentation tasks; wherein the high-level features contain abstract and semantically rich information.

[0015] Further, the aggregation transformer module ATM is represented as:

[0016] F′=Convly(F)

[0017] Q,K,V=W Q F,W K F,W V F

[0018] Q=Agg(Q),K=Agg(K),V=Agg(V)

[0019]

[0020] O=W o Z

[0021] F=Fusion(O,Q)

[0022] Among them, F′ represents the feature map after the pixels are aggregated by the aggregation Transformer module, Convly() represents the convolution layer with a size of 1×1, which is used to compress the number of channels of the input feature map; Q represents the query vector matrix, K represents the key vector matrix, V represents the value vector matrix; W Q represents the linear transformation matrix used to generate the query vector, W K represents the linear transformation matrix used to generate the key vector, W V represents the linear transformation matrix used to generate the value vector; Q′ represents the aggregated query vector matrix, K′ represents the aggregated key vector matrix, and V′ represents the aggregated value vector matrix; Agg(·) represents the aggregation operation; O represents the final output feature map, W O represents the linear transformation matrix, and Z represents the feature map after the feature map is weighted by the attention mechanism weight matrix.

[0023] Furthermore, the self-attention module ESA formula is expressed as:

[0024] Q′ i ,K′ i ,V′ i =W Q′ F i ,W K′ F i ,W V′ F i

[0025]

[0026] F i = Fusion(F i ,Att i )

[0027] F i ″′=Linear(F i ″)

[0028]

[0029] O i =Fusion(F″′,O i FFN )

[0030] Among them, Q′ i ,K′ i ,V′ i They represent the query matrix, key matrix, and value matrix generated by linear transformation in the i-th stage of the backbone network respectively; F i Represents the feature map output by the backbone network at stage i, Att i represents the output of the self-attention mechanism at stage i; F i ″ represents the feature map generated after fusion, Fusion(F i ,Att i ) represents the point-by-point pixel addition fusion function; F i ″′ represents the feature map after linear transformation, and Linear represents the linear transformation function; Represents the output of the feedforward neural network; FFN() represents the feedforward neural network function; Fusion() represents the pixel-by-pixel fusion function, i Represents the final output feature map.

[0031] Furthermore, the FHL formula of the fusion high-level feature and low-level feature module is expressed as:

[0032] F L ″=δ(β(C 1×1 (F L )))

[0033] F HL ′=σ(ρ(δ(C 1×1 (Concat(F L ″,F H ′)))))

[0034]

[0035] F H ″=δ(β(C 1×1 (F H ′)))

[0036]

[0037] Among them, F L ″ represents the processed low-level feature map, δ represents the Relu activation function, β represents the batch normalization function, C 1×1 represents 1×1 convolution, FL represents the low-level feature map, F HL ′ represents the processed high-level feature map, σ represents the Sigmiod activation function, ρ represents the global average pooling, F H ′ represents the high-level feature map, represents channel-by-channel multiplication, F H ″ represents the high-level feature map after further processing, Stands for pixel-by-pixel addition.

[0038] The semantic segmentation method based on bidirectional aggregation Transformer and feature refinement includes the following steps:

[0039] Step 1: Input the original image and preliminarily extract image features;

[0040] Step 2: Aggregate Transformer Module ATM effectively aggregates information around pixels and comprehensively models the complex relationship between pixels and their neighborhoods.

[0041] Step 3, the self-attention module (ESA) uses the self-attention mechanism to refine the feature map;

[0042] In step 4, the semantics of high-level features are used to guide the learning process of low-level features through the FHL module, so that the model can understand and express the content in the image at different levels.

[0043] Step 5: Output the prediction graph.

[0044] Further, the step 2 specifically includes the following steps:

[0045] Step 2.1, given a feature map of the ATM module, After convolutional layers and linear transformations, we get the Q, K, and V matrices;

[0046] Step 2.2, convolve the Q, K, and V matrices with different kernel sizes to aggregate the information of surrounding pixels, so that each pixel contains more local context information; obtain the aggregated matrices Q′, K′, and V′;

[0047] Step 2.3, establish long-distance interdependence through the self-attention mechanism to improve feature distinguishability; use the aggregated matrix Q′, K′, V′ to calculate the self-attention Z is transformed linearly, and finally, it is fused pixel by pixel with the feature map F′ after the pixels are aggregated. The calculation formula is O = W o Z, F″=Fusion(O, Q′).

[0048] Further, the step 3 specifically includes the following steps:

[0049] Step 3.1, the ESA module automatically selects and emphasizes important features and suppresses unimportant features through the self-attention mechanism;

[0050] Step 3.2, give the ESA module a feature map First, after the linear layer, Q′ i ,K′ i ,V′ i =W Q′ F i ,W K′ F i ,W V′ F i , and then it is divided into two paths, one of which goes through self-attention, The other path is directly fused with the attention output, and then the fused feature map passes through the linear layer, and the calculation formula is F i = Fusion(F i ,Att i ), F i =Linear(F i );

[0051] Step 3.3, finally, it is divided into two paths, one path passes through the feedforward neural network FFN layer, and the other path is directly fused with the FFN output; the calculation formula is:

[0052] Further, the step 4 specifically includes the following steps:

[0053] Step 4.1, input the feature map F of the low-level stage L , through C 1×1 The 1×1 convolutional layer pair F L Processing; Apply batch normalization β and ReLU activation function δ to the convolution result to obtain the processed low-level stage feature map F L ″, expressed as: F L ″=δ(β(C 1×1 (F L )));

[0054] Step 4.2, input the feature map F of the high-level stage H ′, use Concat(F L ″,F H ′) The processed low-level feature map F L ″ and the original high-level feature map F H ' is concatenated, and then the concatenated feature map is convolved using 1×1 convolution C 1×1To compress the number of feature map channels and extract feature information; then, the compressed feature map is processed by the nonlinear ReLU activation function δ, and finally the global average pooling ρ and the Sigmoid activation function σ are applied to obtain the processed high-level feature map F HL ′, expressed as: F HL ′=σ(ρ(δ(C 1×1 (Concat(F L ″,F H ′)))));

[0055] Step 4.3, use channel-by-channel multiplication The operation will process the high-level feature map F HL ′ and low-level feature map F L ″ is weighted to obtain F HL ″; F H 'Through C 1×1 The 1×1 convolutional layer represented by processes the fused features; finally, batch normalization β and ReLU activation function δ are applied to obtain the high-level feature map F after further processing H ″, expressed as: F H ″=δ(β(C 1×1 (F H ′)));

[0056] Step 4.4: Use pixel-by-pixel addition The operation will weight the feature F HL ″ and the original high-level feature map F H ″ performs weighted fusion; the final output feature map O is obtained, which is expressed as:

[0057] Compared with the prior art, the present invention has the following advantages:

[0058] The semantic segmentation network (ATFRNet) model based on aggregated Transformer and feature refinement proposed in the present invention can achieve more accurate and efficient semantic segmentation. Based on the aggregated Transformer module, the contextual information around the pixels is effectively utilized, the pixels of the feature maps adjacent to the feature map are aggregated, and long-distance remote relationships are established for the pixels at the contextual information level. Thereby improving the feature expression ability. The feature map output by the encoder is self-attention modeled by the refined feature map module, so that there is a long-distance mutual dependence between the pixels of the feature map, and then the modeled feature map is effectively fused with the feature map of the advanced stage to enhance the intra-class consistency and inter-class distinction, thereby achieving the purpose of the refined feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1It is a schematic diagram of the overall framework structure of the model of the present invention;

[0060] Figure 2 It is a schematic diagram of the structure of the aggregation transformer module ATM;

[0061] Figure 3 It is a schematic diagram of the structure of the self-attention module ESA;

[0062] Figure 4 It is a schematic diagram of the structure of the FHL module that integrates high-level and low-level features;

[0063] Figure 5 It is a schematic diagram of the loss function curve of loss indicator training;

[0064] Figure 6 It is a curve diagram of the mIoU indicator;

[0065] Figure 7 This is a visualization result map on the cityscapes dataset;

[0066] Figure 8 It is an overall flow chart of the method of the present invention. DETAILED DESCRIPTION

[0067] In order to gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description of the present invention. However, the present invention has multiple implementations and is not limited to the specific examples listed herein. The presentation of these examples is intended to deepen the comprehensive understanding of the disclosure of the present invention.

[0068] The semantic segmentation model based on aggregated Transformer and feature refinement provided by the present invention is as follows: Figure 1 Said, including an aggregation Transformer module ATM and a feature refinement module FRM;

[0069] The Aggregate Transformer Module (ATM) consists of a convolutional layer and a Transformer mechanism. The convolutional layer is responsible for local feature extraction, and the Transformer mechanism handles global and long-distance dependencies through an attention mechanism.

[0070] The feature refinement module FRM includes several self-attention modules ESA and several high-level feature fusion and low-level feature modules FHL;

[0071] The Aggregate Transformer Module (ATM) is used to aggregate information around pixels and model the complex relationship between pixels and their neighborhood, so that the model can more accurately understand the semantic content of each part of the image.

[0072] The self-attention module ESA uses the self-attention mechanism to refine the generated feature map, allowing the model to adaptively strengthen or weaken the connection weights between specific pixels at different levels, making it easier to understand the complex semantic structures and associations in the image; the self-attention mechanism allows the model to dynamically allocate attention when processing features, which is conducive to capturing long-distance dependencies between pixels;

[0073] The high-level feature and low-level feature fusion module FHL uses the semantics of high-level features to guide the learning process of low-level features, so that the model can understand and express the content in the image at different levels; it is used to enhance the model's ability to understand image semantics and improve the accuracy and stability of semantic segmentation tasks; the high-level features contain abstract and semantically rich information.

[0074] The Aggregate Transformer Module ATM is represented as:

[0075] F′=Convly(F)

[0076] Q,K,V=W Q F,W K F,W V F

[0077] Q=Agg(Q),K=Agg(K),V=Agg(V)

[0078]

[0079] O=W o Z

[0080] F = Concat(O,Q)

[0081] Among them, F′ represents the feature map after the pixels are aggregated by the aggregation Transformer module, Convly() represents a convolutional layer whose size is composed of several 1×1 convolutions, which is used to compress the number of channels of the input feature map; Q represents the query vector matrix, K represents the key vector matrix, V represents the value vector matrix; W Q represents the linear transformation matrix used to generate the query vector, W K represents the linear transformation matrix used to generate the key vector, W V represents the linear transformation matrix used to generate the value vector; Q′ represents the aggregated query vector matrix, K′ represents the aggregated key vector matrix, and V′ represents the aggregated value vector matrix; Agg(·) represents the aggregation operation; O represents the final output feature map, W O represents the linear transformation matrix, and Z represents the feature map after the feature map is weighted by the attention mechanism weight matrix.

[0082] The self-attention module ESA is expressed as:

[0083] Q′ i ,K′ i ,V′ i =W Q′ F i ,W K′ F i ,W V′ F i

[0084]

[0085] F i = Fusion(F i ,Att i )

[0086] F i ″′=Linear(F i ″)

[0087]

[0088] Among them, Q′ i ,K′ i ,V′ i They represent the query matrix, key matrix, and value matrix generated by linear transformation in the i-th stage of the backbone network respectively; F i Represents the feature map output by the backbone network at stage i, Att i represents the output of the self-attention mechanism at stage i; F i ″ represents the feature map generated after fusion, Fusion(F i ,Att i ) represents the point-by-point pixel addition fusion function; F i ″′ represents the feature map after linear transformation, and Linear represents the linear transformation function; represents the output of the feedforward neural network; FFN() represents the feedforward neural network function; Fusion() represents the pixel-by-pixel fusion function, i Represents the final output feature map.

[0089] The fusion high-level feature and low-level feature module FHL is expressed as:

[0090] F L ″=δ(β(C 1×1 (F L )))

[0091] F HL ′=σ(ρ(δ(C 1×1 (Concat(F L ″,F H ′)))))

[0092]

[0093] F H ″=δ(β(C 1×1 (F H ′)))

[0094]

[0095] Among them, F L represents the low-level feature map, F H ′ represents the high-level feature map, C 1×1 represents 1×1 convolution, β represents batch normalization function, δ represents Relu activation function, ρ represents global average pooling, σ represents Sigmiod activation function, represents channel-by-channel multiplication, Stands for pixel-by-pixel addition.

[0096] Example 1

[0097] The semantic segmentation method based on bidirectional aggregation Transformer and feature refinement includes the following steps:

[0098] Step 1: Input the original image and preliminarily extract image features;

[0099] Step 2: Aggregate Transformer Module ATM effectively aggregates information around pixels and comprehensively models the complex relationship between pixels and their neighborhoods. The detailed structure of the ATM module is shown in Figure 2 shown.

[0100] Step 2.1, given a feature map of the ATM module, After convolutional layers and linear transformations, we get the Q, K, and V matrices;

[0101] Step 2.2, convolve the Q, K, and V matrices with different kernel sizes to aggregate the information of surrounding pixels, so that each pixel contains more local context information; obtain the aggregated matrices Q′, K′, and V′;

[0102] Step 2.3, establish long-distance interdependence through the self-attention mechanism to improve feature distinguishability; use the aggregated matrix Q′, K′, V′ to calculate the self-attention Z is transformed linearly, and finally, it is fused pixel by pixel with the feature map F′ after the pixels are aggregated. The calculation formula is O = W o Z, F″=Fusion(O, Q′).

[0103] Step 3: The self-attention module (ESA) uses the self-attention mechanism to generate feature maps in a refined manner. The structure of the ESA module is shown in the figure. Figure 3 described.

[0104] Step 3.1, the ESA module automatically selects and emphasizes important features and suppresses unimportant features through the self-attention mechanism;

[0105] Step 3.2, give the ESA module a feature map First, after the linear layer, Q′ i ,K′ i ,V′ i =W Q′ F i ,W K′ F i ,W V′ F i , and then it is divided into two paths, one of which goes through self-attention, The other path is directly fused with the attention output, and then the fused feature map passes through the linear layer, and the calculation formula is F i = Fusion(F i ,Att i ), F i =Linear(F i );

[0106] Step 3.3, finally, it is divided into two paths, one path passes through the feedforward neural network FFN layer, and the other path is directly fused with the FFN output; the calculation formula is:

[0107] Step 4: The FHL module uses the semantics of high-level features to guide the learning process of low-level features, so that the model can understand and express the content of the image at different levels. The structure diagram of the FHL module is shown in Figure 4 shown.

[0108] Step 4.1, input the feature map F of the low-level stage L , through C 1×1 The 1×1 convolutional layer pair F L Processing; Apply batch normalization β and ReLU activation function δ to the convolution result to obtain the processed low-level stage feature map F′ L , expressed as: F L ″=δ(β(C 1×1 (F L )));

[0109] Step 4.2, input the feature map F of the high-level stage H ′, use Concat(F L ″,F H ′) The processed low-level feature map F L ″ and the original high-level feature map F H' is concatenated, and then the concatenated feature map is convolved using 1×1 convolution C 1×1 To compress the number of feature map channels and extract feature information; then, the compressed feature map is processed by the nonlinear ReLU activation function δ, and finally the global average pooling ρ and the Sigmoid activation function σ are applied to obtain the processed high-level feature map F HL ′, expressed as: F HL ′=σ(ρ(δ(C 1×1 (Concat(F L ″,F H ′)))));

[0110] Step 4.3, use channel-by-channel multiplication The operation will process the high-level feature map F HL ′ and low-level feature map F L ″ is weighted to obtain F HL ″; F H 'Through C 1×1 The 1×1 convolutional layer represented by processes the fused features; finally, batch normalization β and ReLU activation function δ are applied to obtain the high-level feature map F after further processing H ″, expressed as: F H ″=δ(β(C 1×1 (F H ′)));

[0111] Step 4.4: Use pixel-by-pixel addition The operation will weight the feature F HL ″ and the original high-level feature map F H ″ performs weighted fusion; the final output feature map O is obtained, which is expressed as:

[0112] Step 5: Output the prediction graph.

[0113] Example 2

[0114] In order to ensure the efficiency and stability of the training process, the optimizer and training strategy used are SGD and poly learning rate decay strategy respectively.

[0115] The loss function used is the cross entropy loss function, which measures the difference between the predicted probability distribution and the true label distribution.

[0116] Among them, N represents the total number of samples, C represents the total number of categories, and y i,c is the true label (0 or 1) of the sample, i belongs to category c, is the probability that the model predicts that sample i belongs to category c.

[0117] The SGD optimizer is used, and a polynomial learning rate decay strategy is used. In addition, for fair comparison, a variety of data enhancement techniques are introduced, including random horizontal flipping, random cropping, and random scaling in the range of 0.5 to 2.0.

[0118] When training the Cityscapes dataset, the following key training parameters are set: ietrs = 160000, initial learning rate = 1×10 -2 , weight decay = 5×10 -4 , the image size is cropped to 1024×1024 during training, and the original image size is cropped to 1024×512 during verification, and the batch size of one-time input is 12.

[0119] Example 3 Model Validity

[0120] like Figure 5 and Figure 6 As shown in the figure, the loss function loss curve and mIoU curve during training are visualized respectively; it can be seen from the figure that the loss curve further shows the change of the model's loss on the training set. As the training progresses, the loss value usually decreases gradually, reflecting that the model fits the training data well. Similarly, the model quickly reduces the loss on the training set and can also keep the mIoU indicator rising on the validation set. This trend shows that the model can learn effectively during the training process.

[0121] Example 4 Verifying the accuracy of segmentation on Cityscapes data

[0122] like Figure 7 As shown in the visualization results on the cityscapes dataset, Segforme and Segmenter are visualized and compared visually. Figure 7 The first and second rows show the segmentation effect of this model on the sidewalk category compared to Segformer and Segmenter. This model can more accurately distinguish the sidewalk from other ground and capture fine road textures and boundaries. For the bicycle category, the fourth row shows that this model can accurately identify and segment bicycles, avoiding confusion with pedestrians or buildings, demonstrating efficient object classification and segmentation capabilities. Through detailed visual comparisons, this model has better segmentation effects on the sidewalk, building, and bicycle categories. This model can more accurately distinguish the boundaries of these categories and capture details.

[0123] On the Cityscapes dataset, the segmentation effect of this model on categories such as sidewalks, buildings, and bicycles is significantly better than the existing methods. Table 1 is the quantitative evaluation results of the comparison of semantic segmentation methods on the Cityscapes dataset. From the data in Table 1, it can be seen that the segmentation effect of this model is the best.

[0124] Table 1. Comparison of semantic segmentation methods on the Cityscapes dataset

[0125]

[0126] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. Although the illustrative specific embodiments of the present invention are described above to facilitate the understanding of the present invention by the technical personnel in the field, it should be clear that the present invention is not limited to the scope of the specific embodiments. For the ordinary technical personnel in the field, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A semantic segmentation model based on aggregated Transformer and feature refinement, characterized by: The model includes an aggregation transformer module ATM and a feature refinement module FRM; The Aggregate Transformer Module ATM includes a convolutional layer and a Transformer mechanism, wherein the convolutional layer is responsible for extracting local features, and the Transformer mechanism processes global and long-distance dependencies through an attention mechanism; The feature refinement module FRM includes several self-attention modules ESA and several high-level feature fusion and low-level feature modules FHL; The Aggregate Transformer Module ATM is used to aggregate information around pixels and model the complex relationship between pixels and their neighborhood, so that the model can more accurately understand the semantic content of each part of the image; The self-attention module ESA uses a self-attention mechanism to refine the generated feature map, so that the model can adaptively strengthen or weaken the connection weights between specific pixels at different levels, making it easier to understand the complex semantic structures and associations in the image; wherein the self-attention mechanism allows the model to dynamically allocate attention when processing features, which is conducive to capturing long-distance dependencies between pixels; The high-level feature and low-level feature fusion module FHL uses the semantics of high-level features to guide the learning process of low-level features, so that the model understands and expresses the content in the image at different levels; it is used to enhance the model's ability to understand image semantics and improve the accuracy and stability of semantic segmentation tasks; wherein the high-level features contain abstract and semantically rich information.

2. The semantic segmentation model based on aggregated Transformer and feature refinement technology according to claim 1, characterized in that: The aggregation transformer module ATM is represented as: F￠=Convly(F) Q,K,V=W Q F,W K F,W V F Q=Agg(Q),K=Agg(K),V=Agg(V) O=W o WITH F=Fusion(O,Q) Among them, F′ represents the feature map after the pixels are aggregated by the aggregation Transformer module, Convly() represents the convolution layer with a size of 1×1, which is used to compress the number of channels of the input feature map; Q represents the query vector matrix, K represents the key vector matrix, V represents the value vector matrix; W Q represents the linear transformation matrix used to generate the query vector, W K represents the linear transformation matrix used to generate the key vector, W V represents the linear transformation matrix used to generate the value vector; Q′ represents the aggregated query vector matrix, K′ represents the aggregated key vector matrix, and V′ represents the aggregated value vector matrix; Agg(·) represents the aggregation operation; O represents the final output feature map, W O represents the linear transformation matrix, and Z represents the feature map after the feature map is weighted by the attention mechanism weight matrix.

3. The semantic segmentation model based on aggregated Transformer and feature refinement technology according to claim 1, characterized in that: The self-attention module ESA is expressed as: Q i ′,K i ′,V i ′=W Q′ F i ,W K′ F i ,W V′ F i F i ″=Fusion(F i ,To i ) F i ″′=Linear(F i ″) Oh i FFN =FFN(F i ″′) O i =Fusion(F″′,O i FFN ) Among them, Q i ′,K i ′,V i ′ respectively represent the query matrix, key matrix, and value matrix generated by linear transformation in the i-th stage of the backbone network; F i Represents the feature map output by the backbone network at stage i, Att i represents the output of the self-attention mechanism at stage i; F i ″ represents the feature map generated after fusion, Fusion(F i ,Att i ) represents the point-by-point pixel addition fusion function; F i ″′ represents the feature map after linear transformation, and Linear represents the linear transformation function; represents the output of the feedforward neural network; FFN() represents the feedforward neural network function; Fusion() represents the pixel-by-pixel fusion function, i Represents the final output feature map.

4. The semantic segmentation model based on aggregated Transformer and feature refinement technology according to claim 1, characterized in that: The fusion high-level feature and low-level feature module FHL is expressed as: F L ″=δ(β(C 1×1 (F L ))) F HL ′=σ(ρ(δ(C 1×1 (Concat(F L ″,F H ′))))) F H ″=δ(β(C 1×1 (F H ′))) Among them, F L represents the low-level feature map, F H ′ represents the high-level feature map, C 1×1 represents 1×1 convolution, β represents batch normalization function, δ represents Relu activation function, ρ represents global average pooling, σ represents Sigmiod activation function, represents channel-by-channel multiplication, Stands for pixel-by-pixel addition.

5. A semantic segmentation method based on bidirectional aggregation Transformer and feature refinement, characterized in that: The following steps are involved: Step 1: Input the original image and preliminarily extract image features; Step 2: Aggregate the information around the pixel by using the Aggregate Transformer Module (ATM) to model the complex relationship between the pixel and its neighborhood. Step 3, the self-attention module (ESA) uses the self-attention mechanism to refine the feature map; Step 4: The semantics of high-level features are used to guide the learning process of low-level features through the FHL module, so that the model can understand and express the content in the image at different levels; Step 5: Output the prediction graph.

6. The semantic segmentation method based on bidirectional aggregation Transformer and feature refinement according to claim 5, characterized in that: The step 2 specifically includes the following steps: Step 2.1, given a feature map of the ATM module, After convolutional layers and linear transformations, we get the Q, K, and V matrices. Step 2.2, convolve the Q, K, and V matrices with different kernel sizes to aggregate the information of surrounding pixels, so that each pixel contains more local context information; obtain the aggregated matrices Q′, K′, and V′; Step 2.3, establish long-distance interdependence through the self-attention mechanism to improve feature distinguishability; use the aggregated matrices Q′, K′, V′ to calculate the self-attention Z is transformed linearly, and finally, it is fused pixel by pixel with the feature map F′ after the pixels are aggregated. The calculation formula is O = W o Z, F″=Fusion(O, Q′).

7. The semantic segmentation method based on bidirectional aggregation Transformer and feature refinement according to claim 5, characterized in that: The step 3 specifically comprises the following steps: Step 3.1, the ESA module automatically selects and emphasizes important features and suppresses unimportant features through the self-attention mechanism; Step 3.2, give the ESA module a feature map First, after the linear layer, Q i ′,K i ′,V i ′=W Q′ F i ,W K′ F i ,W V′ F i , and then it is divided into two paths, one of which goes through self-attention, The other path is directly fused with the attention output, and then the fused feature map passes through the linear layer, and the calculation formula is F i = Fusion(F i ,Att i ), F i =Linear(F i ); Step 3.3, finally, it is divided into two paths, one path passes through the feedforward neural network FFN layer, and the other path is directly fused with the FFN output; the calculation formula is:

8. The semantic segmentation method based on bidirectional aggregation Transformer and feature refinement according to claim 5, characterized in that: The step 4 specifically comprises the following steps: Step 4.1, input the feature map F of the low-level stage L , through C 1×1 The 1×1 convolution layer pair F L Processing; Apply batch normalization β and ReLU activation function δ to the convolution result to obtain the processed low-level stage feature map F L ″, expressed as: F L ″=δ(β(C 1×1 (F L ))); Step 4.2, input the feature map F of the high-level stage H ′, use Concat(F L ″,F H ′) The processed low-level feature map F L ″ and the original high-level feature map F H ' is concatenated, and then the concatenated feature map is convolved using 1×1 convolution C 1×1 To compress the number of feature map channels and extract feature information; then, the compressed feature map is processed by the nonlinear ReLU activation function δ, and finally the global average pooling ρ and the Sigmoid activation function σ are applied to obtain the processed high-level feature map F HL ′, expressed as: F HL ′=σ(ρ(δ(C 1×1 (Concat(F L ″,F H ′))))); Step 4.3, use channel-by-channel multiplication The operation will process the high-level feature map F HL ′ and low-level feature map F L ″ is weighted to obtain F HL ″; F H ￠Through C 1×1 The 1×1 convolutional layer represented by processes the fused features; finally, batch normalization β and ReLU activation function δ are applied to obtain the high-level feature map F after further processing H ″, expressed as: F H ″=δ(β(C 1×1 (F H ′))); Step 4.4: Use pixel-by-pixel addition The operation will weight the feature F HL ″ and the original high-level feature map F H ″ performs weighted fusion; the final output feature map O is obtained, which is expressed as:

Images