A lightweight semantic segmentation method and system based on Transformer

Abstract

The present invention discloses a lightweight semantic segmentation method and system based on Transformer. The method extracts features from a pre-acquired image to obtain a feature map F1 of the image to be processed; constructs a feature aggregation module to aggregate channel information to obtain a feature map F2 corresponding to the image to be processed; constructs a Transformer-based bilateral guided aggregation query update layer BGA-QULayer to perform bilateral guided aggregation on the shallow and deep layers of the feature map to obtain semantic feature guidance and semantic feature aggregation, respectively; and performs semantic segmentation on the attention aggregation update feature map to obtain a semantic segmentation result. The present invention aggregates multi-scale information, compensates for edge detail information, strengthens the features of deep semantic guidance, integrates the weights of channel information in the feature network, and avoids information loss during nonlinear activation. At the same time, the query update can effectively help group feature information and improve segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to a semantic segmentation method, specifically a lightweight semantic segmentation method based on Transformer, which can be used in semantic segmentation scenarios. Background Art

[0002] Semantic segmentation can predict the category of each pixel in an image. As an important direction in the field of computer vision, applying semantic segmentation to industrial production, medical imaging, autonomous driving, etc. has important research significance.

[0003] In recent years, the Transformer has significantly outperformed both CNNs and RNNs. Its entire network architecture consists entirely of an attention mechanism (Self-Attention) and a position-wise feed-forward network (Position-wise Feed-Forward Network). One of the main advantages of attention-based models is that they focus on relevant information while ignoring irrelevant information. The attention mechanism is a neural network layer that aggregates feature information from the entire input sequence. Transformers introduce Self-Attention, which updates by scanning each element in the sequence and aggregating the feature information from the entire sequence. Compared to RNNs on long sequences, Transformers can be computed in parallel, making them more suitable. Transformers were first applied to image classification. The Vision Transformer (ViT) is a model for image classification that uses a Transformer-like structure on image patches for image classification tasks. Since the introduction of Vision Transformers by Dosovitskiy et al., the field of semantic segmentation has undergone significant changes, and a growing number of Transformer-based semantic segmentation works have begun to be launched. Although these have achieved good segmentation accuracy, their performance in some specific scenarios is less than satisfactory.

[0004] Most of the current technical solutions are based on the encoder-decoder structure. MobileViT is the first lightweight ViT work that successfully based on the performance of lightweight CNN network. SegFormer uses a hierarchical Transformer Encoder structure and a decoder composed of only a few FCs to achieve more efficient semantic segmentation. TopFormer further studies CNN and ViT, combines their advantages, and uses the pyramid structure commonly used in CNN to quickly generate features from high-resolution images, reduce the amount of calculation, and achieve a good balance between its recognition accuracy and computational cost. SETR's encoder design is similar to that of VIT as the backbone. At the same time, its decoder design uses progressive upsampling to alleviate a large amount of noise, as well as multi-level feature summation similar to the feature pyramid network. The RTFormer proposed a GPU-friendly attention module (GPU-FriendlyAttention), which adopts a cross-attention mechanism for global context information aggregation and uses a fusion structure with different resolutions to eliminate the multi-head mechanism. The SeaFormer proposed a Squeeze-enhanced Axial Transformer, which uses an attention module with axial compression and detail enhancement to build the backbone. At the same time, it designed a lightweight segmentation head, taking into account both speed and accuracy, and achieved further breakthroughs.

[0005] On the server side, Transformer has achieved remarkable results for multi-category semantic segmentation datasets. With the development of Vision Transformer, many researchers are considering using Transformer for semantic segmentation. However, Vision Transformer is computationally intensive and memory intensive. Other Transformer-based works, such as TopFormer, SETR, RTFormer, and SeaFormer lightweight semantic segmentation networks, rely heavily on pre-training and dataset size to achieve good results. Using lightweight semantic segmentation models on mobile devices, especially for high-resolution semantic segmentation tasks, has been difficult to strike a balance between accuracy and speed, and their performance remains unsatisfactory. Summary of the Invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a lightweight semantic segmentation method and system based on Transformer, which extracts features through deep learning methods, aggregates feature information through a feature aggregation module, inputs it into a bilaterally guided aggregation query update layer, performs semantic segmentation through a segmentation head and obtains semantic segmentation results.

[0007] Technical solution: The present invention provides a lightweight semantic segmentation method based on Transformer, which specifically includes the following steps:

[0008] (1) Extract features from the pre-acquired image to obtain a feature map F1 of the image to be processed;

[0009] (2) Construct a feature aggregation module to aggregate channel information and obtain a feature map F2 corresponding to the image to be processed;

[0010] (3) Construct a Transformer-based bilaterally guided aggregation query update layer (BGA-QU Layer) to perform bilateral guided aggregation on the shallow and deep layers of the feature map to obtain semantic feature guidance and semantic feature aggregation respectively;

[0011] (4) Perform semantic segmentation on the attention aggregation updated feature map to obtain the semantic segmentation result.

[0012] Furthermore, the implementation process of step (1) is as follows:

[0013] The MobileNetV2 network is used for feature extraction. The image is input into the network and the standard convolution of Conv3×3 is used to add local context information with stride=2. Then it is input into 4 MobileNetv2 networks, where the stride of the second and fourth MobileNetv2 is 2, and a 1 / 8 feature map F1 is obtained.

[0014] Furthermore, the implementation process of step (2) is as follows:

[0015] A MobileNetv2 and a feature aggregation module FAM are used. MobileNetv2 performs further feature extraction on the image. The feature aggregation module first performs channel expansion and then channel reduction to convert the feature map F∈R H×W×C Input into the regular convolution of Conv1×1 to get F∈R of the expanded channel H×W×C0 , use Dwconv3×3 convolution to aggregate features and reduce the amount of calculation, stride=2, and then perform BN-ReLU6 operation to obtain F∈R H / 2×W / 2×C0 ; Then through the Conv1×1 convolution and BN operation, the output of the deep convolution is projected into the low channel capacity space to obtain F∈R H / 2×W / 2×C1, bypass uses a Dwconv3×3 convolution to collect local spatial information features and expand them to deep features, stride=2, use BN to accelerate network convergence, control overfitting, Conv1×1 convolution to adjust the number of channels, and finally perform an addition operation to obtain a 1 / 16 feature map F2∈R H / 2×W / 2×C1 .

[0016] Furthermore, the bilaterally guided aggregation query update layer BGA-QU Layer in step (3) consists of two parts: a bilaterally guided aggregation module and a query update strategy.

[0017] Furthermore, the bilateral guided aggregation module upsamples the feature map F2 output in step (2) to the same size F2' as 1 / 8 of the feature map F1, and performs a Concat splicing operation with F1 to fuse features, aggregate multi-scale information, and compensate for edge detail information of small targets; then performs Conv1×1 convolution, BN, and ReLU6 operations; F2' performs GAPooling 3×3 convolution, BN, and Conv1×1 convolution, and the result F2" uses Sigmoid operation and performs element-level multiplication operation to obtain a deep semantic guided feature map, GAPooling 3×3 convolution is used as the nonlinear activation layer of the network to reduce the number of parameters and increase the stability and accuracy of the network. The Sigmoid operation integrates the weight of the channel information of the feature network. Finally, Dwconv3×3 convolution aggregates the features, and BN and Conv1×1 convolution are adjusted to update the final query X1. F2" is expanded to the size of F2' through BN, ReLU6, and Broadcast operations and added to F2' to obtain X2, embedding global context information. The above operation formula is expressed as follows:

[0018] F2'=Upsample(F2)

[0019] F1'=ReLU6(BN(Conv(Concat(F2',F1))))

[0020] F2”=Conv(BN(GAPooling(F2')))

[0021] X0=F1'·Sigmoid(F2")

[0022] X1=Conv(BN(Dwconv(X0)))

[0023] F2”'=Broadcast(ReLU6(BN(F2”)))

[0024] X2=F2”'+F2'

[0025] Furthermore, the query update strategy is:

[0026] Use deep semantics to initially guide shallow aggregate information K and V, and then use shallow aggregate information to query the updated shallow aggregate information K and V to help group feature information;

[0027] Semantic relationship guidance: Input K and V of X1 projection and Q of X2 projection into MHSA, add the output result to X2 and perform Layernorm operation to obtain Xm1;

[0028] Feature embedding: Use the feedforward network FFN to process Xm1, add the output result to Xm1 and perform Layernorm operation to obtain Xm1';

[0029] Shallow query update: Input K and V of Xm1' projection and Q of X1 projection into MHSA, add the output result to X1 and perform Layernorm operation to obtain Xm2;

[0030] Update feature embedding: Use the feedforward network FFN to process Xm2, add the output result to Xm2 and perform Layernorm operation to obtain Xm2'. The formula is as follows:

[0031] X m1 =Norm(X2+MHSA(X2,X1,X1))

[0032] X m1 '=Norm(X m1 +FFN(X m1 ))

[0033] X m2 =Norm(X1+MHSA(X1,X m1 ',X m1 '))

[0034] X m2 '=Norm(X m2 +FFN(X m2 )).

[0035] Furthermore, the implementation process of step (4) is as follows:

[0036] The output results after two BGA-QULayer processes are input into the segmentation head, the feature map is semantically segmented, the number of channels is adjusted, and Conv1×1, BN, ReLU6 and Conv1×1 operations are performed to obtain the semantic segmentation results.

[0037] The present invention provides a lightweight semantic segmentation system based on Transformer, comprising:

[0038] The feature extraction module extracts features from the pre-acquired image to obtain a feature map F1 of the image to be processed;

[0039] The feature aggregation module aggregates the channel information to obtain the feature map F2 corresponding to the image to be processed;

[0040] The bilateral guided aggregation query update layer module performs bilateral guided aggregation on the shallow and deep layers of the feature map to obtain semantic feature guidance and semantic feature aggregation respectively;

[0041] The semantic segmentation module inputs the output results of the two BGA-QU Layer processes into the segmentation head, performs semantic segmentation on the feature map, and obtains the semantic segmentation results.

[0042] The present invention provides a device, comprising a memory and a processor, wherein:

[0043] a memory for storing computer programs capable of running on the processor;

[0044] A processor is used to execute the steps of the above-mentioned Transformer-based lightweight semantic segmentation method when running the computer program.

[0045] The present invention provides a storage medium having a computer program stored thereon. When the computer program is executed by at least one processor, the steps of the above-mentioned Transformer-based lightweight semantic segmentation method are implemented.

[0046] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: the feature aggregation module constructed by the present invention effectively aggregates low-level feature information in narrow channels, while collecting local spatial information features in the low-level space and expanding them to deep features; the constructed bilaterally guided aggregation query update layer includes a bilaterally guided aggregation module and a query update. The bilaterally guided aggregation module aggregates multi-scale information, compensates for edge detail information, strengthens the features guided by deep semantics, and integrates the weights of channel information in the feature network to avoid information loss during nonlinear activation. At the same time, the query update can well help group feature information and improve segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Schematic diagram of the lightweight semantic segmentation network structure proposed in this invention;

[0048] Figure 2 This is a schematic diagram of the feature aggregation module structure proposed in the present invention;

[0049] Figure 3 This is a schematic diagram of the bilateral guided aggregate query update layer structure proposed by the present invention;

[0050] Figure 4 This is a schematic diagram of the splitting head structure proposed by the present invention;

[0051] Figure 5 This is the image segmentation effect on the ADE20K dataset. DETAILED DESCRIPTION

[0052] The present invention will be described in further detail below with reference to the accompanying drawings.

[0053] This paper provides a lightweight semantic segmentation method based on Transformer, which includes extracting features from the input image, aggregating channel information using a feature aggregation module to obtain a feature map corresponding to the image to be processed; performing bilateral guided aggregation on the shallow and deep layers of the feature map to obtain semantic feature guidance and semantic feature aggregation, respectively; processing based on the Transformer's attention mechanism to obtain an attention-aggregated updated feature map; and performing semantic segmentation on the attention-aggregated updated feature map to complete the construction of a semantic segmentation network. The specific implementation process is as follows:

[0054] like Figure 1 As shown in the figure, the lightweight semantic segmentation network constructed by the present invention includes MV2 (MobileNetv2), feature aggregation module (FAM), bilateral guided aggregation query update layer (BGA-QU Layer), and segmentation head (Seg Head).

[0055] Step 1: Perform feature extraction on the input image to obtain a feature map F1 corresponding to the image to be processed. This method uses MV2 (MobileNetV2) as the feature extraction network. Unlike conventional convolution, where each convolution kernel operates on every channel of the input image simultaneously, the Depthwise Convolution proposed by MV2 uses one convolution kernel for each channel, and each channel is convolved with only one convolution kernel, which can significantly reduce parameters and computational complexity. The image is input into the network and a standard Conv3×3 convolution is used to add local context information with a stride of 2. The image is then fed into four MV2 modules, with the second and fourth MV2 modules having a stride of 2, to obtain a 1 / 8 feature map F1.

[0056] Step 2: Construct a feature aggregation module. During the feature extraction process, when the number of channels in the low-level feature map is relatively small, all feature information is concentrated in relatively narrow channels. At this time, nonlinear activation such as ReLU will lose a lot of information. This method uses an MV2 (MobileNetv2) and a designed feature aggregation module (Feature Aggregation Module, FAM). MV2 performs further feature extraction of the image, and the feature aggregation module first performs channel expansion and then channel reduction, such as Figure 2 As shown in the figure, it is a schematic diagram of the feature aggregation module, which transforms the feature map F∈R H×W×C Input into the regular convolution of Conv1×1 to get F∈R of the expanded channel H×W×C0 The BN-ReLU6 operation helps to better select features. Dwconv3×3 convolution is used to aggregate features and reduce the amount of calculation. The stride is 2, and then the BN-ReLU6 operation is performed. The obtained F∈R H / 2×W / 2×C0 Then, through the Conv1×1 convolution and BN operation, the output of the deep convolution is projected into the low channel capacity space to obtain F∈R H / 2×W / 2×C1 , bypass uses a Dwconv3×3 convolution to collect local spatial information features and expand them to deep features, stride=2, use BN to accelerate network convergence, control overfitting, Conv1×1 convolution to adjust the number of channels, and finally perform an addition operation to obtain a 1 / 16 feature map F2∈R H / 2×W / 2×C1 .

[0057] Step 3: Construct a Transformer-based bilateral guided aggregation query update layer (BGA-QU Layer), such as Figure 3As shown in the figure, the specific structure of BGA-QU Layer is shown, which consists of two parts: bilateral guided aggregation module and query update strategy. In the bilateral guided aggregation module on the left, the feature map F2 of step 2 output 1 / 16 is upsampled to the same size F2' as the feature map F1 of 1 / 8, and concatenated with F1 to fuse features, aggregate multi-scale information, and make up for the edge detail information of small targets. Then, Conv1×1 convolution, BN (Batch Normalization), and ReLU6 operations are performed, and F2' is GAPooling The result F2" after 3×3 convolution, BN, and Conv1×1 convolution uses Sigmoid operation and element-wise multiplication to obtain a feature map guided by deep semantics. GAPooling 3×3 convolution is used as the nonlinear activation layer of the network to reduce the number of parameters and increase the stability and accuracy of the network. The Sigmoid operation integrates the weights of the channel information of the feature network. Finally, Dwconv 3×3 convolution aggregates the features, and BN and Conv1×1 convolution are adjusted to serve as the final query update X1. F2" is expanded to the size of F2' through BN, ReLU6, and Broadcast operations and added to F2' to obtain X2, embedding global context information. The above operations can be expressed as follows:

[0058] F2'=Upsample(F2)

[0059] F1'=ReLU6(BN(Conv(Concat(F2',F1))))

[0060] F2”=Conv(BN(GAPooling(F2')))

[0061] X0=F1'·Sigmoid(F2")

[0062] X1=Conv(BN(Dwconv(X0)))

[0063] F2”'=Broadcast(ReLU6(BN(F2”)))

[0064] X2=F2”'+F2'

[0065] The purpose of the query update strategy is to use deep semantics to initially guide shallow aggregate information K, V, and then use the shallow aggregate information to query the updated shallow aggregate information K, V to help group feature information. Figure 3As shown, MHSA (Multi-Head Self-Attention) and FFN (Feed-Forward Network) are existing technologies, add is the addition operation, and norm is the Layer norm. Semantic relationship guidance: Input the K and V of the X1 projection and the Q of the X2 projection into the MHSA, add the output result to X2 and perform the Layer norm operation to obtain Xm1. Feature embedding: Use the feedforward network FFN to process Xm1, add the output result to Xm1 and perform the Layer norm operation to obtain Xm1'. Shallow query update: Input the K and V of the Xm1' projection and the Q of the X1 projection into the MHSA, add the output result to X1 and perform the Layer norm operation to obtain Xm2. Update feature embedding: Use the feedforward network FFN to process Xm2, add the output result to Xm2 and perform the Layer norm operation to obtain Xm2'. The above operations can be expressed by the following formula:

[0066] X m1 =Norm(X2+MHSA(X2,X1,X1))

[0067] X m1 '=Norm(X m1 +FFN(X m1 ))

[0068] X m2 =Norm(X1+MHSA(X1,X m1 ',X m1 '))

[0069] X m2 '=Norm(X m2 +FFN(X m2 )).

[0070] Step 4: Repeat step 3 and input the output into Figure 4 In the segmentation head (Seg Head) shown, semantic segmentation is performed according to the adjusted feature map to obtain a semantic segmentation result.

[0071] Deep networks have a larger receptive field and rich semantic information, but have low resolution and lack geometric details. Low-level networks have a smaller receptive field, rich geometric details, high resolution, and less semantic information. The output of step 2 is further input into two MV2s (MobileNetv2), with the second MV2 having a stride of 2, to obtain a 1 / 32 feature map. To combine the advantages of both, two BGA-QU layers are used to process 1 / 16 and 1 / 32 features respectively, and combine them with 1 / 8 features. The processed output is input into the segmentation head (Seg Head), which performs semantic segmentation on the feature map. The number of channels is adjusted, and Conv1×1, BN, ReLU6, and Conv1×1 operations are performed to obtain the semantic segmentation results.

[0072] The present invention provides a lightweight semantic segmentation system based on Transformer, comprising: a feature extraction module, which extracts features from a pre-acquired image to obtain a feature map F1 of the image to be processed; a feature aggregation module, which aggregates channel information to obtain a feature map F2 corresponding to the image to be processed; a bilaterally guided aggregation query update layer module, which performs bilaterally guided aggregation on the shallow layer and deep layer of the feature map to obtain semantic feature guidance and semantic feature aggregation, respectively; and a semantic segmentation module, which inputs the output results processed twice by the BGA-QU layer into a segmentation head, performs semantic segmentation on the feature map, and obtains a semantic segmentation result.

[0073] The present invention also provides an apparatus comprising a memory and a processor, wherein: the memory is used to store a computer program that can be run on the processor; the processor is used to execute the steps of the above-mentioned Transformer-based lightweight semantic segmentation method when running the computer program.

[0074] The present invention also provides a storage medium having a computer program stored thereon, which, when executed by at least one processor, implements the steps of the above-mentioned Transformer-based lightweight semantic segmentation method.

[0075] This implementation uses the Ubuntu 18.04 operating system and Python version 3.7. The computer configuration consists of a 15-core Intel(R) Xeon(R) Platinum 8358P CPU at 2.60GHz, 32GB of memory, and a V100 GPU with 16.00GB of memory. This paper uses the mean intersection-over-union (MIoU) as the evaluation metric for the semantic segmentation task. The batch size is set to 16, the maximum number of iterations is 160,000, and the AdamW optimizer is used with beta1 of 0.9, beta2 of 0.999, and weight decay of 0.01. Polynomial decay is used to adjust the learning rate, with a learning rate of 0.00025. The cross entropy loss function is used. The experiments are validated on the ADE20K dataset, released by the MIT CSAIL Computer Vision Group. It covers various annotations of scenes, objects, and object parts, including over 25k images of complex scenes, with an average of 19.5 instances and 10.5 object classes per image. The dataset includes 20,210 training images, 2,000 validation images, and 3,000 test images.

[0076] Table 1 Comparison on the ADE20K dataset

[0077]

[0078]

[0079] As shown in Table 1, in this dataset, our method is compared with three other currently well-performing lightweight semantic segmentation methods: SeaFormer, TopFormer, and RTFormer. As shown in Table 1, our method outperforms these methods with comparable or fewer FLOPs and latency, achieving higher segmentation accuracy, improving by 3.3%, 1.5%, 3.1%, and 1.7% compared to TopFormer-S, TopFormer-B, RTFormer-S, and SeaFormer-S, respectively.

[0080] The experimental results of the segmentation effect are as follows Figure 5 As shown, in the edge detail segmentation, the present invention can clearly identify the boundary between the tire edge and the car shadow. The segmentation detail effect of the front end of the car shows that the present method is also better than these methods. The results show that the present invention significantly improves the final segmentation effect.

[0081] It should be understood that the above-described specific embodiments of the present invention are merely illustrative or illustrative of the principles of the present invention and do not constitute limitations of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included within the scope of protection of the present invention. In addition, the appended claims are intended to cover all variations and modifications that fall within the scope and metes and bounds of the appended claims, or equivalents thereof.

Claims

1. A lightweight semantic segmentation method based on Transformer, characterized by: The following steps are involved: (1) Extract features from the pre-acquired image to obtain a feature map F1 of the image to be processed; (2) Construct a feature aggregation module to aggregate channel information and obtain a feature map F2 corresponding to the image to be processed; (3) Construct a Transformer-based bilaterally guided aggregation query update layer (BGA-QU Layer) to perform bilateral guided aggregation on the shallow and deep layers of the feature map to obtain semantic feature guidance and semantic feature aggregation respectively; (4) Perform semantic segmentation on the attention aggregation updated feature map to obtain the semantic segmentation result; The implementation process of step (2) is as follows: A MobileNetv2 and a feature aggregation module FAM are used. MobileNetv2 performs further feature extraction on the image. The feature aggregation module first performs channel expansion and then channel reduction to convert the feature map F∈R H×W×C Input into the regular convolution of Conv1×1 to get F∈R of the expanded channel H×W×C0 ; Use Dwconv3×3 convolution to aggregate features and reduce the amount of calculation, stride=2, and then perform BN-ReLU6 operation to obtain F∈R H / 2×W / 2×C0 ; Then, through the Conv1×1 convolution and BN operation, the output of the deep convolution is projected into the low channel capacity space to obtain F∈R H / 2×W / 2×C1 , bypass uses a Dwconv3×3 convolution to collect local spatial information features and expand them to deep features, stride=2, use BN to accelerate network convergence, control overfitting, Conv1×1 convolution to adjust the number of channels, and finally perform an addition operation to obtain a 1 / 16 feature map F2∈R H / 2×W / 2×C1 ; The bilaterally guided aggregation query update layer BGA-QU Layer in step (3) consists of two parts: a bilaterally guided aggregation module and a query update strategy; The bilateral guided aggregation module upsamples the feature map F2 output in step (2) to the same size F2' as 1 / 8 of the feature map F1, and performs a Concat splicing operation with F1 to fuse features, aggregate multi-scale information, and compensate for edge detail information of small targets; then performs Conv1×1 convolution, BN, and ReLU6 operations; After F2' performs GAPooling 3×3 convolution, BN, and Conv1×1 convolution, the result F2" uses Sigmoid operation and element-level multiplication operation to obtain the feature map guided by deep semantics. GAPooling 3×3 convolution is used as the nonlinear activation layer of the network to reduce the number of parameters and increase the stability and accuracy of the network. The Sigmoid operation integrates the weight of the channel information of the feature network. Finally, Dwconv3×3 convolution aggregates the features, and BN and Conv1×1 convolution are adjusted as the final query update X1. F2" performs BN, ReLU6, and Broadcast operations to expand it to the size of F2' and adds it to F2' to obtain X2, embedding global context information. The above operation formula is expressed as follows: F2'=Upsample(F2) F1'=ReLU6(BN(Conv(Concat(F2',F1)))) F2”=Conv(BN(GAPooling(F2'))) X0=F1'·Sigmoid(F2") X1=Conv(BN(Dwconv(X0))) F2”'=Broadcast(ReLU6(BN(F2”))) X2=F2”'+F2'.

2. A lightweight semantic segmentation method based on Transformer according to claim 1, characterized in that: The implementation process of step (1) is as follows: The MobileNetV2 network is used for feature extraction. The image is input into the network and the standard convolution of Conv3×3 is used to add local context information with stride=2. Then it is input into 4 MobileNetv2 networks, where the stride of the second and fourth MobileNetv2 is 2, and a 1 / 8 feature map F1 is obtained.

3. The lightweight semantic segmentation method based on Transformer according to claim 1, characterized in that: The query update strategy is: Use deep semantics to initially guide shallow aggregate information K and V, and then use shallow aggregate information to query the updated shallow aggregate information K and V to help group feature information; Semantic relationship guidance: Input K and V of X1 projection and Q of X2 projection into MHSA, add the output result to X2 and perform Layernorm operation to obtain Xm1; Feature embedding: Use the feedforward network FFN to process Xm1, add the output result to Xm1 and perform the layer norm operation to obtain Xm1'; Shallow query update: Input K and V of Xm1' projection and Q of X1 projection into MHSA, add the output result to X1 and perform Layernorm operation to obtain Xm2; Update feature embedding: Use the feedforward network FFN to process Xm2, add the output result to Xm2 and perform Layernorm operation to obtain Xm2'. The formula is as follows: X m1 =Norm(X2+MHSA(X2,X1,X1)) X m1 '=Norm(X m1 +FFN(X m1 )) X m2 =Norm(X1+MHSA(X1,X m1 ',X m1 ')) X m2 '=Norm(X m2 +FFN(X m2 ))。 4. The lightweight semantic segmentation method based on Transformer according to claim 1, characterized in that The implementation process of step (4) is as follows: The output results after two BGA-QU layer processing are input into the segmentation head, the feature map is semantically segmented, the number of channels is adjusted, and Conv1×1, BN, ReLU6 and Conv1×1 operations are performed to obtain the semantic segmentation results.

5. A Transformer-based lightweight semantic segmentation system using the method according to any one of claims 1 to 4, comprising: The feature extraction module extracts features from the pre-acquired image to obtain a feature map F1 of the image to be processed; The feature aggregation module aggregates the channel information to obtain the feature map F2 corresponding to the image to be processed; The bilateral guided aggregation query update layer module performs bilateral guided aggregation on the shallow and deep layers of the feature map to obtain semantic feature guidance and semantic feature aggregation respectively; The semantic segmentation module inputs the output results of the two BGA-QU Layer processes into the segmentation head, performs semantic segmentation on the feature map, and obtains the semantic segmentation results.

6. A device, characterized in that: comprising a memory and a processor, wherein: a memory for storing computer programs capable of running on the processor; A processor, configured to, when running the computer program, execute the steps of the Transformer-based lightweight semantic segmentation method according to any one of claims 1 to 4.

7. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by at least one processor, implements the steps of the Transformer-based lightweight semantic segmentation method as described in any one of claims 1 to 4.

Images