A method for image semantic segmentation based on TransDeep model

Abstract

A method for image semantic segmentation based on the TransDeep model belongs to the field of image semantic segmentation technology in computer vision, and includes the following steps: 1. Obtain a public data set and perform data preprocessing; 2. Establish a TransDeep model: First, use a backbone network feature extraction module to extract low-level features and high-level features; second, use a dual-branch fusion architecture to process high-level features and low-level features respectively; fuse the semantic feature information of the dual branches through a feature fusion module, and use a linear upsampling module to restore to the desired feature map size; 3. Through a fusion loss function with trainable weights, effectively solve the class imbalance problem in the data set; 4. Use the data set for training to obtain the best network model. Comparative experiments have proved that the present invention has high efficiency and capabilities in feature extraction, context information fusion and segmentation accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image semantic segmentation in computer vision, and in particular relates to an image semantic segmentation method based on a deep semantic segmentation network TransDeep (Deep semantic segmentation network with efficient semantic linker) model integrating an efficient semantic linker. Background Art

[0002] Image semantic segmentation is an important task in computer vision, which aims to segment an image into different regions with semantic information, each of which corresponds to an object or part of an object in the image. With the rise of deep learning, convolutional neural network (CNN) has developed into the mainstream basic neural network model in the field of computer vision, because it can perform high-frequency analysis and processing of images and videos, and has shown excellent performance in multiple visual tasks.

[0003] Deep learning technology has been driving the development of semantic segmentation models since 2007. The fully convolutional neural network in 2014 pioneered the end-to-end segmentation of natural images. The main idea is to improve the image classification network into a semantic segmentation network, and restore the size of the feature map by turning the classifier into an upsampling layer, and perform end-to-end training, marking a major breakthrough in semantic segmentation. Subsequent models such as U-Net, SegNet and DeepLab continued to innovate, introducing encoder-decoder structures, skip connections, and dilated convolutions, greatly improving model performance and efficiency. In recent years, models such as ENet, LinkNet, BiseNet, and DeepLabv3+ have further optimized network structures, feature fusion, and multi-scale processing, bringing higher accuracy, speed, and adaptability to semantic segmentation tasks. Although the above methods have made significant progress in many aspects, they still have problems such as high computational complexity, low accuracy, poor processing of small objects and edges, and weak perception of long-distance pixel relationships.

[0004] Therefore, a new technical solution is urgently needed in the prior art to solve this problem. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an image semantic segmentation method based on the TransDeep model, which is used to solve the technical problems of the existing semantic segmentation model, such as complex calculation, insufficient segmentation accuracy, poor processing of small targets and edges, and weak perception of long-distance pixel relationships.

[0006] A method for image semantic segmentation based on a TransDeep model comprises the following steps, and the following steps are performed sequentially:

[0007] Step 1: Obtain the PASCAL VOC 2012 dataset, divide the dataset into training set, test set and validation set in proportion, and perform preprocessing operations on the image data in the training set, test set and validation set;

[0008] Step 2: Build a TransDeep-based image semantic segmentation model, which is referred to as the TransDeep model. The model includes a backbone network feature extraction module, a global feature encoding block, a residual nested feature enhancement module for high-level features, a residual nested feature enhancement module for low-level features, a feature map fusion module, and a linear upsampling module.

[0009] The backbone network feature extraction module includes a starting convolution layer, an inverted residual block, a 1×1 point-by-point convolution layer, a global average pooling layer, a 1×1 point-by-point convolution layer, and a feature hierarchical processing layer for performing a slicing operation, which are sequentially connected. The backbone network feature extraction module performs preliminary feature extraction to generate two feature layers of high-level features and low-level features;

[0010] Among them, the high-level feature layer first extracts global context features through the global feature encoding block, and then further enhances the feature representation through the high-level feature residual nested feature enhancement module to obtain enhanced high-level features;

[0011] The low-level feature layer strengthens the edge feature information and detail feature information through the low-level feature residual nested feature enhancement module to obtain enhanced low-level features;

[0012] The residual nested feature enhancement module of the high-level feature and the residual nested feature enhancement module of the low-level feature both adopt the residual nested feature enhancement module; the residual nested feature enhancement module is another identical CA coordinate attention module embedded in the external CA coordinate attention module; the CA coordinate attention module is referred to as CA module;

[0013] The working process of the residual nested feature enhancement module is:

[0014] The external CA module performs pooling operations on the input feature maps in two independent directions, X and Y, respectively, to generate two direction-aware feature maps f_h1 and f_w1, and replaces the height dimension and width dimension of the feature map f_w1 to match its dimension with another feature map f_h1, and concatenates the two into the feature map f1; then passes through a convolution block, and applies the Sigmoid activation function to introduce nonlinear features to generate the feature map f1';

[0015] The external CA module separates the feature map f1' along the spatial dimension to generate two feature maps f_h1' and f_w1', and replaces the height dimension and width dimension of the feature map f_w1' to match its dimension with the other feature map f_h1'; then a convolution block is used to calculate the attention weights of the feature map f1' in the X and Y directions, and the Sigmoid activation function is applied to compress the input values ​​between 0 and 1 to generate feature maps f_h1'' and feature maps f_w1'' respectively, and the input feature map of the external CA module is multiplied by the attention weights in the two directions in turn to obtain the feature map f1'' output by the external CA module;

[0016] The feature map f1'' passes through a convolution block to reduce the feature map channel, extract a part of the features output by the external CA module, and generate a new feature map f1''_conv;

[0017] The embedded CA module performs pooling operations on the feature map f1''_conv in two independent directions, X and Y, to generate two direction-aware feature maps f_h2 and f_w2, and replaces the height dimension and width dimension of the feature map f_w2 to match its dimension with another feature map f_h2, and concatenates the two into the feature map f2; then passes through a convolution block, and applies the Sigmoid activation function to introduce nonlinear features to generate the feature map f2';

[0018] The embedded CA module separates the feature map f2' along the spatial dimension to generate two feature maps f_h2' and f_w2', and transforms the feature map f_w2' to match its dimension with the other feature map f_h2'; then a convolution block is used to calculate the attention weights of the feature map f2' in the X and Y directions, and the Sigmoid activation function is applied to compress the input values ​​between 0 and 1 to generate feature maps f_h2'' and f_w2'', respectively. The input feature map f1''_conv of the embedded CA module is multiplied by the attention weights in the two directions in turn to obtain the feature map f2'' output by the embedded CA module;

[0019] The output f2'' of the embedded CA module is fused with the output f1''_conv of the external CA module after convolution to obtain the feature map F'; and then a convolution block is passed to generate the final feature map F'';

[0020] The feature map fusion module concatenates the enhanced high-level features and the enhanced low-level features in the channel dimension through upsampling, and then sequentially passes through two convolution blocks and inputs them into the linear upsampling module;

[0021] The linear upsampling module first maps each channel of the received feature map to a specified category through a convolution layer; then performs bilinear interpolation upsampling on the mapped feature map, and then restores the bilinear interpolation upsampled feature map to the input image size of the backbone network feature extraction module;

[0022] Step 3: Establish a fusion loss function, which combines the equilibrium loss function And the focal loss function , and then introduce the weights corresponding to each loss function, dynamically adjust the weights of each loss function, and enhance the generalization of the TransDeep model; wherein the focus loss function Trainable focusing parameters are used in

[0023] The fusion loss function is: ;

[0024] In the formula, express The loss function is For the i-th training The weight of is the focal loss function in the i-th training The weight of

[0025] Step 4: Use the transfer learning training strategy to train the TransDeep model on the training set to obtain the best TransDeep model;

[0026] Step 5: Use the best TransDeep model to perform image semantic segmentation on the test set to obtain the final segmentation result.

[0027] The specific process of generating two feature layers of high-level features and low-level features by the backbone network feature extraction module is as follows: first, the image data input to the backbone network feature extraction module is subjected to preliminary feature extraction through the starting convolution layer; then, it is further downsampled through a specified number of independent inverted residual blocks in sequence, and each inverted residual block includes a 1×1 dilated convolution layer, a 3×3 depth convolution layer, a 1×1 linear bottleneck convolution layer and a jump connection layer connected in sequence; then, the output feature map of the last inverted residual block is subjected to feature sorting through a 1×1 point-by-point convolution layer, and then the spatial dimension of the feature map is compressed into a one-dimensional feature vector by a global average pooling layer; again, the feature vector is subjected to convolution processing using a 1×1 point-by-point convolution layer; finally, the low-level features obtained after sequential processing by the starting convolution layer and the first several specified number of inverted residual blocks and the high-level features obtained after sequential processing by the starting convolution layer, all inverted residual blocks, 1×1 point-by-point convolution layers, global average pooling layers and 1×1 point-by-point convolution layers are obtained through the slicing operation of the feature hierarchical processing layer.

[0028] The processing process of the global feature encoding block is:

[0029] First, the image is divided into blocks through the Unfold operation, and the dimension is adjusted through the permute transposition function to adapt to the input requirements of the self-attention calculation, and the feature map after the dimension adjustment is obtained;

[0030] Secondly, a multi-head self-attention mechanism is used to perform self-attention calculation on the feature map after dimension adjustment and add it to the feature map after dimension adjustment for residual connection;

[0031] Again, a nonlinear transformation is performed through a multi-layer perceptron, and a residual connection is made with the output of the multi-head self-attention mechanism;

[0032] Then, the dimensions are rearranged through the permute transposition function, and the image is inversely divided into blocks through the Fold folding operation;

[0033] Next, dilated convolution blocks with different dilation rates are used to extract multi-scale feature information in parallel;

[0034] Finally, the feature fusion of multi-scale feature information is performed through channel splicing, and a convolution block adjusts and outputs the feature map.

[0035] Said The formula is:

[0036] ;

[0037] In the formula, is the number of negative samples of the jth class, is the number of positive samples of the jth class, and are the positive sample loss of the jth class and the negative sample loss of the jth class respectively.

[0038] Said The formula is:

[0039] ;

[0040] In the formula, is the weight factor of category t; is the probability that the TransDeep model predicts that category t is a positive category; is the probability that the TransDeep model predicts that category t is a negative category; is the focus parameter in the i-th training.

[0041] The training strategy of transfer learning in step 4 is specifically as follows: the initial weights of the DeeplabV3+ model pre-trained on the PASCAL VOC2012 dataset are used as the initial weights of the TransDeep model, and further training is performed on this basis, so that the TransDeep model can better utilize the features learned by the DeeplabV3+ model, reduce training time and improve segmentation efficiency and accuracy; during the training process, the TransDeep model adopts a feature extraction method and implements a parameter freezing strategy to adapt to the semantic segmentation task;

[0042] The parameter freezing strategy is as follows: the total number of training rounds is set to 100, with the first 50 rounds being the freezing stage and the last 50 rounds being the thawing stage. The batch_size in the thawing stage is reduced to 4, while the batch_size in the freezing stage is set to 8 to avoid the problem of insufficient video memory after thawing. The maximum learning rate is set to 0.007, the minimum learning rate is set to 0.00007, and the learning rate is adjusted by cosine annealing. At the same time, stochastic gradient descent SGD is used as the optimizer of the model, the built-in parameter momentum is set to 0.9, and the weight decay weight_decay parameter is set to 0.0001 to optimize the model parameters and minimize the loss function.

[0043] Through the above design scheme, the present invention can bring the following beneficial effects:

[0044] The TransDeep model adopted in the present invention adopts a lightweight backbone network feature extraction module to quickly extract multi-scale features; the integrated global feature encoding module expands the receptive field and enhances the multi-scale target capture capability through a multi-head self-attention mechanism and a parallel branch design; the spatial position information encoding is enhanced through a coordinate attention embedding mechanism; and the skip connection structure is used to fuse low-level detail features and high-level semantic features to achieve multi-scale feature fusion.

[0045] Among them, the backbone network feature extraction module is used as the front module of feature extraction, and high-level features and low-level features are retained respectively, so that the model can take into account both local detail information and global context information, improve the segmentation accuracy, solve the shortcomings in computational efficiency, and reduce the computational cost of the model.

[0046] Among them, the global feature encoding block adopts a multi-head self-attention mechanism to capture global context dependencies, and combines it with dilated convolutions with different expansion rates to extract multi-scale features, allowing the model to focus on multiple different positions from different subspaces, thereby capturing information on features at different levels, enhancing the model's ability to handle long-distance dependencies and global information modeling capabilities, improving the ability to segment small targets, and demonstrating the model's flexibility in segmenting objects of different scales.

[0047] Among them, after processing the information in the two dimensions of channel and space through the residual nested feature enhancement module, the model's recognition of the semantic information and edge details of the segmentation target is improved, which helps to accurately locate the position of the target object, thereby improving the segmentation performance of the model.

[0048] The present invention also adopts an improved fusion loss function. By combining the two loss functions of Equalization Loss v2 and Focal Loss, it not only improves the model's ability to handle unbalanced data sets, but also enhances the model's generalization performance when facing new and unseen data.

[0049] Data comparison and verification show that the image semantic segmentation method based on the TransDeep model in the present invention has higher efficiency and capability in feature extraction, context information fusion and segmentation accuracy compared with other traditional models. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention is further described below with reference to the accompanying drawings and specific embodiments:

[0051] Figure 1 This is a flow chart of an image semantic segmentation method based on the TransDeep model of the present invention;

[0052] Figure 2 The figure is a diagram of the architecture of the TransDeep model and image processing effect in an image semantic segmentation method based on the TransDeep model of the present invention;

[0053] Figure 3 This is a structural diagram of a backbone network feature extraction module in an image semantic segmentation method based on a TransDeep model of the present invention;

[0054] Figure 4 This is a structural diagram of a global feature encoding block in an image semantic segmentation method based on a TransDeep model of the present invention;

[0055] Figure 5 This is a structural diagram of a residual nested feature enhancement module in an image semantic segmentation method based on a TransDeep model in the present invention. DETAILED DESCRIPTION

[0056] The following is a further detailed description of an image semantic segmentation method based on the TransDeep model of the present invention in conjunction with the accompanying drawings and embodiments of the specification. It should be understood that the embodiments described below are only part of the embodiments of the present invention, not all of them. Example

[0057] like Figure 1 and Figure 2 As shown, the present invention provides an image semantic segmentation method based on the TransDeep model, which specifically comprises the following steps:

[0058] Step 1: Get the PASCAL VOC 2012 dataset, divide the dataset into training set, test set and validation set, ensure that the division process of the dataset follows the appropriate ratio, and perform preprocessing operations on the image data in the dataset;

[0059] The specific processing process of the preprocessing operation is as follows: first, read the image data and the corresponding label image data; then, convert the image and label image from RGB to HSV color space and perform data enhancement, which includes random scaling, horizontally flipping the image with a probability of 50%, filling the redundant part of the image with gray, applying Gaussian blur to the image with a probability of 25%, randomly rotating the image with a probability of 25%, and randomly adjusting the hue, saturation and brightness of the image; then, convert the data-enhanced image into a floating-point array, and convert the image dimension from (H, W, C) to (C, H, W); finally, convert the label image into a one-hot encoding form, and reshape the one-hot encoding vector of each pixel into a three-dimensional tensor with a shape of (H, W, num_classes+1), where num_classes+1 represents the number of object categories in the image including the background, and return to obtain the preprocessed image, label and segmentation mask.

[0060] In this embodiment, the PASCAL VOC 2012 data set is divided into 1464 training sets, 1449 validation sets and 1456 test sets, and the above data preprocessing operations are performed. Wherein, while converting the image to a floating point array, the dimension of the image is converted from (512, 512, 4) to (4, 512, 512). When the label is converted to a one-hot encoding form, the one-hot encoding vector of each pixel is reshaped into a three-dimensional tensor with a shape of (512, 512, 22).

[0061] Step 2: Build a TransDeep-based image semantic segmentation model, which is referred to as the "TransDeep model". The TransDeep model includes a backbone network feature extraction module, a global feature encoding block, a high-level feature residual nested feature enhancement module, a low-level feature residual nested feature enhancement module, a feature map fusion module, and a linear upsampling module. The image after data preprocessing is subjected to feature extraction by the backbone network feature extraction module to generate two feature layers of high-level features and low-level features. The high-level features retain more semantic information, and the low-level features retain more detail information. Then, two branches are used for processing respectively; the two branches include a high-level feature processing branch and a low-level feature processing branch. Among them, the high-level feature processing branch first extracts global context features through the global feature encoding block, and then further enhances the feature representation through the high-level feature residual nested feature enhancement module to focus on important feature information; among them, the low-level feature processing branch strengthens information such as edge and detail features through the low-level feature residual nested feature enhancement module; then, the features processed by the two branches are fused through the feature fusion module; and then upsampling is performed through the linear upsampling module.

[0062] The residual nested feature enhancement module of the high-level feature and the residual nested feature enhancement module of the low-level feature both use the residual nested feature enhancement module; the residual nested feature enhancement module is another identical CA coordinate attention module embedded in the external CA coordinate attention module; the CA coordinate attention module is referred to as the CA module. The working process of the CA module is as follows: first, the 2D channel attention is decomposed into two parallel 1D features, and global average pooling is performed along the horizontal and vertical directions of the feature map. While capturing cross-channel information, the direction perception and position perception information are captured, and then the two direction perception feature maps are merged and processed through the convolution block, and finally the features of the two directions are separated for transposition operation.

[0063] In this embodiment, Figure 3 As shown in the figure, the specific processing process of the backbone network feature extraction module to generate high-level features and low-level features is:

[0064] The image data input to the backbone network feature extraction module is passed through the initial convolutional layer for preliminary feature extraction;

[0065] It is then further downsampled through independent inverted residual blocks arranged in sequence, where the number of inverted residual blocks can be manually adjusted and selected based on the optimal configuration of the model's computational efficiency and performance. In this embodiment, 17 inverted residual blocks are selected. Each inverted residual block includes a 1×1 dilated convolution layer, a 3×3 deep convolution layer, a 1×1 linear bottleneck convolution layer, and a jump connection layer connected in sequence. The inverted residual block expands the number of input channels through a 1×1 dilated convolution, and then efficiently extracts features through a 3×3 deep convolution, and then restores the number of channels to the same order of magnitude as the number of input channels through a 1×1 linear bottleneck convolution, and then adds the input feature map to the output feature map to achieve a residual connection, which is then used as the input of the next inverted residual block, thereby solving the problem of gradient disappearance during training;

[0066] The output feature map of the last inverted residual block is processed through a 1×1 point-by-point convolution layer, and then the global average pooling layer compresses the spatial dimension of the feature map into a one-dimensional feature vector to retain the key information of the feature map.

[0067] The one-dimensional feature vector is then convolved using a 1×1 point-by-point convolution layer, and the low-level features obtained by processing the starting convolution layer and the partially inverted residual block and the high-level features obtained by processing the starting convolution layer, all inverted residual blocks, point-by-point convolution layer, average pooling, and point-by-point convolution layer are obtained through the slicing operation of the feature layer processing layer. In this embodiment, the low-level features obtained after processing the starting convolution layer and the first three inverted residual blocks are selected as raw_L(4, 24,128, 128), and the high-level features obtained after processing the starting convolution layer, all inverted residual blocks, point-by-point convolution layer, average pooling, and point-by-point convolution layer are obtained as raw_H(4, 320, 32, 32).

[0068] Furthermore, the 1×1 dilated convolution layer, 3×3 depth convolution layer and 1×1 linear bottleneck convolution layer of the inverted residual block respectively use ReLU6 as the activation function, thereby avoiding the information loss problem caused by the ReLU activation function in the low-dimensional space.

[0069] Furthermore, the ReLU6 is a restricted ReLU activation function that limits the output of ReLU to between 0 and 6. From a mathematical point of view, ReLU6 is defined as:

[0070] ;

[0071] Where x represents the feature map tensor generated by the 1×1 dilated convolution layer, 3×3 depthwise convolution layer, or 1×1 linear bottleneck convolution layer in the inverted residual block.

[0072] Furthermore, the 1×1 point-by-point convolutional layer combines the information of each input channel to form a new feature representation, which significantly reduces the number of parameters and computational requirements.

[0073] The global feature encoding block is used to extract global context features during high-level feature processing, such as Figure 4 As shown in the figure, the processing process of the global feature encoding block is as follows: first, the image is divided into blocks for independent processing through the Unfold operation to capture detail information in a smaller local area, and the dimension is adjusted through the permute transposition function to adapt to the input requirements of the self-attention calculation; secondly, the multi-head self-attention mechanism is used to perform self-attention calculation and add it to the feature map after the dimension is adjusted to form a residual connection, which captures the dependency between different positions in the feature map and enhances the model's understanding of spatial information; thirdly, a nonlinear transformation is performed through a multi-layer perceptron, and a residual connection is performed again with the input of the multi-layer perceptron; then, the dimension is rearranged through the permute transposition function, and the image block inverse operation is performed through the Fold folding operation; then, the hole convolution blocks with different expansion rates are used to extract multi-scale feature information in parallel, and the hole convolution blocks include a 1×1 convolution block, three 3×3 convolution blocks, and a global average pooling block; finally, feature fusion is performed through channel splicing, and a convolution block is used to adjust the feature map.

[0074] Furthermore, the Unfold operation divides the input feature map into multiple blocks and flattens each block into a one-dimensional vector. The feature map size changes from [B, C, H, W] to , B represents the batch size, C represents the number of channels, H and W represent the length and width of the input feature map respectively, where It indicates the number of blocks obtained after sliding cropping of the feature map according to the convolution kernel size kernel_size. The calculation formula is:

[0075] (2);

[0076] In the formula, It represents the size of the input image, padding represents the zero padding around the input image, dilation represents the dilation ratio of the dilated convolution, kernel_size represents the size of the convolution kernel, stride represents the sliding step of the kernel, and d represents the index of the traversal space dimension. Only the values ​​0 and 1 need to be traversed.

[0077] Furthermore, the processing process of the multi-head self-attention mechanism is as follows: first, the input feature map is processed by a dimension reduction layer, and then enters a linear layer to generate three matrices: a query matrix Q, a key matrix K, and a value matrix V; then, the three matrices are divided into multiple blocks containing partial features, and a scaled dot product attention operation is performed on each block; finally, all the blocks are concatenated, and the original feature map dimension is restored through a linear layer to ensure that the output is consistent with the dimension of the input sequence.

[0078] Furthermore, the scaled dot product attention is calculated by calculating the dot product of the query matrix Q and each key matrix K, and converting the dot product into a probability distribution using the Softmax function to determine the importance of each value, and the output is obtained by the final weighted summation. The calculation formula of the scaled dot product attention operation is:

[0079] (3);

[0080] in denote the query matrix, key matrix and value matrix of the i-th head respectively, is the dimension of the key matrix K vector, represents the output of the i-th head, represents the calculation of the self-attention mechanism, Indicates that the attention weights are normalized. In particular, the scaling factor is introduced Scaling the dot product result to a reasonable range avoids the gradient vanishing or exploding problem.

[0081] In this embodiment, for the high-level feature raw_H(4, 320, 32,32) generated by the backbone network feature extraction module, the specific processing process is as follows:

[0082] The raw_H(4, 320, 32, 32) is divided into blocks for independent processing through the Unfold operation to capture detailed information in a smaller local area, and 3×3 small blocks are extracted from the high-level features. The step size during the extraction process is (1,1), the interpolation rate is (1,1), and the padding is "same" to ensure that the height and width of the output are the same as the input;

[0083] The permute function is used to adjust the feature dimension after segmentation from (4, 2880, 1024) to (4, 1024, 2880) to meet the input requirements of self-attention calculation;

[0084] After feature normalization, a multi-head self-attention mechanism is used for self-attention calculation to capture the dependencies between different positions in the feature map and enhance the model's understanding of spatial information. First, dimensionality reduction is performed to reduce the feature map from (4,1024, 2880) to (4, 1024, 1440), and then a linear layer is used to generate three matrices of size (4, 8, 1024, 180): query matrix Q, key matrix K and value matrix V. The three matrices are then divided into multiple blocks containing partial features. After performing a scaled dot product attention operation on each block, all the blocks are concatenated into (4, 1024, 8, 180), and then reshaped back to the original sequence length (4, 1024, 1440). The original feature map dimension (4, 1024, 2880) is restored through a linear layer to ensure that the output is consistent with the dimension of the input sequence. The output of the multi-head self-attention mechanism is then added to the feature map after the dimension adjustment to (4, 1024, 2880), forming a residual connection, and finally performing a nonlinear transformation through a multi-layer perceptron, and performing a residual connection with the output of the self-attention layer to form a feature map of (4, 1024, 2880);

[0085] The permute function is used to rearrange the dimensions and adjust the feature dimensions from (4, 1024, 2880) to (4, 2880, 1024).

[0086] The image is inversely partitioned by the Fold operation, folding back to the shape of the input image of the unfold operation (4,320, 32, 32). During the operation, the sliding window size is (3,3), the step size is 1, the expansion rate is 1, and the padding is 1;

[0087] Then, we use dilated convolutions with different expansion rates to extract multi-scale feature information in parallel, and then pass through convolution blocks I, II, III, IV, and global average pooling blocks in parallel to generate feature maps raw1_H (4, 256, 32, 32), the input channel of the convolution block I is 4, the output channel is 256, the convolution kernel size is 1×1, the step size is 1, the padding is 0, and the expansion rate is 1; the input channel of the convolution block II is 4, the output channel is 256, the convolution kernel size is 3×3, the step size is 1, the padding is 6, and the expansion rate is 6; the input channel of the convolution block III is 4, the output channel is 256, the convolution kernel size is 3×3, the step size is 1, the padding is 12, and the expansion rate is 12; the input channel of the convolution block IV is 4, the output channel is 256, the convolution kernel size is 3×3, the step size is 1, the padding is 18, and the expansion rate is 18; then the feature fusion is performed by channel splicing to raw2_H(4, 1280, 32,32), and the feature map is adjusted to raw'_H(4, 256, 32, 32), the input channel of the convolution block is 1280, the output channel is 256, the convolution kernel size is 1×1, the stride is 1, and the padding is 0.

[0088] In this embodiment, for the feature map raw'_H(4, 256, 32, 32) generated by the global feature encoding block, the specific processing process is as follows:

[0089] The external CA module performs pooling operations on raw'_H(4, 256, 32, 32) in two independent directions, X and Y, to generate two direction-aware feature maps, f_h1_H(4, 256, 32, 1) and f_w1_H(4, 256, 1, 32), so that the attention module can capture long-range spatial interactions with precise location information, and transform the feature map f_w1_H to (4,256, 32, 1) to match its dimension with another feature map f_h1_H(4, 256, 32, 1), and concatenate the two into f1_H(4, 256, 64, 1); then pass through a convolution block with an input channel of 256, an output channel of 8, a convolution kernel size of 1×1, a step size of 1, and a padding of 0, and then apply the Sigmoid activation function to introduce nonlinear features to generate a feature map f1'_H(4, 8, 64, 1);

[0090] The external CA module separates the feature map f1'_H(4, 8, 64, 1) along the spatial dimension to generate two f_h1'_H(4, 8, 32, 1) and f_w1'_H(4, 8, 32, 1), and converts the feature map f_w1'_H to (4, 8, 1,32); then a convolution block is used to calculate the attention weights of f1'_H(4, 8, 64, 1) in the X and Y directions respectively. The convolution block has 8 input channels, 256 output channels, a convolution kernel size of 1×1, a step size of 1, and a padding of 0. The Sigmoid activation function is applied to compress the input value to between 0 and 1 to generate the feature maps f_h1''_H(4, 256, 32, 1) and f_w1''_H(4, 256, 1, 32), the input feature map raw'_H(4, 256, 32, 32) of the external CA module is multiplied by the attention weights in the two directions in sequence to obtain the feature map f1''_H(4, 256, 32, 32) output by the external CA module;

[0091] The feature map f1''_H(4, 256, 32, 32) is reduced through a convolution block. The input channel of the convolution block is 256, the output channel is 128, the convolution kernel size is 1×1, the step size is 1, and the padding is 0. A part of the external CA features is extracted to generate a new feature map f1''_conv_H(4, 128, 32, 32);

[0092] The embedded CA module performs pooling operations on f1''_conv_H(4, 128, 32, 32) in two independent directions, respectively, to generate two direction-aware feature maps, f_h2_H(4, 128, 32, 1) and f_w2_H(4, 128, 1, 32), so that the attention module can capture long-range spatial interactions with precise location information, and transform the feature map f_w2_H to (4, 128, 32, 1) to match its dimension with another feature map f_h2_H(4, 128, 32, 1), and concatenate the two into f2_H(4, 128, 64, 1); then pass through a convolution block with 128 input channels, 8 output channels, 1×1 convolution kernel size, 1 step size, and 0 padding, and then apply the Sigmoid activation function to introduce nonlinear features to generate the feature map f2'_H(4, 8, 64, 1);

[0093] The embedded CA module separates the feature map f2'_H(4, 8, 64, 1) along the spatial dimension to generate two f_h2'_H(4, 8, 32, 1) and f_w2'_H(4, 8, 32, 1), and converts the feature map f_w2'_H(4, 8, 32, 1) to (4, 8, 1, 32); then a convolution block is used to calculate the attention weights of f2'_H(4, 8, 64, 1) in the X and Y directions respectively. The convolution block has 8 input channels, 128 output channels, a convolution kernel size of 1×1, a step size of 1, and a padding of 0. The Sigmoid activation function is applied to compress the input value to between 0 and 1 to generate feature maps f_h2''_H(4, 128,32, 1) and f_w2''_H(4, 128, 1, 32), the input feature map f1''_conv_H(4, 128,32, 32) of the embedded CA module is multiplied by the attention weights in the two directions in sequence to obtain the feature map f2''_H(4,128, 32, 32) output by the embedded CA module;

[0094] The output f2''_H(4, 128, 32, 32) of the embedded CA module is fused with the output f1''_conv_H(4, 128, 32, 32) of the external CA module after convolution to obtain the feature map F''_H(4, 128, 32, 32); and then a convolution block is passed to generate the final feature map F''_H(4, 256, 32, 32). The convolution block has 128 input channels, 256 output channels, a convolution kernel size of 1×1, a stride of 1, and padding of 0.

[0095] In this embodiment, the specific processing process of the low-level feature raw_L(4, 24, 128,128) generated by the backbone network feature extraction module is as follows:

[0096] (1) The external CA module performs pooling operations on raw_L(4, 24, 128, 128) in two independent directions, X and Y, to generate two direction-aware feature maps, f_h1_L(4, 24, 128, 1) and f_w1_L(4, 24, 1, 128), which enables the attention module to capture long-range spatial interactions with precise location information and transform the feature map f_w1_L to (4, 24, 128, 1) to match its dimension with another feature map f_h1_L(4, 24, 128, 1), and concatenate the two into f1_L(4, 24, 256, 1); then pass through a convolution block (input channel is 24, output channel is 8, convolution kernel size is 1×1, step size is 1, padding is 0), and then apply Sigmoid activation function to introduce nonlinear features to generate feature map f1'_L(4, 8, 256, 1);

[0097] (2) The external CA module separates the feature map f1'_L(4, 8, 256, 1) along the spatial dimension to generate two f_h1'_L(4, 8, 128, 1) and f_w1'_L(4, 8, 128, 1), and converts the feature map f_w1'_L to (4,8, 1, 128); then a convolution block is used to calculate the attention weights of f1'_L(4, 8, 256, 1) in the X and Y directions respectively. The convolution block has 8 input channels, 24 output channels, a convolution kernel size of 1×1, a step size of 1, and a padding of 0. The Sigmoid activation function is applied to compress the input value to between 0 and 1, generating feature maps f_h1''_L(4, 24, 128, 1) and f_w1''_L(4, 24, 1, 128), multiply the input feature map raw_L(4, 24, 128, 128) of the external CA module by the attention weights in the two directions in sequence to obtain the feature map f1''_L(4, 24, 128, 128) output by the external CA module;

[0098] (3) After a convolution block is used to reduce the feature map channel, the input channel of the convolution block is 24, the output channel is 12, the convolution kernel size is 1×1, the step size is 1, and the padding is 0. The features of a part of the internal CA are extracted to generate a new feature map f1''_conv_L(4, 12, 128, 128);

[0099] (4) The embedded CA module performs pooling operations on f1''_conv_L(4, 12, 128,128) in two independent directions, respectively, to generate two direction-aware feature maps, f_h2_L(4, 12, 128, 1) and f_w2_L(4, 12, 1, 128), so that the attention module can capture long-range spatial interactions with precise location information, and transform the feature map f_w2_L to (4, 12, 128, 1) to match its dimension with the other feature map, and concatenate the two into f2_L(4, 12,256, 1); then pass through a convolution block with 12 input channels, 8 output channels, a convolution kernel size of 1×1, a stride of 1, and padding of 0, and then apply the Sigmoid activation function to introduce nonlinear features to generate the feature map f2'_L(4, 8,256, 1);

[0100] (5) The embedded CA module separates the feature map f2'_L(4, 8, 256, 1) along the spatial dimension to generate two f_h2'_L(4, 8, 128, 1) and f_w2'_L(4, 8, 128, 1), and converts the feature map f_w2'_L to (4,8, 1, 128); then a convolution block is used to calculate the attention weights of f2'_L(4, 8, 256, 1) in the X and Y directions respectively. The convolution block has 8 input channels, 12 output channels, a convolution kernel size of 1×1, a step size of 1, and a padding of 0. The Sigmoid activation function is applied to compress the input value to between 0 and 1, generating feature maps f_h2''_L(4, 12, 128, 1) and f_w2''_L(4, 12 1, 128), multiply the input feature map f1''_conv_L(4, 12, 128, 128) of the embedded CA module by the attention weights in the two directions in sequence to obtain the feature map f2''_L(4, 12, 128,128) output by the embedded CA module;

[0101] (6) The output f2''_L(4, 12, 128, 128) of the embedded CA module is fused with the output f1''_conv_L(4, 12, 128, 128) of the external CA module after convolution to obtain the feature map F''_L(4, 12, 128,128); then it passes through a convolution block with 12 input channels, 24 output channels, 1×1 kernel size, 1 step size, and 0 padding to generate the final feature map F''_L(4, 24, 128, 128).

[0102] In this embodiment, the feature map fusion module concatenates F''_H(4, 256, 32, 32) and F''_L(4, 24, 128, 128) after double-branch processing in the channel dimension. The specific processing process is as follows:

[0103] The high-level feature F''_H(4, 256, 32, 32) is passed through the upsampling module to generate F1''_H(4, 256, 128,128), and the low-level feature F''_L(4, 24, 128, 128) is passed through a convolution block with 24 input channels, 48 ​​output channels, 1×1 convolution kernel size, 1 step size, and 0 padding to generate F1''_L(4, 48, 128, 128);

[0104] Concatenate the high-level features F1''_H(4, 256, 128, 128) and the low-level features F1''_L(4, 48, 128, 128) to generate the feature map F(4, 304, 128, 128);

[0105] The feature map F(4, 304, 128, 128) enters convolution block 1 and convolution block 2 in turn to enrich the feature representation and extract local features, generating a feature map F'(4, 256, 128, 128). The input channel of convolution block 1 is 304, the output channel is 256, the convolution kernel size is 3×3, the step size is 1, and the padding is 1; the input channel of convolution block 2 is 256, the output channel is 256, the convolution kernel size is 3×3, the step size is 1, and the padding is 1.

[0106] In this embodiment, the linear upsampling module restores the feature map F'(4, 256, 128, 128) after feature fusion to the original input size. The specific processing process is as follows:

[0107] Each channel of the feature map is mapped to a category through a convolution block, and the output feature map F''(4, 21,128, 128) is output. The input channel of the convolution block is 256, the output channel is 21, the convolution kernel size is 1×1, the step size is 1, and the padding is 0;

[0108] The feature map F''(4, 21, 128, 128) is upsampled by bilinear interpolation to restore the feature map to the same size as the input image of the backbone network feature extraction module in order to generate a segmentation map aligned with the input image of the backbone network feature extraction module. The final output shape is (4, 21, 512, 512), where the value of each pixel represents the probability that the pixel belongs to a certain category.

[0109] Each convolution block in the above steps includes a convolution layer, a normalization layer, a ReLU activation function, and a Dropout regularization layer.

[0110] Step 3: Establish a fusion loss function, which combines the balanced loss function And the focal loss function , through trainable parameters and Dynamically adjust the weights of each loss function to optimize the model's ability to handle unbalanced data sets and enhance model generalization.

[0111] The focal loss function is the unmodified focal loss function Parameters in Modify to trainable parameters , and dynamically adjust the weights of each loss function to optimize the model's ability to handle unbalanced data sets and enhance the generalization of the model. It can give the model the ability to adaptively adjust the degree of attention paid to easy-to-classify and difficult-to-classify samples.

[0112] in, The formula is shown in formula (4):

[0113] (4) ;

[0114] In the formula, is the number of negative samples of the jth class, is the number of positive samples of the jth class, and are the positive sample loss of the jth class and the negative sample loss of the jth class respectively.

[0115] Among them, the unmodified focal loss function The formula is shown in formula (5):

[0116] (5) ;

[0117] In the formula, is the weight factor of category t, which is used to balance the importance of different categories; is the probability that the TransDeep model predicts that category t is a positive category; is the probability that the TransDeep model predicts that category t is a negative category; It is a fixed focusing parameter used to reduce the weight of easy-to-classify samples and increase the weight of difficult-to-classify samples.

[0118] The focal loss function The formula is:

[0119] (6);

[0120] In the formula, Represents the focus parameter in the i-th training. During the back-propagation process, the optimizer automatically adjusts the parameter value.

[0121] Specifically, as shown in formula (6), The adaptive adjustment of enables the TransDeep model to dynamically adjust the focus on easy-to-classify samples according to the current training status, thereby allocating more training resources to difficult-to-classify samples, reducing the loss contribution of easy-to-classify samples and increasing the loss ratio of difficult-to-classify samples, which is especially important for long-tail distribution data sets. Balanced loss function Through the gradient-guided reweighting mechanism, the training process of each category is rebalanced independently and equally, which effectively solves the problem of positive and negative gradient imbalance in long-tail object detection.

[0122] The fusion loss function combines the two loss functions of formula (4) and formula (6), and the fusion loss function is shown in formula (7):

[0123] (7);

[0124] In the formula, for The loss function is For the i-th training The weight of is the focal loss function in the i-th training The weight of

[0125] The fusion loss function is the total loss, and trainable parameters are introduced into the fusion loss function and , so that the TransDeep model can continuously adjust during the training process to optimize the impact of the two loss functions on the total loss and dynamically balance the contribution of the two loss functions to the total loss. The fusion loss function not only improves the TransDeep model's ability to handle unbalanced data sets, but also enhances the model's generalization performance when facing new and unseen data;

[0126] Among them, dynamic balance and The specific strategies of these two loss functions' contribution to the total loss are:

[0127] During the training process of the TransDeep model, It is updated as an additional parameter of the TransDeep model and is usually optimized by gradient descent. The initial value of is usually set to a small positive number. The value of is set to 2;

[0128] and The initial value of can be set according to the scale of the loss function and the characteristics of the problem. In this embodiment, the initial value is 0.5, which means that the two initially contribute equally to the total loss.

[0129] In each iteration, the back-propagation algorithm is used to update the TransDeep model based on its performance on the validation set. and The specific process is to add {'params': [lambda_weight,eta_weight]} in the optimizer settings.

[0130] In the i-th round of training, when When the value of no longer decreases significantly or decreases very slowly, and The optimal solution in the i-th round is used as the optimal solution in the i+1th round. and The initial value of is updated as training continues.

[0131] Step 4: Use the transfer learning training strategy to train the TransDeep model on the training set until the best TransDeep model is obtained; the transfer learning training strategy avoids the random initialization of weights when training the model from scratch, which leads to poor feature extraction effects and other problems. The transfer learning training strategy loads the initial weights pre-trained by the DeeplabV3+ model on the PASCAL VOC 2012 dataset, and fine-tunes the TransDeep model on this basis, so that the TransDeep model can better utilize the features learned by the DeeplabV3+ model, reduce training time and improve segmentation efficiency and accuracy. Specifically:

[0132] First, the DeeplabV3+ model is used to train on the PASCAL VOC 2012 dataset to obtain the Step1.pth weight file. Then, the Step1.pth weight file is loaded using the TransDeep model and fine-tuned on the PASCAL VOC 2012 dataset. The fine-tuning process is the training process, which makes the TransDeep model better adapt to the characteristics of the dataset, reduces training time and improves segmentation efficiency and accuracy.

[0133] During the training process, a parameter freezing strategy is implemented. Specifically, the total number of training rounds is set to 100, with the first 50 rounds being the freezing stage and the last 50 rounds being the thawing stage. The batch_size in the thawing stage is reduced to 4, while the batch_size in the freezing stage is set to 8 to avoid the problem of insufficient video memory after thawing.

[0134] The maximum learning rate of the TransDeep model is set to 0.007, the minimum learning rate is set to 0.00007, and the learning rate is adjusted by cosine annealing. At the same time, stochastic gradient descent SGD is used as the optimizer of the TransDeep model, and the built-in parameter momentum is set to 0.9 and the weight decay parameter weight_decay is set to 0.0001 to optimize the TransDeep model parameters and minimize the loss function.

[0135] Step 5: Perform image semantic segmentation on the test set using the best TransDeep model to obtain the final segmentation result;

[0136] The best TransDeep model is compared with several existing methods on the PASCAL VOC 2012 dataset to verify the feasibility and effectiveness of the TransDeep model.

[0137] Table 1 provides the segmentation performance of various models on the PASCAL VOC 2012 dataset:

[0138]

[0139] From the comparison results, it can be seen that compared with other models, the TransDeep model achieves the best mean intersection-over-union (MIoU) metric on semantic segmentation tasks, which is significantly better than other networks. It not only surpasses traditional convolutional neural networks (such as VGG-16, ResNet38) and many other Transformer-based models (such as SegFormer, ToCo), but also outperforms other networks using the same backbone network backbone (such as PSPNet, DeepLab V3+). Specifically, UNet uses VGG-16 as the backbone, and its performance is relatively low compared to the TransDeep model, with a MIoU that is 15.52% lower, showing that the TransDeep model has a significant performance advantage on the same task; SegFormer is a lightweight segmentation network based on Transformer, and its performance is very close to but lower than the TransDeep model; PSPNet captures global context information through a pyramid pooling module, but even using the same MobileNetV2 as the backbone of the network, its MIoU is 5.93% lower than the TransDeep model. MCTformer has relatively low performance, with MIoU lower than TransDeep model by 12.1%, which indicates that TransDeep model can capture and utilize features more effectively on the same task; FPR is a segmentation network based on feature pyramid, and its performance is 3.72% lower than TransDeep model; ToCo is a segmentation network based on Transformer, and its performance is also 3.32% lower than TransDeep model; DeepLab V3+ uses the same backbone, and TransDeep model has MIoU higher than DeepLabV3+ by 2.1%. The above results show that TransDeep model has high efficiency and capability in feature extraction, context information fusion and segmentation accuracy.

Claims

1. An image semantic segmentation method based on the TransDeep model, characterized by: Includes the following step, And the following steps are carried out in sequence, Step 1: Obtain the PASCAL VOC 2012 dataset, divide the dataset into training set, test set and validation set in proportion, and perform preprocessing operations on the image data in the training set, test set and validation set; Step 2: Build a TransDeep-based image semantic segmentation model, which is referred to as the TransDeep model. The model includes a backbone network feature extraction module, a global feature encoding block, a residual nested feature enhancement module for high-level features, a residual nested feature enhancement module for low-level features, a feature map fusion module, and a linear upsampling module. The backbone network feature extraction module includes a starting convolution layer, an inverted residual block, a 1×1 point-by-point convolution layer, a global average pooling layer, a 1×1 point-by-point convolution layer, and a feature hierarchical processing layer for performing a slicing operation, which are sequentially connected. The backbone network feature extraction module performs preliminary feature extraction to generate two feature layers of high-level features and low-level features; Among them, the high-level feature layer first extracts global context features through the global feature encoding block, and then further enhances the feature representation through the high-level feature residual nested feature enhancement module to obtain enhanced high-level features; The low-level feature layer strengthens the edge feature information and detail feature information through the low-level feature residual nested feature enhancement module to obtain enhanced low-level features; The residual nested feature enhancement module of the high-level feature and the residual nested feature enhancement module of the low-level feature both adopt the residual nested feature enhancement module; the residual nested feature enhancement module is another identical CA coordinate attention module embedded in the external CA coordinate attention module; the CA coordinate attention module is referred to as CA module; The working process of the residual nested feature enhancement module is: The external CA module performs pooling operations on the input feature maps in two independent directions, X and Y, respectively, to generate two direction-aware feature maps f_h1 and f_w1, and replaces the height dimension and width dimension of the feature map f_w1 to match its dimension with another feature map f_h1, and concatenates the two into the feature map f1; then passes through a convolution block, and applies the Sigmoid activation function to introduce nonlinear features to generate the feature map f1'; The external CA module separates the feature map f1' along the spatial dimension to generate two feature maps f_h1' and f_w1', and replaces the height dimension and width dimension of the feature map f_w1' to match its dimension with the other feature map f_h1'; then a convolution block is used to calculate the attention weights of the feature map f1' in the X and Y directions, and the Sigmoid activation function is applied to compress the input values ​​between 0 and 1 to generate feature maps f_h1'' and feature maps f_w1'' respectively, and the input feature map of the external CA module is multiplied by the attention weights in the two directions in turn to obtain the feature map f1'' output by the external CA module; The feature map f1'' passes through a convolution block to reduce the feature map channel, extract a part of the features output by the external CA module, and generate a new feature map f1''_conv; The embedded CA module performs pooling operations on the feature map f1''_conv in two independent directions, X and Y, to generate two direction-aware feature maps f_h2 and f_w2, and replaces the height dimension and width dimension of the feature map f_w2 to match its dimension with another feature map f_h2, and concatenates the two into the feature map f2; then passes through a convolution block, and applies the Sigmoid activation function to introduce nonlinear features to generate the feature map f2'; The embedded CA module separates the feature map f2' along the spatial dimension to generate two feature maps f_h2' and f_w2', and transforms the feature map f_w2' to match its dimension with the other feature map f_h2'; then a convolution block is used to calculate the attention weights of the feature map f2' in the X and Y directions, and the Sigmoid activation function is applied to compress the input values ​​between 0 and 1 to generate feature maps f_h2'' and f_w2'', respectively. The input feature map f1''_conv of the embedded CA module is multiplied by the attention weights in the two directions in turn to obtain the feature map f2'' output by the embedded CA module; The output f2'' of the embedded CA module is fused with the output f1''_conv of the external CA module after convolution to obtain the feature map F'; and then a convolution block is passed to generate the final feature map F''; The feature map fusion module concatenates the enhanced high-level features and the enhanced low-level features in the channel dimension through upsampling, and then sequentially passes through two convolution blocks and inputs them into the linear upsampling module; The linear upsampling module first maps each channel of the received feature map to a specified category through a convolution layer; then performs bilinear interpolation upsampling on the mapped feature map, and then restores the bilinear interpolation upsampled feature map to the input image size of the backbone network feature extraction module; Step 3: Establish a fusion loss function, which combines the equilibrium loss function And the focal loss function , and then introduce the weights corresponding to each loss function, dynamically adjust the weights of each loss function, and enhance the generalization of the TransDeep model; wherein the focus loss function Trainable focusing parameters are used in The fusion loss function is: ; In the formula, express The loss function is For the i-th training The weight of is the focal loss function in the i-th training The weight of Step 4: Use the transfer learning training strategy to train the TransDeep model on the training set to obtain the best TransDeep model; Step 5: Use the best TransDeep model to perform image semantic segmentation on the test set to obtain the final segmentation result.

2. The image semantic segmentation method based on the TransDeep model according to claim 1, characterized in that: The specific process of generating two feature layers of high-level features and low-level features by the backbone network feature extraction module is as follows: first, the image data input to the backbone network feature extraction module is subjected to preliminary feature extraction through the starting convolution layer; then, it is further downsampled through a specified number of independent inverted residual blocks in sequence, and each inverted residual block includes a 1×1 dilated convolution layer, a 3×3 depth convolution layer, a 1×1 linear bottleneck convolution layer and a jump connection layer connected in sequence; then, the output feature map of the last inverted residual block is subjected to feature sorting through a 1×1 point-by-point convolution layer, and then the spatial dimension of the feature map is compressed into a one-dimensional feature vector by a global average pooling layer; again, the feature vector is subjected to convolution processing using a 1×1 point-by-point convolution layer; finally, the low-level features obtained after sequential processing by the starting convolution layer and the first several specified number of inverted residual blocks and the high-level features obtained after sequential processing by the starting convolution layer, all inverted residual blocks, 1×1 point-by-point convolution layers, global average pooling layers and 1×1 point-by-point convolution layers are obtained through the slicing operation of the feature hierarchical processing layer.

3. The image semantic segmentation method based on the TransDeep model according to claim 1, characterized in that: The processing process of the global feature encoding block is: First, the image is divided into blocks through the Unfold operation, and the dimension is adjusted through the permute transposition function to adapt to the input requirements of the self-attention calculation, and the feature map after the dimension adjustment is obtained; Secondly, a multi-head self-attention mechanism is used to perform self-attention calculation on the feature map after dimension adjustment and add it to the feature map after dimension adjustment for residual connection; Again, a nonlinear transformation is performed through a multi-layer perceptron, and a residual connection is made with the output of the multi-head self-attention mechanism; Then, the dimensions are rearranged through the permute transposition function, and the image is inversely divided into blocks through the Fold folding operation; Next, dilated convolution blocks with different dilation rates are used to extract multi-scale feature information in parallel; Finally, the feature fusion of multi-scale feature information is performed through channel splicing, and a convolution block adjusts and outputs the feature map.

4. The image semantic segmentation method based on the TransDeep model according to claim 1, characterized in that: Said The formula is: ； In the formula, is the number of negative samples of the jth class, is the number of positive samples of the jth class, and are the positive sample loss of the jth class and the negative sample loss of the jth class respectively.

5. The image semantic segmentation method based on the TransDeep model according to claim 1, characterized in that: Said The formula is: ； In the formula, is the weight factor of category t; is the probability that the TransDeep model predicts that category t is a positive category; is the probability that the TransDeep model predicts that category t is a negative category; is the focus parameter in the i-th training.

6. The image semantic segmentation method based on the TransDeep model according to claim 1, characterized in that: The training strategy of transfer learning in step 4 is specifically as follows: the initial weights of the DeeplabV3+ model pre-trained on the PASCAL VOC 2012 dataset are used as the initial weights of the TransDeep model, and further training is performed on this basis, so that the TransDeep model can better utilize the features learned by the DeeplabV3+ model, reduce training time and improve segmentation efficiency and accuracy; during the training process, the TransDeep model adopts a feature extraction method and implements a parameter freezing strategy to adapt to the semantic segmentation task; The parameter freezing strategy is as follows: the total number of training rounds is set to 100, with the first 50 rounds being the freezing stage and the last 50 rounds being the thawing stage. The batch_size in the thawing stage is reduced to 4, while the batch_size in the freezing stage is set to 8 to avoid the problem of insufficient video memory after thawing. The maximum learning rate is set to 0.007, the minimum learning rate is set to 0.00007, and the learning rate is adjusted by cosine annealing. At the same time, stochastic gradient descent SGD is used as the optimizer of the model, the built-in parameter momentum is set to 0.9, and the weight decay weight_decay parameter is set to 0.0001 to optimize the model parameters and minimize the loss function.

Images