Mobile agent-oriented boundary information-assisted semantic segmentation method

Abstract

The invention discloses a mobile agent-oriented boundary information-assisted semantic segmentation method. The method comprises the following steps of: preprocessing a data set; constructing a lightweight semantic segmentation model based on boundary information assistance; the constructed network model is trained; calculating a model parameter gradient according to the loss function, performing parameter updating through an AdamW optimizer, applying the network model after parameter updating to the next group of training samples, and repeating the steps until the loss value of the network on the verification set tends to be stable, the model at the moment being a trained model; and inputting the test set into the network model trained in the step 3, and outputting a semantic segmentation graph and a boundary detection graph. According to the method, FLOPs is greatly reduced, the real-time performance of the model is ensured, a semantic segmentation head and an edge detection head are optimized, and high segmentation precision is kept under the condition that the parameter quantity is reduced to a certain extent.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and in particular relates to a semantic segmentation method assisted by boundary information for mobile intelligent bodies. Background Art

[0002] With the rapid development of computer vision and natural language processing technologies, embodied intelligence has become a cutting-edge field in artificial intelligence, with increasingly widespread applications in fields such as robotics, autonomous driving, and virtual reality. Embodied intelligence requires intelligent agents to acquire knowledge and perform tasks through interaction with their physical environment. It emphasizes that intelligent agents must possess a vehicle and learn and adapt through perception, action, and environmental feedback. For intelligent agents to effectively perform tasks, accurate environmental perception is essential, and semantic segmentation is a key component in enabling them to perceive the external environment through visual sensors.

[0003] Semantic segmentation, a key research direction in computer vision, aims to assign semantic labels to each pixel in an image, thereby achieving a detailed understanding of real-world scenes. Since the release of AlexNet in 2012, the semantic segmentation task has shifted from traditional image processing techniques to deep learning-based methods. The most representative pioneering work is the fully convolutional network (FCN) proposed by Jonathan Long et al. in 2015. This method replaces fully connected layers with convolutional layers, enabling the network to accept inputs of arbitrary size and output semantic segmentation maps of the same size as the input. This method achieves pixel-level segmentation and marks the entry of semantic segmentation into the deep learning era. Since then, classic methods have emerged, including the U-Net series, the DeepLab series, and other deep learning methods based on encoder-decoder structures.

[0004] U-Net, originally proposed by Ronneberger et al. in 2015, employs an encoder-decoder architecture. The encoder is responsible for feature extraction and downsamples the image through multiple convolutional and pooling layers, converting the image's high spatial resolution into low-resolution feature maps rich in semantic information. The decoder restores the image's spatial resolution through upsampling and uses skip connections to concatenate feature maps from different levels of the encoder with those from the corresponding decoder layers. This concatenation leverages both the detailed information of low-level features and the semantic information of high-level features, effectively preserving image details. Since then, numerous improvements have been made to U-Net, such as U2Net, which uses a nested U-shaped structure to extract multi-scale features from low to high levels and fuse feature information from different levels. UNet++ builds on U-Net by adding multiple skip connections and deep supervision mechanisms. It also employs a nested U-shaped structure, enabling accurate object segmentation and robust handling of complex backgrounds and blurred boundaries. The UNet series of semantic segmentation methods has good detail preservation capabilities and is more friendly to smaller datasets. It is mainly used in the semantic segmentation of medical images. However, the UNet series of semantic segmentation methods has problems such as weak global information modeling capabilities and limited performance in complex scenarios. Its real-time performance cannot meet the application requirements of mobile devices and intelligent entities.

[0005] The pioneering work in the DeepLab series was published in 2015. This method introduced dilated convolution, which can expand the receptive field without increasing the amount of computation, thereby better utilizing the contextual information of the image. At the same time, DeepLab v1 introduced a fully connected conditional random field (CRF), which improved the image segmentation accuracy. However, this also increased the complexity of the model, resulting in slower inference speed. DeepLab v2 introduced a dilated spatial pyramid pooling module, which can effectively utilize the multi-scale information of the image, but the segmentation effect of small objects is poor. DeepLab v3 improved the dilated spatial pyramid pooling module of DeepLab v2, improved the segmentation accuracy of the model, and removed the CRF module to reduce computational complexity. DeepLab v3+ uses an encoder-decoder structure and uses depthwise separable convolution to reduce computational complexity and improve operational efficiency, but this method still has problems with long inference time and low segmentation accuracy for small objects.

[0006] With the increasing demand for real-time semantic segmentation on embedded and mobile devices, research on lightweight semantic segmentation models has gradually attracted attention. For example, SegFormer achieves lightweight model while maintaining high segmentation accuracy by combining efficient Transformer and CNN. ENet significantly reduces the spatial and temporal complexity of the model by reducing the number of channels in the residual module and introducing dilated convolutions, enabling it to achieve real-time semantic segmentation on resource-constrained embedded devices. The emergence of some general lightweight backbone networks (such as the MobileNet series, EfficientNet series, GhostNet series, and ShuffleNet series, etc.) has also helped semantic segmentation models enter a development stage that is lightweight, efficient, and applicable to embedded devices.

[0007] However, as embodied intelligence systems develop, agents require more lightweight semantic segmentation methods with stable accuracy to support their perception of the external environment. This allows them to achieve real-time perception while freeing up computing power and energy for downstream tasks such as visual and language navigation and intelligent interaction. This requires semantic segmentation models to be as computationally efficient and memory-efficient as possible. While a number of lightweight, real-time semantic segmentation models have emerged, their inference speed and segmentation accuracy still cannot meet these requirements. Summary of the Invention

[0008] To address the problems in the prior art, the present invention provides a semantic segmentation method assisted by boundary information for mobile agents. After inputting the original image, the method outputs a semantic segmentation map and a boundary detection map, as follows:

[0009] Step 1: Dataset preprocessing: divide the processed dataset into training set, validation set, and test set;

[0010] Step 1.1: Use the public semantic segmentation dataset Cityscapes and divide the images in the dataset into training set, validation set, and test set according to the folder names train, val, and test;

[0011] Step 1.2: Convert the JSON files corresponding to the images in the training set into single-channel grayscale images using the official Python script and normalize the grayscale images.

[0012] Step 2: Build a lightweight semantic segmentation model Dual-LiteNet based on boundary information assistance. The model consists of a pre-trained image encoder, a semantic branch, a boundary branch, and a fusion module.

[0013] Step 2.1: Feed the image obtained in step 1.2 into the pre-trained image encoder. The process is expressed as:

[0014] F0=AFF(I), (1)

[0015] Where I represents the original image, AFF(·) represents the pre-trained AFFormer-T model, and F0 represents the image feature obtained after encoding by AFFormer-T;

[0016] Step 2.2: Input F0 into the semantic branch for processing to obtain a semantic feature map. The process is expressed as:

[0017] F S =f Seg (F0), (2)

[0018] Where, f Seg (·) represents semantic branch, F S is a semantic feature;

[0019] The specific process is:

[0020] In step 2.2.1, the feature map F0 obtained in step 2.1 is sent to the lightweight void spatial convolution pooling pyramid ASPP module and the efficient channel attention ECA module for processing. The process is expressed as follows:

[0021]

[0022] Where, f ASPP (·) indicates the ASPP module, f ECA (·) indicates ECA module; F ASPP represents the features processed by the ASPP module, F ECA Represents the features processed by the ECA module;

[0023] The ASPP module has three branches: the dilated convolution branch, the 1×1 convolution branch, and the global pooling branch.

[0024] The dilated convolution branch uses three dilated convolutions with different expansion rates to process features. The process is expressed as:

[0025]

[0026] Where, f C1 (·),f C2 (·) and f C3 (·) represents the dilated convolution operations with three dilation rates; F r1 、F r2 and F r3 Respectively represent the image features after three types of hole convolution processing;

[0027] The 1×1 convolution branch is expressed as:

[0028] F 1×1 =Conv 1×1 (F0), (5)

[0029] Where, Conv 1×1 (·) represents a 1×1 convolution operation, F 1×1 is the feature obtained after processing;

[0030] The global pooling branch uses a cascade of pooling, convolution, and upsampling modules to obtain image-level semantic information and align feature sizes with other branches. The process is expressed as:

[0031] F Pool =f Up (Conv 1×1 (f GAP (F0))), (6)

[0032] Where, f GAP (·), is the global average pooling operation, f Up (·) is the upsampling operation; the pooled feature obtained is F Pool ;

[0033] All features are concatenated in the channel dimension, and then a 1×1 convolution is used to integrate the information and reduce the dimension to obtain the output feature F ASPP , the process is expressed as:

[0034] F ASPP =Conv 1×1 (Concat(F r1 , F r2 , F r3 , F 1×1 , F Pool )) (7)

[0035] Where Concat(·) represents the concatenation operation;

[0036] In step 2.2.2, the ECA module includes global average pooling (GAP), one-dimensional convolution with adaptive kernel size, and Sigmoid activation function. The steps of this module are as follows:

[0037] Perform a global average pooling operation on F0, and the process is expressed as:

[0038] F GAP =f GAP (F0), (8)

[0039] Where, f GAP (·) represents the global pooling operation, F GAP is the processed feature;

[0040] One-dimensional convolution with adaptive kernel size according to F GAP To select the size of the convolution kernel K, the calculation formula is as follows:

[0041]

[0042] Where, the function ψ(C) represents the formula for calculating the size of the convolution kernel, b and γ are both constants, and C is F GAP The number of channels;

[0043] After a one-dimensional convolution operation with an adaptive kernel size, the feature F is obtained. AC , use the Sigmoid activation function to extract the weight, multiply the obtained weight by F0 point by point to get the output feature F S , the process is expressed as:

[0044]

[0045] Where, f AKC (·) represents a one-dimensional convolution operation with adaptive kernel size, Represents a point-by-point multiplication operation;

[0046] In step 2.3, F0 is input into the boundary branch for processing to obtain the boundary feature map. The process is expressed as:

[0047] F B =f Bdy (F0), (11)

[0048] Where, f Bdy (·) represents the edge branch, F B is the edge feature;

[0049] The specific process is:

[0050] Step 2.3.1, the features Send it to the Ghost module for processing. C, H, and W represent the number of channels, length, and width of the feature map respectively. The specific steps are as follows:

[0051] The channel is compressed by a 3×3 convolution, and the process is expressed as:

[0052] F C =Conv 3×3 (F0),

[0053] Where, is the feature after convolution processing, Represents a tensor with m channels, H length, and W width, Conv 3×3 (·) represents a 3×3 convolution operation; m is F CThe number of channels is s is the feature generation ratio;

[0054] Using depth-wise separable convolution on the feature map of each channel, we get features, the process is expressed as:

[0055]

[0056] Where, is the i-th depthwise separable convolution operation, F C The characteristics of the i-th channel in, is the corresponding output feature;

[0057] The feature F C and all Splicing along the channel dimension to obtain the output feature F G , the process is expressed as:

[0058]

[0059] Step 2.3.2: The feature F obtained in step 2.3.1 G It is sent to the dual-branch attention DA module for further processing. The process is expressed as:

[0060]

[0061] Where, F DA represents the features processed by the DA module, f CA (·) represents the channel attention mechanism, f SA (·) represents the spatial attention mechanism, It is a point-by-point multiplication operation;

[0062] Step 2.4: Fuse the two features obtained after processing the semantic branch and the edge branch;

[0063] Semantic feature F S and edge features F B After passing through a CBAM module, the enhanced semantic features F′ are obtained. S and enhanced boundary features F′ B , the process is expressed as:

[0064]

[0065] Where, f CBA (·) indicates the processing performed by the CBAM module;

[0066] F′ S and F′ BThrough cross attention fusion, the final semantic segmentation map M is obtained seg ; The process is expressed as:

[0067] M seg =f CRA (F′ S , F′ B ), (17)

[0068] Where, f CRA (·) indicates cross-attention processing;

[0069] Step 3: Train the network model constructed in step 2. Calculate the model parameter gradients based on the loss function, update the parameters using the AdamW optimizer, and use the updated network model for the next set of training samples. Repeat this process until the network loss on the validation set stabilizes. The model is now considered trained.

[0070] A cross-entropy-based contrast loss is used for the semantic segmentation map, and a binary cross-entropy loss is used for the edge. The loss function L is expressed as:

[0071] L=λ1L ctr (M seg , Lab seg )+λ2L bce (F B , L abb ), (18)

[0072] Where λ1 and λ2 are hyperparameters, L ctr (·,·) represents the contrast loss calculation, L bce (·,·) represents the binary cross entropy calculation, Lab seg and Lab b Represent semantic segmentation labels and edge labels respectively;

[0073] Among them, the contrast loss L based on cross entropy ctr The calculation process of (·,·) is as follows: let the semantic segmentation map M seg The feature vector of each pixel i is z i , the category label is y i :

[0074]

[0075] Where, L cont represents semantic contrast loss, P(i) represents the set of positive samples of the same class as pixel i, p is the number of positive samples, N(i) represents the set of negative samples of different classes from pixel i, τ is the temperature coefficient, N is the total number of pixels, L ctrIt represents the final contrast loss after weighted addition with the cross entropy loss, α is a hyperparameter, L CE (·,·) represents the cross entropy loss calculation, z i Represents the current pixel, z p Indicates that z i Pixels of the same category, z n Indicates that z i Different categories of pixels;

[0076] Step 4: Input the test set into the network model trained in step 3 and output the semantic segmentation map and boundary detection map;

[0077] In the testing phase, the model parameters trained in the training phase are loaded into the Dual-LiteNet model. After the initial image is input, the semantic segmentation map and boundary detection map are output.

[0078] In step 1.2 of a specific embodiment of the present invention, the grayscale image is normalized according to the ImageNet mean [0.485, 0.456, 0.406] and the standard deviation [0.229, 0.224, 0.225].

[0079] In step 2.2.1 of another specific embodiment of the present invention, f C1 、f C2 and f C3 Represents dilation rates of 6, 12, and 18, respectively;

[0080] In step 2.2.2 of another specific embodiment of the present invention, b is set to 1, and γ is set to 2;

[0081] In step 2.3.1 of another specific embodiment of the present invention, s is 2.

[0082] In step 3 of another specific embodiment of the present invention, λ1 and λ2 are both set to 1, τ is set to 0.01, and α is set to 0.5.

[0083] The present invention provides a semantic segmentation method assisted by boundary information for mobile agents, which alleviates the balance problem between real-time performance and accuracy of existing lightweight semantic segmentation methods used in resource-constrained situations such as mobile agents.

[0084] The present invention uses AFFormer-T pre-trained on the ImageNet dataset as the feature extraction network, and adopts a dual-branch structure including a semantic segmentation branch and an edge detection branch. The semantic segmentation branch introduces an optimized lightweight atrous spatial convolutional pyramid pooling (ASPP) module and an efficient channel-spatial attention (Convolutional Block Attention Module, CBAM) to extract semantic features in parallel, and uses dynamic convolution to adjust the fusion weight of the two; the edge detection branch uses a more lightweight and efficient Ghost module, and introduces a dual-branch attention block (DAB) module to better extract edge features; the present invention improves the adaptive fusion module, using lightweight spatial-channel attention to capture features in the spatial dimension, thereby improving the accuracy of edge detection and semantic segmentation.

[0085] This paper proposes for the first time a lightweight dual-branch semantic segmentation and boundary detection algorithm. This method innovatively designs two network branches. In the semantic branch, a lightweight ASPP module and an ECA module are introduced in parallel as the backbone network for the first time, making full use of the multi-scale features and channel features of the image, and using a convolution module to dynamically learn the weights of the two, so that the model can learn features that are more conducive to semantic segmentation; a lightweight edge detection module is designed, and the Ghost module and the DA module are introduced to maintain model accuracy while reducing FLOPs and reducing computational overhead; a contrastive learning training strategy is introduced to make the prediction results sufficiently close to the positive samples without model increment, thereby improving model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 1 is a network structure diagram of the boundary information-assisted semantic segmentation method for mobile intelligent agents of the present invention;

[0087] Figure 2 This is the network structure diagram of the ASPP module mentioned in the semantic segmentation method assisted by boundary information for mobile intelligent agents of the present invention;

[0088] Figure 3 This is the network structure diagram of the ECA module mentioned in the boundary information-assisted semantic segmentation method for mobile intelligent agents of the present invention;

[0089] Figure 4 This is a network structure diagram of the multi-scale non-local attention module mentioned in the boundary information-assisted semantic segmentation method for mobile intelligent agents of the present invention;

[0090] Figure 5 This is a network structure diagram of the EAF module in the semantic segmentation method assisted by boundary information for mobile agents of the present invention;

[0091] Figure 6 This is the result diagram of Example 2 of the semantic segmentation method assisted by boundary information for mobile intelligent agents of the present invention. DETAILED DESCRIPTION

[0092] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0093] The present invention is a semantic segmentation method assisted by boundary information of mobile intelligent bodies. Figure 1 As shown in Figure 2, after inputting the original image, the semantic segmentation map and boundary detection map are output.

[0094] Please follow the steps below to implement it:

[0095] Step 1: Dataset preprocessing: divide the processed dataset into training set, validation set, and test set;

[0096] Step 1.1: Taking the public semantic segmentation dataset Cityscapes as an example, divide the images in the dataset into training set, validation set, and test set according to the folder names train, val, and test;

[0097] In step 1.2, convert the JSON files corresponding to the images in the training set into single-channel grayscale images using the official Python script, and normalize the grayscale images to, for example, the ImageNet mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225].

[0098] Step 2: Build a lightweight semantic segmentation model Dual-LiteNet based on boundary information. The model mainly consists of a pre-trained image encoder, a semantic branch, a boundary branch, and a fusion module.

[0099] Step 2.1: Feed the image obtained in step 1.2 into the pre-trained image encoder. The process can be expressed as:

[0100] F0=AFF(I) (1)

[0101] Where I represents the original image, AFF(·) represents the pre-trained AFFormer-T model, and F0 represents the image feature obtained after encoding by AFFormer-T;

[0102] Step 2.2: Input F0 into the semantic branch for processing (this process is well known to those skilled in the art) to obtain a semantic feature map. This process can be expressed as:

[0103] F S =f Seq (F0), (2)

[0104] Where, f Seg (·) represents semantic branch, F s It is a semantic feature.

[0105] The specific process of step 2.2 is:

[0106] Step 2.2.1, as Figure 2 As shown in Figure 2, the feature map F0 obtained in step 2.1 is sent to the lightweight atrous spatial convolutional pyramid pooling (ASPP) module and the efficient channel attention (ECA) module for processing. The process can be expressed as:

[0107]

[0108] Where, f ASPP (·) indicates the ASPP module, f ECA (·) indicates ECA module; F ASPP represents the features processed by the ASPP module, f ECA Represents the features processed by the ECA module.

[0109] Among them, Figure 2 As shown in Figure 3, the ASPP module has three branches: the dilated convolution branch, the 1×1 convolution branch, and the global pooling branch.

[0110] The dilated convolution branch uses three dilated convolutions with different expansion rates to process features. The process can be expressed as:

[0111]

[0112] Where, f C1 (·),f C2 (·) and f C3 (·) denotes dilation operations with dilation rates of 6, 12, and 18, respectively; F r1 、F r2 and F r3 Represent the image features after three types of hole convolution processing.

[0113] The convolution processing features used by the 1×1 convolution branch can compress the feature dimension and retain local details. The process can be expressed as:

[0114] F 1×1 =Conv 1×1 (F0), (5)

[0115] Where, Conv 1×1(·) represents a 1×1 convolution operation, F 1×1 are the features obtained after processing.

[0116] The global pooling branch uses a cascade of pooling, convolution, and upsampling modules to obtain image-level semantic information and align feature sizes with other branches. The process can be expressed as:

[0117] F Pool =f Up (Conv 1×1 (f GAP (F0))), (6)

[0118] Where, f GAP (·), is the global average pooling operation, f Up (·) is the upsampling operation; the final pooling feature is F Pool .

[0119] Finally, all features are concatenated in the channel dimension, and a 1×1 convolution is used to integrate the information and reduce the dimension to obtain the output feature F ASPP , the process can be expressed as:

[0120] F ASPP =Conv 1×1 (Concat(F r1 ,F r2 ,F r3 ,F 1×1 ,F Pool )) (7)

[0121] Where Concat(·) represents the concatenation operation.

[0122] Step 2.2.2, as Figure 3 As shown in the figure, the ECA module is a lightweight channel attention mechanism whose structure includes global average pooling (GAP), one-dimensional convolution with adaptive kernel size, and Sigmoid activation function. The steps of this module are as follows:

[0123] First, perform a global average pooling operation on F0. The process can be expressed as:

[0124] F GAP =f GAP (F0), (8)

[0125] Where, f GAP (·) represents the global pooling operation, F GAP The processed features.

[0126] Then, the one-dimensional convolution with adaptive kernel size is performed according to F GAP To select the size of the convolution kernel K, the calculation formula is as follows:

[0127]

[0128] In the formula, the function ψ(C) represents the calculation formula of the convolution kernel size, b and γ are both constants, b is set to 1, γ is set to 2; C is F GAP The number of channels.

[0129] After a one-dimensional convolution operation with an adaptive kernel size, the feature F is obtained. AC , and use the Sigmoid activation function to extract the weights, and finally multiply the obtained weights by F0 point by point to get the final output feature F S , the process can be expressed as:

[0130]

[0131] Where, f AKC (·) represents a one-dimensional convolution operation with adaptive kernel size, Represents a point-wise multiplication operation.

[0132] In step 2.3, F0 is input into the boundary branch for processing to obtain a boundary feature map (this process is well known to those skilled in the art). The process can be expressed as:

[0133] F B =f Bdy (F0), (11)

[0134] Where, f Bdy (·) represents the edge branch, F B is the edge feature.

[0135] The specific process of step 2.3 is:

[0136] Step 2.3.1, as Figure 4 As shown, the features Send it to the Ghost module for processing. C, H, and W represent the number of channels, length, and width of the feature map respectively. The specific steps are as follows:

[0137] The channel is compressed by a 3×3 convolution, which can be expressed as:

[0138] F C =Conν 3×3 (F0),

[0139] Where, is the feature after convolution processing, Represents a tensor with m channels, H length, and W width, Conv 3×3 (·) represents a 3×3 convolution operation; m is F CThe number of channels is s is the feature generation ratio, and in the present invention, s is, for example, 2.

[0140] Using depth-wise separable convolution on the feature map of each channel, we get The process can be expressed as:

[0141]

[0142] Where, is the i-th depthwise separable convolution operation, F C The characteristics of the i-th channel in, is the corresponding output feature.

[0143] Then the feature F C and all Splicing along the channel dimension to obtain the final output feature F G , the process can be expressed as:

[0144]

[0145] Step 2.3.2: The feature F obtained in step 2.3.1 G It is sent to the dual attention (DA) module for further processing. The process can be expressed as:

[0146]

[0147] Where, F DA represents the features processed by the DA module, f CA (·) represents the channel attention mechanism, f SA (·) represents the spatial attention mechanism, It is a point-by-point multiplication operation.

[0148] Step 2.4, as Figure 5 As shown, the two features obtained after processing the semantic branch and the edge branch are fused. First, the semantic feature F S and edge features F B After passing through a CBAM module, the enhanced semantic features F′ are obtained. S and enhanced boundary features F′ B , the process can be expressed as:

[0149]

[0150] Where, f CBA (·) indicates the processing performed by the CBAM module, which is well known to those skilled in the art.

[0151] Then F′ S and F′ B Through cross attention fusion, the final semantic segmentation map M is obtained seg The process can be expressed as:

[0152] M seg =f CRA (F′ S , F′ B ), (17)

[0153] Where, f CRA (·) represents cross-attention processing, which is well known to those skilled in the art.

[0154] Step 3: Train the network model constructed in step 2. Calculate the model parameter gradient according to the loss function, update the parameters through the AdamW optimizer, and use the updated network model for the next set of training samples. Repeat this process until the network loss value on the validation set stabilizes. The model is now considered a trained model.

[0155] A cross-entropy-based contrast loss is used for the semantic segmentation map, and a binary cross-entropy loss is used for the edge. The loss function L is expressed as:

[0156] L=λ1L ctr (M seg , Lab seg )+λ2L bce (F B , Lab b ), (18)

[0157] Where λ1 and λ2 are hyperparameters, L ctr (·,·) represents the contrast loss calculation, L bce (·,·) represents the binary cross entropy calculation, Lab seg and Lab b Represent semantic segmentation labels and edge labels respectively;

[0158] Among them, the contrast loss L based on cross entropy ctr The calculation process of (·,·) is as follows: let the semantic segmentation map M seg The feature vector of each pixel i is z i , the category label is y i :

[0159]

[0160] Where, L contrepresents semantic contrast loss, P(i) represents the set of positive samples of the same class as pixel i, p is the number of positive samples, N(i) represents the set of negative samples of different classes from pixel i, τ is the temperature coefficient, N is the total number of pixels, L ctr It represents the final contrast loss after weighted addition with the cross entropy loss, α is a hyperparameter, L CE (·,·) represents the cross entropy loss calculation (the calculation method is well known to those skilled in the art), z i Represents the current pixel, z p Indicates that z i Pixels of the same category, z n Indicates that z i Different categories of pixels;

[0161] Step 4: Input the test set into the network model trained in step 3 and output the semantic segmentation map and boundary detection map.

[0162] In the testing phase, the model parameters trained in the training phase are loaded into the Dual-LiteNet model. After inputting the initial image, the semantic segmentation map and boundary detection map can be output. The feature propagation process is the same as the forward propagation process in the training phase.

[0163] Example 1

[0164] Table 1: Comparison of indicators between the method of the present invention and the baseline method

[0165]

[0166] After comparison, the objective evaluation indicators of the proposed method on the Cityscapes dataset are generally better than the baseline method, especially for categories with rich edge textures, such as truck, bus, train and motorcycle, the segmentation precision and accuracy are greatly improved.

[0167] Example 2

[0168] like Figure 6 The figure shows the experimental results of the method of the present invention. It can be seen that the method can effectively achieve the two tasks of semantic segmentation and boundary detection.

[0169] Implementation Example 3

[0170] Table 2: Ablation experiment results of the effectiveness of the contrastive learning strategy in this invention

[0171]

[0172] In Table 2, wo indicates that the contrastive learning strategy is not used, and w indicates that the contrastive learning strategy is used. It can be seen that after using the contrastive learning strategy, the method of the present invention has been improved in all three evaluation indicators. The present invention uses the pre-trained AFFormer-T as the feature extraction network, and uses a dual-branch structure including a semantic segmentation branch and an edge detection branch. The semantic segmentation branch introduces an optimized lightweight ASPP module and efficient channel-spatial attention to extract semantic features in parallel, and uses dynamic convolution to adjust the fusion weights of the two; the edge detection branch introduces a more lightweight and efficient Ghost module, and introduces a bidirectional attention module to better extract edge features; finally, this method improves the adaptive fusion module, so that the lightweight spatial-channel attention can capture the features of the spatial dimension and improve the accuracy of edge detection and semantic segmentation.

Claims

1. A semantic segmentation method assisted by boundary information for mobile agents, which inputs the original image and outputs a semantic segmentation map and a boundary detection map, characterized in that: The details are as follows: Step 1: Dataset preprocessing: divide the processed dataset into training set, validation set, and test set; Step 1.1: Use the public semantic segmentation dataset Cityscapes and divide the images in the dataset into training set, validation set, and test set according to the folder names train, val, and test; Step 1.2: Convert the JSON files corresponding to the images in the training set into single-channel grayscale images using the official Python script and normalize the grayscale images. Step 2: Build a lightweight semantic segmentation model Dual-LiteNet based on boundary information assistance. The model consists of a pre-trained image encoder, a semantic branch, a boundary branch, and a fusion module. Step 2.1: Feed the image obtained in step 1.2 into the pre-trained image encoder. The process is expressed as: F0=AFF(I) (1) Where I represents the original image, AFF(·) represents the pre-trained AFFormer-T model, and F0 represents the image feature obtained after encoding by AFFormer-T; Step 2.2: Input F0 into the semantic branch for processing to obtain a semantic feature map. The process is expressed as: F S =f Seg (F0), (2) Where, f Seg (·) represents semantic branch, F S is a semantic feature; The specific process is: In step 2.2.1, the feature map F0 obtained in step 2.1 is sent to the lightweight void spatial convolution pooling pyramid ASPP module and the efficient channel attention ECA module for processing. The process is expressed as follows: Where, f ASPP (·) indicates the ASPP module, f ECA (·) indicates ECA module; F ASPP represents the features processed by the ASPP module, F ECA Represents the features processed by the ECA module; The ASPP module has three branches: the dilated convolution branch, the 1×1 convolution branch, and the global pooling branch. The dilated convolution branch uses three dilated convolutions with different expansion rates to process features. The process is expressed as: Where, f C1 (·),f C2 (·) and f C3 (·) represents the dilated convolution operations with three dilation rates; F r1 、F r2 and F r3 Respectively represent the image features after three types of hole convolution processing; The 1×1 convolution branch is expressed as: F 1×1 =Conv 1×1 (F0), (5) Where, Conv 1×1 (·) represents a 1×1 convolution operation, F 1×1 is the feature obtained after processing; The global pooling branch uses a cascade of pooling, convolution, and upsampling modules to obtain image-level semantic information and align feature sizes with other branches. The process is expressed as: F Pool =f Up (Conv 1×1 (f GAP (F0))), (6) Where, f GAP (·), is the global average pooling operation, f Up (·) is the upsampling operation; the pooled feature obtained is F Pool ; All features are concatenated in the channel dimension, and then a 1×1 convolution is used to integrate the information and reduce the dimension to obtain the output feature F ASPP , the process is expressed as: F ASPP =Conv 1×1 (Concat(F r1 ,F r2 ,F r3 ,F 1×1 ,F Pool )) (7) Where Concat(·) represents the concatenation operation; In step 2.2.2, the ECA module includes global average pooling (GAP), one-dimensional convolution with adaptive kernel size, and Sigmoid activation function. The steps of this module are as follows: Perform a global average pooling operation on F0, and the process is expressed as: F GAP =f GAP (F0), (8) Where, f GAP (·) represents the global pooling operation, F GAP is the processed feature; One-dimensional convolution with adaptive kernel size according to F GAP To select the size of the convolution kernel K, the calculation formula is as follows: Where, the function ψ(C) represents the formula for calculating the size of the convolution kernel, b and γ are both constants, and C is F GAP The number of channels; After a one-dimensional convolution operation with an adaptive kernel size, the feature F is obtained. AC , use the Sigmoid activation function to extract the weight, multiply the obtained weight by F0 point by point to get the output feature F S , the process is expressed as: Where, f AKC (·) represents a one-dimensional convolution operation with adaptive kernel size, Represents a point-by-point multiplication operation; In step 2.3, F0 is input into the boundary branch for processing to obtain the boundary feature map. The process is expressed as: F B =f Bdy (F0), (11) Where, f Bdy (·) represents the edge branch, F B is the edge feature; The specific process is: Step 2.3.1, the features Send it to the Ghost module for processing. C, H, and W represent the number of channels, length, and width of the feature map respectively. The specific steps are as follows: The channel is compressed by a 3×3 convolution, and the process is expressed as: Where, is the feature after convolution processing, Represents a tensor with m channels, H length, and W width, Conv 3×3 (·) represents a 3×3 convolution operation; m is F C The number of channels is s is the feature generation ratio; Using depth-wise separable convolution on the feature map of each channel, we get features, the process is expressed as: Where, is the i-th depthwise separable convolution operation, F C The characteristics of the i-th channel in, is the corresponding output feature; The feature F C and all Splicing along the channel dimension to obtain the output feature F G , the process is expressed as: Step 2.3.2: The feature F obtained in step 2.3.1 G It is sent to the dual-branch attention DA module for further processing. The process is expressed as: Where, F DA represents the features processed by the DA module, f CA (·) represents the channel attention mechanism, f SA (·) represents the spatial attention mechanism, It is a point-by-point multiplication operation; Step 2.4: Fuse the two features obtained after processing the semantic branch and the edge branch; Semantic feature F S and edge features F B After passing through a CBAM module, the enhanced semantic features F′ are obtained. S and enhanced boundary features F′ B , the process is expressed as: Where, f CBA (·) indicates the processing performed by the CBAM module; F′ S and F′ B Through cross attention fusion, the final semantic segmentation map M is obtained seg ; The process is expressed as: M seg =f CRA (F′ S ,F′ B ), (17) Where, f CRA (·) indicates cross-attention processing; Step 3: Train the network model constructed in step 2. Calculate the model parameter gradients based on the loss function, update the parameters using the AdamW optimizer, and use the updated network model for the next set of training samples. Repeat this process until the network loss on the validation set stabilizes. The model is now considered trained. A cross-entropy-based contrast loss is used for the semantic segmentation map, and a binary cross-entropy loss is used for the edge. The loss function L is expressed as: L=λ1L ctr (M seg ,Lab seg )+λ2L bce (F B ,Lab b ), (18) Where λ1 and λ2 are hyperparameters, L ctr (·,·) represents the contrast loss calculation, L bce (·,·) represents the binary cross entropy calculation, Lab seg and Lab b Represent semantic segmentation labels and edge labels respectively; Among them, the contrast loss L based on cross entropy ctr The calculation process of (·,·) is as follows: let the semantic segmentation map M seg The feature vector of each pixel i is z i , the category label is y i : Where, L cont represents semantic contrast loss, P(i) represents the set of positive samples of the same class as pixel i, p is the number of positive samples, N(i) represents the set of negative samples of different classes from pixel i, τ is the temperature coefficient, N is the total number of pixels, L ctr It represents the final contrast loss after weighted addition with the cross entropy loss, α is a hyperparameter, L CE (·,·) represents the cross entropy loss calculation, z i Represents the current pixel, z p Indicates that z i Pixels of the same category, z n Indicates that z i Different categories of pixels; Step 4: Input the test set into the network model trained in step 3 and output the semantic segmentation map and boundary detection map; In the testing phase, the model parameters trained in the training phase are loaded into the Dual-LiteNet model. After the initial image is input, the semantic segmentation map and boundary detection map are output.

2. A semantic segmentation method assisted by boundary information for mobile agents, characterized in that: In step 1.2, the grayscale image is normalized according to the ImageNet mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225].

3. A semantic segmentation method assisted by boundary information for mobile agents, characterized in that: In step 2.2.1, f C1 、f C2 and f C3 They represent dilation rates of 6, 12, and 18, respectively.

4. A semantic segmentation method assisted by boundary information for mobile agents, characterized in that: In step 2.2.2, b is set to 1 and γ is set to 2.

5. A semantic segmentation method assisted by boundary information for mobile agents, characterized in that: In step 2.3.1, s is 2.

6. A semantic segmentation method assisted by boundary information for mobile agents, characterized in that: In step 3, λ1 and λ2 are both set to 1, τ is set to 0.01, and α is set to 0.5.

Images