A multi-modal image semantic segmentation method based on shape and semantic enhancement

Abstract

The application discloses a kind of based on shape and semantic enhancement multi-modal image semantic segmentation method, belong to image processing technical field, including the following steps: from visible light sensor and thermal infrared sensor respectively acquire RGB image and Thermal image, and image alignment is carried out by algorithm, then unify image size, construct data set;Multi-modal image semantic segmentation network model based on shape and semantic enhancement is constructed;Model training is carried out to multi-modal image semantic segmentation network model, and the network model that training is completed and performance is good is saved;RGB sensor and Thermal sensor real-time acquisition RGB image and Thermal image, input current training completed and the network model of good performance, carries out semantic segmentation to current image.The application has the characteristics of plug and play, can be combined with existing semantic segmentation network, effectively improve image segmentation performance, can be popularized to automatic driving, medical image analysis and other application scenarios, and considerable economic value is expected to be created.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a multi-modal image semantic segmentation method based on shape and semantic enhancement. BACKGROUND

[0002] Semantic segmentation is to label each pixel in an image with a class, which belongs to dense prediction. It plays an important role in many practical applications, such as autonomous driving, medical image analysis, geographic information analysis, etc. With the popularity of autonomous driving and human-assisted driving, semantic segmentation of urban road scenes has attracted more attention. In recent years, with the rapid development of deep learning technology, the performance of semantic segmentation using convolutional neural networks (CNN) has exceeded that of traditional semantic segmentation algorithms in a large number of tasks. However, the current mainstream semantic segmentation mostly uses RGB images captured by regular cameras, or uses RGB images and depth images captured by depth cameras. Both of these two ways are easily affected by lighting conditions, that is, when the lighting conditions of the environment are too bright or too dark, the performance of semantic segmentation will be greatly reduced. In order to further improve the performance of semantic segmentation and thus improve the safety of autonomous vehicles, recent research has paid more attention to using thermal images to further supplement RGB images and reduce the information loss caused by too bright or too dark environments. Thermal images can provide good imaging information under harsh lighting conditions. It can reflect the thermal radiation of any object with a temperature higher than absolute zero, thereby supplementing the RGB image with rich semantic information and contour information.

[0003] Sun et al.'s RTFNet network designed a novel decoder module to recover feature resolution. This decoder block is mainly designed based on the residual structure and stacks multiple convolutional layers. However, this structure significantly increases the number of model parameters and computational cost, and is difficult to transfer to other models. Guo et al.'s MLFNet uses multi-level jump connections to flexibly fuse features at all levels, further utilizing the contextual information of the features. It also performs channel concatenation of features from each level of the decoder for fusion in the final decoder block. Zhou et al.'s EGFNet network uses prior edge information and semantic information to perform multi-task deep monitoring of the network, generating high-quality segmentation results. It uses boundary labels to perform deep supervision of each layer of the encoder, and simultaneously uses boundary labels and semantic labels to supervise the model. Zhou et al.'s MFFENet and GMNet both implement class shape and edge supervision constraints. The difference is that the former fuses multi-level features of the encoder, thus providing richer feature information to the decoder. The latter uses hierarchical supervision of multi-level features of the encoder, imposing differentiated constraints on feature information. Both use shape and edge labels to optimize the decoder. The aforementioned networks do not focus on the overall information of the segmented object, ignore the relationships between pixels, and fail to effectively strengthen the regional connections of the semantically segmented object. Although MFFENet and GMNet both employ shape supervision, it significantly damages the semantic information of the segmented object.

[0004] Furthermore, most existing RGB-thermal (RGB-T) multimodal image semantic segmentation networks focus on multimodal feature fusion, employing various attention mechanisms to improve the fusion performance of different modalities. These methods largely concentrate on the encoder, neglecting the recovery of encoded features after fusion. Only RTFNet and MLFNet explicitly optimize the decoder's decoding blocks, but this optimization is not effectively applicable to other networks. Other methods only employ simple deep supervision at the decoder level while optimizing the encoder, failing to propose effective and targeted optimization strategies for the decoder. Since the network's encoder uses downsampling to extract high-level features, the decoder bears the heavy responsibility of recovering feature information. If the decoder fails to mitigate the information loss caused by downsampling, the segmentation results will suffer from incomplete or incorrect segmentation objects. On the other hand, existing RGB-T semantic segmentation networks often use cross-entropy loss for network training. This results in a lack of connection between the predicted result of each pixel and its surrounding pixels, making it difficult to effectively maintain the geometry of the segmentation result. This leads to segmentation object confusion and blurred segmentation edges. These problems significantly reduce the usability of semantic segmentation in real-world scenarios. By improving the decoder, and building upon an advanced encoder structure, we believe that superior RGB-T segmentation can be achieved. Summary of the Invention

[0005] To address the issue that most current multimodal image segmentation networks focus on the encoder and neglect the recovery of fused encoded features, this invention proposes a multimodal image semantic segmentation method based on shape and semantic enhancement. An effective decoder is designed, and the network is subjected to multi-stage deep supervision using a symbolic distance map. At the same time, channel attention is used to enhance semantic information, further improving the semantic extraction capability of the decoder and maintaining good segmentation shape.

[0006] The technical solution of the present invention is as follows:

[0007] A multimodal image semantic segmentation method based on shape and semantic enhancement includes the following steps:

[0008] Step 1: Acquire RGB and Thermal images from the RGB and Thermal sensors respectively, align the images using a calibration algorithm, and then uniformly resize the images by cropping; this forms dataset Q, which is then divided into training set Q. train and test set Q test ;

[0009] Step 2: Construct a multimodal image semantic segmentation network model based on shape and semantic enhancement;

[0010] Step 3: Train the multimodal image semantic segmentation network model and save the trained and high-performing network model.

[0011] Step 4: The RGB and Thermal sensors acquire RGB and Thermal images in real time, input them into the currently trained and high-performing network model, and perform semantic segmentation on the current image.

[0012] Furthermore, the decoder of the multimodal image semantic segmentation network is divided into a multi-stage shape supervision branch and a semantic supervision branch. The semantic supervision branch contains multiple channel-level semantic enhancement decoder blocks CLSE-D, and each CLSE-D contains a channel-level semantic enhancement module CLSE-M and a decoder block D. j The multi-stage shape supervision branch uses a symbolic distance map for supervision to preserve the geometric shape information of the segmentation results.

[0013] Furthermore, the specific workflow of the multimodal image semantic segmentation network model is as follows:

[0014] Step 2.1: Input the RGB and Thermal image data acquired by the sensor into the RGB encoder and Thermal encoder respectively. The modal fusion module fuses the image data in the encoder to obtain the features. C represents the number of feature channels, H represents the height of the feature, and W represents the width of the feature;

[0015] Step 2.2: In the semantic supervision branch, obtain the output of each decoder layer through the CLSE-D decoder block. As shown in equation (1),

[0016]

[0017] In the formula, This indicates the output of the previous channel-level semantic enhancement decoder block; CLSE-M(*) indicates the channel-level semantic enhancement module; D j This represents the j-th decoder block; This represents the output features of a channel-level semantic enhancement decoder block;

[0018] The specific working process of the CLSE-D decoder block is as follows:

[0019] In the CLSE-M module, the input features are first... The features are obtained by applying a global max pooling (GMP) and a global average pooling (GAP) method respectively, and then summing the outputs for each corresponding pixel. Next, a 1×1 convolutional layer (Conv) is used for dimensionality reduction to obtain the features. Where τ = 16, then the dimensions are increased by using the ReLU activation function and a 1×1 convolutional layer to obtain the features. Finally, the weighting coefficients W are obtained using the Sigmoid activation function as shown in equation (2). n The specific calculation is shown in equation (3); the obtained W n Input features We get the weighted multiplication and summation. As shown in equation (4); finally, the decoder block D of the original network is passed. j Obtain the output characteristics of CLSE-D As shown in equation (5);

[0020]

[0021] Where x represents the pixel value of each input feature;

[0022]

[0023] in, This indicates adding each element one by one;

[0024]

[0025] in, This indicates element-wise multiplication;

[0026]

[0027] Step 2.3: In the multi-stage shape supervision branch, decode the multi-stage features of the semantic supervision branch respectively. Perform deep supervision; first, the features or Dimensionality reduction is performed using a 1×1 convolution to obtain a single-channel feature, which is then passed through a standard batch normalization (BN) layer and a stochastically corrected linear unit activation function (RReLU). If the input is a feature... It also needs to be compared with the output features of the previous layer. Add them together; finally, perform double upsampling using bilinear interpolation Up2 to obtain the result. As shown in equations (6) and (7),

[0028]

[0029]

[0030] Furthermore, the specific process of step 3 is as follows:

[0031] Step 3.1: Use the training set Q trainThe model was trained with a learning rate of 0.01, using an exponential function to reduce the learning rate, and 200 iterations were performed. The optimizer used a Momentum of 0.9 and a weight decay of 0.0005. The data augmentation strategy and batch size were consistent with the network settings in the existing techniques for insertion.

[0032] Step 3.2: Through training, the output features obtained from the semantic supervision branch are... P is obtained through the Softmax function as shown in equation (8). seg Then, a weighted CE loss is calculated with the semantic label Q to obtain the semantic loss value L. seg As shown in equation (9); the output features obtained from the shape supervision branch P is obtained through the Tanh activation function. shape Then, the mean absolute error loss value is calculated by comparing it with the symbol distance map label S as shown in equation (10), and the shape loss value L is obtained. shape As shown in equation (11); finally, the two calculated loss values ​​are weighted and added together to obtain the total loss value L. total As shown in equation (12),

[0033]

[0034] Where K is the number of categories, x k P represents the predicted value for the k-th category. seg This represents the prediction result obtained from the semantic supervision branch;

[0035]

[0036] Where M represents the width of the input image, N represents the height of the input image, m represents the x-coordinate index of the image, n represents the y-coordinate index of the image, and x... mn Let (m,n) represent the image pixel value corresponding to coordinates (m,n), Q represent the semantic label for semantic segmentation, W represent the weight corresponding to each category, and L represent the image pixel value corresponding to coordinates (m,n). seg This represents the loss value obtained from the semantic supervision branch;

[0037]

[0038] Where p and q represent different pixels in the semantic label, and ||*||2 represents the Euclidean norm. This indicates that the infimum is defined by the boundary of the target object. Ω represents the boundary of the target object. in and Ω out Let S(p) represent the internal and external regions of the target object, respectively, and let S(p) represent the result of the symbolic distance map at point p.

[0039]

[0040] Among them, P shape This represents the prediction result obtained from the shape supervision branch, where S represents the symbol distance map label;

[0041] L total =λ1×L seg +λ2×L shape (12)

[0042] Where λ1 and λ2 represent L respectively seg and L shape The corresponding weights;

[0043] Step 3.3: Through iterative training, save the parameters of the network model with the best results;

[0044] Step 3.4, using the test set Q test The trained model is used to make predictions to obtain semantic segmentation results. If the prediction results correspond to the test set, it proves that the currently trained network model has good performance, and the current model is saved. Otherwise, the process is repeated until a network model with good performance is obtained.

[0045] The beneficial technical effects of this invention are as follows:

[0046] This invention explicitly proposes an optimization strategy for the decoder of an RGB-T semantic segmentation network. The proposed network, as a plug-and-play module, effectively improves the performance of existing networks. The symbolic distance map used has less impact on the semantic information of the decoder compared to the binary ground truth used in existing networks. Furthermore, multi-stage depth supervision is employed to more effectively improve the shape preservation capability of the decoder. Finally, channel attention is used to reduce the impact of shape supervision on semantic information, improving the decoder's ability to extract category semantic information. The proposed method is plug-and-play and can be combined with existing semantic segmentation networks to effectively improve image segmentation performance. This invention can effectively achieve pixel-level semantic segmentation of multimodal images (such as RGB-Thermal and RGB-Depth), and can be extended to applications such as autonomous driving and medical image analysis, with the expectation of creating considerable economic value. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the construction and training of the multimodal image semantic segmentation model based on shape and semantic enhancement according to the present invention;

[0048] Figure 2 This is a block diagram of the overall structure of the multimodal semantic segmentation network model for shape and semantic enhancement in this invention;

[0049] Figure 3 forFigure 2 A schematic diagram of the structure of the mid-channel semantic enhancement module. Detailed Implementation

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0051] As attached Figure 1 As shown, the present invention includes the following steps:

[0052] Step 1: Acquire RGB and Thermal images from a visible light sensor (RGB sensor) and a thermal infrared sensor (Thermal sensor), respectively. Align the images manually or using a calibration algorithm, and then crop them to obtain images of uniform size (640x480). This forms dataset Q. Divide dataset Q into training set Q. train and test set Q test .

[0053] Step 2: Construct a multimodal image semantic segmentation network model M based on shape and semantic enhancement. t ;

[0054] Based on the original RGB-T semantic segmentation model, inserting the method of this invention can effectively enhance the shape and category semantic information of the segmentation results. (Appendix) Figure 2 The overall structure of the proposed network model is summarized. The method of this invention is primarily used as a plug-and-play module in different networks to improve the feature recovery capability at the decoder end. First, the decoder end of the network is divided into a multi-stage shape supervision branch and a semantic supervision branch (structure as shown in the attached diagram). Figure 2 (As shown). Among them,

[0055] The semantic supervision branch contains multiple CLSE-D decoder blocks (channel-level semantic enhancement decoder blocks), each CLSE-D containing a module CLSE-M (channel-level semantic enhancer block) and decoder block D. j (j represents the i-th decoder block). In this embodiment of the invention, four decoder blocks are provided, namely D1, D2, D3, and D4. The CLSE-M module structure is shown in the attached figure. Figure 3 As shown, decoder block D j It maintains the same structure as the original network.

[0056] In the multi-stage shape supervision branch, to establish relationships between pixels, the shape information of the categories is strengthened. Specifically, the geometric shape information of the segmentation result is preserved by using a signed distance map for supervision. Because the signed distance map can capture the contours and distance information of geometric objects, the transition of edge distances is very smooth, making the shape supervision of the model in the shallow layers of the network more gentle. Compared with the supervision of binary mask map labels, it has less impact on the semantic information of shallow features. At the same time, in order to enhance the decoder's ability to recover features, multi-stage deep supervision of the semantic supervision branch is also performed through the shape supervision branch. However, using only the shape supervision branch will not only reduce the semantic information of the categories, but also fail to enhance the category features that are already difficult to identify. To solve this problem and further optimize the segmentation results, channel attention is inserted into the semantic supervision branch (structure as attached). Figure 3 (As shown) This enhances the semantic information of the segmentation results, thereby strengthening the geometric information of the segmentation results. This allows the semantic supervision branch to not only optimize the category geometry but also segment objects that are difficult to identify in certain scenes, resulting in better segmentation performance. The combination of the latter and the former not only enhances the connections between pixels but also further improves the decoder's utilization of the fused encoded features. The network will be described in detail below.

[0057] As attached Figure 2 As shown, the specific workflow of this model is as follows:

[0058] Step 2.1: Input the RGB and Thermal image data acquired by the sensor into the RGB encoder and Thermal encoder respectively. The modal fusion module fuses the image data in the encoder to obtain the features. C represents the number of channels of the feature (1024), H represents the height of the feature (30), and W represents the width of the feature (40). Its encoder can be the encoder of any multimodal fusion network.

[0059] Step 2.2: In the semantic supervision branch, obtain the output of each decoder layer through the CLSE-D decoder block. As shown in equation (1).

[0060]

[0061] In the formula, This indicates the output of the previous channel-level semantic enhancement decoder block; CLSE-M(*) indicates the channel-level semantic enhancement module; D j This represents the j-th decoder block; This represents the output features of the channel-level semantic enhancement decoder block.

[0062] The specific working process of the CLSE-D decoder block is as follows:

[0063] As attached Figure 3 As shown, in the CLSE-M module, the input features are first... The features are obtained by applying a global max pooling (GMP) and a global average pooling (GAP) method respectively, and then summing the outputs for each corresponding pixel. Next, a 1×1 convolutional layer (Conv) is used for dimensionality reduction to obtain the features. Where τ = 16, then the dimensions are increased by using the ReLU activation function and a 1×1 convolutional layer to obtain the features. Finally, the weighting coefficients W are obtained through the Sigmoid activation function (as shown in equation (2)). n As shown in equation (3). The obtained W n Input features We get the weighted multiplication and summation. As shown in equation (4). Finally, the decoder block D of the original network is passed through... j Obtain the output characteristics of CLSE-D As shown in equation (5).

[0064]

[0065] Where x represents the feature The value of each pixel.

[0066]

[0067] in, This indicates adding each element individually.

[0068]

[0069] in, This indicates element-wise multiplication.

[0070]

[0071] Step 2.3: In the multi-stage shape supervision branch, decode the multi-stage features of the semantic supervision branch respectively. Perform deep supervision. First, analyze the features... or Dimensionality reduction is performed using a 1×1 convolution to obtain a single-channel feature, which is then passed through a standard batch normalization (BN) layer and a stochastically corrected linear unit activation function (RReLU). If the input is a feature... It also needs to be compared with the output features of the previous layer. Add them together. Finally, perform a double upsampling using bilinear interpolation Up2 to obtain the result. As shown in equations (6) and (7).

[0072]

[0073]

[0074] Step 3: Train the multimodal image semantic segmentation network model and save the trained and high-performing network model. The specific process is as follows:

[0075] Step 3.1: Use the training set Q train The model was trained with a learning rate of 0.01, which was then reduced exponentially for 200 iterations. The optimizer used a momentum of 0.9 and weight decay of 0.0005. The data augmentation strategy and batch size were consistent with the network settings in the existing techniques being used.

[0076] Step 3.2: Through training, the output features obtained from the semantic supervision branch are... P is obtained through the Softmax function (as shown in Equation (8)). seg Then, a weighted CE (cross-entropy) loss is calculated with the label Q (semantic label) to obtain the semantic loss value L. seg As shown in equation (9); the output features obtained from the shape supervision branch P is obtained through the Tanh activation function. shape Then, the L1 (mean absolute error) loss value is calculated with the label S (symbol distance map) (as shown in equation (10)) to obtain the shape loss value L. shape As shown in equation (11); finally, the two calculated loss values ​​are weighted and added together to obtain the total loss value L. total As shown in equation (12),

[0077]

[0078] Where K is the number of categories (9), x k P represents the predicted value for the k-th category. seg This represents the prediction result obtained from the semantic supervision branch.

[0079]

[0080] Where M represents the width of the input image (640), N represents the height of the input image (480), m represents the x-coordinate index of the image, n represents the y-coordinate index of the image, and x... mn Let (m,n) represent the image pixel value corresponding to coordinates (m,n), Q represent the semantic label of semantic segmentation, W represent the weight corresponding to each category (e.g., 1, 10, 10, 10, 10, 10, 10, 10), and L represent the image pixel value corresponding to coordinates (m,n). seg This represents the loss value obtained from the semantic supervision branch.

[0081]

[0082] Where p and q represent different pixels in the semantic label, The boundary of the target object is represented by ||*||2, which represents the Euclidean norm. Ω represents the infimum, which is the boundary of the target object. in and Ω out Let S(p) represent the internal and external regions of the target object, respectively, and let S(p) represent the result of the symbolic distance map at point p.

[0083]

[0084] Among them, P shape This represents the prediction result obtained from the shape supervision branch, and S represents the symbol distance map label.

[0085] L total =λ1×L seg +λ2×L shape (12)

[0086] Where λ1 and λ2 represent L respectively seg and L shape The corresponding weights are 1 and 0.5.

[0087] Step 3.3: Through iterative training, save the parameters of the network model with the best results.

[0088] Step 3.4, using the test set Q test For the trained model M t Make a prediction to obtain the semantic segmentation result. If the prediction result corresponds to the test set, it proves that the currently trained network model has good performance and saves the current model. Otherwise, return to retraining until a network model with good performance is obtained.

[0089] Step 4: The RGB and Thermal sensors acquire RGB and Thermal images in real time, input them into the currently trained and high-performing network model, and perform semantic segmentation on the current image.

[0090] Compared to existing RGB-T semantic segmentation networks, this invention proposes an effective and targeted optimization strategy for the decoder. It primarily employs a symbolic distance map for multi-stage depth supervision of the decoder, effectively improving its ability to preserve category shapes and establishing connections between pixels. Compared to binary mask image labels, this approach has less impact on the decoder's semantics. Since shape supervision alone reduces the semantic information of semantic branches, this invention proposes the CLSE-M module to mitigate the impact of shape supervision branches on semantic branches and further improve the decoder's semantic information extraction capability.

[0091] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A multimodal image semantic segmentation method based on shape and semantic enhancement, characterized in that, Includes the following steps: Step 1: Acquire RGB and Thermal images from the RGB and Thermal sensors respectively, align the images using a calibration algorithm, and then uniformly resize the images by cropping; this forms dataset Q, which is then divided into training set Q. train and test set Q test ; Step 2: Construct a multimodal image semantic segmentation network model based on shape and semantic enhancement; The decoder of the multimodal image semantic segmentation network is divided into a multi-stage shape supervision branch and a semantic supervision branch. The semantic supervision branch contains multiple channel-level semantic enhancement decoder blocks CLSE-D, and each CLSE-D contains a channel-level semantic enhancement module CLSE-M and a decoder block D. j The multi-stage shape supervision branch uses a signed distance map for supervision to preserve the geometric shape information of the segmentation results. The specific workflow of the multimodal image semantic segmentation network model is as follows: Step 2.1: Input the RGB and Thermal image data acquired by the sensor into the RGB encoder and Thermal encoder respectively. The modal fusion module fuses the image data in the encoder to obtain the features. C represents the number of feature channels, H represents the height of the feature, and W represents the width of the feature; Step 2.2: In the semantic supervision branch, obtain the output of each decoder layer through the CLSE-D decoder block. As shown in equation (1), (1) In the formula, This represents the output of the previous channel-level semantic enhancement decoder block; This indicates a channel-level semantic enhancement module; This represents the j-th decoder block; This represents the output features of a channel-level semantic enhancement decoder block; The specific working process of the CLSE-D decoder block is as follows: In the CLSE-M module, the input features are first... The features are obtained by applying a global max pooling (GMP) and a global average pooling (GAP) method respectively, and then summing the outputs for each corresponding pixel. Next, a 1×1 convolutional layer (Conv) is used for dimensionality reduction to obtain the features. ,in Then, the dimensions are increased by using the ReLU activation function and a 1×1 convolutional layer to obtain the features. Finally, the weighting coefficients are obtained through the Sigmoid activation function as shown in equation (2). The specific calculation is shown in equation (3); the obtained Input features We get the weighted multiplication and summation. As shown in equation (4); finally, the decoder block of the original network is passed through... Obtain the output characteristics of CLSE-D As shown in equation (5); (2) Where x represents the feature Each pixel value; (3) in, This indicates adding each element one by one; (4) in, This indicates element-wise multiplication; (5) Step 2.3: In the multi-stage shape supervision branch, decode the multi-stage features of the semantic supervision branch respectively. Perform deep supervision; first, the features or Dimensionality reduction is performed using a 1×1 convolution to obtain a single-channel feature, which is then passed through a standard batch normalization (BN) layer and a stochastically corrected linear unit activation function (RReLU). If the input is a feature... It also needs to be compared with the output features of the previous layer. Add them together; finally, perform double upsampling using bilinear interpolation Up2 to obtain the result. As shown in equations (6) and (7), (6) (7) Step 3: Train the multimodal image semantic segmentation network model and save the trained and high-performing network model. Step 4: The RGB and Thermal sensors acquire RGB and Thermal images in real time, input them into the currently trained and high-performing network model, and perform semantic segmentation on the current image.

2. The multimodal image semantic segmentation method based on shape and semantic enhancement according to claim 1, characterized in that, The specific process of step 3 is as follows: Step 3.1: Use the training set Q train The model was trained with a learning rate of 0.01, using an exponential function to reduce the learning rate, and 200 iterations were performed. The optimizer used a Momentum of 0.9 and a weight decay of 0.0005. The data augmentation strategy and batch size were consistent with the network settings in the existing techniques for insertion. Step 3.2: Through training, the output features obtained from the semantic supervision branch are... The softmax function shown in equation (8) is used to obtain Then, a weighted CE loss is calculated by combining the semantic label Q with the result to obtain the semantic loss value. As shown in equation (9); the output features obtained from the shape supervision branch Obtained through the Tanh activation function Then, the mean absolute error loss value is calculated by comparing it with the symbol distance map label S as shown in equation (10), and the shape loss value is obtained. As shown in equation (11); finally, the two calculated loss values ​​are weighted and summed to obtain the total loss value. As shown in equation (12), (8) Where K is the number of categories, This represents the predicted value for the k-th category. This represents the prediction result obtained from the semantic supervision branch; (9) Where M represents the width of the input image, N represents the height of the input image, m represents the x-coordinate index of the image, n represents the y-coordinate index of the image, and x... mn This represents the image pixel value corresponding to coordinates (m, n). Semantic tags representing semantic segmentation. This represents the weight corresponding to each category. This represents the loss value obtained from the semantic supervision branch; (10) Where p and q represent different pixels in the semantic label, Describes the Euclidean norm. This indicates that the infimum is defined by the boundary of the target object. Indicates the boundary of the target object. and These represent the internal and external regions of the target object, respectively. This represents the result of the symbolic distance plot at point p; (11) in, This represents the prediction result obtained from the shape supervision branch. Indicates the symbol distance map label; (12) in, and They represent and The corresponding weights; Step 3.3: Through iterative training, save the parameters of the network model with the best results; Step 3.4, using the test set Q test The trained model is used to make predictions to obtain semantic segmentation results. If the prediction results correspond to the test set, it proves that the currently trained network model has good performance, and the current model is saved. Otherwise, the process is repeated until a network model with good performance is obtained.

Images