Image semantic segmentation method, electronic device, and computer-readable storage medium

Abstract

The application discloses an image semantic segmentation method, an electronic device and a computer readable storage medium, comprising: obtaining a target image and an image gradient map thereof; inputting the target image into a semantic segmentation network to obtain shallow feature maps and deep feature maps through an encoder; the semantic segmentation network comprises a cascaded encoder, a semantic-boundary double-branch decoder and an output layer; the feature fusion of multiple feature maps is performed through the semantic fusion branch of the decoder to obtain a semantic segmentation prediction result; the boundary information is obtained based on the image gradient map and the multiple feature maps through the boundary refinement branch of the decoder, and the feature fusion of the multiple feature maps is performed based on the boundary information to obtain a boundary prediction result; and the semantic segmentation prediction result and the boundary prediction result are fused through the output layer to obtain a semantic segmentation image of the target image. The application can further depict the object boundary in the semantic segmentation prediction result, thereby improving the accuracy and segmentation effect of semantic segmentation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to image semantic segmentation methods, electronic devices, and computer-readable storage media. Background Technology

[0002] With the rapid development of computer vision algorithms, semantic segmentation has gradually become an important aspect of image understanding in image processing and machine vision technologies. The purpose of semantic segmentation is to predict the semantic category of each pixel in an image by training on the image's content.

[0003] Currently, semantic segmentation algorithms based on convolutional neural networks have achieved increasingly higher accuracy. However, high-precision models are often accompanied by complex network structures and large amounts of computational resources, which can severely impact network runtime and inference speed. In existing technologies, to reduce the computational resources required for semantic segmentation, lightweight networks are typically used to achieve a balance between inference speed and segmentation accuracy. For example, depthwise separable convolutions are used to reduce computational load, and the final output features of the encoder are used as input features of the decoder to obtain the final predicted segmentation map of the image.

[0004] However, the decoders of the aforementioned lightweight networks typically only use semantic and / or channel information from a single layer of the encoder for fusion, without fully extracting and fusing spatial boundary information. They are insensitive to details in the image, which affects the network's segmentation performance and makes it impossible to accurately segment the boundaries of the target, thus impacting the final segmentation result. Summary of the Invention

[0005] The main technical problem addressed by this application is to provide an image semantic segmentation method, electronic device, and computer-readable storage medium that can solve the problem that existing technologies cannot accurately segment target boundaries.

[0006] To address the aforementioned technical problems, the first technical solution adopted in this application is to provide an image semantic segmentation method, comprising: acquiring a target image and an image gradient map of the target image; inputting the target image into a semantic segmentation network, and extracting features from the target image through the encoder of the semantic segmentation network to obtain shallow feature maps and deep feature maps of the target image; wherein, the semantic segmentation network includes a cascaded encoder, a semantic-boundary dual-branch decoder, and an output layer; and inputting the shallow feature maps and deep feature maps into the semantic fusion branch of the semantic-boundary dual-branch decoder to perform semantic fusion on the shallow feature maps and deep feature maps. Semantic information and channel information are fused to obtain semantic segmentation prediction results. Image gradient maps, shallow feature maps, and deep feature maps are input into the boundary refinement branch of the semantic-boundary dual-branch decoder to obtain boundary information based on these maps. The spatial information of the shallow and deep feature maps is then fused based on this boundary information to obtain boundary prediction results. The semantic segmentation prediction results and boundary prediction results are input into the output layer, where they are fused to obtain the semantic segmentation image of the target image.

[0007] To solve the above-mentioned technical problems, the second technical solution adopted in this application is to provide an electronic device, including: a memory for storing program data, wherein when the stored program data is executed, the steps in the image semantic segmentation method as described in any of the above claims are implemented; and a processor for executing program instructions stored in the memory to implement the steps in the image semantic segmentation method as described in any of the above claims.

[0008] To solve the above-mentioned technical problems, the third technical solution adopted in this application is to provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps in the image semantic segmentation method as described in any of the above claims.

[0009] The beneficial effects of this application are as follows: Unlike existing technologies, this application provides an image semantic segmentation method, electronic device, and computer-readable storage medium. By using a semantic fusion branch in a semantic-boundary dual-branch decoder to fuse semantic and channel information in shallow and deep feature maps, it can fully integrate semantic features from different layers, thereby enhancing the accuracy of the semantic segmentation network in target recognition and localization, and enabling the semantic segmentation prediction results to possess multi-scale semantic information. Furthermore, by using a boundary refinement branch to obtain boundary information based on image gradient maps, shallow feature maps, and deep feature maps, and then using this boundary information to fuse spatial information from shallow and deep feature maps, it can fully learn and fuse boundary features contained in different layers, thereby enhancing the semantic segmentation network's ability to recognize target contours and improving the discrimination of edges between targets, thus improving the accuracy of target boundary segmentation in the boundary prediction results. By fusing the semantic segmentation prediction results with the boundary prediction results, it is possible to further characterize the object boundaries in the semantic segmentation prediction results based on a better fusion of image information, thereby improving the accuracy of semantic segmentation and ultimately enhancing the segmentation effect. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is an architecture diagram of one embodiment of the semantic segmentation network of this application;

[0012] Figure 2 This is an architecture diagram of one implementation of the ultra-efficient cross-stage non-bottleneck residual module of this application;

[0013] Figure 3 yes Figure 2 Architecture diagram of one implementation of the decomposed residual module;

[0014] Figure 4 yes Figure 1 Architecture diagram of the first fusion unit in one implementation method;

[0015] Figure 5 yes Figure 4 Architecture diagram of one implementation of the mid-channel attention module;

[0016] Figure 6 yes Figure 4 Architecture diagram of one implementation of the bidirectional spatial attention module;

[0017] Figure 7yes Figure 1 Architecture diagram of the second fusion unit in one implementation method;

[0018] Figure 8 This is a flowchart illustrating one implementation method of the semantic segmentation network training method of this application;

[0019] Figure 9 This is a flowchart illustrating the first embodiment of the image semantic segmentation method of this application;

[0020] Figure 10 This is a flowchart illustrating the second embodiment of the image semantic segmentation method of this application;

[0021] Figure 11 This is a flowchart illustrating the third embodiment of the image semantic segmentation method of this application;

[0022] Figure 12 This is a flowchart illustrating the fourth embodiment of the image semantic segmentation method of this application;

[0023] Figure 13 This is a schematic diagram of one embodiment of the image semantic segmentation apparatus of this application;

[0024] Figure 14 This is a schematic diagram of the structure of one embodiment of the electronic device of this application;

[0025] Figure 15 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “said,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless otherwise clearly indicated above. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0028] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0029] It should be understood that the terms "comprising," "including," or any other variations used herein are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0030] This application first provides a semantic segmentation network.

[0031] Specifically, please refer to Figure 1 , Figure 1 This is an architecture diagram of one embodiment of the semantic segmentation network of this application. In this embodiment, the semantic segmentation network 100 includes a cascaded encoder 10, a semantic-boundary dual-branch decoder, and an output layer 30.

[0032] In this embodiment, encoder 10 is used for image feature extraction, semantic-boundary dual-branch decoder 20 is used to restore the feature image size, and output layer 30 is used to output the final segmentation result.

[0033] In this embodiment, the encoder 10 includes three feature extraction units 11 and a downsampling module 12. The downsampling module 12 is used to reduce the resolution of the image, and the feature extraction units 11 are used to extract image features.

[0034] In this embodiment, the semantic-boundary dual-branch decoder includes a semantic fusion branch 21, a boundary refinement branch 22, and an upsampling module. The upsampling module is used to reduce the image resolution. The semantic fusion branch 21 is used to generate semantic segmentation prediction results. The boundary refinement branch 22 is used to generate boundary prediction results.

[0035] The semantic fusion branch 21 includes a first fusion unit 211 and a pointwise convolution unit 212, and the boundary refinement branch 22 includes a second fusion unit 221 and a pointwise convolution unit 222.

[0036] Understandably, by fusing the output features (boundary prediction result) of the boundary refinement branch 22 with the output features (semantic segmentation prediction result) of the semantic fusion branch 21, the object boundaries in the semantic segmentation prediction result can be further characterized on the basis of better fusing image information, thereby improving the segmentation accuracy of the semantic segmentation network 100 and thus improving the segmentation effect of the semantic segmentation network 100 on the image.

[0037] In this embodiment, the encoder 10 includes three stages, with each set of feature extraction units 11 representing one extraction stage. Specifically, the first set of feature extraction units 11 is used to extract low-level features (shallow features), while the second and third sets of feature extraction units 11 are used to extract high-level features (deep features).

[0038] The first set of feature extraction units 11 includes three Extremely Efficient Cross Stage Non-bottleneck Residual Modules (EECSNB), the second set of feature extraction units 11 includes two EECSNBs, and the third set of feature extraction units 11 includes eight EECSNBs.

[0039] Specifically, please refer to Figure 2 and Figure 3 , Figure 2 This is an architecture diagram of one implementation method of the ultra-efficient cross-stage non-bottleneck residual module of this application. Figure 3 yes Figure 2 Architecture diagram of one implementation of the decomposition residual module.

[0040] In this embodiment, the target image is downsampled and used as the input image of the encoder 10 of the semantic segmentation network 100. The input image is first input into the first set of feature extraction units 11 of the encoder 10. After receiving the input image, the EECSNB in ​​the first set of feature extraction units 11 first performs channel splitting on the feature channels of the input image to divide the feature channels into two parts, and sends one part into the left branch and the other part into the right branch in parallel.

[0041] Understandably, the splitting method provided in this implementation directly sets the splitting method in the network architecture and does not use pointwise convolution (1×1 convolution, 1×1 Conv). Therefore, no training is required, no parameters are added, and the computational complexity of the model can be reduced.

[0042] In this embodiment, both the left and right branches include a 3×1 and a 1×3 asymmetric convolutional unit, respectively, i.e., a decomposed convolutional unit. The left branch extracts image features through the decomposed convolutional unit and then performs batch normalization (BN) on the network layer. The right branch extracts image features through the factorized residual block (FR block) and then performs batch normalization and nonlinear activation on the network layer, using the rectified linear unit (ReLU) activation function.

[0043] In existing technologies, a ReLU activation function is applied after each network layer. However, the extensive use of ReLU activation functions can lead to information loss or feature corruption. In this embodiment, ReLU activation functions are applied only after the FR Block in the right branch and after the decomposed convolutional unit in the left branch. This approach enhances the network's representational capabilities through nonlinear data while avoiding information loss caused by excessive use of ReLU activation functions.

[0044] In this implementation, the left branch uses a set of 3×1 and 1×3 asymmetric convolutional units to extract local features (shallow features) of the input image. These local features need to be comprehensive but not very deep. The right branch uses FRBlock and another set of 3×1 and 1×3 asymmetric convolutional units to extract deeper semantic features (deep features) of the input image.

[0045] The FR Block includes two sets of 3×1 and 1×3 asymmetric convolutional units, which can extract deeper features using the two sets of decomposition convolutions. In order to make up for the lack of expressive power of decomposition convolutions, the second set of asymmetric convolutions in the FR Block is a dilated convolutional unit. Dilated convolutions can expand the receptive field of the image, so that the FR Block can adaptively extract features of the corresponding scale at each network stage.

[0046] In this context, the receptive field refers to the size of the input layer region corresponding to an element in the output of a given layer. A larger receptive field contains more contextual relationships, enabling the extraction of more semantic features and thus including more semantic information in the output feature map. Dilated convolution refers to injecting holes into the convolutional region to increase the receptive field. Dilated convolution only includes the aperture coefficient r, where r represents the number of holes, which can be 0 or any integer greater than 0. Understandably, when r is 0, dilated convolution is equivalent to ordinary depthwise convolution; only when r is greater than 0 is it considered dilated convolution.

[0047] The FR Block primarily extracts deep features. To prevent the FR Block from missing some shallow features, the right branch sets up a set of 3×1 and 1×3 asymmetric convolutional units after the FR Block to extract shallow features more comprehensively, thereby compensating for the potential shortcomings of the right branch in terms of feature representation.

[0048] In this embodiment, after the features of the input image are extracted by the left and right branches, the features extracted by the left and right branches are concatenated, and then added pixel by pixel.

[0049] Here, pixel-by-pixel addition refers to adding each pixel in the feature map obtained from the left branch to each pixel in the feature map obtained from the right branch.

[0050] Furthermore, after addition, a channel shuffle is introduced to facilitate information exchange between the left and right branches. Channel shuffle involves grouping the feature channels obtained from the left and right branches, then evenly distributing the groups to restore the total number of channels.

[0051] Understandably, from the perspective of convolution methods, the EECSNB provided in this implementation does not use grouped convolution or 1×1 convolution, and from the perspective of feature splitting and feature rearrangement, it does not introduce more parameters, thus greatly reducing the computational complexity and memory access of the model. Furthermore, since EECSNB also includes FR Blocks, it can enhance the feature representation capability of the overall module by enhancing the extraction of deep features.

[0052] In this embodiment, after feature extraction by the first set of feature extraction units 11, the number of channels in the resulting shallow feature map is equal to the number of convolutional units (convolutional kernels) in the first set of feature extraction units 11. Further, the resulting shallow feature map is input to the second set of feature extraction units 11 for feature extraction, and the number of channels in the resulting new shallow feature map is equal to the number of convolutional units in the second set of feature extraction units 11. Further, the new shallow feature map is input to the third set of feature extraction units 11 for feature extraction, and the number of channels in the resulting deep feature map is equal to the number of convolutional units in the third set of feature extraction units 11.

[0053] Specifically, each convolutional unit extracts features from the input image once, resulting in an H×W feature map, where H represents the height and W represents the width. The feature maps extracted by multiple convolutional units are stacked together to form a shallow or deep feature map, whose feature dimensions are represented as C×H×W, where C represents the number of feature channels (or the total number of feature maps used for stacking). A more concrete representation can be found in [link to relevant documentation]. Figure 5 The final structure of the channel attention map C is obtained.

[0054] For each pixel, the feature map extracted by each convolutional unit includes a portion of its features. The feature maps extracted by all convolutional units need to be stacked together to obtain detailed image features based on multiple feature channels.

[0055] In this embodiment, after the semantic-boundary dual-branch decoder obtains the corresponding feature map from each extraction stage of the encoder 10, it fuses the semantic information and channel information contained in different layers extracted by the feature extraction units 11 of each stage of the encoder 10 through the semantic fusion branch 21 to generate a semantic segmentation prediction result; and it fuses the boundary information contained in different layers extracted by each feature extraction unit 11 of the encoder 10 through the boundary refinement branch 22 to generate a boundary prediction result.

[0056] Specifically, after fusing the semantic and channel information contained in each layer through the first fusion unit 211 in the semantic fusion branch 21, the number of channels is reduced by the pointwise convolution unit 212 to transform the number of channels in the decoded feature map into the required number of segmentation categories, thereby generating a semantic segmentation prediction result. Similarly, after fusing the boundary information contained in each layer through the second fusion unit 221 in the boundary refinement branch 22, the number of channels is reduced by the pointwise convolution unit 222 to transform the number of channels in the decoded feature map into the required number of boundary categories, thereby generating a boundary prediction result.

[0057] In this embodiment, the first fusion unit 211 is a fusion bilateral global attention block (ABGAB), which includes a channel attention module (CAM), a bilateral spatial attention module (BSAM), and a decomposed convolution unit.

[0058] Specifically, please refer to Figure 4 , Figure 5 and Figure 6 , Figure 4 yes Figure 1 Architecture diagram of the first fusion unit in one implementation method. Figure 5 yes Figure 4 Architecture diagram of one implementation of the mid-channel attention module. Figure 6 yes Figure 4 Architecture diagram of one implementation of the bidirectional spatial attention module.

[0059] In this embodiment, the semantic fusion branch 21 upsamples the deep feature map obtained from the encoder 10 so that the resolution of the upsampled deep feature map is consistent with that of the obtained shallow feature map. Then, the upsampled deep feature map and the shallow feature map are stacked, and the stacked image is used as the input image of the first fusion unit 211. The input image is then fed into the channel attention module (CAM), the bidirectional spatial attention module (BSAM), and a set of 3×1 and 1×3 decomposition convolutional units of the first fusion unit 211.

[0060] In this embodiment, the Channel Attention (CAM) module can model the dependencies between channels, adaptively adjust the importance of each channel, and assign greater weights to important channels and smaller weights to secondary features. That is, starting from global information, it selectively amplifies valuable feature channels and suppresses useless feature channels, thereby improving the performance of the network.

[0061] Specifically, suppose the input feature map of the input channel attention module CAM is F. in ∈R C×H×W Where C represents the number of channels in the feature map, H represents the height of the feature map, and W represents the width of the feature map. For the input feature map F... in A global average pooling operation is performed to aggregate the spatial information of each channel into the corresponding channel's feature vector C'. The formula for calculating the feature vector C' is as follows:

[0062]

[0063] Among them, F in (i,j) represents the value of the input feature in the i-th row and j-th column, and ∑ is the summation function.

[0064] After global average pooling, the input feature map F in The dimension changes from C×H×W to C×1×1. Then, the feature vector C' is fed into a 1×1 convolution for single-point feature extraction, followed by normalization using a Sigmoid activation function, and finally averaged to obtain the channel attention map C' (i.e., the channel weights). The value of each channel attention map C' remains between (0,1). The formula for calculating the channel attention map C' is as follows:

[0065] C”=δ(f 1×1 *C')

[0066] Among them, f 1×1 This represents a 1×1 convolution, δ represents the Sigmoid activation function, and * represents the convolution operation.

[0067] In this embodiment, the input feature map F in The input is fed into 3×1 and 1×3 decomposed convolutional units to process the input feature map F. in Perform asymmetric convolution, and denote the feature map extracted by the convolution as F. m .

[0068] Furthermore, the feature map F extracted by convolution is... m The channel attention map C is multiplied pixel-by-pixel to obtain the channel-weighted feature map F. c Among them, the channel-weighted feature map F c The calculation formula is as follows:

[0069]

[0070] in, This indicates pixel-by-pixel multiplication.

[0071] Wherein, pixel-wise multiplication represents the feature map F extracted by convolution. m Each pixel in the input is multiplied by each pixel in the channel attention map C.

[0072] Understandably, this embodiment stacks channels from different network layers of varying depths together as input to the Channel Attention Module (CAM), maximizing the utilization of channel information from each network layer in the encoder 10. Furthermore, by re-encoding the importance of each channel using the CAM, the semantic segmentation network 100's ability to recognize targets can be effectively improved.

[0073] In this embodiment, the Bidirectional Spatial Attention (BSAM) module decomposes traditional spatial attention into horizontal and vertical spatial attention to capture long-range correlations (i.e., long-range dependent positional relationships) between pixels in the horizontal and vertical directions, respectively. This results in bidirectional spatial attention maps with different weights in two different dimensions, namely, a horizontal spatial weight matrix and a vertical spatial weight matrix. When the bidirectional spatial attention map is fused with the backbone network, it can provide feature guidance to high-level features, guiding them to learn more important and complete object regions.

[0074] Specifically, suppose the input feature map of the bidirectional spatial attention module BSAM is F. in ∈RC×H×W The input feature map F in The data is fed into a 1×1 convolution for single-point feature extraction. Then, the features extracted by the convolution are subjected to horizontal pooling and vertical pooling respectively, resulting in a horizontal matrix G. h With the vertical matrix G v , where the horizontal matrix G h With the vertical matrix G v The calculation formulas are as follows:

[0075]

[0076]

[0077] Among them, f 1×1 ∑ represents a 1×1 convolution, * represents the convolution operation, and ∑ is the summation function.

[0078] Then the horizontal matrix G h With the vertical matrix G v After normalizing each signal using a Sigmoid activation function, the average value is taken to obtain the horizontal spatial attention map S. h With vertical spatial attention map S v (that is, the horizontal space weight matrix S) h With vertical space weight matrix S v ), where the horizontal spatial attention map S h With vertical spatial attention map S v The calculation formulas are as follows:

[0079] S h =δ(G h )

[0080] S v =δ(G v )

[0081] Where δ represents the Sigmoid activation function.

[0082] Furthermore, the horizontal spatial attention map S h With vertical spatial attention map S v The feature maps F are weighted by the channels respectively. c Pixel-wise multiplication followed by pixel-wise summation yields a spatially weighted feature map. This spatially weighted feature map is then combined with the feature map F extracted by convolution. m The pixels are added one by one to obtain the output feature map F of the first fusion unit 211. cs The output feature map F cs The calculation formula is as follows:

[0083]

[0084] in, This indicates pixel-by-pixel multiplication. This indicates that each pixel is added together.

[0085] Furthermore, the output feature map F of the first fusion unit 211 cs The data is fed into pointwise convolutional unit 212 for 1×1 convolution to convert the output feature map F. cs The number of channels is transformed into the required number of segmentation categories, thereby generating semantic segmentation prediction results.

[0086] Specifically, for a network, the feature maps output by the shallow layers contain more boundary and detail information, while the feature maps output by the higher layers contain more detailed semantic information. Semantic information is beneficial for pixel classification, while boundary and detail information enables more refined and accurate classification of target boundaries. Both shallow and deep features retain key spatial location information, but shallow features contain richer spatial information than deep features.

[0087] Understandably, this embodiment stacks channels from different network layers of varying depths together as input to the bidirectional spatial attention module (BSAM), maximizing the utilization of spatial information from each network layer in the encoder 10. Furthermore, by optimizing the traditional spatial attention method into horizontal and vertical spatial attention methods, it is possible to better capture positional relationships with long-distance dependencies, thereby improving the target localization capability of the semantic segmentation network 100.

[0088] Understandably, the first fusion unit 211 (ABGAB) provided in this embodiment can generate a channel attention map using the rich channel features after stacking, and thereby encode the input feature map F. in The importance of each channel is emphasized, thereby improving the model's classification accuracy. Simultaneously, a horizontal spatial attention map S is extracted from the stacked features. h With vertical spatial attention map S v It can utilize richer location and channel information to encode the input feature map F in The importance of each pixel is determined, thereby improving the localization accuracy of the semantic segmentation network 100.

[0089] In this embodiment, the second fusion unit 221 is an Efficient Spatial Attention Aggregation Block (ESAAB), which includes a bidirectional spatial attention module BSAM and a decomposed convolution unit.

[0090] Specifically, please refer to Figure 7, Figure 7 yes Figure 1 Architecture diagram of the second fusion unit in the first implementation method.

[0091] In this embodiment, the boundary refinement branch 22 upsamples the deep feature map obtained from the encoder 10 so that the resolution of the upsampled deep feature map is consistent with that of the obtained shallow feature map. Then, the upsampled deep feature map and the shallow feature map are stacked, and the stacked image is used as the input image of the second fusion unit 221. The input image is input into the bidirectional spatial attention module BSAM and a set of 3×1 and 1×3 decomposition convolution units of the second fusion unit 221, respectively.

[0092] Specifically, suppose the input feature map to the second fusion unit 221 is F in ∈R C×H×W In this embodiment, in order to make the boundary refinement branch 22 more accurately characterize the boundary information of pixels, the Canny edge detection algorithm is also used to detect the target image to obtain the image gradient map of the target image, and the image gradient map is compared with the input feature map F. in The concatenated images are then fed into a 1×1 convolution to extract single-point features, maximizing the extraction of boundary information for each pixel.

[0093] Furthermore, the feature map including boundary information is input into the bidirectional spatial attention module BSAM, so that the extracted features are horizontally pooled and vertically pooled by the bidirectional spatial attention module BSAM to obtain horizontal and vertical matrices. Then, the horizontal and vertical matrices are normalized by a sigmoid activation function and the average value is taken to obtain horizontal and vertical spatial attention maps containing rich boundary information.

[0094] Specifically, when horizontal and vertical spatial attention maps containing rich boundary information are fused with the backbone network, boundary-related features in the high-level feature maps can be enhanced, while non-boundary-related features can be suppressed, thereby making the boundaries of the target clearer.

[0095] In this embodiment, the input image F in The input is fed into 3×1 and 1×3 decomposition convolution units to perform asymmetric convolution on the input image, resulting in the feature map F extracted by the convolution. m Then, the horizontal spatial attention map and the vertical spatial attention map, which contain rich boundary information, are respectively convolved with the feature map F extracted by convolution. m Pixel-wise multiplication followed by pixel-wise summation yields a spatially weighted feature map. This spatially weighted feature map is then combined with the feature map F extracted by convolution. m The pixels are added one by one to obtain the output feature map F of the second fusion unit 221.ss .

[0096] Furthermore, the output feature map F of the second fusion unit 221 ss The input is a pointwise convolutional unit 222, where a 1×1 convolution is performed to convert the output feature map F. ss The number of channels is transformed into the required number of boundary categories, thereby generating boundary prediction results.

[0097] Understandably, the second fusion unit 221 (ESAAB) provided in this embodiment can obtain the boundary information contained in different layers of the encoder 10 by using the image gradient maps of the input image and the target image, and fuse the boundary features contained in different layers.

[0098] Furthermore, the semantic segmentation prediction result is multiplied pixel by pixel with the boundary prediction result to obtain a boundary-weighted feature map. Then, the boundary-weighted feature map is added pixel by pixel with the semantic segmentation prediction result to obtain a segmentation image that incorporates the boundary features. Subsequently, bilinear interpolation is used to upsample the segmentation image that incorporates the boundary features to restore its resolution to be consistent with the input image, thereby generating the semantic segmentation image of the semantic segmentation network 100.

[0099] Understandably, this embodiment can assist the semantic fusion branch 21 in learning more accurate target boundary features by introducing a boundary refinement branch 22. At the same time, under the interaction between the bidirectional spatial attention module BSAM and the image gradient map in the boundary refinement branch 22, the semantic segmentation network 100 can further enhance its ability to recognize target contours and the distinguishability of edges between targets, so that the shape and boundary of the target in the final semantic segmentation image are clearer and the classification is more accurate.

[0100] Please see Figure 8 , Figure 8 This is a flowchart illustrating one embodiment of the training method for the semantic segmentation network of this application. In this embodiment, the training method includes:

[0101] S11: Obtain the training dataset; the training dataset includes multiple training images, pre-annotated semantic segmentation information, and boundary segmentation information generated based on the semantic segmentation information.

[0102] In this embodiment, for the pre-annotated semantic segmentation information, the boundary is divided by obtaining the response values ​​of different targets at their contours, and one-hot encoding is used to distinguish between the boundary and non-boundary to obtain the boundary segmentation information.

[0103] S12: Based on the pre-annotated semantic segmentation information, the image gradient map of each training image is obtained using the edge detection algorithm.

[0104] S13: Input the training image into the encoder of the preset segmentation model to obtain the shallow feature map and deep feature map of the training image.

[0105] S14: Input the shallow and deep feature maps of the training image into the semantic fusion branch of the preset segmentation model to obtain the semantic segmentation prediction result of the training image.

[0106] S15: Input the shallow feature map, deep feature map, and image gradient map of the training image into the boundary refinement branch of the preset segmentation model to obtain the boundary segmentation prediction result of the training image.

[0107] S16: Calculate the segmentation loss function between the semantic segmentation prediction result of the training image and the pre-annotated semantic segmentation information.

[0108] In this embodiment, the cross-entropy loss function (CEloss) is used to calculate the error between the semantic segmentation prediction result and the pre-annotated semantic segmentation information, i.e., the segmentation loss function L between the two. ce .

[0109] S17: Calculate the boundary loss function between the boundary segmentation prediction result of the training image and the pre-labeled boundary segmentation information.

[0110] In this embodiment, the binary cross-entropy loss function (BCEloss) is used to calculate the error between the boundary segmentation prediction result of the training image and the pre-labeled boundary segmentation information, i.e., the boundary loss function Lbce.

[0111] When calculating BCEloss, a Sigmoid activation function needs to be added to reduce L. bce The predicted values ​​are mapped to the range (0,1).

[0112] Specifically, since the main loss during training comes from the segmentation loss function L ce Boundary loss function L bce It is only used as an auxiliary function, so the boundary loss function Lbce needs to be normalized.

[0113] S18: Calculate the total loss function of the preset segmentation model based on the segmentation loss function and the boundary loss function.

[0114] In this embodiment, the total loss function L total The calculation formula is as follows:

[0115] L total =Lce +σ×L bce

[0116] Where σ is the boundary loss function L bce The weighting coefficient σ has a value of (0,1).

[0117] S19: Use the total loss function to back-train the model parameters of the preset segmentation model to obtain the semantic segmentation network.

[0118] In this embodiment, the model parameters of the preset segmentation model are back-trained using the total loss function to calculate the gradient values ​​of all model parameters and back-update the parameter values ​​of the preset segmentation model to achieve the purpose of optimizing the model and thus obtaining an optimized semantic segmentation network.

[0119] Please see Figure 9 , Figure 9 This is a flowchart illustrating the first embodiment of the image semantic segmentation method of this application. In this embodiment, the image semantic segmentation method is implemented through the aforementioned semantic segmentation network, and the image semantic segmentation method includes:

[0120] S21: Obtain the target image and its gradient map.

[0121] In this embodiment, the target image is the image after preprocessing the image to be processed.

[0122] The images to be processed are either images captured by surveillance cameras and uploaded, or images uploaded by users.

[0123] Preprocessing refers to randomly cropping, randomly flipping horizontally or vertically, and randomly scaling (0.5 to 2 times) the image to be processed.

[0124] In this embodiment, the Canny edge detection algorithm is used to detect the target image in order to obtain the image gradient map of the target image.

[0125] Image gradients calculate the rate of change in an image, essentially calculating the difference between adjacent pixels in the current direction. Edges of an image exhibit larger grayscale changes and thus larger gradient values; conversely, smoother areas show smaller grayscale changes and correspondingly smaller gradient values. Generally, image gradients calculate edge information.

[0126] S22: Input the target image into the semantic segmentation network, and extract features from the target image through the encoder of the semantic segmentation network to obtain shallow feature maps and deep feature maps of the target image; wherein, the semantic segmentation network includes a cascaded encoder, a semantic-boundary dual-branch decoder and an output layer.

[0127] In this embodiment, the target image is first downsampled by 8 times by the downsampling module in the semantic segmentation network to obtain an image with a resolution of 1 / 8 of the target image, which is then used as the input image of the encoder in the semantic segmentation network.

[0128] Downsampling can be used to reduce the dimensionality of features while retaining effective information, and can avoid overfitting to some extent.

[0129] In this embodiment, the input image is input to the encoder of the semantic segmentation network, and the input image is extracted by multiple feature extraction units in the encoder to obtain the shallow feature map and deep feature map of the target image.

[0130] Among them, the deep feature map is the feature map after the shallow feature map has been downsampled and then extracted by the next set of feature extraction units.

[0131] In this embodiment, each feature extraction unit includes a different number of extremely efficient cross-stage non-bottleneck residual modules (EECSNB), which can greatly reduce the computational complexity and memory access volume of the model while effectively extracting features.

[0132] S23: Input the shallow feature map and the deep feature map into the semantic fusion branch of the semantic-boundary dual-branch decoder to perform feature fusion of semantic information and channel information in the shallow feature map and the deep feature map to obtain the semantic segmentation prediction result.

[0133] In this embodiment, the semantic fusion branch of the semantic-boundary dual-branch decoder of the semantic segmentation network obtains shallow and deep feature maps from different network layers at the encoder end. It then upsamples the deep feature maps and stacks the upsampled deep feature maps with the shallow feature maps to obtain the input image. The channel attention mechanism and the bidirectional spatial attention mechanism are used to complete the reselection and combination of features in the input image. Finally, a 1×1 convolution is used to obtain the semantic segmentation prediction result.

[0134] Understandably, the channel attention module in the semantic fusion branch can utilize the rich channel features after stacking to generate channel attention maps, and use these maps to encode the importance of each channel in the input image, thereby improving the accuracy of target classification. Simultaneously, the bidirectional spatial attention module in the semantic fusion branch, by extracting horizontal and vertical spatial attention maps from the stacked features, can utilize richer positional and channel information to encode the importance of each pixel in the input image, thereby improving the accuracy of target localization.

[0135] S24: Input the image gradient map, shallow feature map, and deep feature map into the boundary refinement branch of the semantic-boundary dual-branch decoder to obtain boundary information based on the image gradient map, shallow feature map, and deep feature map. Then, based on the boundary information, perform feature fusion on the spatial information of the shallow feature map and the deep feature map to obtain the boundary prediction result.

[0136] In this embodiment, the boundary refinement branch of the semantic-boundary dual-branch decoder of the semantic segmentation network obtains shallow and deep feature maps from different network layers at the encoder end. It then upsamples the deep feature maps and stacks the upsampled deep feature maps with the shallow feature maps to obtain the input image. The input image is then concatenated with the obtained image gradient map. A bidirectional spatial attention mechanism is used to reselect and combine the features in the concatenated image. Finally, a 1×1 convolution is used to obtain the boundary prediction result.

[0137] Understandably, the boundary refinement branch can utilize the image gradient maps of the input image and the target image to obtain the boundary information contained in different layers of the encoder, and fuse the boundary features contained in different layers.

[0138] S25: Input the semantic segmentation prediction result and the boundary prediction result into the output layer. The output layer fuses the semantic segmentation prediction result and the boundary prediction result to obtain the semantic segmentation image of the target image.

[0139] In this embodiment, the semantic segmentation prediction result and the boundary prediction result are input to the output layer. The output layer multiplies the semantic segmentation prediction result and the boundary prediction result pixel by pixel to obtain the boundary-weighted feature map. Then, the boundary-weighted feature map is added to the semantic segmentation prediction result pixel by pixel to obtain a segmentation image that incorporates the boundary features. Subsequently, the segmentation image that incorporates the boundary features is upsampled to restore its resolution to be consistent with the input image, thereby generating a semantic segmentation image and outputting it.

[0140] Understandably, this implementation can assist the semantic fusion branch in learning more accurate target boundary features by introducing a boundary refinement branch, thereby further enhancing the semantic segmentation network's ability to recognize target contours and the distinguishability of edges between targets.

[0141] Unlike existing technologies, this implementation method uses a semantic fusion branch in the semantic-boundary dual-branch decoder to fuse semantic and channel information from shallow and deep feature maps. This fully integrates semantic features from different layers, thereby enhancing the accuracy of the semantic segmentation network in target recognition and localization, and giving the semantic segmentation prediction results multi-scale semantic information. Furthermore, a boundary refinement branch obtains boundary information based on image gradient maps, shallow feature maps, and deep feature maps, and fuses spatial information from shallow and deep feature maps based on this boundary information. This fully learns and integrates boundary features from different layers, thereby enhancing the semantic segmentation network's ability to recognize target contours and improving the discrimination of edges between targets, thus increasing the accuracy of target boundary segmentation in the boundary prediction results. By fusing the semantic segmentation prediction results with the boundary prediction results, the object boundaries in the semantic segmentation prediction results can be further characterized based on a better fusion of image information, thereby improving the accuracy of semantic segmentation and ultimately enhancing the segmentation effect.

[0142] Please see Figure 10 , Figure 10 This is a flowchart illustrating a second embodiment of the image semantic segmentation method of this application. In this embodiment, the image semantic segmentation method is implemented through the aforementioned semantic segmentation network, and the image semantic segmentation method includes:

[0143] S301: Obtain the target image and the image gradient map of the target image.

[0144] Please refer to the description in S21 for the specific process, which will not be repeated here.

[0145] S302: Input the target image into the semantic segmentation network, and extract features from the target image through the encoder of the semantic segmentation network to obtain the shallow feature map and deep feature map of the target image.

[0146] Please refer to the description in S22 for the specific process, which will not be repeated here.

[0147] S303; Upsample the deep feature map through the semantic fusion branch, and stack the upsampled deep feature map with the shallow feature map to obtain a stacked feature map.

[0148] In this embodiment, after the semantic fusion branch of the semantic segmentation network obtains shallow and deep feature maps from different network layers of the encoder, it upsamples the deep feature maps to restore their resolution to that of the shallow feature maps. Figure 1 Then stack the two together.

[0149] In this embodiment, the number of channels in the stacked feature map is the sum of the number of channels in the shallow feature map and the deep feature map.

[0150] S304: Obtain the channel weights of each feature channel in the stacked feature map using the channel attention mechanism; and obtain the horizontal and vertical spatial weight matrices of the stacked feature map using the bidirectional spatial attention mechanism.

[0151] In this embodiment, obtaining the channel weight of each feature channel in the stacked feature map using the channel attention mechanism means that the channel attention module in the semantic fusion branch is used to model the dependency relationship between channels in the stacked feature map, so as to adaptively adjust the importance of each channel and give greater weight to important channels and less weight to secondary features. That is, starting from global information, valuable feature channels are selectively amplified and useless feature channels are suppressed.

[0152] Specifically, the channel attention module performs global pooling on the stacked feature map, aggregating the spatial information of each channel into the feature vector of the corresponding channel to output the feature vector corresponding to each feature channel in the stacked feature map. Further, a 1×1 convolution is performed on the feature vector, and then the convolutional feature map is normalized using the Sigmoid activation function and averaged to obtain the channel weight for each channel, keeping the value of each channel weight between (0,1).

[0153] In this embodiment, obtaining the horizontal and vertical spatial weight matrices of the stacked feature map using the bidirectional spatial attention mechanism means that the bidirectional spatial attention module in the semantic fusion branch captures the long-distance correlation between pixels in the stacked feature map from the horizontal and vertical directions respectively, so as to obtain bidirectional spatial attention maps with different weights in two different dimensions, namely the horizontal and vertical spatial weight matrices.

[0154] In this matrix, each parameter in the horizontal spatial weight matrix represents the horizontal weight of the corresponding pixel, and each parameter in the vertical spatial weight matrix represents the vertical weight of the corresponding pixel.

[0155] Specifically, the bidirectional spatial attention module performs 1×1 convolution on the stacked feature maps, and then performs horizontal pooling and vertical pooling on the features extracted by the convolution to obtain horizontal and vertical matrices. The horizontal and vertical matrices are then normalized by a Sigmoid activation function and averaged to obtain the horizontal spatial attention map and the vertical spatial attention map.

[0156] S305: Decompose the stacked feature map into a convolution to obtain a fused feature map, and use channel weights to weight the feature channels in the fused feature map to obtain a channel-weighted feature map.

[0157] In this embodiment, the stacked feature map is input into 3×1 and 1×3 decomposition convolution units to perform asymmetric convolution on the stacked feature map to obtain a fused feature map.

[0158] Understandably, compared to 3×3 convolution, decomposition can greatly reduce the number of parameters and computational cost.

[0159] Furthermore, each feature channel and its corresponding feature weight in the fused feature map are multiplied to weight each feature channel, thus obtaining a channel-weighted feature map.

[0160] S306: The pixels in the channel weighted feature map are horizontally and vertically weighted using the horizontal and vertical spatial weight matrices of the stacked feature map, respectively, to obtain the first horizontally weighted feature map and the first vertically weighted feature map.

[0161] In this embodiment, the horizontal spatial attention map of the stacked feature map is multiplied pixel by pixel with the channel weighted feature map to obtain the first horizontal weighted feature map, and the vertical spatial attention map of the stacked feature map is multiplied pixel by pixel with the channel weighted feature map to obtain the first vertical weighted feature map.

[0162] S307: Add the first horizontal weighted feature map to the first vertical weighted feature map to obtain the first bidirectional spatial weighted feature map, and add the first bidirectional spatial weighted feature map to the fused feature map pixel by pixel to obtain the first output map.

[0163] S308: Use pointwise convolution to map the segmentation categories to the first output image to obtain a semantic segmentation prediction result with the number of channels equal to the number of segmentation categories.

[0164] In this embodiment, the segmentation category refers to the semantic category of each pixel in the first output image, that is, the object type to which each pixel belongs.

[0165] S309: Use the boundary refinement branch to upsample the deep feature map, and stack the upsampled deep feature map with the shallow feature map to obtain the stacked feature map.

[0166] In this embodiment, the boundary refinement branch at the decoder end of the semantic segmentation network obtains shallow and deep feature maps from different network layers at the encoder end, and then upsamples the deep feature maps to restore their resolution to that of the shallow feature maps. Figure 1 Then stack the two together.

[0167] S310: Concatenate the image gradient map with the stacked feature map to obtain a concatenated feature map.

[0168] In this embodiment, the image gradient map is concatenated with the stacked feature map in order to better characterize the boundary information of pixels using the image gradient map.

[0169] S311: Perform pointwise convolution on the cascaded feature map to extract the boundary information of each pixel in the cascaded feature map, and use a bidirectional spatial attention mechanism to obtain the horizontal and vertical spatial weight matrices of the cascaded feature map after pointwise convolution.

[0170] In this embodiment, obtaining the horizontal and vertical spatial weight matrices of the cascaded feature map after point-by-point convolution using the bidirectional spatial attention mechanism means that the bidirectional spatial attention module in the boundary refinement branch captures the long-distance correlation between pixels in the cascaded feature map after point-by-point convolution from the horizontal and vertical directions respectively, so as to obtain bidirectional spatial attention maps with different weights in two different dimensions.

[0171] The horizontal and vertical weight matrices of the cascaded feature maps after pointwise convolution contain rich boundary information, which can be used to enhance boundary-related features in high-level feature maps and suppress non-boundary-related features, thereby making the boundaries of objects clearer.

[0172] S312: Using the horizontal and vertical weight matrices of the concatenated feature maps after pointwise convolution, the pixels in the fused feature map are horizontally and vertically weighted respectively to obtain the second horizontally weighted feature map and the second vertically weighted feature map.

[0173] In this embodiment, the horizontal spatial weight matrix of the concatenated feature map after pointwise convolution is multiplied pixel by pixel with the fused feature map to obtain the second horizontally weighted feature map, and the vertical spatial weight matrix of the concatenated feature map after pointwise convolution is multiplied pixel by pixel with the fused feature map to obtain the second vertically weighted feature map.

[0174] S313: Add the second horizontal weighted feature map to the second vertical weighted feature map to obtain the second bidirectional spatial weighted feature map, and add the second bidirectional spatial weighted feature map to the fused feature map pixel by pixel to obtain the second output map.

[0175] S314: Use pointwise convolution to perform boundary class mapping on the second output map to obtain the boundary prediction result where the number of channels is equal to the number of boundary classes.

[0176] In this embodiment, the boundary category refers to the object boundary category to which each pixel in the second output image belongs.

[0177] S315: Input the semantic segmentation prediction result and the boundary prediction result into the output layer. The output layer fuses the semantic segmentation prediction result and the boundary prediction result to obtain the semantic segmentation image of the target image.

[0178] In this embodiment, the semantic segmentation prediction result and the boundary prediction result are input to the output layer. The output layer multiplies the semantic segmentation prediction result and the boundary prediction result pixel by pixel to obtain a boundary-weighted feature map. Then, the boundary-weighted feature map is added to the semantic segmentation prediction result pixel by pixel to obtain a segmentation image that incorporates boundary features. Subsequently, the segmentation image that incorporates boundary features is upsampled to restore its resolution to be consistent with the input image, thereby generating a semantic segmentation image and outputting it.

[0179] Understandably, this implementation can assist the semantic fusion branch in learning more accurate target boundary features by introducing a boundary refinement branch, thereby further enhancing the semantic segmentation network's ability to recognize target contours and the distinguishability of edges between targets.

[0180] Unlike existing technologies, the semantic fusion branch provided in this embodiment includes a channel attention module and a bidirectional spatial attention module. It can generate a channel attention map using the rich channel features after stacking, and encode the importance of each channel in the stacked feature map, thereby improving the accuracy of target classification. Simultaneously, by extracting horizontal and vertical spatial attention maps from the stacked feature map, it can encode the importance of each pixel in the stacked feature map using richer positional and channel information, thereby improving the accuracy of target localization. Furthermore, the boundary refinement branch provided in this embodiment includes a bidirectional spatial attention module, which can obtain boundary information contained in different layers using the stacked feature map and the image gradient map of the target image, and fuse the boundary features contained in different layers. The boundary prediction result output by the boundary refinement branch is fused with the semantic segmentation prediction result output by the semantic fusion branch. This allows for better fusion of image information and further characterization of object boundaries in the semantic segmentation prediction result, thereby improving the accuracy of semantic segmentation and ultimately enhancing the segmentation effect.

[0181] Please see Figure 11 , Figure 11 This is a flowchart illustrating the third embodiment of the image semantic segmentation method of this application. In this embodiment, the image semantic segmentation method is implemented through the aforementioned semantic segmentation network, and is specifically used to obtain semantic segmentation prediction results.

[0182] This image semantic segmentation method includes:

[0183] S401: Obtain the target image and its gradient map.

[0184] Please refer to the description in S21 for the specific process, which will not be repeated here.

[0185] S402: Input the target image into the semantic segmentation network, downsample the target image and use it as the input image of the encoder, use the encoder to downsample the input image by a preset factor and extract features to obtain the first feature map.

[0186] In this embodiment, the target image is first downsampled eight times by the downsampling module in the semantic segmentation network to obtain an image with a resolution of 1 / 8 of the target image, which is then used as the input image of the encoder in the semantic segmentation network.

[0187] In this embodiment, the preset multiple is two times. In other embodiments, the preset multiple may be other multiples, and this application does not limit this.

[0188] Specifically, the input image is fed into the encoder of the semantic segmentation network, and the input image is downsampled by a factor of two, so that the resolution of the downsampled image becomes half that of the input image. Then, the three extremely efficient cross-stage non-bottleneck residual modules EECSNB in ​​the first set of feature extraction units are used to extract features from the downsampled image to obtain the first feature map.

[0189] The first feature map has 16 channels and a resolution that is half that of the input image.

[0190] In the first set of feature extraction units, the dilated coefficient r in all three EECSNBs is 0. Understandably, the first set of feature extraction units is used to comprehensively extract shallow features and does not require a large receptive field, thus eliminating the need for dilated convolutions.

[0191] S403: Perform downsampling and feature extraction on the first feature map by a preset multiple to obtain the second feature map; wherein, both the first feature map and the second feature map are shallow feature maps.

[0192] In this embodiment, after downsampling the first feature map by a factor of two, the two extremely efficient cross-stage non-bottleneck residual modules EECSNB in ​​the second set of feature extraction units are used to extract features from the downsampled image to obtain the second feature map.

[0193] The second feature map has 64 channels and a resolution that is half that of the first feature map.

[0194] In the first set of feature extraction units, the dilated coefficients r in both EECSNBs are 0. Understandably, the second set of feature extraction units is also used to extract shallow features and does not require a large receptive field, thus eliminating the need for dilated convolutions.

[0195] S404: Perform downsampling and feature extraction on the second feature map by a preset multiple to obtain the third feature map; wherein, the third feature map is a deep feature map.

[0196] In this embodiment, after downsampling the second feature map by a factor of two, the eight extremely efficient cross-stage non-bottleneck residual modules EECSNB in ​​the third group of feature extraction units are used to extract features from the downsampled image to obtain the third feature map.

[0197] The third feature map has 128 channels and half the resolution of the second feature map.

[0198] Among them, the aperture coefficients r in the eight EECSNBs of the third feature extraction unit are 1, 2, 5, 9, 2, 5, 9, and 17, respectively.

[0199] In other embodiments, the aperture coefficient r of the eight EECSNBs in the third group of feature extraction units can also be other integers greater than 0, which is not limited in this application.

[0200] Understandably, the third set of feature extraction units is used to extract deep features, which requires a larger receptive field to extract more semantic features, thus requiring dilated convolution.

[0201] S405: Upsample the third feature map by a preset multiple using the semantic fusion branch, and stack the upsampled third feature map with the second feature map to obtain the fourth feature map.

[0202] In this embodiment, the first fusion unit (ABGAB) in the semantic fusion branch of the decoder obtains the third feature map from the output of the third set of feature extraction units of the encoder and the second feature map from the output of the second set of feature extraction units of the encoder. Then, it upsamples the third feature map by a factor of two to restore its resolution to that of the second feature map. Figure 1 Then, stack the two together to obtain the fourth feature map.

[0203] The fourth feature map has 192 channels.

[0204] S406: Obtain the channel weights of each feature channel in the fourth feature map using the channel attention mechanism; and obtain the horizontal and vertical spatial weight matrices of the fourth feature map using the bidirectional spatial attention mechanism.

[0205] For details, please refer to the description in S305, which will not be repeated here.

[0206] S407: Perform decomposition and convolution on the fourth feature map to obtain the fifth feature map.

[0207] In this embodiment, the fourth feature map is input into 3×1 and 1×3 decomposition convolution units to perform asymmetric convolution on the fourth feature map to obtain the fifth feature map.

[0208] The fifth feature map has 64 channels.

[0209] Specifically, since the fourth feature map is obtained by simply stacking the upsampled third feature map and the second feature map, when convolving the fourth feature map, the additional channels in the upsampled third feature map that are greater than those in the second feature map need to be fused to reduce the number of channels to the same level as the second feature map. Figure 1 Then, the feature channels in the upsampled third feature map are connected to the feature channels in the second feature map to fuse the deep features into the shallow features.

[0210] S408: Use channel weights to perform channel weighting on the feature channels in the fifth feature map to obtain the channel-weighted fifth feature map.

[0211] In this embodiment, each feature channel in the fifth feature map is multiplied by its corresponding feature weight to weight each feature channel, thereby obtaining the channel-weighted fifth feature map.

[0212] S409: Use the horizontal spatial weight matrix of the fourth feature map to perform horizontal weighting on the pixels in the channel-weighted fifth feature map to obtain the horizontally weighted fifth feature map; and use the vertical spatial weight matrix of the fourth feature map to perform vertical weighting on the pixels in the channel-weighted fifth feature map to obtain the vertically weighted fifth feature map.

[0213] In this embodiment, the horizontal spatial weight matrix of the fourth feature map is multiplied pixel-by-pixel with the channel-weighted fifth feature map to obtain the horizontally weighted fifth feature map. Similarly, the vertical spatial weight matrix of the fourth feature map is multiplied pixel-by-pixel with the channel-weighted fifth feature map to obtain the vertically weighted fifth feature map.

[0214] S410: Add the horizontally weighted fifth feature map to the vertically weighted fifth feature map, and then add the latter to the fifth feature map to obtain the sixth feature map; wherein the resolution and number of channels of the sixth feature map are the same as those of the second feature map.

[0215] In this embodiment, the horizontally weighted fifth feature map is added to the vertically weighted fifth feature map to obtain a channel-weighted and bidirectional spatially weighted feature map. Then, the channel-weighted and bidirectional spatially weighted feature map is added to the fifth feature map pixel by pixel to obtain the sixth feature map.

[0216] The sixth feature map has 64 channels and a resolution that is half that of the first feature map.

[0217] S411: Upsample the sixth feature map by a preset factor, stack the upsampled sixth feature map with the first feature map, and repeat the above channel weighting and spatial weighting steps to obtain the seventh feature map; wherein the resolution and number of channels of the seventh feature map are the same as those of the first feature map.

[0218] In this embodiment, the second first fusion unit (ABGAB) in the semantic fusion branch obtains the first feature map from the output of the first set of feature extraction units of the encoder, and performs a 2x upsampling on the sixth feature map output by the first first fusion unit to restore the resolution of the sixth feature map to the same level as the first feature map. Figure 1 The upsampled sixth feature map is then stacked with the first feature map.

[0219] Furthermore, the channel weights and bidirectional spatial weights of the stacked image are obtained by using the channel attention module and bidirectional spatial attention module in the semantic fusion branch. The stacked image is decomposed and convolved, and the channel weights and bidirectional spatial weights are used to perform channel weighting and bidirectional spatial weighting on the feature image after decomposition and convolution. Then, the weighted feature map and the feature image after decomposition and convolution are added pixel by pixel to obtain the seventh feature map.

[0220] The seventh feature map has 16 channels and the same resolution as the first feature map. Figure 1 To.

[0221] S412: Use pointwise convolution to map the segmentation categories to the seventh feature map to obtain a semantic segmentation prediction result with the number of channels equal to the number of segmentation categories.

[0222] S413: Input the image gradient map, shallow feature map, and deep feature map into the boundary refinement branch of the semantic-boundary dual-branch decoder to obtain boundary information based on the image gradient map, shallow feature map, and deep feature map. Then, based on the boundary information, perform feature fusion on the spatial information of the shallow feature map and the deep feature map to obtain the boundary prediction result.

[0223] For details, please refer to the descriptions in S24 and S309 to S314, which will not be repeated here.

[0224] S414: Input the semantic segmentation prediction result and the boundary prediction result into the output layer. The output layer fuses the semantic segmentation prediction result and the boundary prediction result to obtain the semantic segmentation image of the target image.

[0225] For details, please refer to the descriptions in S25 and S315, which will not be repeated here.

[0226] Understandably, this implementation method, by fusing the features of each feature extraction stage in the encoder through the two first fusion units in the semantic fusion branch, can make full use of the multi-angle semantic information in the deep and shallow features, which helps the decoder to identify the target category, thereby improving the overall network's ability to identify the target and thus improving the final inference accuracy.

[0227] Understandably, introducing a boundary refinement branch can help the semantic fusion branch learn more accurate target boundary features, thereby further characterizing the object boundaries in the semantic segmentation prediction results, thus improving the accuracy of semantic segmentation and consequently enhancing the segmentation effect.

[0228] Please see Figure 12 , Figure 12 This is a flowchart illustrating the fourth embodiment of the image semantic segmentation method of this application. In this embodiment, the image semantic segmentation method is implemented through the aforementioned semantic segmentation network, and is specifically used to obtain boundary prediction results.

[0229] This image semantic segmentation method includes:

[0230] S501: Obtain the target image and the image gradient map of the target image.

[0231] Please refer to the description in S21 for the specific process, which will not be repeated here.

[0232] S502: Input the target image into the semantic segmentation network, downsample the target image and use it as the input image of the encoder, use the encoder to downsample the input image by a preset factor and extract features to obtain the first feature map.

[0233] Please refer to the description in S402 for the specific process, which will not be repeated here.

[0234] S503: The first feature map is downsampled by a preset factor and its features are extracted to obtain the second feature map; wherein, both the first feature map and the second feature map are shallow feature maps.

[0235] For details, please refer to the description in S403, which will not be repeated here.

[0236] S504: Perform downsampling and feature extraction on the second feature map by a preset multiple to obtain the third feature map; wherein, the third feature map is a deep feature map.

[0237] For details, please refer to the description in S404; it will not be repeated here.

[0238] S505: Input the shallow feature map and the deep feature map into the semantic fusion branch of the semantic-boundary dual-branch decoder to perform feature fusion of semantic information and channel information in the shallow feature map and the deep feature map to obtain the semantic segmentation prediction result.

[0239] For details, please refer to the descriptions in S405 to S412, which will not be repeated here.

[0240] S506: Upsample the third feature map by a preset multiple using the boundary refinement branch, and stack the upsampled third feature map with the second feature map to obtain the fourth feature map.

[0241] In this embodiment, the preset multiple is two times. In other embodiments, the preset multiple may be other multiples, and this application does not limit this.

[0242] In this embodiment, the first second fusion unit (ESAAB) in the boundary refinement branch of the decoder obtains the third feature map from the output of the third set of feature extraction units of the encoder, and the second feature map from the output of the second set of feature extraction units of the encoder. Then, it upsamples the third feature map by a factor of two to restore its resolution to that of the second feature map. Figure 1 Then, stack the two together to obtain the fourth feature map.

[0243] The fourth feature map has 192 channels.

[0244] S507: Concatenate the image gradient map with the fourth feature map to obtain a concatenated feature map.

[0245] In this embodiment, the image gradient map is concatenated with the fourth feature map in order to better characterize the boundary information of pixels using the image gradient map.

[0246] S508: Perform pointwise convolution on the cascaded feature map to extract the boundary information of each pixel in the cascaded feature map, and use a bidirectional spatial attention mechanism to obtain the horizontal and vertical spatial weight matrices of the cascaded feature map after pointwise convolution.

[0247] For details, please refer to the description in S311, which will not be repeated here.

[0248] S509: The pixels in the fifth feature map are horizontally weighted using the horizontal spatial weight matrix of the concatenated feature map after pointwise convolution, resulting in a fifth feature map with redefined horizontal boundaries; and the pixels in the fifth feature map are vertically weighted using the vertical spatial weight matrix of the concatenated feature map after pointwise convolution, resulting in a fifth feature map with redefined vertical boundaries.

[0249] In this embodiment, the horizontal spatial weight matrix of the concatenated feature map after pointwise convolution is multiplied pixel by pixel with the fifth feature map to obtain the fifth feature map after the horizontal boundary is redefined, and the vertical spatial weight matrix of the concatenated feature map after pointwise convolution is multiplied pixel by pixel with the fifth feature map to obtain the fifth feature map after the vertical boundary is redefined.

[0250] S510: Add the fifth feature map after redefining the horizontal boundary to the fifth feature map after redefining the vertical boundary, and then add it to the fifth feature map to obtain the eighth feature map; wherein the resolution and number of channels of the eighth feature map are the same as those of the second feature map.

[0251] S511: Upsample the eighth feature map by a preset factor, stack the upsampled eighth feature map with the first feature map, and repeat the above steps of extracting boundary information and spatial weighting to obtain the ninth feature map; wherein the resolution and number of channels of the ninth feature map are the same as those of the first feature map.

[0252] In this embodiment, the second second fusion unit (ESAAB) in the boundary refinement branch obtains the first feature map from the output of the first set of feature extraction units of the encoder, and performs a 2x upsampling on the eighth feature map output by the first second fusion unit to restore the resolution of the eighth feature map to that of the first feature map. Figure 1 The upsampled eighth feature map is then stacked with the first feature map.

[0253] Furthermore, the bidirectional spatial attention module in the boundary refinement branch is used to obtain the bidirectional spatial weight matrix of the stacked image, and the stacked image is decomposed and convolved. The bidirectional spatial weight is used to perform bidirectional spatial weighting on the feature image after decomposition and convolution. Then, the weighted feature image is added pixel by pixel to the feature image after decomposition and convolution to obtain the ninth feature map.

[0254] S512: Use pointwise convolution to perform boundary category mapping on the ninth feature map to obtain the boundary prediction result with the number of channels equal to the number of boundary categories.

[0255] S513: The semantic segmentation prediction result and the boundary prediction result are multiplied pixel by pixel through the output layer to obtain the boundary-weighted feature map. Then, the boundary-weighted feature map is added pixel by pixel to the semantic segmentation prediction result to obtain the tenth feature map.

[0256] S514: The tenth feature map is upsampled by a preset multiple using bilinear interpolation to restore the resolution of the upsampled tenth feature map to be consistent with the resolution of the input image, thus obtaining a semantic segmentation image.

[0257] Understandably, this implementation method, by fusing the boundary features of each feature extraction stage in the encoder through two second fusion units in the boundary fusion branch, can make full use of the spatial information in the deep and shallow features, which helps the decoder recover the lost details, thereby enhancing the overall network's ability to recognize the target contour and the distinguishability of the edges between targets, and thus improving the final segmentation accuracy.

[0258] Understandably, introducing a boundary refinement branch can help the semantic fusion branch learn more accurate target boundary features, thereby further characterizing the object boundaries in the semantic segmentation prediction results, thus improving the accuracy of semantic segmentation and consequently enhancing the segmentation effect.

[0259] Correspondingly, this application provides an image semantic segmentation apparatus.

[0260] Please see Figure 13 , Figure 13 This is a schematic diagram of one embodiment of the image semantic segmentation apparatus of this application. In this embodiment, the image semantic segmentation apparatus 60 includes an acquisition module 61, a feature extraction module 62, a semantic fusion module 63, a boundary refinement module 64, and a semantic segmentation image acquisition module 65.

[0261] The acquisition module 61 is used to acquire the target image and the image gradient map of the target image.

[0262] The feature extraction module 62 is used to input the target image into the semantic segmentation network, and extract features from the target image through the encoder of the semantic segmentation network to obtain shallow feature maps and deep feature maps of the target image; wherein, the semantic segmentation network includes a cascaded encoder, a semantic-boundary dual-branch decoder and an output layer.

[0263] The semantic fusion module 63 is used to input the shallow feature map and the deep feature map into the semantic fusion branch of the semantic-boundary dual-branch decoder, so as to perform feature fusion on the semantic information and channel information in the shallow feature map and the deep feature map to obtain the semantic segmentation prediction result.

[0264] The boundary refinement module 64 is used to input the image gradient map, shallow feature map and deep feature map into the boundary refinement branch of the semantic-boundary dual-branch decoder to obtain boundary information based on the image gradient map, shallow feature map and deep feature map, and to perform feature fusion on the spatial information of shallow feature map and deep feature map based on the boundary information to obtain the boundary prediction result.

[0265] The semantic segmentation image acquisition module 65 is used to input the semantic segmentation prediction result and the boundary prediction result into the output layer, and then fuse the semantic segmentation prediction result and the boundary prediction result through the output layer to obtain the semantic segmentation image of the target image.

[0266] For details of the process, please refer to the relevant textual descriptions in S21~S25, S301~S315, S401~S414, and S501~S514, which will not be repeated here.

[0267] Unlike existing technologies, this application uses a semantic fusion module 63 to fuse semantic and channel information from shallow and deep feature maps. This fully integrates semantic features from different layers, thereby enhancing the accuracy of the semantic segmentation network in target recognition and localization, and giving the semantic segmentation prediction results multi-scale semantic information. Furthermore, a boundary refinement module 64 obtains boundary information based on image gradient maps, shallow feature maps, and deep feature maps, and fuses spatial information from shallow and deep feature maps based on this boundary information. This fully learns and integrates boundary features from different layers, thereby enhancing the semantic segmentation network's ability to recognize target contours and improving the discrimination of edges between targets, thus increasing the accuracy of target boundary segmentation in the boundary prediction results. Finally, a semantic segmentation image acquisition module 65 fuses the semantic segmentation prediction results with the boundary prediction results. This further refines the object boundaries in the semantic segmentation prediction results based on better image information fusion, thereby improving the accuracy of semantic segmentation and ultimately enhancing the segmentation effect.

[0268] Correspondingly, this application provides an electronic device.

[0269] Please see Figure 14 , Figure 14 This is a schematic diagram of one embodiment of the electronic device of this application. For example... Figure 14 As shown, in this embodiment, the electronic device 70 includes a memory 71 and a processor 72.

[0270] In this embodiment, the memory 71 is used to store program data, and when the program data is executed, it implements the steps in the image semantic segmentation method described above; the processor 72 is used to execute the program instructions stored in the memory 71 to implement the steps in the image semantic segmentation method described above.

[0271] Specifically, processor 72 controls itself and memory 71 to implement the steps in the image semantic segmentation method described above. Processor 72 can also be referred to as a CPU (Central Processing Unit). Processor 72 may be an integrated circuit chip with signal processing capabilities. Processor 72 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 72 can be implemented using multiple integrated circuit chips.

[0272] Unlike existing technologies, this implementation uses processor 72 to fuse semantic and channel information from shallow and deep feature maps. This fully integrates semantic features from different layers, thereby enhancing the accuracy of the semantic segmentation network in target recognition and localization, and giving the semantic segmentation prediction results multi-scale semantic information. Furthermore, boundary information is obtained based on image gradient maps, shallow feature maps, and deep feature maps. Based on this boundary information, spatial information from the shallow and deep feature maps is fused. This fully learns and integrates boundary features from different layers, thereby enhancing the semantic segmentation network's ability to recognize target contours and improving the discrimination of edges between targets, thus increasing the accuracy of target boundary segmentation in the boundary prediction results. By fusing the semantic segmentation prediction results with the boundary prediction results, the object boundaries in the semantic segmentation prediction results can be further characterized based on a better fusion of image information, thereby improving the accuracy of semantic segmentation and ultimately enhancing the segmentation effect.

[0273] Correspondingly, this application provides a computer-readable storage medium.

[0274] Please see Figure 15 , Figure 15 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application.

[0275] The computer-readable storage medium 80 includes a computer program 801 stored on it. When executed by the aforementioned processor, the computer program 801 implements the steps in the image semantic segmentation method described above. Specifically, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 80. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium 80 and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 80 includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0276] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0277] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0278] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0279] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0280] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0281] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. An image semantic segmentation method, characterized in that, include: Obtain the target image and its image gradient map; The target image is input into the semantic segmentation network, and the encoder of the semantic segmentation network extracts features from the target image to obtain shallow and deep feature maps. The semantic segmentation network includes a cascaded encoder, a semantic-boundary dual-branch decoder, and an output layer. The encoder includes three sets of feature extraction units, each set including at least one highly efficient cross-stage non-bottleneck residual module. This module performs channel splitting on the input image, dividing the feature channels into two parts and feeding them into the left and right branches respectively. The features processed by the left and right branches are then cascaded and added, and information exchange between the two branches is achieved through channel rearrangement. The left branch includes a set of asymmetric convolutional units for extracting shallow features; the right branch includes a decomposition residual module and another set of asymmetric convolutional units, the decomposition residual module being used to extract deep features. The shallow feature map and the deep feature map are input into the semantic fusion branch of the semantic-boundary dual-branch decoder to perform feature fusion on the semantic information and channel information in the shallow feature map and the deep feature map to obtain a semantic segmentation prediction result. This includes: upsampling the deep feature map through the semantic fusion branch, and stacking the upsampled deep feature map with the shallow feature map to obtain a stacked feature map. The channel weights of each feature channel in the stacked feature map are obtained using a channel attention mechanism; and the horizontal and vertical spatial weight matrices of the stacked feature map are obtained using a bidirectional spatial attention mechanism. The stacked feature map is decomposed and convolved to obtain a fused feature map, and the feature channels in the fused feature map are weighted using the channel weights to obtain a channel-weighted feature map. The pixels in the channel weighted feature map are weighted horizontally and vertically using the horizontal and vertical spatial weight matrices of the stacked feature map, respectively, to obtain a first horizontally weighted feature map and a first vertically weighted feature map. The first horizontally weighted feature map is added to the first vertically weighted feature map to obtain the first bidirectional spatially weighted feature map, and the first bidirectional spatially weighted feature map is added to the fused feature map pixel by pixel to obtain the first output map; By performing segmentation category mapping on the first output image using pointwise convolution, the semantic segmentation prediction result with the number of channels equal to the number of segmentation categories is obtained; and... The image gradient map, the shallow feature map, and the deep feature map are input into the boundary refinement branch of the semantic-boundary dual-branch decoder to obtain boundary information based on the image gradient map, the shallow feature map, and the deep feature map. Based on the boundary information, the spatial information of the shallow feature map and the deep feature map is fused to obtain the boundary prediction result. The semantic segmentation prediction result and the boundary prediction result are input into the output layer, and the semantic segmentation prediction result and the boundary prediction result are fused by the output layer to obtain the semantic segmentation image of the target image.

2. The image semantic segmentation method according to claim 1, characterized in that, The step of obtaining the channel weights of each feature channel in the stacked feature map using the channel attention mechanism includes: Perform global pooling on the stacked feature map to output a corresponding feature vector for each feature channel in the stacked feature map; After performing pointwise convolution on each feature vector, the convolutional feature vector is mapped using an activation function to obtain the channel weights of each feature channel. The step of obtaining the horizontal and vertical spatial weight matrices of the stacked feature map using a bidirectional spatial attention mechanism includes: After performing pointwise convolution on the stacked feature maps, horizontal pooling and vertical pooling are performed on the feature maps extracted by convolution to obtain horizontal and vertical matrices. The horizontal matrix and the vertical matrix are mapped using the activation function to obtain the horizontal spatial weight matrix and the vertical spatial weight matrix of the stacked feature map.

3. The image semantic segmentation method according to claim 2, characterized in that, The step of inputting the image gradient map, the shallow feature map, and the deep feature map into the boundary refinement branch of the semantic-boundary dual-branch decoder to obtain boundary information based on the image gradient map, the shallow feature map, and the deep feature map, and then performing feature fusion on the spatial information of the shallow feature map and the deep feature map based on the boundary information to obtain the boundary prediction result includes: The deep feature map is upsampled using the boundary refinement branch, and the upsampled deep feature map is stacked with the shallow feature map to obtain the stacked feature map; The image gradient map is concatenated with the stacked feature map to obtain a concatenated feature map. The concatenated feature map is convolved pointwise to extract the boundary information of each pixel in the concatenated feature map, and the horizontal and vertical spatial weight matrices of the concatenated feature map after pointwise convolution are obtained by using the bidirectional spatial attention mechanism. The pixels in the fused feature map are weighted horizontally and vertically using the horizontal and vertical weight matrices of the concatenated feature map after pointwise convolution, respectively, to obtain a second horizontally weighted feature map and a second vertically weighted feature map. The second horizontally weighted feature map is added to the second vertically weighted feature map to obtain the second bidirectional spatially weighted feature map. The second bidirectional spatially weighted feature map is then added pixel by pixel to the fused feature map to obtain the second output map. By performing boundary category mapping on the second output map using pointwise convolution, the boundary prediction result is obtained with the number of channels equal to the number of boundary categories.

4. The image semantic segmentation method according to claim 1, characterized in that, The step of inputting the target image into the semantic segmentation network and extracting features from the target image through the encoder to obtain shallow feature maps and deep feature maps of the target image includes: The target image is input into the semantic segmentation network. The target image is downsampled and used as the input image of the encoder. The encoder is used to downsample the input image by a preset factor and extract features to obtain a first feature map. The first feature map is downsampled by the preset factor and its features are extracted to obtain a second feature map; wherein, both the first feature map and the second feature map are shallow feature maps; The second feature map is downsampled by the preset factor and its features are extracted to obtain a third feature map; wherein the third feature map is the deep feature map.

5. The image semantic segmentation method according to claim 4, characterized in that, The step of inputting the shallow feature map and the deep feature map into the semantic fusion branch of the semantic-boundary dual-branch decoder to perform feature fusion of the semantic information in the shallow feature map and the deep feature map with the output channel information to obtain the semantic segmentation prediction result includes: The semantic fusion branch is used to upsample the third feature map by the preset multiple, and the upsampled third feature map is stacked with the second feature map to obtain the fourth feature map; The channel weights of each feature channel in the fourth feature map are obtained using a channel attention mechanism; and the horizontal and vertical spatial weight matrices of the fourth feature map are obtained using a bidirectional spatial attention mechanism. The fourth feature map is subjected to the decomposition convolution to obtain the fifth feature map; The feature channels in the fifth feature map are weighted using the channel weights to obtain the channel-weighted fifth feature map. The pixels in the channel-weighted fifth feature map are horizontally weighted using the horizontal spatial weight matrix of the fourth feature map to obtain the horizontally weighted fifth feature map; and the pixels in the channel-weighted fifth feature map are vertically weighted using the vertical spatial weight matrix of the fourth feature map to obtain the vertically weighted fifth feature map. The horizontally weighted fifth feature map is added to the vertically weighted fifth feature map, and then added to the fifth feature map again to obtain the sixth feature map; wherein the resolution and number of channels of the sixth feature map are the same as those of the second feature map; The sixth feature map is upsampled by the preset multiple, and the upsampled sixth feature map is stacked with the first feature map. The above channel weighting and spatial weighting steps are repeated to obtain the seventh feature map. The resolution and number of channels of the seventh feature map are the same as those of the first feature map. The seventh feature map is mapped to segmentation categories using pointwise convolution to obtain the semantic segmentation prediction result with the number of channels equal to the number of segmentation categories.

6. The image semantic segmentation method according to claim 5, characterized in that, The step of inputting the image gradient map, the shallow feature map, and the deep feature map into the boundary refinement branch of the semantic-boundary dual-branch decoder to obtain boundary information based on the image gradient map, the shallow feature map, and the deep feature map, and then performing feature fusion on the spatial information of the shallow feature map and the deep feature map based on the boundary information to obtain the boundary prediction result includes: The third feature map is upsampled by the preset multiple using the boundary refinement branch, and the upsampled third feature map is stacked with the second feature map to obtain the fourth feature map; The image gradient map is concatenated with the fourth feature map to obtain a concatenated feature map. The concatenated feature map is convolved pointwise to extract the boundary information of each pixel in the concatenated feature map, and the horizontal and vertical spatial weight matrices of the concatenated feature map after pointwise convolution are obtained by using the bidirectional spatial attention mechanism. The pixels in the fifth feature map are horizontally weighted using the horizontal spatial weight matrix of the concatenated feature map after pointwise convolution, resulting in a fifth feature map with redefined horizontal boundaries; and the pixels in the fifth feature map are vertically weighted using the vertical spatial weight matrix of the concatenated feature map after pointwise convolution, resulting in a fifth feature map with redefined vertical boundaries. The fifth feature map after the horizontal boundary is redefined is added to the fifth feature map after the vertical boundary is redefined, and then added to the fifth feature map to obtain the eighth feature map; wherein the resolution and number of channels of the eighth feature map are the same as those of the second feature map; The eighth feature map is upsampled by the preset multiple, and the upsampled eighth feature map is stacked with the first feature map. The steps of extracting the boundary information and spatial weighting are repeated to obtain the ninth feature map. The resolution and number of channels of the ninth feature map are the same as those of the first feature map. By performing boundary category mapping on the ninth feature map using pointwise convolution, the boundary prediction result is obtained with the number of channels equal to the number of boundary categories.

7. The image semantic segmentation method according to claim 6, characterized in that, The step of inputting the semantic segmentation prediction result and the boundary prediction result into the output layer, and fusing the semantic segmentation prediction result and the boundary prediction result through the output layer to obtain the semantic segmentation image of the target image includes: The semantic segmentation prediction result and the boundary prediction result are multiplied pixel by pixel through the output layer to obtain a boundary-weighted feature map. The boundary-weighted feature map is then added pixel by pixel to the semantic segmentation prediction result to obtain the tenth feature map. The tenth feature map is upsampled by the preset multiple using bilinear interpolation to restore the resolution of the upsampled tenth feature map to be consistent with the resolution of the input image, thereby obtaining the semantic segmentation image.

8. The image semantic segmentation method according to claim 1, characterized in that, The semantic fusion branch in the semantic-boundary dual-branch decoder includes a first fusion unit and a pointwise convolution unit; wherein, the first fusion unit includes a channel attention module, a bidirectional spatial attention module, and a decomposition convolution unit; The boundary refinement branch in the semantic-boundary dual-branch decoder includes a second fusion unit and a pointwise convolution unit; wherein, the second fusion unit includes a bidirectional spatial attention module and a decomposition convolution unit.

9. The image semantic segmentation method according to claim 8, characterized in that, The semantic segmentation network is trained using labeled images, and the training method includes: Obtain a training dataset; the training dataset includes multiple training images, pre-annotated semantic segmentation information, and boundary segmentation information generated based on the semantic segmentation information; Based on the pre-annotated semantic segmentation information, the image gradient map of each training image is obtained using an edge detection algorithm; The training image is input into the encoder of a preset segmentation model to obtain the shallow feature map and deep feature map of the training image; The shallow and deep feature maps of the training image are input into the semantic fusion branch of the preset segmentation model to obtain the semantic segmentation prediction result of the training image; and, The shallow feature map, deep feature map, and image gradient map of the training image are input into the boundary refinement branch of the preset segmentation model to obtain the boundary segmentation prediction result of the training image. Calculate the segmentation loss function between the semantic segmentation prediction result of the training image and the pre-annotated semantic segmentation information; and, The boundary loss function is calculated between the boundary segmentation prediction result of the training image and the pre-labeled boundary segmentation information. Calculate the total loss function of the preset segmentation model based on the segmentation loss function and the boundary loss function; The semantic segmentation network is obtained by back-training the model parameters of the preset segmentation model using the total loss function.

10. An electronic device, characterized in that, include: A memory for storing program data, which, when executed, implements the steps of the image semantic segmentation method as described in any one of claims 1 to 9; A processor is configured to execute program instructions stored in the memory to implement the steps of the image semantic segmentation method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image semantic segmentation method as described in any one of claims 1 to 9.

Images