A semantic segmentation method with enhanced boundary perception

Abstract

The present invention discloses a semantic segmentation method for enhancing boundary perception, which belongs to the field of semantic segmentation technology, and mainly includes an encoding path and a decoding path. The encoding path is composed of 5 encoding modules, each encoder encodes the multi-level semantic information of the target area in the whole image, and the convolution operations of different scales in the encoding module obtain the multi-scale information of the target area; it uses pooling operation to effectively aggregate contextual semantics; the decoding path is mainly composed of four modules, each decoding module, under the guidance of the attention embedding module AEM, aggregates and refines the information flow of different branches, the graph convolution module captures the feature information of large-scale irregular areas, and the attention embedding module generates complementary spatial details to better model the encoding features. The present invention adopts the above-mentioned semantic segmentation method for enhancing boundary perception, and has a more accurate segmentation effect.

Description

Technical Field

[0001] The present invention relates to the technical field of semantic segmentation, and in particular to a semantic segmentation method for enhancing boundary perception. Background Art

[0002] Computer vision semantic segmentation aims to correctly classify the objects in the image and segment them to obtain precise contour shapes. It is a classic task in the field of computer vision and has important application value in the fields of information-based intelligent industry, industrial intelligence, and autonomous driving. It has become an important prerequisite for subsequent visual tasks. With the rapid development of deep learning technology, semantic segmentation tasks are also breaking through into new areas step by step. However, the images collected in reality usually have problems such as complex image structure, insufficient target saliency, and low image quality. It is particularly important to be able to extract rich and accurate target features in panoramic images.

[0003] With the development of image processing technology, image analysis has gone through three stages of development: image processing technology, machine learning, and deep learning. Methods based on image processing technology usually include three steps: image preprocessing, feature extraction, and target region extraction. Among them, image preprocessing includes graying, filtering, binarization, etc., the purpose of which is to remove noise in the image, enhance image contrast, and obtain the segmented target region using methods such as threshold analysis, edge detection, and Hough transform. On the other hand, machine learning-based methods achieve target segmentation by training classifiers. These methods usually include three steps: collecting data sets, extracting features, and training classifiers. Among them, commonly used classifiers are support vector machines (SVM) and convolutional neural networks (CNN), and commonly used feature extraction methods are image segmentation and feature selection. However, considering the inherent difficulty of pixel-level image annotation, and the complex semantic structure and category imbalance exhibited by most images, it is still a challenge to perform detailed analysis of images through traditional data-driven deep learning methods.

[0004] Most of the existing semantic segmentation methods based on deep learning are suitable for natural images, but for images with low light and unclear light-dark contrast, they often cannot accurately segment the boundary between the target and the background, resulting in poor segmentation accuracy. In addition, since the basic CNN can only operate in a fixed-size square area with structured features, most target areas present irregular unstructured features. Therefore, it is still a difficult task to effectively extract foreground objects in images. Summary of the invention

[0005] The purpose of the present invention is to provide a semantic segmentation method with enhanced boundary perception, which has a more accurate segmentation effect.

[0006] To achieve the above object, the present invention provides a semantic segmentation method for enhancing boundary perception, including an encoding path and a decoding path;

[0007] The encoding path includes the following sub-steps:

[0008] S1, the initial feature extraction module extracts features from the entire image and obtains a convolution feature map;

[0009] S2, inputting the convolution feature map obtained in step S1 into four encoding modules in sequence to obtain module feature maps of the four encoding modules;

[0010] S3, embed the convolution feature map obtained in step S1 into the four module feature maps in sequence to obtain four module total feature maps;

[0011] The decoding path consists of the following sub-steps:

[0012] S4, inputting the total feature map of the fourth module outputted in step S3 into the third decoding module, obtaining a refined feature map through convolution, wherein each pixel point of the refined feature map constitutes a topological map, and the critical points of the refined feature map constitute an adjacency matrix;

[0013] S5, reconstruct and refine the feature map generated by the attention embedding module AEM to reconstruct a set of reduced-dimensional feature maps and a set of projection matrices to meet the input requirements of the multi-scale graph reasoning module MsGRM and the requirements of graph node feature aggregation;

[0014] S6, the topological map and adjacency matrix obtained in step S4, a set of reduced-dimensional feature maps obtained in step S5, and a set of projection matrices are used as input, and the feature map is reconstructed using graph convolution and attention embedding module AEM;

[0015] S7. The reconstructed feature map obtained in step S6 is sequentially passed to the second decoding module, the first decoding module, the 0th decoding module, and the same goes for other decoding modules.

[0016] The present invention includes a third decoding module, a second decoding module, a first decoding module, and a zeroth decoding module. In each decoding module, a reconstructed feature map is obtained through the method described in step 6 and transmitted in a densely connected manner.

[0017] Preferably, the encoding path consists of an initial feature extraction module and four identical encoding modules;

[0018] The initial feature extraction module includes a convolution module with a convolution kernel of 7×7 and two convolution modules with a convolution kernel of 3×3.

[0019] Preferably, in step S1, the initial feature extraction module extracts features in the entire image to obtain a convolution feature map, and the calculation process is shown in formula (1):

[0020]

[0021] Where f0 represents the features that pass through the first layer of the initial feature extraction module; f1 represents the features that pass through the second and third layers of the initial feature extraction module; f′0 represents the features that pass through the fourth layer of the initial feature extraction module; σ(·) represents the activation function; σ s (·) represents the ReLU activation function; BN(·) represents the batch normalization layer; Conv 3×3 (.) and Conv 7×7 (.) represent 3×3 and 7×7 convolution operations respectively; Represents the addition operation of the matrix; MaxPool 3×3 (.) represents a 3×3 maximum pooling operation.

[0022] Preferably, the specific operations of step S2 are as follows:

[0023] The convolution feature map obtained in step S1 is input into the first encoding module, and then each encoding module inputs the feature map output by the previous layer into a set of consecutive 3×3 convolutions to obtain the first scale feature map in the image;

[0024] The feature map output by the previous layer is input into the 1×1 convolution in parallel to obtain the second-scale feature map, and then the two feature maps of different scales are fused. The fusion method is shown in the following formula:

[0025]

[0026] Among them, f fus represents the fused feature map; UP2() represents the bilinear interpolation upsampling operation; FC(·) represents the fully connected layer; represents a convolution with dimension increased by 1×1; f v Represents the feature map from the previous layer; σ r (·) represents the Leaky ReLU activation function; represents a convolution with a dimension reduction of 1×1; θ ρ represents the attention coefficient; H′ represents the height after reconstruction; W′ represents the width after reconstruction; H represents the original height; h ρ represents the high dimension after squeezing; w ρ represents the wide dimension after squeezing; v represents the scale set; l represents the low scale; h represents the high scale;

[0027] The fused feature map f fusAfter passing through an activation layer, the information flow is then transferred to the next layer through a skip link using a residual method. The calculation is shown in formula (3):

[0028]

[0029] Among them, f′ fus represents f fus The squeezed fusion feature map obtained after the convolution operation; f s represents f′ fus and f fus Module feature map after residual structure combination; Δ ★ (·) represents two layers of consecutive 3×3 convolution operations.

[0030] Preferably, the specific operations of step S3 are as follows:

[0031] The f0 extracted by the initial feature extraction module in step S1 is embedded into each encoding module, and finally the module total feature map is generated in sequence through the four encoding modules. The embedding process of f0 is shown in formula (4):

[0032]

[0033] in, represents the total feature map of the module after the feature transformation module ConvTM; Cat(·) represents feature concatenation; s represents the current encoding module.

[0034] Preferably, the decoding path is composed of four decoding modules with the same structure, each decoding module includes a feature transformation module ConvTM composed of convolution, an attention embedding module AEM and a multi-scale graph reasoning module MsGRM;

[0035] Among them, the feature transformation module ConvTM converts and up- and down-samples the total feature maps of different scale modules output by the corresponding encoding module to ensure that the feature maps input to the decoder remain consistent in size. The conversion is shown in formula (5):

[0036]

[0037] in, express The first scale feature map after downsampling; represents the second scale feature map; express The third scale feature map after downsampling; express The fourth-scale feature map after convolution activation and upsampling; DownS(·) represents the downsampling operation; UP(·) represents the nearest neighbor interpolation upsampling operation;

[0038] The attention embedding module AEM is used to establish the interaction between high-level and low-level semantic features. The input feature set is {f1…f N}, where N is the number of features, and the attention embedding module AEM obtains the attention fusion feature f a As shown in formula (6):

[0039]

[0040] Among them, α represents the attention mechanism; att(·) represents the attention operation; f a represents the generated attention fusion feature; n represents the current feature index; f n Indicates the current feature; f N Indicates all features;

[0041] The multi-scale graph reasoning module MsGRM uses node information transmission and convergence functions to obtain the global semantics and spatial details of the target. The multi-scale graph reasoning module MsGRM uses the topological graph method to collect and transmit node features of different scales, form interactions between these nodes, and establish long-term dependencies to strengthen the features.

[0042] Preferably, in step S4, the topology diagram is: ε={ε1…ε M} represents the set of edges between nodes, represents the set of graph nodes, and the adjacency matrix A is the critical point of the constructed topological graph, as shown in formula (7):

[0043]

[0044] Among them, v i and v j Represents a graph node; when node v i and v j Associated, or v i and v j If the node is the same, the node is a self-loop, and the value of the adjacency matrix A ij Defined as 1, when there is no association between these nodes, the boundary power value is set to 0, that is, A ij =0.

[0045] Preferably, the specific operations of step S5 are:

[0046] Reconstruct and refine the feature map generated by the attention embedding module AEM, and generate two scale feature maps of the attention embedding module AEM through convolutions of different scales. and Perform operations to reconstruct a set of dimensionality reduction feature maps and And a set of projection matrices and The reconstruction process is shown in formula (8):

[0047]

[0048] in, Represents a set of reduced-dimensional feature maps of the first scale; Represents a set of reduced-dimensional feature maps of the second scale; Represents the set of reconstructed feature maps of the first scale; Represents the set of reconstructed feature maps of the second scale; δ ↓ (·) represents the reconstruction function of dimensionality reduction; δ(·) represents the projection function, which converts the corresponding dimensionality reduction feature map With the projection matrix Perform matrix multiplication to obtain the node feature map At the same time, Laplace regularization is used to optimize the adjacency matrix A in step S4, and the feature maps of nodes at different scales are The optimization of the adjacency matrix A is as follows:

[0049]

[0050] in, A set of feature maps representing nodes of different scales; is matrix multiplication; I is the identity matrix; represents the adjacency matrix after Laplace re-regularization.

[0051] Preferably, the specific operations of step S6 are:

[0052] The generated node feature map Provide a graph convolution GCN module to implement node feature aggregation;

[0053] First, the multi-scale graph reasoning module MsGRM again performs the two-scale feature graphs generated by the attention embedding module AEM in step S5 Reconstruct and get the inverse projection matrix

[0054] Next, we inverse the projection matrix Convolutional feature maps with graph Multiply it, convert it back to the original hidden space, and use the convolution layer to fuse it to get the feature map

[0055] Finally, the generated A pixel-by-pixel addition strategy is used to add to the initial generation of the attention embedding module AEM In this paper, a new graph convolution feature is generated. in, and The calculation is shown in formula (10):

[0056]

[0057] in, Indicates the features corresponding to different scales; Represents features generated by AEM; represents the adjacency matrix after Laplace adjustment; represents the degree matrix of the adjacency matrix A; Θ s Represents the weighted matrix; the new graph convolution feature finally obtained The generation method is shown in formula (11):

[0058]

[0059] Preferably, in step S7, prior knowledge is introduced to increase the representation of low-level semantics. In the decoding path, the information flow of the decoding module has not only layer-by-layer connections, but also cross-layer connections. The cross-layer transmission of the information flow is performed in the form of residual connections. The total feature map f of the graph reasoning module finally generated by the multi-scale graph reasoning module MsGRM is MsGRM It is expressed as:

[0060]

[0061] in, Represents the feature map of the first scale; Represents the feature map of the second scale; Represents the total feature maps of the decoding modules at four scales generated by the four decoding modules.

[0062] Therefore, the present invention adopts the above-mentioned semantic segmentation method for enhancing boundary perception, and its technical effects are as follows:

[0063] (1) The U-shaped network structure allows for feature extraction at more scales, and the cleverly constructed graph convolution makes the segmentation network more friendly to irregular target regions. When applied to image analysis, the proposed framework can better accommodate unstructured features in the image, making it easier to extract relevant targets and exclude irrelevant backgrounds;

[0064] (2) In the model, richer image features can be extracted through the scale transformation of the encoding module and transmitted to each decoding module through dense links, thereby enhancing the reusability of features;

[0065] (3) We deploy a graph convolutional network in the decoding path to analyze large irregular regions and combine it with the SCSE (Concurrent Spatia and Channel Squeeze and Excitation) attention mechanism to capture complementary details, resulting in a more robust and accurate decoder;

[0066] (4) The model of the present invention is experimentally verified to maintain advanced results while being as lightweight as possible in a curated dataset;

[0067] (5) By comparing the results of the existing advanced segmentation algorithms on the same data set, the method of the present invention has a more accurate segmentation effect on images with insufficient target saliency and low image quality.

[0068] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a network overall structure diagram of a semantic segmentation method for enhancing boundary perception of the present invention; wherein, Figure 1 (a) is the overall structure diagram of the algorithm; Figure 1 (b) is the structural diagram of the coding module; Figure 1 (c) is the structural diagram of the decoding module;

[0070] Figure 2 It is a visual graph of the algorithm reasoning results. DETAILED DESCRIPTION

[0071] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0072] Embodiment 1

[0073] The present invention provides a semantic segmentation method for enhancing boundary perception, using U-Net as the backbone structure, improving the reuse and transferability of features, and using convolution operations of different sizes in the encoding part to make the feature scale richer. It is worth noting that in the decoding path of the network, graph convolution is added to replace a convolution layer. Compared with traditional convolutions that can only operate in a square area of ​​a fixed size, graph convolution can capture a wider range of node relationships and capture more contextual information. The form of graph nodes makes it more flexible and sensitive when facing complex boundaries. In addition, an attention embedding module (AEM) is added to each decoding module to aggregate and refine the information flows of different branches. Without pre-training and post-processing, the network model still shows excellent results and great potential.

[0074] This method mainly includes an encoding path and a decoding path, such as Figure 1(a). Specifically include:

[0075] S1, the encoding path consists of 5 encoding modules, each encoder is as follows Figure 1 As shown in (b), the multi-level semantic information of the target area in the entire image is encoded, and the convolution operations of different scales in the encoding module obtain the multi-scale information of the target area, namely the feature map. The encoding path uses pooling operations to effectively aggregate contextual semantics. The decoding path mainly consists of four modules (see details in Figure 1 (c)).

[0076] S11, the encoding path consists of 5 encoding modules, including an initial feature extraction module and four identical encoder group layers, where the initial feature module includes a convolution module with a convolution kernel of 7×7 and two convolution modules with a convolution kernel of 3×3, and a convolution feature map is obtained. The specific calculation process is shown in formula (1):

[0077]

[0078] Where f0 represents the features that pass through the first layer of the initial feature extraction module; f1 represents the features that pass through the second and third layers of the initial feature extraction module; f′0 represents the features that pass through the fourth layer of the initial feature extraction module; σ(·) represents the activation function; σ s (·) represents the ReLU activation function; BN(·) represents the batch normalization layer; Conv 3×3 (.) and Conv 7×7 (.) represent 3×3 and 7×7 convolution operations respectively; Represents the addition operation of the matrix; MaxPool 3×3 (.) represents a 3×3 maximum pooling operation.

[0079] S12: Input the convolution feature map obtained in step S11 into the first encoding module. Then, the internal structure of each encoding module is as follows: Figure 1 As shown in (b), the feature map output by the previous layer is input into a set of continuous 3×3 convolutions to obtain local features in the image to highlight the difference between the target area and the background. At the same time, in order to prevent the features in more details from being ignored, the feature map output by the previous layer is input into a 1×1 convolution in parallel, and then the features of two different scales are fused. The fusion method is shown in the following formula:

[0080]

[0081] After the feature map output by the fusion of features of different scales passes through an activation layer, in order to prevent the gradient explosion or disappearance caused by the too deep network structure, the residual method is used to transfer the information flow to the next layer through the skip link. The specific calculation is shown in formula (3), where Δ★ (·) represents a set of consecutive 3×3 convolution operations. σ(·) represents the activation function relu:

[0082]

[0083] S13. Considering that rich underlying semantic information is conducive to the representation of physical properties such as target size and appearance, the large-scale feature f0 extracted by the initial feature module is embedded into each encoding module, thereby prompting the network to obtain more detailed encoding features. Finally, useful encoding features f are generated in turn through four encoding modules. s The embedding process of large-scale features is shown in formula (4):

[0084]

[0085] in, represents the total feature map of the module after the feature transformation module ConvTM; Cat(·) represents feature concatenation; s represents the current encoding module.

[0086] S2. Each decoding module aggregates and refines multi-scale information under the guidance of the Attention Embedding Module (AEM). The graph convolution module captures the feature information of large-scale irregular regions. The Attention Embedding Module generates complementary spatial details to better model the encoded features.

[0087] The decoding path consists of four decoders with the same structure, namely, a feature transformation module (Convolution Transform Module, ConvTM) composed of convolutions, an attention embedding module (Attention Embedding Module, AEM) and a multi-scale graph reasoning module (Multi-scale Graph Reasoning Module, MsGRM).

[0088] (1) ConvTM: ConvTM mainly processes the different scale features f output by the corresponding encoding module s The conversion and up- and down-sampling are performed to ensure that the feature maps input to the decoder are consistent in size, effectively preventing problems such as loss, explosion, and disappearance of detailed information and gradients caused by excessive up- and down-sampling in the network. For example, the transformation of the third decoding module is shown in equation (5). The detailed structure of the third decoding module is shown in Figure 1 (c) as shown:

[0089]

[0090] in, express The first scale feature map after downsampling; represents the second scale feature map; express The third scale feature map after downsampling; express The fourth-scale feature map after convolution activation and upsampling; DownS(·) represents the downsampling operation; UP(·) represents the nearest neighbor interpolation upsampling operation;

[0091] (2) AEM: It is used to establish the interaction between high-level and low-level semantic features. In simple terms, it embeds low-level semantic information into high-level features and embeds high-level discriminative features into low-level features. Assume that the input feature set is {f1…f N}, where N is the number of features. The fusion f of AEM proposed in the present invention a The characteristics are shown in formula (6):

[0092]

[0093] Among them, α represents the attention mechanism; att(·) represents the attention operation; f a represents the generated attention fusion feature; n represents the current feature index; f n represents the current feature, f N In this way, the underlying features embed contextual information while retaining spatial details, improving feature representation and facilitating the segmentation of target areas in the image.

[0094] (3) MsGRM: MsGRM mainly uses node information transmission and convergence functions to obtain the global semantics and spatial details of the target. Traditional convolution algorithms can only operate within a square area of ​​fixed size and cannot effectively extract the features of irregular targets in the image. MsGRM uses a topological graph method to collect and transmit node features of different scales, forming interactions between these nodes and establishing long-term dependencies to strengthen the features. This module adopts the method of topological reconstruction and multi-scale feature embedding nodes, which helps to improve the network's ability to learn the attributes of the boundary pixels of the target area, as well as more effective learning through graph nodes. The global and contextual spatial details further enhance the characteristics of the features.

[0095] The specific decoding paths include:

[0096] S21. Nodes of the constructed topology graph

[0097] The feature map outputted in step S13 is inputted into the decoder, and a smaller feature map is obtained through convolution. Each pixel point of the smaller feature map forms a topological map. The topological map forms ε={ε1…ε M} represents the set of edges between nodes, that is, the relationship between elements in the image. Represents the set of graph nodes. The adjacency matrix A is the critical point of the constructed topological graph, as shown in formula (7):

[0098]

[0099] Among them, v i and v j Represents a graph node. When the node v i and v j Associated, or v i and v j If the node is the same, the node is a self-loop, and the value of the adjacency matrix A ij is defined as 1. When there is no association between these nodes, we set the boundary power value to 0, i.e. A ij =0.

[0100] S22. Reconstruct and refine the feature map generated by AEM to meet the input requirements of MsGRM and the requirements of graph node feature aggregation

[0101] MsGRM extracts global semantics by aggregating and transferring these node features. The feature map generated by AEM is reconstructed and refined to meet the input requirements of MsGRM and the requirements of graph node feature aggregation. Figure 1 (c) shows a set of feature maps generated by convolution of different scales for AEM and Perform operations to reconstruct a set of dimensionality reduction feature maps and And a set of projection matrices and The reconstruction process is shown in equation (8):

[0102]

[0103] in, Represents a set of reduced-dimensional feature maps of the first scale; Represents a set of reduced-dimensional feature maps of the second scale; Represents the set of reconstructed feature maps of the first scale; Represents the set of reconstructed feature maps of the second scale; δ ↓ (·) represents the reconstruction function of dimensionality reduction; δ(·) represents the projection function, which converts the corresponding dimensionality reduction feature map With the projection matrix Perform matrix multiplication to obtain the node feature map At the same time, Laplace regularization is used to optimize the adjacency matrix A in step S4, and the feature maps of nodes at different scales are The optimization of the adjacency matrix A is as follows:

[0104]

[0105] in, A set of feature maps representing nodes of different scales; is matrix multiplication; I is the identity matrix; represents the adjacency matrix after Laplace re-regularization.

[0106] S23. Reconstruct feature maps using graph convolution and AEM

[0107] The generated node feature map Provide a graph convolution (GCN) module to achieve node feature aggregation and obtain better global and contextual semantic details, including:

[0108] First, the feature map generated by AEM in step S22 is again Reconstruct and get the inverse projection matrix

[0109] Next, we inverse the projection matrix Convolutional feature maps with graph Multiply them and transform them back to the original hidden space, and use the convolution layer to fuse them to get the feature map

[0110] Finally, in order to obtain better feature representation, the generated feature map Adopt pixel-by-pixel addition strategy to add to the feature map initially generated by AEM In this paper, a new graph convolution feature is generated. Among them, multi-scale and The calculation is shown in formula (10):

[0111]

[0112] in, Leakyrelu represents the activation function, which adds a small negative slope to relu, so that when the input is negative, the activation function is not 0 and has a smaller derivative than relu, avoiding the death of neurons and accelerating the convergence of the neural network. represents the degree matrix of the adjacency matrix A, Θ represents the weighting matrix. The new graph convolution feature finally obtained is The generation method is shown in the following formula (11):

[0113]

[0114] S24, cross-layer connection of information flow of decoding module

[0115] MsGRM obtains multi-scale node features by constructing topological graphs of different scales, making full use of multi-scale information, while better processing global and contextual details through node transfer functions. In addition, the introduction of prior knowledge increases the representation of low-level semantics. On the decoding path, the information flow of the decoding module can be connected across layers, ensuring the stable transmission of detailed information to the greatest extent. However, in order to reduce redundant information and alleviate problems such as gradient disappearance, the cross-layer transmission of information flow is carried out using residual connections. The feature map f finally generated by MsGRM MsGRM It can be expressed as:

[0116]

[0117] in, Represents the feature map of the first scale; Represents the feature map of the second scale; Represents the total feature maps of the decoding modules at four scales generated by the four decoding modules.

[0118] Table 1 Comparison of different segmentation methods on the same dataset

[0119] method Backbone network FLOPs↓ aACC↑ mIOU↑ APCNet ResNet101 5.1 75.22 13.90 DeepLabv3+ ResNet101 1.8 81.17 14.46 MobileNetv3 MobileNetV3 2.1 77.24 16.54 UNet FCN 1.0 82.42 17.41 OCRNet HRNetV2p-W48 4.7 77.32 29.25 UperNet Swin-B 6.9 80.04 27.23 SETRNaive ViT-L 3.8 78.66 18.65 Segformer MIT-B5 13.9 80.78 16.06 U-GRNets (this invention) ResNet34 6.8 93.68 64.16

[0120] Therefore, the present invention adopts the above-mentioned semantic segmentation method with enhanced boundary perception, which has a more accurate segmentation effect. The final effect diagram of the algorithm is as follows: Figure 2 As shown, from left to right are the original Figure 1 ,Effect Figure 1 ,original Figure 2 ,Effect Figure 2 , original picture three, effect picture three.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A semantic segmentation method with enhanced boundary perception, characterized in that: Includes encoding path and decoding path; The encoding path includes the following sub-steps: S1, the initial feature extraction module extracts features from the entire image and obtains a convolution feature map; S2, inputting the convolution feature map obtained in step S1 into four encoding modules in sequence to obtain module feature maps of the four encoding modules; S3, embed the convolution feature map obtained in step S1 into the four module feature maps in sequence to obtain four module total feature maps; The decoding path consists of four decoding modules with the same structure. Each decoding module includes a feature transformation module ConvTM composed of convolution, an attention embedding module AEM and a multi-scale graph reasoning module MsGRM. The decoding path includes the following sub-steps: S4, inputting the total feature map of the fourth module outputted in step S3 into the third decoding module, and obtaining a refined feature map through convolution, wherein each pixel point of the refined feature map constitutes a topological map, and the critical points of the refined feature map constitute an adjacency matrix; S5, reconstruct and refine the feature map generated by the attention embedding module AEM to reconstruct a set of reduced-dimensional feature maps and a set of projection matrices to meet the input requirements of the multi-scale graph reasoning module MsGRM and the requirements of graph node feature aggregation; S6, the topological map and adjacency matrix obtained in step S4, the set of reduced-dimensional feature maps obtained in step S5, and a set of projection matrices are used as input, and the set of reduced-dimensional feature maps obtained in step S5 are reconstructed using graph convolution and attention embedding module AEM; S7. The reconstructed feature map obtained in step S6 is passed to the second decoding module, the first decoding module, and the 0th decoding module in sequence.

2. The semantic segmentation method with enhanced boundary perception according to claim 1, characterized in that: The encoding path consists of an initial feature extraction module and four identical encoding modules; The initial feature extraction module includes a convolution module with a convolution kernel of 7×7 and two convolution modules with a convolution kernel of 3×3.

3. The semantic segmentation method with enhanced boundary perception according to claim 2, characterized in that: In step S1, the initial feature extraction module extracts features from the entire image to obtain a convolution feature map. The calculation process is shown in formula (1): Among them, f0 represents the features of the first layer of the initial feature extraction module; f m represents the features of the second and third layers of the initial feature extraction module; f'0 represents the features of the fourth layer of the initial feature extraction module; σ(·) represents the activation function; σ s (·) represents the ReLU activation function; BN(·) represents the batch normalization layer; Conv 3×3 (.) and Conv 7×7 (.) represents 3×3 and 7×7 convolution operations respectively; ⊕ represents matrix addition operation; MaxPool 3×3 (.) represents a 3×3 maximum pooling operation.

4. The semantic segmentation method with enhanced boundary perception according to claim 3, characterized in that: The specific operations of step S2 are as follows: The convolution feature map obtained in step S1 is input into the first encoding module, and then each v+1th encoding module inputs the feature map obtained by the vth encoding module into a set of consecutive 3×3 convolutions to obtain the first scale feature map in the image; The feature map obtained by the vth encoding module is input into the 1×1 convolution in parallel to obtain the second-scale feature map, and then the two feature maps of different scales are fused. The fusion method is shown in the following formula: Among them, f fus represents the fused feature map; UP2() represents the bilinear interpolation upsampling operation; FC(·) represents the fully connected layer; represents a convolution with dimension increased by 1×1; f v represents the feature map obtained by the vth encoding module; σ r (·) represents the Leaky ReLU activation function; represents a convolution with a dimension reduction of 3×3; θ ρ represents the attention coefficient; H' represents the height after reconstruction; W' represents the width after reconstruction; H represents the original height; h ρ represents the high dimension after squeezing; w ρ represents the wide dimension after squeezing; v represents the scale set; l represents the low scale; h represents the high scale; The fused feature map f fus After passing through an activation layer, the information flow is then transferred to the next layer through a skip link using a residual method. The calculation is shown in formula (3): Among them, f' fus represents f fus The squeezed fusion feature map obtained after the convolution operation; f s Indicates f' fus and f fus Module feature map after residual structure combination; Δ ＊ (·) represents two layers of consecutive 3×3 convolution operations.

5. The semantic segmentation method with enhanced boundary perception according to claim 4, characterized in that: The specific operation of step S3 is as follows: The f0 extracted by the initial feature extraction module in step S1 is embedded into each encoding module, and finally the module total feature map is generated in sequence through the four encoding modules. The embedding process of f0 is shown in formula (4): in, represents the total feature map of the module after the feature transformation module ConvTM; Cat(·) represents feature concatenation; s represents the current encoding module.

6. The semantic segmentation method with enhanced boundary perception according to claim 5, characterized in that: The decoding path consists of four decoding modules with the same structure. Each decoding module includes a feature transformation module ConvTM composed of convolution, an attention embedding module AEM and a multi-scale graph reasoning module MsGRM. Among them, the feature transformation module ConvTM converts and up- and down-samples the total feature maps of different scale modules output by the corresponding encoding module to ensure that the feature maps input to the decoder remain consistent in size. The conversion is shown in formula (5): in, express The first scale feature map after downsampling; represents the second scale feature map; express The third scale feature map after downsampling; express The fourth-scale feature map after convolution activation and upsampling; DownS(·) represents the downsampling operation; UP(·) represents the nearest neighbor interpolation upsampling operation; The attention embedding module AEM is used to establish the interaction between high-level and low-level semantic features. The input feature set is Among them, N is the number of features, and the attention embedding module AEM obtains the attention fusion feature f a As shown in formula (6): Among them, α represents the attention mechanism; att(·) represents the attention operation; f a represents the generated attention fusion feature; n represents the current feature index; f n Indicates the current feature; Indicates all features; The multi-scale graph reasoning module MsGRM uses node information transmission and convergence functions to obtain the global semantics and spatial details of the target. The multi-scale graph reasoning module MsGRM uses the topological graph method to collect and transmit node features of different scales, form interactions between these nodes, and establish long-term dependencies to strengthen the features.

7. The semantic segmentation method with enhanced boundary perception according to claim 6, characterized in that: In step S4, the topology diagram is: ε={ε1…ε M } represents the set of edges between nodes, represents the set of graph nodes, and the adjacency matrix A is the critical point of the constructed topological graph, as shown in formula (7): Among them, v i and v j Represents a graph node; when node v i and v j Associated, or v i and v j If the node is the same, the node is a self-loop, and the value of the adjacency matrix A ij Defined as 1, when there is no association between these nodes, the boundary power value is set to 0, that is, A ij =0.

8. The semantic segmentation method with enhanced boundary perception according to claim 7, characterized in that: The specific operations of step S5 are: Reconstruct and refine the feature map generated by the attention embedding module AEM, and generate two scale feature maps of the attention embedding module AEM through convolutions of different scales. and Perform operations to reconstruct a set of dimensionality reduction feature maps and And a set of projection matrices and The reconstruction process is shown in formula (8): in, Represents a set of reduced-dimensional feature maps of the first scale; Represents a set of reduced-dimensional feature maps of the second scale; Represents the set of reconstructed feature maps of the first scale; Represents the set of reconstructed feature maps of the second scale; δ ↓ (·) represents the reconstruction function of dimensionality reduction; δ(·) represents the projection function, which converts the corresponding dimensionality reduction feature map With the projection matrix Perform matrix multiplication to obtain the node feature map At the same time, Laplace regularization is used to optimize the adjacency matrix A in step S4, and the feature maps of nodes at different scales are The optimization of the adjacency matrix A is as follows: in, A set of feature maps representing nodes of different scales; is matrix multiplication; I is the identity matrix; represents the adjacency matrix after Laplace re-regularization.

9. The semantic segmentation method with enhanced boundary perception according to claim 8, characterized in that: The specific operations of step S6 are: The generated node feature map Provide a graph convolution GCN module to implement node feature aggregation to obtain graph convolution feature mapping First, the multi-scale graph reasoning module MsGRM again performs the two-scale feature graphs generated by the attention embedding module AEM in step S5 Reconstruct and get the inverse projection matrix Next, the inverse projection matrix Convolutional feature maps with graph Multiply it, convert it back to the original hidden space, and use the convolution layer to fuse it to get the feature map Finally, the generated A pixel-by-pixel addition strategy is used to add to the initial generation of the attention embedding module AEM In this paper, a new graph convolution feature is generated. in, and The calculation is shown in formula (10): in, Indicates the features corresponding to different scales; Represents features generated by AEM; represents the adjacency matrix after Laplace adjustment; represents the degree matrix of the adjacency matrix A; Θ s Represents the weighted matrix; the new graph convolution feature finally obtained The generation method is shown in formula (11):

10. The semantic segmentation method with enhanced boundary perception according to claim 9, characterized in that: In step S7, prior knowledge is introduced to increase the representation of low-level semantics. In the decoding path, the information flow of the decoding module has not only layer-by-layer connections, but also cross-layer connections. The cross-layer transmission of the information flow is carried out in the form of residual connections. The total feature map f of the graph reasoning module finally generated by the multi-scale graph reasoning module MsGRM is MsGRM It is expressed as: in, Represents the feature map of the first scale; Represents the feature map of the second scale; Represents the total feature maps of the decoding modules at four scales generated by the four decoding modules.

Images