River and lake coastal building image segmentation method based on fusion CNN and Transform
By introducing dynamic residual shorting module, multi-scale feature channel cross-fusion Transformer and dual-feature channel fusion attention in the U-Net network, the problems of feature information loss and semantic differences in the prior art are solved, and high-precision segmentation of building images along river and lake shores is achieved.
Patent Information
- Application Number
- CN202411899267.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-04-15
AI Technical Summary
Existing deep learning algorithms are difficult to accurately extract and segment buildings along river and lake coasts in complex background environments within the scope of river and lake supervision. This is mainly due to the loss of feature information, semantic differences between encoder and decoder features, and poor feature fusion during upsampling.
The image segmentation method of river and lake coastal buildings based on fusion CNN and Transformer is adopted. By constructing dynamic residual short-circuit modules, multi-scale feature channels cross-fusion Transformer and dual-feature channel fusion attention, the U-Net network architecture is improved, feature information loss, semantic differences are reduced, and feature fusion accuracy is improved.
The model's ability to capture multi-scale feature information is improved, the impact of complex building shapes and scale differences on the segmentation results is reduced, the ability to describe building edge details is enhanced, the influence of water mirror reflection and shore-side ships on the model segmentation accuracy is reduced, and high-precision architectural image segmentation is achieved.
Smart Images
Figure CN119832239A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method for segmenting river and lake shore building images based on fusion of CNN and Transformer. Background Art
[0002] The phenomenon of building flood-blocking buildings within the management scope of rivers and lakes is particularly common. The lack of water space will reduce the self-regulation ability of water bodies, which will in turn cause problems such as flood discharge difficulties. In order to alleviate the encroachment of flood-blocking buildings on water areas, current river and lake regulators determine the distribution of buildings along rivers and lakes through manual annotation based on remote sensing images, and dynamically monitor water-adjacent buildings by comparing remote sensing images of different periods. However, there are a large number of buildings within the management scope of rivers and lakes, and the manual annotation method is time-consuming and occupies a lot of human resources, which is difficult to meet the needs of daily river and lake supervision.
[0003] With the development of deep learning technology, it has become possible to extract buildings from remote sensing images quickly and efficiently. At present, deep learning algorithms based on convolutional neural networks (CNN) and Transformer have been applied to a certain extent in the field of building segmentation in remote sensing images, but related research mainly focuses on foreign cities, towns and other areas where buildings are concentrated and there is less environmental interference. However, faced with complex background environments such as mirror reflections of water bodies and ships near the shore with texture features similar to buildings within the regulatory scope of rivers and lakes, current deep learning algorithms are difficult to accurately extract and segment water-adjacent buildings. The main reasons are as follows: (1) Building segmentation networks are often based on U-Net. The traditional convolution linear transmission used in the downsampling process is not conducive to the transmission of complex feature information, and simple and multiple convolution operations will lose some of the original upper-level feature information. This loss will continue to accumulate between features of multiple layers and different scales, thereby affecting the model's capture of multi-scale feature information; (2) There is a certain degree of semantic difference between the features of the encoder and decoder at the same level. It is difficult to reduce this difference by simply copying the jump connection, which leads to a decrease in model performance; (3) Directly splicing features of the same scale during upsampling is difficult to fuse their semantic information, and will also cause ambiguity in the upsampling process, affecting the retention of detailed information of the building target. Summary of the invention
[0004] Purpose of the invention: In order to overcome the shortcomings of the background technology, the present invention discloses a method for image segmentation of buildings along rivers and lakes based on the fusion of CNN and Transformer.
[0005] Technical solution: The method for segmenting river and lake shore building images based on the fusion of CNN and Transformer disclosed in the present invention comprises the following steps:
[0006] S1. Obtain remote sensing images of buildings within the river and lake management area;
[0007] S2, crop the acquired remote sensing images, annotate the building targets, and then construct the data set;
[0008] S3, based on the U-Net network architecture, combined with the dynamic residual short-circuit module, multi-scale feature channel cross-fusion Transformer, and dual feature channel fusion attention, a river and lake shore building image segmentation model is constructed;
[0009] S4. The output results are stitched and restored to obtain the segmentation result of the original image, and building image segmentation is performed on the actual river and lake supervision remote sensing image.
[0010] Among them, S2 includes
[0011] S2-1. Screen remote sensing images with buildings;
[0012] S2-2, cropping each remote sensing image into 512×512 pixels, and for rows and columns whose edge positions are less than 512 pixels, forming a cropping area with the reciprocal 512 pixels;
[0013] S2-3, using an image annotation tool to manually annotate the cropped image;
[0014] S2-4. Divide the labeled images into training set, validation set and test set in a ratio of 8:1:1.
[0015] Furthermore, S3 specifically includes:
[0016] S3-1, using the U-Net network as the basic network, including an encoder and decoder with a depth of five layers, and four skip connections connecting the first four feature maps respectively;
[0017] S3-2, after the image T0 is input, a dynamic residual short-circuit module is added to expand the feature map channel, and the output feature map is T1; after each maximum pooling layer of the encoder structure, a dynamic residual short-circuit module is added, and the output feature maps are T2, T3, T4, and T5 respectively;
[0018] S3-3, the output features are skipped through the multi-scale feature channel cross-fusion Transformer, and the output feature maps O1, O2, O3, O4 and O5 are output;
[0019] S3-4, take O1~O5 as the input of the decoder; put the feature maps O5 and O4 at the same level, perform feature fusion through dual feature channel fusion attention, and then output it as feature map Y4 through bilinear interpolation and dynamic residual short-circuit module; take feature map Y4 and O3 as input, perform feature fusion through dual feature channel fusion attention, and then output it as feature map Y3 through bilinear interpolation and dynamic residual short-circuit module; take feature map Y3 and O2 as input, perform feature fusion through dual feature channel fusion attention, and then output it as feature map Y2 through bilinear interpolation and dynamic residual short-circuit module; take feature map Y2 and O1 as input, perform feature fusion through dual feature channel fusion attention, and then output it as feature map Y1 through bilinear interpolation and dynamic residual short-circuit module;
[0020] S3-5, taking the feature map Y1 as input, after convolution and Sigmoid function, finally obtaining the segmentation result of the input image;
[0021] S3-6. For each image, the loss function L is calculated using the following formula:
[0022] L=L Dice +L Focal
[0023] Among them, L Dice is the Dice loss function; L Focal is the Focal loss function.
[0024] Furthermore, the dynamic residual short-circuit module in S3-2 includes a dynamic convolution block and a residual module;
[0025] The dynamic convolution block is based on K linear functions Perform dynamic nonlinear aggregation; the calculation formula of the dynamic convolution block is:
[0026]
[0027] y = ReLU(W T (x)x+b(x))
[0028] Among them, π k is the kth linear function The attention weight ranges from [0, 1], and the sum of the weights is 1; W is the weight matrix; b is the bias vector ReLU; is the activation function; x is the input feature; y is the output feature;
[0029] The residual module is used to combine the dynamic convolution block and the traditional convolution. The image T0 is processed through two paths. The first path is first processed by a 3×3 dynamic convolution block, then normalized by Batch Norm, activated by the ReLU function, and then processed by 3×3 convolution, and then processed by Batch Norm and ReLU again. The second path is that T0 is first processed by a 1×1 dynamic convolution block, and then normalized by Batch Norm. Finally, the outputs of the two paths are concatenated and ReLU activation is performed to obtain the feature map T1 of the next layer as the output.
[0030] Furthermore, S3-3 is specifically:
[0031] Take the feature maps T1, T2, T3, T4 and T5 as input, normalize them respectively and then perform feature fusion. The formula is:
[0032] S=Concat[δ(T1), δ(T2), δ(T3), δ(T4), δ(T5)];
[0033] Among them, δ() is the LayerNorm normalization operation; Concat[] is the feature fusion operation; S is the set embedding layer;
[0034] Normalize the sequence T i As the input of the multi-head channel attention mechanism, and with five different input sequences T i and 1 set embedding layer S to transform into Q i , K, V matrix, the conversion formula is:
[0035] Q i =δ(T i )×W Qi
[0036] K=S×W K
[0037] V=S×W V
[0038] Among them, W Qi , W K , W V are the weights of different inputs;
[0039] The input matrix is further processed to obtain the output matrix MHCA of the channel attention mechanism:
[0040]
[0041] Among them, ψ() is the Instance Norm normalization function; σ() is the Sigmoid function; n is the number of feature inputs;
[0042] After the following formula is repeatedly processed four times, the feature maps O1, O2, O3, O4 and O5 are obtained as the output of the multi-scale feature channel cross-fusion Transformer:
[0043] O i =MHCA i +MLP[δ(Q i +MHCA i )];
[0044] Among them, MLP is a multi-layer perceptron operation.
[0045] Furthermore, the dual feature channel fusion attention in S3-4 specifically includes:
[0046] Take two feature maps X at the same level i , Y i As input, the two feature maps are globally averaged and pooled to obtain the feature vector f(X i ) and f(Y i ), and then use a single linear layer and Sigmoid function to construct the channel feature map; the two feature maps ζ(X i ) and ζ(Y i ), they are calibrated, activated and re-fused with the initial feature maps to obtain the output of dual-feature channel fusion attention;
[0047] The formula for dual feature channel fusion attention is as follows:
[0048]
[0049] G(κ)=σ[ζ(X i )+ζ(Y i )]×(X i +Y i )
[0050] Among them, κ is the input feature; G() is the output feature; k is the feature of the kth channel; H is the number of pixels in the vertical direction of the feature map; W is the number of pixels in the horizontal direction of the feature map; κ is the input amount of each formula; α is the weight of a single linear layer; σ() is the activation function, which represents the importance of the channel.
[0051] Beneficial effects: Compared with the prior art, the advantages of the present invention are:
[0052] (1) By constructing a dynamic residual short-circuit module to replace the traditional downsampling process, the loss of feature information in the downsampling process is greatly reduced, and the transmission ability of nonlinear feature information is improved, thereby improving the model's ability to capture multi-scale feature information, which minimizes the interference of complex building shapes and building scale differences on the segmentation results;
[0053] (2) By constructing a multi-scale feature channel cross-fusion Transformer, the semantic difference between the multi-scale features of the encoder and decoder is reduced; by constructing a dual-feature channel fusion attention, the semantic difference between the output features of the multi-scale feature channel cross-fusion Transformer and the decoder features is reduced, and two same-scale features are better integrated, which improves the model's ability to describe the edge details of the building and reduces the impact of features similar to building features such as water mirror reflections and ships on the shore on the model segmentation accuracy;
[0054] (3) The deep learning method is applied to river and lake supervision work scenarios with complex backgrounds, which has the advantages of high accuracy and good building segmentation integrity in building segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 is a basic flow chart of the present invention;
[0056] Figure 2 A schematic diagram of the structure of the deep learning framework designed for the present invention;
[0057] Figure 3 A schematic diagram of the structure of a dynamic residual short-circuit module constructed in the present invention;
[0058] Figure 4 A schematic diagram of the structure of the multi-scale feature channel cross-fusion Transformer constructed by the present invention;
[0059] Figure 5 A schematic diagram of the structure of the dual-feature channel fusion attention constructed for the present invention;
[0060] Figure 6 This is a typical sample example of the HRI Building dataset;
[0061] Figure 7 This is the building segmentation result of the ablation experiment;
[0062] Figure 8 This is the building segmentation result of the comparative experiment. DETAILED DESCRIPTION
[0063] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0064] A river and lake shore building image segmentation method based on fusion of CNN and Transformer, comprising the following steps:
[0065] S1. Obtain remote sensing images of buildings within the management scope of rivers and lakes.
[0066] S2, cropping the remote sensing image obtained in step S1, and marking the building targets, thereby constructing a data set;
[0067] The specific steps include:
[0068] Screening remote sensing images for the presence of buildings;
[0069] Each remote sensing image was cropped to 512 × 512 pixels. For rows and columns with edge positions less than 512 pixels, the cropping area was formed by the reciprocal 512 pixels.
[0070] Use the image annotation tool to manually annotate the cropped image;
[0071] The labeled images are divided into training set, validation set and test set in the ratio of 8:1:1.
[0072] S3. Based on the U-Net network architecture, a dynamic residual short-circuit module, multi-scale feature channel cross-fusion Transformer, and dual feature channel fusion attention are combined to build a river and lake shore building image segmentation model; specifically, the following steps are included:
[0073] A. The U-Net network is used as the basic network; the U-Net network includes an encoder and decoder with a depth of five layers, and four skip connections that connect the first four feature maps respectively;
[0074] B. After the image T0 is input, a dynamic residual short-circuit module is added to expand the feature map channel, and the output feature map is T1; after each maximum pooling layer of the encoder structure, a dynamic residual short-circuit module is added, and the output feature maps are T2, T3, T4, and T5 respectively; the dynamic residual short-circuit module specifically includes the following contents:
[0075] Dynamic residual short-circuit module, including dynamic convolution block and residual module;
[0076] The dynamic convolution block is based on K linear functions Perform dynamic nonlinear aggregation; the calculation formula of the dynamic convolution block is:
[0077]
[0078] y = ReLU(W T (x)x+b(x))
[0079] Among them, πk is the kth linear function The attention weight ranges from [0, 1], and the sum of the weights is 1; W is the weight matrix; b is the bias vector ReLU; is the activation function; x is the input feature; y is the output feature;
[0080] The residual module is used to combine the dynamic convolution block and the traditional convolution. The image T0 is processed through two paths. The first path is first processed by a 3×3 dynamic convolution block, then normalized by Batch Norm, activated by the ReLU function, and then processed by 3×3 convolution, and then processed by Batch Norm and ReLU again. The second path is that T0 is first processed by a 1×1 dynamic convolution block, and then normalized by Batch Norm. Finally, the outputs of the two paths are concatenated and ReLU activated to obtain the feature map T1 of the next layer as the output.
[0081] C. The output features obtained in step B are skipped through the multi-scale feature channel cross-fusion Transformer:
[0082] Take the feature maps T1, T2, T3, T4 and T5 as input, normalize them respectively and then perform feature fusion. The formula is:
[0083] S=Concat[δ(T1), δ(T2), δ(T3), δ(T4), δ(T5)];
[0084] Among them, δ() is the Layer Norm normalization operation; Concat[] is the feature fusion operation; S is the set embedding layer;
[0085] Normalize the sequence T i As the input of the multi-head channel attention mechanism, and with five different input sequences T i and 1 set embedding layer S to transform into Q i , K, V matrix, the conversion formula is:
[0086] Q i =δ(T i )×W Qi
[0087] K=S×W K
[0088] V=S×W V Among them, W Qi , W K , W V are the weights of different inputs;
[0089] The input matrix is further processed to obtain the output matrix MHCA of the channel attention mechanism:
[0090]
[0091] Among them, ψ() is the Instance Norm normalization function; σ() is the Sigmoid function; n is the number of feature inputs;
[0092] After the following formula is repeatedly processed four times, the feature maps O1, O2, O3, O4 and O5 are obtained as the output of the multi-scale feature channel cross-fusion Transformer:
[0093] O i =MHCA i +MLP[δ(Q i +MHCA i )];
[0094] Among them, MLP is a multi-layer perceptron operation;
[0095] D. Take the input features O1~O5 obtained in step C as the input of the decoder; place the feature maps O5 and O4 at the same level, perform feature fusion through dual feature channel fusion attention, and then output as feature map Y4 through bilinear interpolation and dynamic residual short-circuit module; take feature map Y4 and O3 as input, perform feature fusion through dual feature channel fusion attention, and then output as feature map Y3 through bilinear interpolation and dynamic residual short-circuit module; take feature map Y3 and O2 as input, perform feature fusion through dual feature channel fusion attention, and then output as feature map Y2 through bilinear interpolation and dynamic residual short-circuit module. Take feature map Y2 and O1 as input, perform feature fusion through dual feature channel fusion attention, and then output as feature map Y1 through bilinear interpolation and dynamic residual short-circuit module; the dual feature channel fusion attention specifically includes the following contents:
[0096] Dual feature channel fusion attention is used to combine two feature maps X at the same level. i , Y i As input, the two feature maps are globally averaged and pooled to obtain the feature vector f(X i ) and f(Y i ), and then use a single linear layer and Sigmoid function to construct the channel feature map; the two feature maps ζ(X i ) and ζ(Y i ), they are calibrated, activated and re-fused with the initial feature maps to obtain the output of dual-feature channel fusion attention; the formula of dual-feature channel fusion attention is as follows:
[0097]
[0098] G(κ)=σ[ζ(Xi )+ζ(Y i )]×(X i +Y i )
[0099] Among them, K is the input feature; G() is the output feature; k is the feature of the kth channel; H is the number of pixels in the vertical direction of the feature map; W is the number of pixels in the horizontal direction of the feature map; κ is the input of each formula; α is the weight of a single linear layer; σ() is the activation function, which represents the importance of the channel;
[0100] E. Take the feature map Y1 output in step D as input, and after 3×3 convolution and Sigmoid function, finally get the segmentation result of the input image;
[0101] F. For each image, the loss function L is calculated using the following formula:
[0102] L=L Dice +L Focal
[0103] Among them, L Dice is the Dice loss function; L Focal is the Focal loss function;
[0104] S4. The output results are stitched and restored to obtain the segmentation result of the original image, and building image segmentation is performed on the actual river and lake supervision remote sensing image.
[0105] like Figure 1 As shown in the figure, by collecting remote sensing images along the river and lake, after cropping and manual annotation, input into the deep learning framework designed by the present invention for training, a building image segmentation model is obtained. The remote sensing images to be segmented are cropped and input into the trained model to obtain the building image segmentation result, and finally the original scale remote sensing images are restored by image stitching. Among them, the deep learning framework designed by the present invention is as follows Figure 2 As shown in the figure, the network takes a remote sensing image of a specific scale as input, downsamples it through a dynamic residual short-circuit module in the encoder to obtain feature maps of different scales; inputs the feature maps of different scales into the multi-scale feature channel cross-fusion Transformer, and outputs the input feature map of the decoder; in the decoder, it is gradually upsampled through dual feature channel fusion attention, bilinear interpolation and dynamic residual short-circuit module, and finally obtains the segmentation result through 3×3 convolution and Sigmoid activation function.
[0106] This embodiment uses the HRIBuilding building dataset within the river and lake management scope collected and labeled in the river and lake supervision work scenario to evaluate the accuracy of building extraction by this method, and fully verifies the effectiveness of the various modules and deep learning framework proposed in this method through ablation experiments and comparative tests.
[0107] (1) Dataset description
[0108] The HRI Building dataset mainly includes remote sensing image data collected along the shores of Hongze Lake and Gaoyou Lake in Jiangsu Province in the second half of 2023. The data collection equipment is a DJI M30 series drone equipped with an M30T camera. The data are all three-channel RGB images of 3840×2160 pixels. In addition, the dataset also includes drone aerial RGB images of some river supervision work scenarios in Jiangsu Province in 2023, with a wealth of building samples within the scope of river and lake management. Figure 6 As shown in the figure, the dataset includes a variety of samples from river and lake supervision work scenarios, such as waterside buildings, different degrees of water mirror reflection, different numbers of ships near the shore, buildings with different distribution densities, and building images under different lighting conditions.
[0109] In order to make full use of remote sensing images and highlight the segmentation effect of the model, the original images were cropped to 512 × 512 pixels. For rows and columns with edge positions less than 512, the reciprocal 512 pixels were used to form the cropping area. Finally, images without buildings were proposed, and 817 images were retained as the data set.
[0110] (2) Accuracy evaluation index
[0111] In order to reasonably and objectively evaluate the segmentation performance of each CNN and Transformer model on the building dataset, this embodiment uses six indicators based on the confusion matrix, including Precision, Recall, F1 value, Kappa coefficient, Intersection over Union (IoU) and Overall Accuracy (OA). The calculation formulas of the above indicators are as follows:
[0112]
[0113]
[0114] Wherein, TP is the total number of pixels that are actually buildings and the segmentation result is also the building; FP is the total number of pixels that are actually background but the segmentation result is the building; FN is the total number of pixels that are actually buildings but the segmentation result is the background; TN is the total number of pixels that are actually background and the segmentation result is also the background; po is the number of pixels correctly segmented for each class divided by the total number of pixels; pe is the product of the actual number of pixels and the number of segmented pixels corresponding to all categories divided by the square of the total number of pixels.
[0115] (3) Hyperparameter setting
[0116] The model was trained on Google Colaboratory platform with A100 (40G memory) GPU, python 3.7 and Pytorch deep learning framework. The total number of training epochs is 100.
[0117] Since a small amount of data will lead to poor network approximation and underfitting, the images of the HRI Building dataset are augmented, mainly by vertical mirroring and rotation of 90°, 180° and 270°. The 3268 images obtained are randomly divided into training set, validation set and test set in a ratio of 8:1:1. The first 50 epochs of training are the freezing stage, freezing the backbone feature network. This backbone freezing method can reduce the training time. The last 50 epochs are the thawing stage. The optimizer is the Adam optimizer, the batch size is set to 2, the initial learning rate is 1e-4, and the learning rate is decayed in combination with the cosine annealing strategy to skip the local optimum.
[0118] (4) Ablation experiment
[0119] In order to evaluate the effectiveness of the dynamic residual short circuit module (RDSC), multi-scale feature channel cross fusion Transformer (MCCT) and dual feature channel fusion attention (DCF), ablation experiments were performed on the HRI Building dataset. The basic model adopts a simplified U-Net network with a depth of 5 layers. In order to better reflect the building segmentation effects of different modules and network models in various working scenarios, the following six scenarios are set: A. With strong water mirror reflection interference; B. With weak water mirror reflection interference; C. The building is adjacent to water but has no water mirror reflection; D. Complex distribution buildings with good lighting; E. Simple distribution buildings under dim conditions; F. Complex distribution buildings under dim conditions. Figure 7 In the figure, (a) is the original image after cropping; (b) is the label result of the original image; (c) is the basic network without adding any modules; (d) is the model after adding the RDSC module, verifying the effectiveness of RDSC; (e) is the model after adding the RDSC and MCCT modules, verifying the effectiveness of MCCT; (f) is the model after adding the RDSC, MCCT and DCF modules, verifying the effectiveness of the DCF module.
[0120] From Table 1 and Figure 7It can be seen that after adding the RDSC module, the improvement of each model index exceeds 1%, which can greatly reduce the erroneous extraction and segmentation of ships in scenario C, and increase the segmentation accuracy and completeness of multi-scale buildings in other scenarios; after further adding the MCCT module, various evaluation indicators have also been improved, the model's erroneous segmentation of ships has been further reduced, and the erroneous segmentation of strong water body mirror reflections has been more concentrated. The building targets missed in the left box of scenario D can be detected, and the segmentation accuracy and completeness of buildings in all scenarios are further improved; after further adding the DCF module, various evaluation indicators are still greatly improved, eliminating the influence of strong water body mirror reflections in scenario A and ships in scenario C on building segmentation. At the same time, the edges and internal details of buildings in each scenario are further portrayed, and the accuracy and completeness of building segmentation are further improved.
[0121] Table 1 Ablation test evaluation results on HRI Building dataset
[0122]
[0123] (5) Comparative experiment
[0124] The visualization segmentation results of the method of the present invention and the existing eight methods are shown in Figure 8 shown. Figure 8 (a) is the cropped image; (b) is the building annotation result of image (a); (c) is the result of the FCN method; (d) is the result of the U-Net method; (e) is the result of the U-Net++ method; (f) is the result of the Swin-UNet method; (g) is the result of the ACC-UNet method; (h) is the result of the CSC-UNet method; (i) is the result of the UCTransNet method; (j) is the result of the DTA-UNet method; (k) is the result of the method proposed in this invention.
[0125] From Table 2 and Figure 8 It can be seen that compared with other methods, the method of the present invention shows better building segmentation performance than other methods. The method of the present invention can not only basically eliminate the influence of water body mirror reflection, but also eliminate the influence of ships on building segmentation, and improve the integrity of building segmentation results within the scope of river and lake management, as well as the edge and internal details of the building.
[0126] Table 2 Comparative experimental evaluation results on the HRI Building dataset
[0127]
[0128]
Claims
1. A river and lake shore building image segmentation method based on fusion of CNN and Transformer, characterized in that: The following steps are involved: S1. Obtain remote sensing images of buildings within the river and lake management area; S2, crop the acquired remote sensing images, annotate the building targets, and then construct the data set; S3, based on the U-Net network architecture, combined with the dynamic residual short-circuit module, multi-scale feature channel cross-fusion Transformer, and dual feature channel fusion attention, a river and lake shore building image segmentation model is constructed; S4. The output results are stitched and restored to obtain the segmentation result of the original image, and building image segmentation is performed on the actual river and lake supervision remote sensing image.
2. The method for segmenting river and lake shore building images based on fusion of CNN and Transformer according to claim 1 is characterized in that: S2 includes S2-1. Screen remote sensing images with buildings; S2-2, cropping each remote sensing image into 512×512 pixels, and for rows and columns whose edge positions are less than 512 pixels, forming a cropping area with the reciprocal 512 pixels; S2-3, using an image annotation tool to manually annotate the cropped image; S2-4. Divide the labeled images into training set, validation set and test set in a ratio of 8:1:
1.
3. The river and lake shore building image segmentation method based on fusion of CNN and Transformer according to claim 1 is characterized in that: S3 specifically includes: S3-1, using the U-Net network as the basic network, including an encoder and decoder with a depth of five layers, and four skip connections connecting the first four feature maps respectively; S3-2, after the image T0 is input, a dynamic residual short-circuit module is added to expand the feature map channel, and the output feature map is T1; after each maximum pooling layer of the encoder structure, a dynamic residual short-circuit module is added, and the output feature maps are T2, T3, T4, and T5 respectively; S3-3, the output features are skipped through the multi-scale feature channel cross-fusion Transformer, and the output feature maps O1, O2, O3, O4 and O5 are output; S3-4, take O1~O5 as the input of the decoder; put the feature maps O5 and O4 at the same level, perform feature fusion through dual feature channel fusion attention, and then output it as feature map Y4 through bilinear interpolation and dynamic residual short-circuit module; take feature map Y4 and O3 as input, perform feature fusion through dual feature channel fusion attention, and then output it as feature map Y3 through bilinear interpolation and dynamic residual short-circuit module; take feature map Y3 and O2 as input, perform feature fusion through dual feature channel fusion attention, and then output it as feature map Y2 through bilinear interpolation and dynamic residual short-circuit module; take feature map Y2 and O1 as input, perform feature fusion through dual feature channel fusion attention, and then output it as feature map Y1 through bilinear interpolation and dynamic residual short-circuit module; S3-5, taking the feature map Y1 as input, after convolution and Sigmoid function, finally obtaining the segmentation result of the input image; S3-6. For each image, the loss function L is calculated using the following formula: L=L Dice +L Focal Among them, L Dice is the Dice loss function; L Focal is the Focal loss function.
4. The method for segmenting river and lake shore building images based on fusion of CNN and Transformer according to claim 3 is characterized in that: The dynamic residual short-circuit module in S3-2 includes a dynamic convolution block and a residual module; The dynamic convolution block is based on K linear functions Perform dynamic nonlinear aggregation; the calculation formula of the dynamic convolution block is: y=ReLU(W T (x)x+b(x)) Among them, π k is the kth linear function The attention weight ranges from [0,1], and the sum of weights is 1; W is the weight matrix; b is the bias vector ReLU; is the activation function; x is the input feature; y is the output feature; The residual module is used to combine the dynamic convolution block and the traditional convolution. The image T0 is processed through two paths. The first path is first processed by a 3×3 dynamic convolution block, then normalized by Batch Norm, activated by the ReLU function, and then processed by 3×3 convolution, and then processed by Batch Norm and ReLU again. The second path is that T0 is first processed by a 1×1 dynamic convolution block, and then normalized by Batch Norm. Finally, the outputs of the two paths are concatenated and ReLU activation is performed to obtain the feature map T1 of the next layer as the output.
5. The river and lake shore building image segmentation method based on fusion of CNN and Transformer according to claim 3 is characterized in that: S3-3 is specifically: Take the feature maps T1, T2, T3, T4 and T5 as input, normalize them respectively and then perform feature fusion. The formula is: S=Concat[δ(T1), δ(T2), δ(T3), δ(T4), δ(T5)]; Among them, δ( ) is the Layer Norm normalization operation; Concat[] is the feature fusion operation; S is the set embedding layer; Normalize the sequence T i As the input of the multi-head channel attention mechanism, and with five different input sequences T i and 1 set embedding layer S to transform into Q i , K, V matrix, the conversion formula is: Q i =δ(T i )×W Qi K=S×W K H=S×W V Among them, W Qi , W K , W V are the weights of different inputs; The input matrix is further processed to obtain the output matrix MHCA of the channel attention mechanism: Among them, ψ( ) is the Instance Norm normalization function; σ( ) is the Sigmoid function; n is the number of feature inputs; After the following formula is repeatedly processed four times, the feature maps O1, O2, O3, O4 and O5 are obtained as the output of the multi-scale feature channel cross-fusion Transformer: O i =MHCA i +MLP[δ(Q i +MHCA i )]; Among them, MLP is a multi-layer perceptron operation.
6. The river and lake shore building image segmentation method based on fusion of CNN and Transformer according to claim 3 is characterized in that: The dual feature channel fusion attention in S3-4 specifically includes: Take two feature maps X at the same level i , Y i As input, the two feature maps are globally averaged and pooled to obtain the feature vector f(X i ) and f(Y i ), and then use a single linear layer and Sigmoid function to construct the channel feature map; the two feature maps ζ(X i ) and ζ(Y i ), they are calibrated, activated and re-fused with the initial feature maps to obtain the output of dual-feature channel fusion attention; The formula for dual feature channel fusion attention is as follows: G(κ)=σ[ζ(X i )+ζ(Y i )]×(X i +Y i ) Among them, κ is the input feature; G( ) is the output feature; k is the feature of the kth channel; H is the number of pixels in the vertical direction of the feature map; W is the number of pixels in the horizontal direction of the feature map; κ is the input of each formula; α is the weight of a single linear layer; σ( ) is the activation function, which represents the importance of the channel.