Railway line tree and lake segmentation method based on remote sensing image

Abstract

The application discloses a railway along the line forest and lake segmentation method based on remote sensing images, comprising: preparing forest and lake data sets; constructing a semantic segmentation network model; using the obtained data set to train the network model in the established semantic segmentation network model; for the image to be segmented, using the trained network model to predict and generate a label image. The application innovatively designs a multi-branch residual weighted module, a context feature refinement module and a high-low feature fusion module, and applies these modules to the image recognition and segmentation of the forest and lake along the railway, effectively improves the image segmentation precision, realizes the fine extraction of the typical topography along the railway, accurately identifies the forest and lake along the railway, can improve the wind speed prediction precision of the environment along the railway, and then comprehensively considers the influence of the terrain and the wind speed, constructs a perfect foreign matter invasion prevention system, and provides protection for the safe operation of the railway.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, and relates to semantic segmentation algorithm technology, in particular to a railway line forest and lake segmentation method based on remote sensing images. BACKGROUND

[0002] By the end of 2021, the total mileage of national railways has broken through 150,000 kilometers, of which more than 40,000 kilometers are high-speed railways. China's railways, especially China's high-speed railways, have become an important infrastructure and a national artery, and play a key role in the comprehensive transportation system. Such a complex railway transportation line and passenger volume poses a severe challenge to the safety of China's railway operation. As one of the important reasons affecting the safe operation of railways, foreign matter intrusion is mostly caused by floating objects such as plastic greenhouses, mulching films, plastic bags in residential areas, and color steel tiles on construction sites, which intrude into the railway buildings under the action of strong winds and affect the safety of train operation. However, the complex terrain along the railway is the main reason for the strong wind. Therefore, in view of the problem of foreign matter intrusion, it is necessary to accurately identify the terrain along the railway to improve the accuracy of wind speed prediction, thereby preventing the occurrence of foreign matter intrusion and forming a railway emergency plan focusing on "prevention-avoidance-rescue" as the core, and focusing on the "prevention" aspect.

[0003] The identification of the terrain along the railway can be classified into the field of land cover classification. According to relevant information, some scholars have attempted to introduce semantic segmentation into the research of land cover classification. However, the problem of land cover classification algorithm based on semantic segmentation is that the model cannot restore the detailed information of the remote sensing image after upsampling due to the small feature resolution after downsampling. Moreover, the boundaries of the targets are mostly winding and tortuous, which poses a major challenge to the segmentation process. In addition, when fusing features during upsampling, the current mainstream semantic segmentation algorithm cannot effectively filter out useless or even contradictory feature information, so the land cover classification algorithm based on semantic segmentation still needs to improve the feature extraction, accurate positioning of target boundaries, and upsampling feature fusion of the model.

[0004] Accurate identification of forests and lakes along the railway can improve the accuracy of wind speed prediction of the environment along the railway, and then comprehensively consider the influence of terrain and wind speed to build a perfect foreign matter intrusion prevention system to protect the safe operation of railways. Therefore, identifying forests and lakes along the railway and accurately dividing them are important prerequisites for building a foreign matter intrusion prevention system. SUMMARY

[0005] The purpose of the present application is to overcome the deficiencies in the prior art, and to provide a railway line forest and lake segmentation method based on remote sensing images, which realizes the fine extraction of typical terrain along the railway.

[0006] Technical solution: To achieve the above object, the application provides a railway along the line forest and lake segmentation method based on remote sensing image, comprising the following steps:

[0007] S1: making forest and lake data set;

[0008] S2: constructing semantic segmentation network model;

[0009] S3: using the data set obtained in step S1, training the network model in the semantic segmentation network model established in step S2;

[0010] S4: for the image to be segmented, using the trained network model to predict and generate the label image of forest and lake.

[0011] Further, the data set making process in step S1 is as follows: the challenging data set is very important for remote sensing image change detection task, considering that there are few public data sets for forest and lake along the railway, the application selects pictures related to forest and lake from LandCover data set, and integrates them into a new data set, because the image size in the original data set is 9000*9000*3 (length*width*channel) pixels, limited by GPU, the original picture is uniformly cut into 256*256*3 (length*width*channel), finally 2500 images of 256*256*3 (length*width*channel) size are obtained, the cutting mode is sliding cover cutting, then all the pictures are divided into training set (2000 pictures) and verification set (500 pictures) with the ratio of 4:1, wherein the label of forest is green, the label of lake is gray, and the label of background is black.

[0012] Further, the semantic segmentation network model in step S2 includes an encoder part and a decoder part, the encoder part includes a double-branch feature extraction structure, a context feature refinement (CFR) module, and a plurality of bottleneck modules (BB);

[0013] The double-branch feature extraction structure includes an improved ResNet34 backbone network and a multi-branch residual weighted (MBRW) module, which adopts layer-by-layer downsampling mode to extract channel information and spatial information respectively, and fuses the outputs of the same level ResNet34 backbone block and the multi-branch residual weighted module (MBRW) together to establish the connection between different pixel points;

[0014] The context feature refinement module is used to extract context information, and the feature Figure One is divided into two, as the input of the two branches of the decoder;

[0015] The bottleneck module (BB) and the fusion operation are used for feature enhancement and fusion of different feature layer information;

[0016] The decoder part includes a high-low feature fusion module (HLFF) and a residual module, the high-low feature fusion module (HLFF) is used to fully fuse the features at each level, restore the original size of the feature map layer by layer, and finally use a residual module for channel ladder refinement at the output end.

[0017] Further, feature extraction is very important in the process of semantic segmentation, and selecting an appropriate feature extraction network can effectively extract feature information. The design of the residual block in ResNet allows the establishment of horizontal connections between different layers, which helps to pass and integrate channel information in the network. Moreover, the introduction of residual connection alleviates the problem of gradient disappearance, which helps to preserve the integrity of channel information. The present application selects ResNet34 as the backbone network. Compared with deeper ResNet models, ResNet34 has fewer parameter quantities, which makes it more lightweight, but it can still effectively extract channel information. Compared with the original ResNet34, the present application makes some modifications to it. Considering that the boundary information of forests and lakes is complex, a larger down-sampling rate may also lead to the loss of some detailed information. Therefore, the improved ResNet34 network is used in the semantic segmentation network model, the improved ResNet34 network includes 5 backbone blocks, and the down-sampling of the last backbone block is removed, so that the image size of the last layer and the second-to-last layer is the same. The improved ResNet34 network adopts 16 times down-sampling as a whole, and the channel number is the same as that of the original ResNet34 network.

[0018] Further, the bottleneck module in the semantic segmentation network model includes bottleneck module a and bottleneck module b, which are used for down-sampling of the MBRW branch and feature enhancement at each level, respectively. The bottleneck module is composed of two 1x1 convolution and 3x3 convolution, wherein the 1x1 convolution is used to adjust the channel number, and the 3x3 convolution is used to strengthen feature extraction, which can reduce the parameter quantity and computational complexity of the network, while maintaining high model performance, making the model more generalizable and robust. When the step of the 3x3 convolution is 2, it also plays a role in down-sampling.

[0019] Further, the design concept of the multi-branch residual weighted module MBRW is: since lakes and forests exhibit different appearances and features at different scales, it is necessary to use multi-scale feature extraction to obtain more global and local spatial information. The whole module adopts the idea of multi-scale convolution, and the use of different size convolution kernels can better capture the detailed information of the image and improve the network's understanding ability of large image regions. However, considering that the network depth is not enough, the model may have underfitting problem, therefore, the present application uses small size convolution kernel stacking to achieve the same receptive field as large size convolution kernel, which ensures the depth of MBRW while realizing multi-scale convolution.

[0020] The design and operation method of the multi-branch residual weighting module MBRW is as follows:

[0021] A1: The module as a whole adopts a multi-branch structure, and each branch extracts image features in the form of a bottleneck structure; in each branch, the channel number is first compressed to 1 / 2 by using a 1x1 convolution, then a plurality of 3x3 convolutions are used to obtain feature information of different receptive fields, and finally a 1x1 convolution is used to adjust the channels; and the output of the last 3x3 convolution of the previous branch is the input of the next branch, which greatly reduces the calculation amount of the module. At the same time, a pyramid split attention mechanism (Pyramid Split Attention) is embedded after the last 3x3 convolution of each branch, which can help the model focus better on features at different scales; the mathematical expression of each branch is as follows:

[0022] X = Conv1d(x)

[0023] Y n = K n *(K n-1 *(...K2*(K1*X))))n = 1,2,3

[0024] M n = Conv1d(Psa(Y n ))n = 1,2,3

[0025] Where x represents the input tensor, X represents the output result of the first convolution; K n represents the nth 3x3 convolution; Y n represents the output of the last 3x3 convolution in the nth branch; M n represents the output result after fusion of all branches; Psa represents the use of the pyramid split attention mechanism, and Conv1d represents the one-dimensional convolution operation;

[0026] A2: After adding the outputs of all branches, adjusting the channels by using a 1x1 convolution, and performing residual connection with the input image, a global context vector is multiplied, and the vector is obtained by performing global average pooling on the output of the first branch and then performing Sigmoid activation;

[0027] The module output calculation formula is as follows:

[0028] P = Conv1d(concat(Y1, Y2, Y3))

[0029] W = Sigmoid(GAP(Y1))

[0030] output MBRW = (X+P) x W

[0031] wherein, Conv1d represents a one-dimensional convolution operation; concat represents connection of multiple tensors in the row channel dimension; W represents a weight vector; Sigmoid represents a nonlinear activation function Sigmoid; GAP represents a global average pooling operation; output MBRW represents the output of the MBRW.

[0032] Further, the design concept of the context feature refinement module is that: in the identification process of the forest and the lake, since the boundary often presents an irregular shape and is interfered by many similar backgrounds, the context refinement module is proposed. The module utilizes the global average pooling and the global maximum pooling, can learn the mutual relationship between the channels of the feature map, and adaptively adjusts the weight of the channel feature, so that the model pays more attention to important feature information. The convolution of different scales and the strip pooling not only refine the boundary information and improve the boundary segmentation accuracy, but also more comprehensively perceive the context semantic information of the input image, so as to improve the utilization efficiency of information.

[0033] The design and operation method of the context feature refinement module are as follows:

[0034] B1: first, adjust the channel by using 1*1 convolution, use different scale convolutions in the left half to obtain different scale context information, improve the expression ability of the model, then use strip pooling to refine and extract the edge information in the feature map, help the network to learn more fine boundary feature information, and improve the performance of the network, finally splice and output the results;

[0035] The calculation formula of the left half is as follows:

[0036] A=Conv1d(x)

[0037]

[0038]

[0039] B i =B i1 +B i2

[0040] C=Conv1d(concat(B1,B2))

[0041] wherein, Conv1d represents a one-dimensional convolution operation; A represents the output of the first 1*1 convolution of the module; Conv represents a convolution of different scales; k i represents the size of the convolution kernel or the pooling kernel; Avg represents a strip pooling of different sizes; B ijrepresents the output of the i-th convolution and the j-th strip pooling; concat represents the concatenation operation; C represents the output of the left half of the module;

[0042] B2: the right half of the model respectively performs global average pooling and global maximum pooling on the input feature map to obtain the average value and the maximum value of each channel, then uses a 1x1 convolution with a Relu nonlinear transformation for dimension reduction, and then uses a 1x1 convolution with a Sigmoid activation function for dimension increase to obtain the weight of each channel, and finally combines and outputs the two weights;

[0043] The combination of global average pooling and global maximum pooling can retain the rich information of the input feature map, and the two 1x1 convolutions use dimension reduction and dimension increase operations to reduce the calculation amount and the parameter amount and learn the linear relationship between the channels; the final output result of the module is calculated as follows:

[0044] V1 = Sigmoid(Conv1d(σ(Conv1d(GAP(A)))))

[0045] V2 = Sigmoid(Conv1d(σ(Conv1d(MAX(A)))))

[0046] output CFR = C x (V1 + V2)

[0047] Wherein, A represents the output of the first 1x1 convolution of the module; sigma represents the nonlinear activation function Relu, GAP represents global average pooling, MAX represents global maximum pooling; V1 represents the weight obtained by global average pooling; V2 represents the weight obtained by global maximum pooling; output CFR represents the output of the CFR.

[0048] Further, the design concept of the high-low feature fusion module is that: in the semantic segmentation task, the main role of the decoding end is to decode the high-level semantic information extracted in the encoder into low-level pixel information of the original input image. However, in the process of extracting the features of the trees and the lake, due to the tree shadow, the reflection of the lake surface and the complex background image, it is easy to cause ambiguity of the decoder, that is, it is difficult to accurately determine the trees and the lake. Moreover, in the encoding process, the structure of the double-branch feature extraction adopted by the present application extracts rich semantic information from each branch, and if the semantic information of the two branches is simply fused together in the decoding process, some effective information will be discarded as redundant information. Therefore, the present application proposes a high-low feature fusion module.

[0049] The high-low feature fusion module has three inputs, two of which are low-level semantic features of the same level, respectively from the output of the ResNet34 backbone block and the MBRW, and the other is a high-level semantic feature of the adjacent layer, including the output of the previous level CFR module or the output of the previous level HLFF module. The operation method of the high-low feature fusion module is as follows:

[0050] C1: In the high-low feature fusion module, a double-branch mode is adopted for each input feature layer; for the two low-level semantic information, they are fully fused, and for the low-level features output by the MBRW module, the semantic information in the first branch is refined by a 3*3 convolution layer; the 1*3 and 3*1 strip convolution in the second branch can better capture the horizontal and vertical features in the feature map; for the low-level features output by the ResNet34 backbone block, a 3*3 dilated convolution is introduced in the first branch to refine the feature layer while increasing the receptive field area, which is added to the first branch of the MBRW to realize multi-scale fusion; the second branch of the ResNet34 uses a channel attention mechanism to obtain a weight, which is multiplied by the output result of the second branch of the MBRW, so that the key features obtain a higher weight, improving the accuracy of the model; finally, the two results are added, and after adjusting the channels by a 1*1 convolution, the fused low-level semantic information is obtained;

[0051] C2: For the high-level semantic information, the first branch is weighted by a channel attention and multiplied by the fused low-level semantic information to ignore the noise and unnecessary details in the low-level semantic information; in the second branch, the high-level semantic information is upsampled to the same size as the low-level semantic information, and the channel number is adjusted by a 1*1 convolution, and the features are further extracted by a 3*3 convolution, and finally the output of the first branch is added to obtain the final result of the module.

[0052] Further, the training of the network model in step S3 comprises three parts of forward propagation, backward propagation and gradient descent in turn, and the training process is as follows:

[0053] D1: Forward propagation: transmit the input picture data to the model to obtain an output result;

[0054] D2: Backward propagation: the output result of the model and the true value are reversely calculated by a loss function to obtain the parameter partial derivative of each neuron;

[0055] D3: Gradient descent: after obtaining the partial derivative of each neuron in the backward propagation process, the model weight is finally updated by the gradient descent process, which descends along the inverse of the derivative of the maximum direction of the current position, and the descending step is determined by the learning rate.

[0056] Further, the loss function in step D2 is selected as a cross-entropy loss function, and the calculation formula is as follows:

[0057]

[0058] where y is the output tensor of the x network, and class is the real label.

[0059] The present application is based on the following research results: through the study of historical encroachment events of railways, wind speed data of weather stations and remote sensing images along the line, the two landforms that have the greatest impact on wind speed along the railway are forests and lakes. The present application takes forests and lakes as typical landforms to study, extracts forests and lakes in complex backgrounds, takes other targets in the background as black, and gives forests and lakes different colors, so as to mark forests and lakes along the railway in the remote sensing image.

[0060] Research shows that there are several problems to be solved in this scheme:

[0061] Problem one: the remote sensing image along the railway contains objects or landscapes of different scales, and the difference between the scales can be large. This leads to the model being unable to identify the same target of different scales, resulting in missed detection of the target and affecting the segmentation accuracy. Therefore, the different scale differences of forests, lakes and backgrounds need to be solved.

[0062] Problem two: in the process of identifying forests and lakes, the boundaries may be fuzzy or confused. For example, some forest boundaries may intersect or overlap with lawns, making it difficult for the model to accurately divide the boundaries. Improving the boundary ambiguity problem is a key challenge to improve the accuracy of the model.

[0063] Problem three: in some cases, the terrain along the railway needs to distinguish complex scenes, such as residential areas, lawns, wastelands, etc. These scenes have complex textures, shapes and spatial structures, and some have similar features to the identified targets, which can easily cause false detection of the target. Therefore, redundant background information needs to be removed in the identification process.

[0064] Therefore, the present application proposes the following three innovations in view of the above three main problems:

[0065] Innovation one: in view of problem one, a multi-branch residual weighted module is designed innovatively, which uses multiple 3x3 convolutions instead of large size convolutions to extract multi-scale feature information, ensures the amount of calculation while increasing the depth of the network, realizes multi-scale feature extraction, and introduces an attention mechanism to focus on important information in the feature map, improving the accuracy of the model.

[0066] Innovation two: in view of problem two, a context feature refinement module is designed innovatively, which fully extracts context information, introduces a strip pooling to refine the segmentation edges of forests and lakes, and a sub-channel attention mechanism to focus on important information in the map, making the model more effective in detecting boundary information.

[0067] Innovation three: in response to problem three, a high-low feature fusion module is designed innovatively, a multi-level feature fusion mode is adopted, low-level semantic information is fully fused, high-level features with rich semantics are used to guide low-level feature classification, redundant information in low-level semantic information is removed, and interference of complex background is avoided.

[0068] Beneficial effects: compared with the prior art, the present application innovatively designs a multi-branch residual weighted module, a context feature refinement module and a high-low feature fusion module, and applies these modules to image recognition and segmentation of trees and lakes along the railway, effectively improves the image segmentation precision, realizes fine extraction of trees and lakes along the railway, improves the wind speed prediction precision of the environment along the railway, and further comprehensively considers the influence of terrain and wind speed to build a perfect foreign matter invasion prevention system, and provides protection for safe operation of the railway. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 The network structure framework diagram of the semantic segmentation network model is shown in the figure.

[0070] Figure 2 The network structure framework diagram of the bottleneck module is shown in the figure.

[0071] Figure 3 The network structure diagram of the multi-branch residual weighted module is shown in the figure.

[0072] Figure 4 The network structure diagram of the context feature refinement module is shown in the figure.

[0073] Figure 5 The network structure diagram of the high-low feature fusion module is shown in the figure.

[0074] Figure 6 The label image of the trees and lakes obtained in the embodiment is shown in the figure. DETAILED DESCRIPTION

[0075] The present application will be further illustrated in conjunction with the drawings and specific embodiments, and it should be understood that these embodiments are only used to illustrate the present application and not used to limit the scope of the present application, and after reading the present application, various equivalent modifications of the present application by those skilled in the art fall within the scope defined by the appended claims.

[0076] As shown in the figure, Figure 1 The present application provides a railway tree and lake segmentation method based on remote sensing images, which comprises the following steps:

[0077] S1: make a tree and lake data set:

[0078] In this embodiment, pictures related to forests and lakes are selected from the LandCover dataset and integrated into a new dataset. Due to the GPU limitation, the original pictures are uniformly cropped to 256x256x3 (lengthxwidthxchannel) from 9000x9000x3 (lengthxwidthxchannel) in size. Finally, 2500 images of 256x256x3 (lengthxwidthxchannel) are obtained. The cropping method is sliding uncapped cutting. Then, all the pictures are divided into a training set (2000 pictures) and a validation set (500 pictures) at a ratio of 4:1. The label of the forest is green, the label of the lake is gray, and the label of the background is black.

[0079] S2: Constructing a semantic segmentation network model

[0080] As shown in Figure 1 , the semantic segmentation network model includes an encoder part (Encoder) and a decoder part (Decoder). The encoder part includes a double-branch feature extraction structure, a context feature refinement module (CFR), and seven bottleneck modules (BB).

[0081] The double-branch feature extraction structure includes an improved ResNet34 backbone network and a multi-branch residual weighted (MBRW) module. In this embodiment, the improved ResNet34 backbone block is a 5-layer structure composed of Layer1-Layer5. The down-sampling of Layer5 is removed, so that the image size of Layer5 and Layer4 is the same. The improved ResNet34 network adopts 16 times down-sampling, and the channel number is the same as the original ResNet34 network. The multi-branch residual weighted module is a 4-layer structure composed of MBRW1-MBRW4, each layer down-samples by 2 times. The ResNet34 backbone network and the multi-branch residual weighted module use layer-by-layer down-sampling to extract channel information and spatial information, respectively, and fuse the outputs of the same level of ResNet34 backbone network and the multi-branch residual weighted module to establish the connection between different pixel points.

[0082] As shown in Figure 2 , the bottleneck module (BB) and the fusion operation are used for feature enhancement and fusion of different feature layer information. In this embodiment, the bottleneck module includes bottleneck module a Figure 2 and bottleneck module b Figure 2b) in the b), respectively, for the downsampling of the MBRW branch and the feature enhancement of each level, the bottleneck module is composed of two 1x1 convolutions and 3x3 convolutions, wherein the 1x1 convolution is used to adjust the number of channels, and the 3x3 convolution is used to enhance feature extraction, which can reduce the number of network parameters and computational complexity while maintaining high model performance, making the model more generalizable and robust. When the 3x3 convolution step is 2, it also plays a role in downsampling.

[0083] As shown in Figure 3 , the design and operation method of the multi-branch residual weighting module MBRW is as follows:

[0084] A1: The whole module adopts a multi-branch structure, and each branch extracts image features in the form of a bottleneck structure; in each branch, first use 1x1 convolution to compress the number of channels to 1 / 2 of the original, then use multiple 3x3 convolutions to obtain feature information of different receptive fields, and finally use 1x1 convolution to adjust the channel; and the output of the last 3x3 convolution of the previous branch is the input of the next branch, which greatly reduces the computational amount of the module. At the same time, Pyramid Split Attention is embedded after the last 3x3 convolution of each branch, which can help the model focus better on features at different scales; the mathematical expression of each branch is as follows:

[0085] X = Conv1d(x)

[0086] Y n = K n *(K n-1 *(...K2*(K1*X))))n = 1,2,3

[0087] M n = Conv1d(Psa(Y n ))n = 1,2,3

[0088] Where x represents the input tensor, X represents the output result of the first convolution; K n represents the nth 3x3 convolution; Y n represents the output of the last 3x3 convolution in the nth branch; M n represents the output result after fusion of all branches; Psa represents the use of Pyramid Split Attention, and Conv1d represents one-dimensional convolution operation.

[0089] A2: After adding the outputs of all branches, adjust the channel through 1x1 convolution, then perform residual connection with the input image, and finally multiply a global context vector, which is obtained by performing global average pooling on the output of the first branch and then activating it through Sigmoid;

[0090] The module output calculation formula is as follows:

[0091] P = Conv1d(concat(Y1, Y2, Y3))

[0092] W = Sigmoid(GAP(Y1))

[0093] output MBRW = (X + P) x W

[0094] Wherein, Conv1d represents a one-dimensional convolution operation; concat represents the connection of multiple tensors in the row channel dimension; W represents a weight vector; Sigmoid represents a nonlinear activation function Sigmoid; GAP represents a global average pooling operation; output MBRW represents the output of the MBRW module.

[0095] As Figure 4 shown, the context feature refinement module is used to extract context information and divide the features Figure One into two, as the input of the two branches of the decoder.

[0096] The design and operation method of the context feature refinement module are as follows:

[0097] B1: First, adjust the channel with 1x1 convolution, use different scale convolutions in the left half to obtain different scale context information, improve the expression ability of the model, then use strip pooling to refine and extract the edge information in the feature map, help the network learn more fine boundary feature information, and improve the performance of the network, finally splice the results and output;

[0098] The calculation formula of the left half is as follows:

[0099] A = Conv1d(x)q

[0100]

[0101]

[0102] B i = B i1 + B i2

[0103] C = Conv1d(concat(B1, B2))

[0104] Wherein, Conv1d represents a one-dimensional convolution operation; A represents the output of the first 1x1 convolution of the module; Conv represents a convolution of different scales; k idenotes the size of the convolution kernel or the pooling kernel; Avg denotes the strip pooling of different sizes; concat denotes the concatenation operation; C denotes the output of the left half of the module;

[0105] B2: The right half of the model performs global average pooling and global maximum pooling on the input feature map respectively, obtains the average value and the maximum value of each channel, then uses a 1x1 convolution with a Relu nonlinear transformation for dimension reduction, and then uses a 1x1 convolution with a Sigmoid activation function for dimension increase to obtain the weight of each channel, and finally combines the two weights and outputs;

[0106] The combination of global average pooling and global maximum pooling can retain the rich information of the input feature map, and the two 1x1 convolutions use dimension reduction and dimension increase operations to reduce the calculation amount and the parameter amount and learn the linear relationship between channels; The final output result of the module is calculated as follows:

[0107] V1 = Sigmoid(Conv1d(σ(Conv1d(GAP(A)))))

[0108] V2 = Sigmoid(Conv1d(σ(Conv1d(MAX(A)))))

[0109] output CFR = C x (V1 + V2)

[0110] Wherein, A denotes the output of the first 1x1 convolution of the module; σ denotes the nonlinear activation function Relu, GAP denotes global average pooling, MAX denotes global maximum pooling; V1 denotes the weight obtained by global average pooling; V2 denotes the weight obtained by global maximum pooling; output CFR denotes the output of the CFR.

[0111] The decoder part includes a high-low feature fusion module (HLFF) and a residual module, the decoder part uses the high-low feature fusion module (HLFF) to fully fuse the features of each level, restores the original size of the feature map layer by layer, and finally uses a residual module at the output end to refine the channel ladder.

[0112] As shown in Figure 5 , the high-low feature fusion module has three inputs, two of which are low-level semantic features of the same level, respectively from the output of the ResNet34 backbone block and the MBRW, and the other is a high-level semantic feature of the adjacent layer, including the output of the previous level CFR module or the output of the previous level HLFF module, and the running method of the high-low feature fusion module is as follows:

[0113] C1: In the high-low feature fusion module, a double branch mode is adopted for each input feature layer; for two low-level semantic information, they are fully fused, and the low-level features output by the MBRW module are refined in the first branch through a 3x3 convolution layer; the 1x3 and 3x1 strip convolution in the second branch can better capture the horizontal and vertical features in the feature map; for the low-level features output by the ResNet34 backbone block, a 3x3 dilated convolution is introduced in the first branch to refine the features while increasing the receptive field area, which is added to the first branch of the MBRW to realize multi-scale fusion; the second branch of the ResNet34 uses the channel attention mechanism to obtain a weight, which is multiplied by the output result of the second branch of the MBRW, so that the key features obtain a higher weight and improve the accuracy of the model; finally, the two results are added, and after adjusting the channels through a 1x1 convolution, the fused low-level semantic information is obtained;

[0114] C2: For high-level semantic information, the first branch is multiplied by the fused low-level semantic information after using the channel attention weighting to ignore the noise and unnecessary details in the low-level semantic information; in the second branch, the high-level semantic information is upsampled to the same size as the low-level semantic information, and the channel number is adjusted by a 1x1 convolution, and the features are further extracted by a 3x3 convolution, and finally added to the output of the first branch to obtain the final result of the module.

[0115] S3: Using the data set obtained in step S1, the network model is trained in the semantic segmentation network model established in step S2:

[0116] The training of the network model includes three parts of forward propagation, backward propagation and gradient descent in turn, and the training process is as follows:

[0117] D1: Forward propagation: transmit the input picture data to the model to obtain an output result;

[0118] D2: Backward propagation: the output result of the model and the true value are inversely calculated through the loss function to obtain the parameter partial derivative of each neuron;

[0119] D3: Gradient descent: after obtaining the partial derivative of each neuron in the backward propagation process, the model weight is finally updated through the gradient descent process, which descends along the inverse of the derivative of the maximum direction of the current position, and the descending step is determined by the learning rate.

[0120] Based on the above training method, the following conditions are used for training in this embodiment:

[0121] The experimental platform selects Nvidia GeForce RTX 3060 and the Pytorch environment. The step learning rate (StepLR) strategy is adopted in the training process, and the learning rate is reduced with the increase of the training number to achieve better training effect. The initial learning rate is set to 0.001, the decay coefficient is 0.98, the learning rate is updated every 3 times of training, and a total of 300 times of training are performed. The learning rate calculation formula is as follows:

[0122]

[0123] wherein is the learning rate of the Nth training, is the initial learning rate, β is the decay coefficient, and s is the update interval.

[0124] The loss function selects the cross-entropy loss function, and the calculation formula is as follows:

[0125]

[0126] wherein is the output tensor of the x network, and class is the real label.

[0127] Since the Adam algorithm has a certain range of learning rate after bias correction in each iteration, the parameters are relatively stable, so the Adam algorithm is used as the optimizer in the training process, wherein the learning rate (lr) parameter is set to 0.001, betas=(0.9, 0.999), and the weight decay is 1e-4.

[0128] S4: For the image to be segmented, the trained network model is used for prediction and label image of greenhouse and dust screen is generated;

[0129] In this embodiment, a picture to be segmented is input into the trained semantic segmentation network model, and the label image of forest and lake is output after the trained weight extracts features and performs encoding and decoding operation, as shown in Figure 6 .

[0130] In order to verify the actual effect of the method of the application, the segmentation performance of the picture is tested and evaluated, as follows:

[0131] The pixel accuracy (PA), class average pixel accuracy (MPA), F1, weighted intersection over union (FWIOU), and average intersection over union (MIOU) evaluation indexes are selected to evaluate the performance of the model in actual application. The formula of various indexes is as follows.

[0132]

[0133]

[0134]

[0135]

[0136]

[0137] wherein k represents the number of the two categories of forest and lake (excluding background), P ii represents the number of pixels belonging to category i and predicted as category i, p ij represents the number of pixels belonging to category i but predicted as category j, p ji represents the number of pixels belonging to category j but predicted as category i.

[0138] The model provided by the present application and the existing model were tested in the embodiment, and the specific test data are shown in Table 1.

[0139] Table 1

[0140] Method PA MPA F1 MIOU FWIOU HRNet 89.85 89.85 87.63 78.54 82.33 PSPNet 90.29 90.29 88.78 79.63 83.79 UNet 90.60 90.60 88.68 79.87 83.75 FCN-16 90.74 90.74 88.42 80.03 84.44 DeeplabV3+ 91.27 91.27 89.63 81.74 85.16 BiseNet 91.39 91.39 90.18 81.85 86.98 DenseASPP 92.57 92.22 90.85 82.28 87.72 Ours 94.85 93.39 90.68 84.77 89.21

[0141] As can be seen from Table 1, the model of the present application is superior to the existing model in the five evaluation index data of pixel accuracy (PA), category average pixel accuracy (MPA), F1, weighted intersection over union (FWIOU) and average intersection over union (MIOU), which verifies that the present application can effectively improve the segmentation precision of the image and realize fine extraction of forests and lakes along the railway.

Claims

1. A method for segmenting trees and lakes along a railway based on remote sensing images, characterized in that, The method comprises the following steps: S1: making a forest and lake data set; S2: constructing a semantic segmentation network model; S3: training the network model in the semantic segmentation network model established in step S2 by using the data set obtained in step S1; S4: for an image to be segmented, using the trained network model to make a prediction and generating a label image; The semantic segmentation network model in step S2 comprises an encoder part and a decoder part, the encoder part comprises a double-branch feature extraction structure, a context feature refinement (CFR) module and a plurality of bottleneck modules (BB); The double-branch feature extraction structure comprises an improved ResNet34 main block and a multi-branch residual weighted module (MBRW), adopts a layer-by-layer down-sampling manner, extracts channel information and spatial information respectively, and fuses the outputs of the ResNet34 main block and the MBRW of the same level together; The context feature refinement module is used for extracting context information and dividing the feature map into two parts as inputs of two branches of the decoder; The bottleneck module BB and the fusion operation are used for feature enhancement and fusion of information of different feature layers; The decoder part comprises a high-low feature fusion module (HLFF) and a residual module, the decoder part uses the high-low feature fusion module HLFF to fully fuse features of different levels, restores the original size of the feature map layer by layer, and finally uses a residual module at the output end to refine the channel ladder; The design and operation method of the multi-branch residual weighted module are as follows: A1: the module as a whole adopts a multi-branch structure, each branch extracts image features in the form of a bottleneck structure; in each branch, first, 1x1 convolution is used to compress the channel number to 1 / 2 of the original, then a plurality of 3x3 convolutions are used to obtain feature information of different receptive fields, and finally, 1x1 convolution is used to adjust the channel; and the output of the last 3x3 convolution of the previous branch is the input of the next branch, and at the same time, pyramid segmentation attention mechanism is embedded after the last 3x3 convolution of each branch; A2: after adding the outputs of all branches, the channel is adjusted through 1x1 convolution, then residual connection is performed with the input image, and finally a global context vector is multiplied, the vector is obtained by performing global average pooling on the output of the first branch and then performing Sigmoid activation; The design and operation method of the context feature refinement module are as follows: B1: first, the channel is adjusted by 1x1 convolution, the left half uses different scale convolutions to obtain rich context information, then edge information in the feature map is extracted by using strip pooling for refinement, helping the network to learn more fine boundary feature information, and finally the results are spliced and output; B2: the right half of the model performs global average pooling and global maximum pooling on the input feature map respectively to obtain the average value and the maximum value of each channel, then a 1x1 convolution with a Relu nonlinear transformation is used for dimension reduction, and then a 1x1 convolution with a Sigmoid activation function is used for dimension increase to obtain the weight of each channel, and finally the two weights are combined and output.

2. The method according to claim 1, wherein, The data set in the step S1 is made by selecting pictures of forest and lake from the LandCover data set, integrating the pictures into a new data set, uniformly cutting the original pictures by sliding uncovered cutting, and dividing all the pictures into a training set and a verification set, wherein the label of the forest is green, the label of the lake is gray, and the label of the background is black.

3. The method of claim 1, wherein the method is characterized by: The improved ResNet34 network in the semantic segmentation network model comprises five backbone blocks, the downsampling of the last backbone block is removed, and the improved ResNet34 network adopts 16 times of overall downsampling and has the same number of channels as the original ResNet34 network.

4. The method of claim 1, wherein the method is characterized by: The bottleneck module in the semantic segmentation network model comprises a bottleneck module a and a bottleneck module b, and is respectively used for downsampling of the MBRW branch and feature enhancement of each level, the bottleneck module is composed of two 1x1 convolution and 3x3 convolution, wherein the 1x1 convolution is used for adjusting the number of channels, and the 3x3 convolution is used for enhancing feature extraction, and when the step of the 3x3 convolution is 2, the 3x3 convolution also plays a role of downsampling.

5. The method of claim 1, wherein the method further comprises: The mathematical expression of each branch in the multi-branch residual weighted module is as follows: X = Conv1d(x) Y n = K n *(K n-1 *(...K2*(K1*X)))n = 1,2,3 M n = Conv1d(Psa(Y n ))n = 1,2,3 wherein x represents an input tensor, X represents the result of the first convolution output; K n represents the nth 3x3 convolution; Y n represents the output of the last 3x3 convolution in the nth branch; M n represents the output result after fusion of all branches; Psa represents the use of a pyramid split attention mechanism, and Conv1d represents a one-dimensional convolution operation; The output calculation formula of the multi-branch residual weighted module is as follows: P = Conv1d(concat(Y1, Y2, Y3)) W = Sigmoid(GAP(Y1)) output MBRW = (X+P) x W wherein Conv1d denotes a one-dimensional convolution operation; concat denotes concatenation of multiple tensors along the channel dimension; W denotes a weight vector; Sigmoid denotes a nonlinear activation function Sigmoid; GAP denotes a global average pooling operation; output MBRW denotes the output of the MBRW.

6. The method of claim 1, wherein the method further comprises: The calculation formula of the left half part in the context feature refinement module is as follows: A = Conv1d(x) B i = B i1 + B i2 C = Conv1d(concat(B1, B2)) wherein, Conv1d represents a one-dimensional convolution operation; A represents the output of the first 1x1 convolution of the module; Conv represents a convolution of different scales; k i represents the size of the convolution kernel or the pooling kernel; Avg represents a strip pooling of different sizes; B ij represents the output of the i-th convolution and the j-th strip pooling; concat represents a concatenation operation; C represents the output of the left half of the module; The final output result calculation process of the context feature refinement module is as follows: V1 = Sigmoid(Conv1d(sigma(Conv1d(GAP(A))))) V2 = Sigmoid(Conv1d(sigma(Conv1d(MAX(A))))) output CFR = C x (V1 + V2) wherein A represents the output of the first 1x1 convolution of the module; σ represents a nonlinear activation function Relu, GAP represents global average pooling, MAX represents global maximum pooling; V1 represents the weight obtained by global average pooling; V2 represents the weight obtained by global maximum pooling; output CFR represents the output of the CFR module.

7. The method of claim 1, wherein the method further comprises: The high-low feature fusion module has three inputs, two of which are low-level semantic features of the same level, respectively from the output of the ResNet34 backbone block and the MBRW, and the other is a high-level semantic feature of the adjacent level, including the output of the previous CFR module or the output of the previous HLFF module, and the operation method of the high-low feature fusion module is as follows: C1: In the high-low feature fusion module, double-branch mode is adopted for each input feature layer; for the two low-level semantic information, it is fully fused; For the low-level features output by the MBRW module, the semantic information in the first branch is refined by a 3x3 convolution layer; the 1x3 and 3x1 strip convolution in the second branch can better capture the horizontal and vertical features in the feature map; For the low-level features output by the ResNet34 backbone block, the first branch introduces a 3x3 dilated convolution to refine the feature layer and increase the receptive field area, and adds it to the first branch of the MBRW to realize multi-scale fusion; The second branch of ResNet34 uses channel attention mechanism to obtain a weight, which is multiplied by the output result of the second branch of MBRW, so that the key features obtain higher weight and improve the accuracy of the model; Finally, the two results are added, and after adjusting the channel by 1x1 convolution, the fused low-level semantic information is obtained; C2: For high-level semantic information, the first branch is multiplied by the fused low-level semantic information after using channel attention weighting to ignore the noise and unnecessary details in the low-level semantic information; In the second branch, the high-level semantic information is upsampled to the same size as the low-level semantic information, and the channel number is adjusted by 1x1 convolution, and the feature is further extracted by 3x3 convolution, and finally the output of the first branch is added to obtain the final result of the module.

8. The method of claim 1, wherein the method further comprises: The training of the network model in step S3 includes three parts of forward propagation, backward propagation and gradient descent in turn, and the training process is as follows: D1: Forward propagation: transmit the input picture data to the model to obtain an output result; D2: Backward propagation: the obtained output result of the model and the true value are reversely calculated by the loss function to obtain the parameter partial derivative of each neuron; D3: Gradient descent: after obtaining the partial derivative of each neuron in the backward propagation process, the model weight is finally updated by the gradient descent process, which descends along the inverse of the derivative of the maximum direction of the current position, and the descending step is determined by the learning rate.

9. The method of claim 8, wherein the method further comprises: The loss function in step D2 is cross-entropy loss function, and the calculation formula is as follows: Where x is the output tensor of the network, and class is the true label.

Images