A deep learning-based inpainting method for remote sensing image segmentation

Abstract

This invention proposes a deep learning-based remote sensing image segmentation repair method, belonging to the field of remote sensing images. The invention specifically comprises: preparing a remote sensing image dataset and constructing a road scene semantic segmentation network model; inputting high-resolution remote sensing road images from the remote sensing image dataset into the trained network model to perform scene semantic segmentation of the remote sensing road map; searching for breakpoints using an improved eight-neighborhood breakpoint detection algorithm; classifying the searched breakpoints into regular breakpoints and special breakpoints; curve fitting regular breakpoints using a polynomial fitting method, enlarging or thickening special breakpoints using morphological dilation to achieve repair of the overall road image breakpoints; and morphologically eroding the burrs or rough edges around the breakpoints in the overall road image. This invention solves the problem of road breakpoints and lack of connectivity caused by neglecting some local information due to the complexity of road scenes.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing images, and in particular to a remote sensing image segmentation and repair method based on deep learning. Background Art

[0002] Road recognition in remote sensing images is a necessary step for many applications, such as geographic information system updates, land use monitoring, and image registration. The accuracy of remote sensing image extraction not only significantly impacts the recognition of other objects, such as vehicles and buildings, but is also a crucial metric for other applications, such as natural disaster warning and unmanned vehicle path planning. Therefore, remote sensing image road extraction technology is crucial in this field.

[0003] Computer vision encompasses image classification, object detection, semantic segmentation, and video analysis. Semantic segmentation is currently a hot research topic. Krizhevsky et al., targeting pixel-level semantic segmentation of images, won first place in the 2012 ILSVRC competition with their eight-layer AlexNet. This led to renewed interest in convolutional neural networks (CNNs) and even deep learning. FCNs are an evolution of CNNs. Traditional CNNs generate feature maps after convolution, mapping them to fixed-length feature vectors. FCNs, on the other hand, discard the fully connected layers and perform upsampling to restore the original image size.

[0004] In recent years, a large number of CNN models with diverse and rich structures have been proposed for road areas. The first attempts were made by Mnih and Hinton using deep learning technology. They proposed a method for detecting road areas from high-resolution aerial images using restricted Boltzmann machines. Saito et al. proposed using convolutional neural networks to directly extract buildings and roads from raw remote sensing images. Cheng et al. proposed a cascaded end-to-end convolutional neural network (CasNet) to simultaneously handle road detection and centerline extraction tasks, and proposed a refinement algorithm to obtain a smooth and complete road centerline network. Wang et al. proposed methods such as the unet network for high-resolution remote sensing image road extraction that takes connectivity into account.

[0005] These methods have made great improvements in remote sensing image segmentation, but they still cannot effectively solve the problem of information loss caused by occlusion and shadows on the road. Considering that some road features are not obvious, such as parking lot roads, country field roads, tree occlusion and shadows, a large number of difficult-to-process fine boundary pixels will be generated when extracting road areas.

[0006] To sum up, the existing technical problem is that the existing remote sensing image processing method still cannot solve the problem of information loss caused by occlusion and shadows on the road. In addition, considering that some road features are not obvious, such as parking lot roads, country roads, tree occlusions and shadows, a large number of difficult-to-process fine boundary pixels will be generated when extracting road areas. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention proposes a remote sensing image segmentation repair method based on deep learning.

[0008] A remote sensing image segmentation repair method based on deep learning, comprising the following steps:

[0009] Step 1: Create a remote sensing image dataset;

[0010] Step 2: Build a road scene semantic segmentation network model and train the network model;

[0011] Step 3: Input the high-resolution remote sensing road images in the remote sensing image dataset into the trained network model to perform scene semantic segmentation of the remote sensing road map, and output the remote sensing road image after road scene semantic extraction;

[0012] Step 4: Binarize the final output image of the road scene semantic segmentation network to obtain a black and white road image composed of multiple pixels;

[0013] Step 5: Refine the black and white road image composed of multiple pixels into a single pixel road image;

[0014] Step 6: Use the improved eight-neighborhood breakpoint detection algorithm to search for breakpoints;

[0015] Step 7: Use kmeans clustering algorithm to cluster the searched breakpoints, and divide the clustered breakpoints into regular breakpoints and special breakpoints;

[0016] Step 8: Use polynomial fitting to perform curve fitting on regular breakpoints, and use morphological expansion to enlarge or thicken special breakpoints, thus repairing the breakpoints of the entire road image.

[0017] Step 9: Use the morphological corrosion method to remove the burrs around the breakpoints of the overall road image.

[0018] Preferably, the images in the remote sensing image dataset are divided into image blocks with a resolution of 512*512, image enhancement processing is performed on the image blocks in the remote sensing image dataset, and the processed remote sensing image dataset is used as a training set of the road scene semantic segmentation network model to train the model.

[0019] Preferably, the road scene semantic segmentation network model includes: an encoder network framework and a decoder network framework, the encoder framework is divided into five layers, the first layer extracts shallow semantic information through the ordinary convolution layer FE module, and the second to fifth layers extract high-level semantic information through grouped convolution; the grouped convolution block includes two parts, squeezing and excitation, the squeezing operation encodes the spatial features on each channel into a global feature map; the excitation operation linearly transforms the global feature map obtained by the squeezing operation through a fully connected layer, and restores the number of channels to the original number of channels through the ReLU activation layer and the fully connected layer to obtain the encoded remote sensing road image; the decoder framework uses the convolution vector operation spatial attention mechanism and the channel attention mechanism to extract global mixed feature information from the remote sensing road image, promotes semantic information fusion through dense jump links, obtains the global mixed features of the image, and uses a 3×3 convolution layer to compress the global mixed features of the image to a single channel to obtain the final output image.

[0020] Preferably, an improved eight-neighborhood breakpoint detection algorithm is used to search for breakpoints, with the road center pixel set as p1 and the eight adjacent pixel points in the surrounding areas set as p2→p9:

[0021] 1): When p2+p3+p4+p5+p6+p7+p8+p9=1, p1 is the breakpoint;

[0022] 2) When p2+p3+p4+p5+p6+p7+p8+p9≥2, and there is only one 0→1 pattern transition in the clockwise direction centered on p1 within the eight-neighborhood, then p1 is a breakpoint, where 0 is the background pixel and 1 is the target pixel.

[0023] Preferably, the kmeans clustering algorithm is used to cluster the searched breakpoints, and the clustered breakpoints are divided into regular breakpoints and special breakpoints, specifically including:

[0024] S1: Randomly select k initial points in the single-pixel road image as the center points of the cluster of the kmeans clustering algorithm;

[0025] S2: For each breakpoint, assign it to the cluster closest to the center point;

[0026] S3: For each breakpoint in a cluster, calculate the average distance of all breakpoints assigned to the cluster as the new center point of the cluster;

[0027] S4: Define the loss function as the sum of squared errors between each breakpoint and the center point of the cluster to which it belongs. Repeat S2-S3 until the loss function converges and the cluster no longer changes.

[0028] S5: Calculate the distance between the center points of each cluster and find the minimum value m, then calculate the distance M between the sample points in each cluster and its center point lk, divide the breakpoints where M lk ≥m into special breakpoints, and divide the breakpoints where M lk <m into regular breakpoints.

[0029] Furthermore, define the loss function as the sum of the squared errors of each sample from the center point of its belonging cluster. The expression of the loss function is:

[0030]

[0031] where N is the total number of breakpoints in the cluster, x i is the i-th breakpoint, C i is the cluster to which x i belongs, is the center point corresponding to the cluster, and Cost represents the loss function.

[0032] Preferably, fit the breakpoints of the same type with a polynomial curve. The expression of the polynomial curve fitting is:

[0033]

[0034] where x i represents the breakpoint of the i-th order road, w i represents the coefficient of the i-th order breakpoint, M represents the order of the high-order polynomial, and b represents the linear correction value of the error of the breakpoint fitting curve.

[0035] Preferably, make the width of the special breakpoints larger through dilation operation. The width is controlled by the size of the structural element used. Select a linear structural element to process the special breakpoints of the remote sensing image. The expression of the dilation operation of A by B is:

[0036]

[0037] where, represents A being dilated by B, B represents the linear structural element, A represents the remote sensing image to be processed containing breakpoint information, AΘB represents A being eroded by B, and C represents the complement set.

[0038] Preferably, eliminate or weaken the fine edges or burrs in the image through erosion operation, and shrink the boundary of the image inward. The mathematical expression of the erosion operation of A by B is:

[0039]

[0040] where AΘB represents A being eroded by B, B represents the linear structural element, A represents the remote sensing image to be processed containing breakpoint information, and x represents the translation distance.

[0041] The beneficial effects of the present invention are as follows: the present invention segments the road remote sensing image, extracts the semantic information of the image, binarizes the image, and performs breakpoint search to achieve breakpoint repair of the road remote sensing image. This solves the problems of large-scale missing road detail information and many difficult-to-process fine boundary features caused by the existence of high resolution, rich semantic information, tree occlusion, shadows, and some lack of obvious road feature information in remote sensing images. It also solves the problem of neglecting some local information due to the complexity of the road scene, resulting in road breakpoints and lack of connectivity. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Flow chart of the specific solution of the present invention

[0043] Figure 2 Improved unet++ network structure diagram for this invention

[0044] Figure 3 SEResNext module of the present invention

[0045] Figure 4 This is the channel and spatial attention mechanism diagram of the present invention

[0046] Figure 5 Schematic diagram of the 0 to 1 mode conversion of the present invention DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] like Figure 1 As shown, the present invention provides a remote sensing image segmentation repair method based on deep learning, and the specific steps include:

[0049] Step 1: Create a remote sensing image dataset;

[0050] Step 2: Build a road scene semantic segmentation network model and train the network model;

[0051] Step 3: Input the high-resolution remote sensing road images in the remote sensing image dataset into the trained network model to perform scene semantic segmentation of the remote sensing road map, and output the remote sensing road image after road scene semantic extraction;

[0052] Step 4: Binarize the final output image of the road scene semantic segmentation network to obtain a black and white road image composed of multiple pixels;

[0053] Step 5: Refine the black and white road image composed of multiple pixels into a single pixel road image;

[0054] Step 6: Use the improved eight-neighborhood breakpoint detection algorithm to search for breakpoints;

[0055] Step 7: Use kmeans clustering algorithm to cluster the searched breakpoints, and divide the clustered breakpoints into regular breakpoints and special breakpoints;

[0056] Step 8: Use polynomial fitting to perform curve fitting on regular breakpoints, and use morphological expansion to enlarge or thicken special breakpoints, thus repairing the breakpoints of the entire road image.

[0057] Step 9: Use the morphological corrosion method to remove the burrs around the breakpoints of the overall road image.

[0058] Preferably, the 1,171 high-resolution images in the Massachusetts road dataset are divided into 30,000 image blocks with a resolution of 512*512 to form a remote sensing image dataset. Image enhancement processing is performed on the image blocks in the remote sensing image dataset, and the processed remote sensing image dataset is used as a training set for the road scene semantic segmentation network model to train the model.

[0059] This paper proposes an improved encoder and decoder network framework of unet++ network to extract road areas. The network framework mainly includes four parts: FE shallow information extraction module, high-level semantic information extraction module, global hybrid feature extraction module and dense jump connection module. The encoder essentially classifies and analyzes the low-level pixel values ​​of the image to obtain high-level semantic information, such as Figure 2 As shown in the figure, the improved unet++ network framework structure is given;

[0060] Preferably, the specific steps of scene semantic segmentation of road remote sensing images include:

[0061] S1: Input the images in the remote sensing image dataset into the road scene semantic segmentation network model, extract the shallow semantic information of the image through the FE module, and extract the high-level semantic information of the image through group convolution;

[0062] S2: Through the global hybrid feature module of the decoder, the shallow semantic information and the high-level semantic information are fused through upsampling operations, and then dense skip connections are performed to obtain the global hybrid features of the image;

[0063] S3: Use a 3×3 convolutional layer to compress the global mixed features of the image into a single channel to obtain the final output image of the road scene semantic segmentation network;

[0064] S4: Semantic segmentation metrics are used to accurately evaluate the segmentation accuracy of the final output image of the road scene semantic segmentation network.

[0065] The encoder in this invention is divided into five layers. The first layer is a common convolutional layer FE module to extract shallow semantic information. The second to fifth layers extract high-level semantic information in the form of grouped convolution SEResNext blocks, such as Figure 3 As shown in Figure 2, the structure of the SEResNext module is given.

[0066] The grouped convolution SEResNext block includes two parts: squeezing and excitation. Squeezing corresponds to the global average pooling operation, which encodes the entire spatial feature on each channel into a global feature, thereby obtaining a larger perception field of view. The process is shown in the following mathematical formula:

[0067]

[0068] Among them, C represents the channel, H and W represent the size of the feature map, S represents the output after the pooling operation, and U C (i, j) represents a pixel point of the input feature map;

[0069] The excitation operation first performs a linear transformation on the squeeze operation feature map through a fully connected layer, compressing the number of channels to C / r, where r is set to 16 in this invention;

[0070] Then the number of channels is restored to the original number of channels through the ReLU activation layer and the fully connected layer:

[0071] P1=f c2 (δ(f c1 (S C )))

[0072] Among them, f c1 and f c2 represents the fully connected layer, δ represents the ReLU activation function, and P1 represents the output of the channel attention unit.

[0073] In order to extract roads more accurately, this paper uses SEResNext, which combines ResNeXt and SENet, as the basis of the encoder:

[0074]

[0075] Among them, M is the output of SEResNext, X is the initial input of the module, T is a series of dimensionality reduction and convolution processing, C is divided into 32 groups, P i Output of the channel attention mechanism.

[0076] The decoder extracts global mixed feature information from the remote sensing road image; the decoder upsamples the reduced feature map, and the present invention uses interpolation to perform upsampling, and then performs convolution processing on the upsampled image. The decoder collects this semantic information and maps the same object to the corresponding pixel point, and each object is represented by a different color;

[0077] The output of the remote sensing road image after encoding by the five-layer encoder is input into the global hybrid feature module of the decoder. It is then iteratively processed to fuse the shallow semantic information and deep semantic information of the image extracted by the encoder. Then, the output of each layer of the encoder in the previous step is combined to collect the semantic information of the remote sensing image and output the image by mapping the same object to the corresponding pixel points.

[0078] The encoder output images of the fifth layer and the fourth layer are used as the input of the decoder of the fourth layer, and the output is a remote sensing road image with 256 channels and a resolution of 32*32.

[0079] The output image of the third-layer encoder, the output image of the fourth-layer encoder, the output image of the intermediate global hybrid feature module, and the output image of the fourth-layer decoder are all used as the input image of the third-layer decoder. The output is a remote sensing road image with 128 channels and a resolution of 64*64.

[0080] The second-layer encoder output image, the third-layer encoder output image, the intermediate global hybrid feature module output image, and the third-layer decoder output image are all used as the second-layer decoder input image, and the output is a remote sensing road image with 64 channels and a resolution of 128*128;

[0081] The output image of the first-layer encoder, the output image of the second-layer encoder, the output image of the intermediate global mixed feature module, and the output image of the second-layer decoder are all used as the input image of the first-layer decoder. The output is a remote sensing road image with 64 channels and a resolution of 256*256.

[0082] The present invention uses conv+Attention (conv is a convolution vector operation, Attenton is a type of public attention mechanism) to construct the decoder, where Attention is the SESattention mechanism (a public module in Attention), which combines the spatial attention mechanism and the channel attention mechanism.

[0083] The spatial attention mechanism: the feature map undergoes a 1×1×C convolution, retaining the length and width of the feature map, with the number of channels being 1, and then passes through the sigmoid function:

[0084] P2=σ(C(I))

[0085] P=P1+P2

[0086] Among them, P2 represents the output of the spatial attention unit, C represents the ordinary 1×1 convolution layer, I represents the input feature map, σ represents the sigmoid operation, and P represents the output of the channel and spatial attention mechanism module. P1 and P2 represent the outputs of the channel attention unit and the spatial attention unit, respectively.

[0087] like Figure 4 As shown in the figure, the channel and spatial attention mechanisms are given, and the global mixing module is divided into two branches. One branch passes through the channel and spatial attention mechanism, and the other passes through two convolutions and then passes through the channel and spatial attention mechanism, and the two matrices are added together.

[0088] The remote sensing road image is gradually upsampled to fuse deep and shallow features to ensure accurate target segmentation. Each layer in the module receives all previous layers as its additional input. Each layer is connected and spliced ​​with all previous layers in the channel dimension, and each layer is spliced ​​with the upsampling layer, followed by dense jump connections.

[0089] Each layer module not only accepts all previous layers as input, but also connects with all previous layers in the channel dimension. It also connects with the upsampling layer to make comprehensive use of the low-complexity features of the shallow layer, resulting in a smooth decision function with better generalization performance, which also helps improve road extraction accuracy:

[0090]

[0091] In the above formula, H(·) is a convolution operation followed by an activation function, D(·) and U(·) represent the downsampling layer and upsampling layer respectively, and [] represents the cascade layer, j indicates the number of skip connections in the same layer, and x is the output after channel splicing.

[0092] The final output image of the road scene semantic segmentation network is obtained by compressing it into a single channel using a 3×3 convolutional layer. Finally, the remote sensing road image with 64 channels and a resolution of 256*256 output in the previous step is compressed into a single channel with a resolution of 512*512 through a 3×3 convolutional layer. This remote sensing road image serves as the remote sensing road image segmentation result of the final road scene semantic segmentation module.

[0093] Compared to existing decoders, ours comprehensively utilizes shallow, low-complexity features to produce a smooth decision function with improved generalization performance and excellent resistance to overfitting. Furthermore, feature reuse improves the accuracy of extracting shadows and occluded roads, and dense upsampling of skip connections captures non-obvious roads and evaluation metrics, resulting in improvements over the UNET algorithm.

[0094] In order to accurately evaluate the segmentation accuracy of the remote sensing image road segmentation model, the present invention adopts the commonly used indicators in semantic segmentation for evaluation: R is the ratio of the number of correctly predicted road pixels to the number of all road pixels, P is the ratio of the number of correctly predicted road and background pixels to the number of pixels in the entire remote sensing image, F1-score is the comprehensive evaluation index of recall rate and precision rate, P is the ratio of the number of correctly predicted road and background pixels to the number of pixels in the entire remote sensing image, and F1-score is the comprehensive evaluation index of recall rate and precision rate. r 、P p 、F1-score calculation formula is:

[0095]

[0096]

[0097] Among them, TP is the number of pixels with correct road prediction; TN is the number of pixels with correct background prediction; FN is the number of pixels with incorrect background information; FP is the number of pixels with incorrect road information.

[0098] Input the high-resolution remote sensing image road dataset of the training set into the established road scene semantic segmentation network module to train the model;

[0099] The training of the road scene semantic segmentation network model includes: using the Dice loss function instead of the standard cross entropy of the BCE loss function for model training, and using the BCE+Diceloss joint loss function method to train the segmentation network, which alleviates the problem of uneven distribution of road pixels and background pixels and more accurately segments unclear roads. The expression of the training model is:

[0100]

[0101] Among them, GT represents the ground truth of the input image, P represents the prediction result of the network output, || represents the sum of matrix elements, and |GT∩P| represents the number of elements shared by GT and P.

[0102] The breakpoint repair algorithm process specifically includes:

[0103] Image binarization: The process of setting the grayscale value of pixels on a remote sensing road image to 0 or 255, which is the process of making the entire image appear in obvious black and white.

[0104] Road skeleton: Refine the roads composed of multiple pixels into single-pixel roads, which preserves road connectivity and removes redundant information, which is beneficial for road breakpoint detection.

[0105] Searching for road breakpoints: This paper proposes an improved eight-neighborhood breakpoint detection algorithm. With the judgment point as the center, the algorithm searches for the sum of the pixel values ​​in eight adjacent directions to determine whether a road is a breakpoint. This improved eight-neighborhood breakpoint detection algorithm can detect breakpoints in a wider range of situations (including those with other fragmented interference information), effectively overcoming the limitations of existing search algorithms that can only search for isolated points, internal points, and endpoints. It also has stronger anti-interference performance against noisy information points.

[0106] The specific method for determining the breakpoint condition is as follows: let a road pixel be p1, and its neighboring eight-direction pixel points be p2→p9:

[0107] When p2+p3+p4+p5+p6+p7+p8+p9=1, p1 is the breakpoint;

[0108] When p2+p3+p4+p5+p6+p7+p8+p9≥2, and the 0→1 pattern appears only once in the eight-neighborhood (clockwise direction), (0 is the background pixel point, 1 is the target pixel point), the 0→1 pattern conversion means that in the eight-neighborhood with p1 as the center point, the two adjacent neighborhoods 0 are converted to 1, such as Figure 5 As shown, a schematic diagram of the 0 to 1 mode conversion is shown.

[0109] Preferably, the kmeans clustering algorithm is used to cluster the searched breakpoints, and the clustered breakpoints are divided into regular breakpoints and special breakpoints, specifically including:

[0110] S1: Randomly select k initial points in the single-pixel road image as the center points of the cluster of the kmeans clustering algorithm;

[0111] S2: For each breakpoint, assign it to the cluster closest to the center point;

[0112] S3: For each breakpoint in a cluster, calculate the average distance of all breakpoints assigned to the cluster as the new center point of the cluster;

[0113] S4: Define the loss function as the sum of squared errors between each breakpoint and the center point of the cluster to which it belongs. Repeat S2-S3 until the loss function converges and the cluster no longer changes.

[0114] S5: Calculate the distance between the center points of each cluster and find the minimum value m, then calculate the distance M between the sample points in each cluster and its center point lk (l represents the lth cluster, k represents the kth sample point in the cluster), M lk The breakpoints ≥m are classified as special breakpoints, M lkThe breakpoints of m are divided into regular breakpoints to avoid the situation that several special breakpoints are divided into one cluster due to the small breakpoint density in a certain area or a special breakpoint is divided into the regular breakpoint cluster.

[0115] Furthermore, the loss function is defined as the sum of the squared errors of each sample from the center point of its belonging cluster, and the expression of the loss function is:

[0116]

[0117] where N is the total number of breakpoints in the cluster, x i is the i-th breakpoint, and C i is the cluster to which x i belongs, is the center point corresponding to the cluster, and Cost represents the loss function.

[0118] Considering the curvature of the road, the method of polynomial fitting is used to perform curve fitting on the regular breakpoints to repair the road breakpoints:

[0119]

[0120] In the above formula, x represents the input of the road breakpoint, W i represents the coefficient, M represents the order of the high-order polynomial, and b represents the constant term.

[0121] Handling special breakpoints: It is a breakpoint, but there is no matching point around it to match. In this invention, the method of morphological dilation and erosion is used to connect such breakpoints.

[0122] Among them, the dilation operation can make the object being operated larger or thicker, and the width of its thickening is controlled by the size of the used structural element. In this invention, a linear structural element is selected, and the expression of A being dilated by B is:

[0123]

[0124] In the above formula, represents A being dilated by B.

[0125] The erosion operation is a process of eliminating or weakening the fine fringes or burrs in the image. In essence, it is a process of shrinking the boundary of the image inward. The mathematical expression of A being eroded by B is:

[0126]

[0127] In the above formula, AΘB represents A being eroded by B.

[0128] Among them, when multiple breakpoints need to be connected, each time a breakpoint is connected, the above-designed breakpoint algorithm is run once, and the output result is sent to the breakpoint connection algorithm until all breakpoints are connected.

[0129] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A remote sensing image segmentation repair method based on deep learning, characterized in that: The following steps are involved: Step 1: Create a remote sensing image dataset; Step 2: Build a road scene semantic segmentation network model and train the network model; The road scene semantic segmentation network model includes: an encoder network framework and a decoder network framework. The encoder framework is divided into five layers. The first layer extracts shallow semantic information through an ordinary convolutional layer FE module, and the second to fifth layers extract high-level semantic information through grouped convolution. The grouped convolution includes two parts: squeezing and excitation. The squeezing operation encodes the spatial features on each channel into a global feature map. The excitation operation linearly transforms the global feature map obtained by the squeezing operation through a fully connected layer, and restores the number of channels to the original number of channels through a ReLU activation layer and a fully connected layer to obtain an encoded remote sensing road image. The decoder framework uses a convolutional vector operation spatial attention mechanism and a channel attention mechanism to extract global mixed feature information from the remote sensing road image, promotes semantic information fusion through dense jump links, obtains the global mixed features of the image, and uses a 3×3 convolution layer to compress the global mixed features of the image into a single channel to obtain the final output image. Step 3: Input the high-resolution remote sensing road images in the remote sensing image dataset into the trained network model to perform scene semantic segmentation of the remote sensing road map, and output the remote sensing road image after road scene semantic extraction; Step 4: Binarize the final output image of the road scene semantic segmentation network to obtain a black and white road image composed of multiple pixels; Step 5: Refine the black and white road image composed of multiple pixels into a single pixel road image; Step 6: Use the improved eight-neighborhood breakpoint detection algorithm to search for breakpoints; Step 7: Use kmeans clustering algorithm to cluster the searched breakpoints, and divide the clustered breakpoints into regular breakpoints and special breakpoints; Step 8: Use polynomial fitting to perform curve fitting on regular breakpoints, and use morphological expansion to enlarge or thicken special breakpoints, thus repairing the breakpoints of the entire road image. Step 9: Use the morphological corrosion method to remove the burrs around the breakpoints of the overall road image.

2. The method for repairing remote sensing image segmentation based on deep learning according to claim 1, characterized in that: The images in the remote sensing image dataset are divided into image blocks with a resolution of 512*512, and image enhancement processing is performed on the image blocks in the remote sensing image dataset. The processed remote sensing image dataset is used as the training set of the road scene semantic segmentation network model to train the model.

3. The method for repairing remote sensing image segmentation based on deep learning according to claim 1, characterized in that: The improved eight-neighborhood breakpoint detection algorithm is used to search for breakpoints. The center pixel of the road is set as p1, and the eight adjacent pixel points in the surrounding areas are set as p2→p9: 1): When p2+p3+p4+p5+p6+p7+p8+p9=1, p1 is the breakpoint; 2) When p2+p3+p4+p5+p6+p7+p8+p9≥2, and there is only one 0→1 pattern transition in the clockwise direction centered on p1 within the eight-neighborhood, then p1 is a breakpoint, where 0 is the background pixel and 1 is the target pixel.

4. The method for repairing remote sensing image segmentation based on deep learning according to claim 1, characterized in that: The kmeans clustering algorithm is used to cluster the searched breakpoints, and the clustered breakpoints are divided into regular breakpoints and special breakpoints, including: S1: Randomly select k initial points in the single-pixel road image as the center points of the cluster of the kmeans clustering algorithm; S2: For each breakpoint, assign it to the cluster closest to the center point; S3: For each breakpoint in a cluster, calculate the average distance of all breakpoints assigned to the cluster as the new center point of the cluster; S4: Define the loss function as the sum of squared errors between each breakpoint and the center point of the cluster to which it belongs. Repeat S2-S3 until the loss function converges and the cluster no longer changes. S5: Calculate the distances between the center points of each cluster and find the minimum value m, and then calculate the distances M between the sample points in each cluster and their center points lk , and divide the breakpoints where M lk ≥m into special breakpoints, and the breakpoints where M lk <m into regular breakpoints.

5. The method for repairing remote sensing image segmentation based on deep learning according to claim 4, characterized in that: The loss function is defined as the sum of squared errors between each sample and the center point of the cluster to which it belongs. The loss function expression is: Where N is the total number of cluster break points, x i is the i-th breakpoint, C i is x i The cluster to which it belongs, is the center point corresponding to the cluster, and Cost represents the loss function.

6. The method for repairing remote sensing image segmentation based on deep learning according to claim 1, characterized in that: Use a polynomial curve to fit the same type of breakpoints. The polynomial curve fitting expression is: Among them, x i represents the breakpoint of the i-th order path, w i represents the coefficient of the i-th order breakpoint, M represents the order of the higher-order polynomial, and b represents the linear correction value of the breakpoint fitting curve error.

7. The method for repairing remote sensing image segmentation based on deep learning according to claim 1, characterized in that: The width of the special breakpoint is enlarged by the dilation operation. The width is controlled by the size of the structure element used. The linear structure element is selected to process the special breakpoints of the remote sensing image. The expression of the dilation operation of A by B is: in, It means that A is expanded by B, B is a linear structure element, A is a remote sensing image to be processed containing breakpoint information, AΘB is A is eroded by B, and C is the inverse complement.

8. The method for repairing remote sensing image segmentation based on deep learning according to claim 1, characterized in that: The erosion operation is used to eliminate or weaken the fine edges or burrs in the image and shrink the image boundary inward. The mathematical expression of the erosion operation of A by B is: Where AΘB means A is eroded by B, B represents a linear structure element, A represents the remote sensing image to be processed containing breakpoint information, and x represents the translation distance.

Images