Method and system for enhancing road connectivity of remote sensing image based on conditional mask

Through the conditional mask and loss function optimization framework, the problem of road extraction difficulties in remote sensing images is solved, road connectivity and boundary clarity are improved, and it is suitable for high-resolution remote sensing images.

CN119068184AActive Publication Date: 2024-12-03CHENGDU UNIVERSITY OF TECHNOLOGY

Patent Information

Application Number
CN202410825874.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-25
Publication Date
2024-12-03
Estimated Expiration
2044-06-25

AI Technical Summary

Technical Problem

There are difficulties in extracting roads in high-resolution remote sensing images, including complex backgrounds, unstable lighting conditions, difficult distinction caused by small road spacing, and road breakage problems, affecting the connectivity and accuracy of the road network.

Method used

Using a semantic segmentation optimization framework based on condition masks, the channel fusion layer and regional continuity loss are designed, combined with focus loss, and the semantic segmentation model is optimized to enhance road connectivity.

Benefits of technology

It effectively improves the accuracy of the semantic segmentation network, improves the road disconnection situation, provides clearer road boundaries and better spatial consistency, and is suitable for any convolutional semantic segmentation network, improving the road connectivity of remote sensing images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068184A_ABST
    Figure CN119068184A_ABST
Patent Text Reader

Abstract

The invention discloses a method and a system for enhancing road connectivity of a remote sensing image based on conditional masks. A remote sensing road prediction image of an original semantic segmentation network is used as extra supervision. Semantic information with road deep attributes is provided for a network through an accurate road backbone extraction module and a label image conditional mask strategy. A channel fusion layer is designed, and aims to efficiently fuse semantic information of an original input image and a predicted label image and adaptively fit an original network feature extraction mode. In addition, a regional continuity loss function is proposed. And the loss function endows the breakpoint part with a relatively high weight value, so that the model pays attention to pixel points of road disconnection in prediction, and the road connectivity is optimized. According to the method, the performance of the network in a remote sensing image road semantic segmentation task can be enhanced under the condition that only a small amount of parameters and calculation amount are increased.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for enhancing road connectivity of remote sensing images based on conditional masks. Background Art

[0002] Remote sensing image semantic segmentation is the process of pixel-level classification of high-resolution images obtained by remote sensing technology, with the aim of understanding the type and location of objects in the image. In this task, whether it is an urban or rural object scene, it is inseparable from the extraction of the road network, so accurate segmentation of roads is crucial. Road segmentation can help traffic management departments monitor and analyze road network conditions, identify traffic congestion points, optimize traffic signal control, and improve road use efficiency. In addition, by monitoring the expansion and changes of roads in remote sensing images, the impact of human activities on the natural environment can be evaluated, providing support for the formulation of ecological protection and sustainable development strategies. However, there are often four problems in the extraction of roads in high-resolution remote sensing images.

[0003] First, roads are usually embedded in extremely complex backgrounds, including multiple object elements such as buildings, trees, vehicles, pedestrians, etc. The presence of these elements greatly increases the difficulty of the segmentation task.

[0004] Second, the lighting conditions when the satellite acquires the image have a significant impact on the visualization of roads in remote sensing images. Changes in natural lighting conditions (such as sunlight intensity, time, and cloud cover) will affect the brightness and contrast of the image, which in turn affects the recognition and segmentation of roads.

[0005] Third, in a high-density road environment, the spatial intervals between roads are small, and adjacent roads may merge with each other in remote sensing images and be difficult to distinguish.

[0006] Most importantly, road wear, repair or the use of different materials will cause visual breaks in the image, while irregularities in the street network, such as dead ends, roundabouts or multi-level overpasses, also cause breaks in the continuous road network in the image. Such breaks make the extracted roads unsuitable for remote sensing image map detection and road condition analysis. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide a method for enhancing the road connectivity of remote sensing images based on conditional masks, which uses a semantic segmentation optimization framework of conditional masks to enhance the connectivity in remote sensing road prediction images. The method provides a semantic segmentation training optimization framework for the road disconnection problem.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] The method for enhancing the road connectivity of remote sensing images based on conditional masks provided by the present invention comprises the following steps:

[0010] Step S1, select a basic semantic segmentation network, train the network until convergence, and use the inference result of the network as a predicted image;

[0011] Step S2, conditionally masking the predicted image obtained in step S1 according to different strategies to remove structural information in the predicted image;

[0012] Step S3, designing a channel fusion layer to fuse the conditional mask prediction image obtained in step S2 with the input original image into a three-channel image with clear road boundary semantic information;

[0013] Step S4, using the three-channel image obtained in step S3 as the training set input of the same basic semantic segmentation network, using regional continuity loss and focal loss to calculate the loss and back propagate and update the parameters until the network converges to obtain the optimal semantic segmentation algorithm model;

[0014] Step S5: Use the convolutional semantic segmentation network trained in step S4 to perform prediction, and use the obtained predicted image as the input of step S2. Repeat steps S2-S5 to obtain the final semantic segmentation result image with excellent continuity.

[0015] Furthermore, the specific operations of basic semantic segmentation network training in step S1 are:

[0016] Step S11, randomly selecting training samples and verification samples of the basic semantic segmentation network from the total samples; randomly selecting 80% of the total samples as training samples of the network and 20% as verification samples;

[0017] Step S12, traverse all training samples, randomly select one of the three enhancement methods to process the image and add it to the total training samples;

[0018] Step S13: Use the loss function to iteratively train the basic semantic segmentation network and update the gradient until the network converges relatively.

[0019] Furthermore, the specific operation of conditional masking according to different strategies in step S2 is:

[0020] Step S21, respectively selecting the space domain and the road skeleton domain to conditionally mask the label image;

[0021] Step S22, using three strategies of random mask, block-by-block mask and chessboard mask to edit the predicted image;

[0022] Step S23, extracting a road skeleton in the predicted image, randomly selecting a pixel point in the road skeleton and using a matrix block to cover the road corresponding to the pixel point;

[0023] Furthermore, the specific operation of conditional masking according to different strategies in step S2 is:

[0024] Step S21 selects a spatial mask strategy to destroy the structural information of the road, prompting the model to learn deep representations from limited information. The road backbone mask is selected to simulate various road damage scenarios in the image.

[0025] Step S22 selects three strategies: random mask, block-by-block mask and chessboard mask. The random mask evenly divides the road label image into regular non-overlapping small blocks according to the pixel size, then samples each small block and randomly masks the image blocks according to different proportions; the block-by-block mask randomly generates rectangular blocks and pixel coordinates with different aspect ratios, and masks the label image based on the coordinates; the chessboard mask generates a mask grid covering the entire image according to the determined grid size and area.

[0026] Step S23 creates a 7×7 cross convolution kernel to extract the skeleton of each road, then selects the skeleton points in the internal area of ​​the image from the coordinate points, randomly generates a rectangle size, and finally creates a mask on the corresponding coordinates.

[0027] Furthermore, the specific fusion operation in step S3 is:

[0028] The channel fusion layer consists of an adaptive multi-head attention module and a fully connected layer with dilated convolution. The predicted grayscale image after conditional masking and the input original RGB image will be used as the two inputs of the channel fusion layer through the convolutional attention mechanism module respectively, and then converted into a three-channel image through the self-attention mechanism; the multi-layer perceptron with dilated convolution adaptively retains the spatial information of the original image and provides the model with clear road boundary semantic information.

[0029] Further, the specific model training operation in step S4 is:

[0030] Step S41 uses the three-channel image after the channel fusion layer as the input of the basic semantic segmentation network;

[0031] Step S42 initializes the basic semantic segmentation network and retrains the semantic segmentation network;

[0032] Step S43 uses regional continuity loss combined with focal loss to update the model weights until the network converges.

[0033] Further, the regional continuity loss calculation process in step S43 is:

[0034] The regional continuity loss first uses morphological opening and closing operations and binary image thinning operations to simplify the model to predict the road structure of the image and detect breakpoints, calculates the Euclidean distance between each breakpoint to update the weight map, and makes the model focus on the disconnected road area. Then, a differentiable soft-IOU loss function is used for calculation and multiplied by the weight map to finally obtain the regional continuity loss.

[0035] Further, the iterative training operation in step S5 is:

[0036] Step S51 uses the trained semantic segmentation model for reasoning;

[0037] Step S52 conditionally masks the predicted image obtained by inference;

[0038] Step S53 repeats steps S3-S5 described in claim 1, iterates the training 5 times, and obtains the final semantic segmentation result image.

[0039] Furthermore, in step S1, a total of 3216 original remote sensing images with a resolution of 512×512 are obtained, including two types of targets.

[0040] The system for enhancing the road connectivity of remote sensing images based on conditional masks provided by the present invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. The method is implemented when the processor executes the program.

[0041] The beneficial effects of the present invention are:

[0042] The method for enhancing the road connectivity of remote sensing images based on conditional masks provided by the present invention can effectively improve the accuracy of semantic segmentation networks and improve the road disconnection in remote sensing prediction images. The method is suitable for any convolutional semantic segmentation network and is based on conditional masks to enhance the road connectivity of remote sensing images.

[0043] The present invention conditionally edits the predicted image to remove the information that needs to be learned and optimized in road segmentation, thereby prompting the basic semantic segmentation model to learn deep representations from limited information. Masks based on remote sensing image space (random masks, block-by-block masks, grid masks) will eliminate most of the redundancy in the image and provide a more complex image representation for the model; masks based on remote sensing road network skeletons can effectively break the structural information such as boundaries in the original label image, while retaining semantic context information to a large extent.

[0044] This method fuses the conditionally masked predicted grayscale image with the input original image through the channel fusion layer, aiming to fit the road occlusion scene and remove the wrong road. The channel fusion layer includes a convolutional attention module and a multi-head self-attention module. The convolutional attention module will retain the precise location information in the grayscale image and the original image respectively, and establish a long-term spatial dependency relationship; the multi-head self-attention will connect the feature information of the two to enhance the clarity of the road boundary and realize the interaction of structural information. In order to enhance the model's attention to the road breakpoints in the predicted image, the model will use regional continuity loss and focal loss to update parameters.

[0045] This method is a general framework that can be used in any convolution-based semantic segmentation network training to improve the network's performance in high-resolution remote sensing semantic segmentation tasks. It is universal for remote sensing image semantic segmentation problems. In addition, this method is based on convolutional neural networks and can process remote sensing images more efficiently, providing accurate location information for urban planning, land use, and environmental monitoring.

[0046] This method provides a method for enhancing the road connectivity of remote sensing images based on conditional masks. This method edits the predicted image obtained after training the basic semantic segmentation network and removes some structural information in the feature map. It is then used as an additional channel and incorporated into the original remote sensing image as an input image for training through the channel fusion layer to connect the fragmented roads. At the same time, this method combines the proposed regional continuity loss with the focal loss to enhance the network's attention to the coordinate pixels of the disconnected roads. As a new learning strategy, this method enables the model to learn road connection patterns and remove erroneous roads by restoring damaged encodings during training.

[0047] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0049] Figure 1 Flowchart of the method for enhancing road connectivity of remote sensing images based on conditional masks.

[0050] Figure 2 An overview of the structure of the method for enhancing road connectivity in remote sensing images based on conditional masks.

[0051] Figure 3 Visualize an example graph for the random masking strategy.

[0052] Figure 4 An example diagram for visualizing the block-by-block masking strategy.

[0053] Figure 5 Visualize an example graph for the grid mask strategy.

[0054] Figure 6 This is an example diagram of the extracted road backbone visualization.

[0055] Figure 7 An example image showing the road skeleton masking strategy visualization.

[0056] Figure 8 This is an example diagram of the channel fusion layer structure.

[0057] Fig. 9 This is an example diagram of the overall iterative training operation.

[0058] Fig.10 Coloring image of the road prediction result after inputting high-resolution remote sensing image into the network. DETAILED DESCRIPTION

[0059] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0060] The method for enhancing the road connectivity of remote sensing images based on conditional masks provided in this embodiment is a complex computational process involving multiple stages from data preparation to iterative training of deep learning models. In the data preparation stage, this embodiment trains the basic semantic segmentation network to convergence, and then uses the network weights to predict the road segmentation map, and performs conditional mask operations in different fields based on the road skeleton pixel positions in the segmentation map, including random masking, block-by-block masking, grid masking in the spatial field, and road skeleton masking strategies in the skeleton field. The masked image will be used to create a training data set, in which the specific pixel positions of the road in the image are masked, thereby simulating the situation of being blocked by clouds and vehicles in real scenes.

[0061] Semantic segmentation of high-resolution remote sensing images can determine the land cover type on each image pixel in turn, providing rich semantic and location information for terrain observation and land use. In recent years, with the introduction of convolutional neural networks and Transformer, more and more researchers have used deep neural networks to extract ground feature information from high-resolution images. However, in the road segmentation task of high-resolution remote sensing images, since the roads in the images are often obscured by shadows of trees or buildings, the extraction of roads by the semantic segmentation network is often discontinuous. Therefore, in order to overcome the many difficulties in the semantic segmentation method of high-resolution remote sensing road images, this embodiment provides a method for enhancing the connectivity of remote sensing image roads based on conditional masks.

[0062] This embodiment uses a road connectivity optimization framework applicable to any convolutional network, aiming to improve the network semantic segmentation performance through additional supervision of mask operations. The framework conditionally masks the original segmentation results predicted by the semantic segmentation network to remove structural information. The masked grayscale image and the original remote sensing image will be used as input to the channel fusion layer to enhance the network's semantic expression ability. To increase the model's attention to disconnected areas, this implementation uses regional continuity loss. The road connectivity optimization framework iteratively enhances the topological structure of the semantic segmentation network, resulting in more accurate predictions, clearer boundaries, and better spatial consistency.

[0063] like Figure 1 , Figure 2 As shown, this example discloses a method for enhancing the road connectivity of remote sensing images based on conditional masks. The method uses a conditional mask semantic segmentation optimization framework to enhance the connectivity in remote sensing road prediction images. The steps are as follows:

[0064] Step S1, selecting a basic semantic segmentation network, training the network until convergence, and using the inference result of the network as a predicted image; the basic semantic segmentation network in this embodiment is a basic convolutional semantic segmentation network;

[0065] Step S2, conditionally masking the predicted image obtained in step S1 according to different strategies, removing the structural information in the predicted image, and obtaining a conditional mask prediction image;

[0066] Step S3, designing a channel fusion layer to fuse the conditional mask prediction image obtained in step S2 with the input original image into a three-channel image with clear road boundary semantic information;

[0067] Step S4, using the three-channel image obtained in step S3 as the training set input of the same basic semantic segmentation network, using regional continuity loss and focal loss to calculate the loss and back propagate and update the parameters until the network converges to obtain the optimal semantic segmentation algorithm model;

[0068] Step S5: Use the convolutional semantic segmentation network trained in step S4 to perform prediction, and use the obtained predicted image as the input of step S2. Repeat steps S2-S5 to obtain the final semantic segmentation result image with excellent continuity.

[0069] In this embodiment, the specific operations for performing basic semantic segmentation network training in step S1 are as follows:

[0070] Step S11, randomly selecting training samples and verification samples of the basic semantic segmentation network from the total samples; randomly selecting 80% of the total samples as training samples of the network and 20% as verification samples;

[0071] Step S12, traverse all training samples, randomly select one of the three enhancement methods to process the image and add it to the total training samples;

[0072] Step S13: Use the loss function to iteratively train the basic semantic segmentation network and update the gradient until the network converges relatively.

[0073] In this embodiment, the specific operations for conditional masking according to different strategies in step S2 are as follows:

[0074] Step S21, respectively selecting the space domain and the road skeleton domain to conditionally mask the label image;

[0075] Step S22, using three strategies of random mask, block-by-block mask and chessboard mask to edit the predicted image;

[0076] Step S23, extracting a road skeleton in the predicted image, randomly selecting a pixel point in the road skeleton and using a matrix block to cover the road corresponding to the pixel point;

[0077] The step S21 is specifically as follows:

[0078] The spatial mask strategy is chosen to destroy the structural information of the road, forcing the model to learn deep representations from limited information. The road backbone mask is chosen to simulate various road damage scenarios in the image.

[0079] The step S22 is specifically as follows:

[0080] Three strategies are selected: random mask, block-by-block mask and chessboard mask. The random mask divides the road label image into regular non-overlapping small blocks according to the pixel size, then samples each small block and randomly masks the image blocks according to different proportions; the block-by-block mask randomly generates rectangular blocks and pixel coordinates with different aspect ratios, and masks the label image based on the coordinates; the chessboard mask generates a mask grid covering the entire image according to the determined grid size and area.

[0081] The step S23 is specifically as follows:

[0082] Create a 7×7 cross convolution kernel to extract the skeleton of each road, then select the skeleton points in the internal area of ​​the image from the coordinate points, randomly generate the rectangle size, and finally create a mask at the corresponding coordinates.

[0083] like Figure 3 , Figure 4 , Figure 5 As shown, Figure 3 is the predicted image obtained by the random mask strategy, Figure 4 is the predicted image obtained by the block-by-block masking strategy, Figure 5 It is the image obtained after grid masking, where (a)(b)(c) represent the original image, label image and mask prediction image respectively. Figure 6 The extracted road skeleton graph is shown. Figure 7 is the predicted image obtained by road skeleton mask. Figure 8 As shown, Figure 8 Schematic diagram of the channel fusion layer structure.

[0084] In this embodiment, the channel fusion layer in step S3 is specifically:

[0085] The channel fusion layer includes an adaptive multi-head attention module and a fully connected layer;

[0086] The adaptive multi-head attention module includes a convolutional attention mechanism and a self-attention mechanism;

[0087] The convolutional attention mechanism enhances the convolutional neural network's focus on important features by combining spatial and channel attention, thereby improving the performance of the model;

[0088] The self-attention mechanism captures long-range dependencies in the model sequence by allowing each element to interact with other elements in the sequence and dynamically adjusting the weights;

[0089] The fully connected layer includes a multi-layer perceptron with a hole convolution and a convolution stream composed of a stack of lightweight convolution operations. Multiple convolution operations effectively enhance the network feature extraction capability.

[0090] The multi-layer perceptron enables the model to learn complex patterns and features of input data through multiple fully connected layers and nonlinear activation functions;

[0091] The convolutional flow combines convolutional neural networks and flow models to perform probabilistic modeling and generation of data, thereby improving the accuracy of feature extraction by the model;

[0092] The convolutional attention mechanism in the adaptive multi-head attention module takes the conditionally masked predicted grayscale image and the input original RGB image as two inputs.

[0093] In this example, the spatial information of the feature maps is aggregated using average pooling and maximum pooling operations to obtain the descriptors of the two feature maps. The two descriptors will be sent to the multi-layer perceptron (using the same terminology as the previous expression) and the output features will be combined by element-wise summation. The calculation formula can be expressed as:

[0094]

[0095] Where, σ represents the Sigmoid function;

[0096] W1 and W0 represent shared parameters;

[0097] represents the spatial context descriptor after average pooling;

[0098] Represents the spatial context descriptor after maximum pooling;

[0099] MLP stands for Multi-Layer Perceptron.

[0100] Subsequently, the channel information of the features is aggregated through a continuous pooling operation, and then connected to a standard convolutional layer to finally generate a two-dimensional spatial attention image; the continuous pooling operation in this embodiment uses two continuous pooling operations;

[0101] The calculation formula can be expressed as:

[0102]

[0103] Where, σ represents the Sigmoid function;

[0104] represents the spatial context descriptor after average pooling;

[0105] Represents the spatial context descriptor after maximum pooling;

[0106] f 7×7 Represents a convolution operation with a convolution kernel of 7.

[0107] The self-attention mechanism in the adaptive multi-head attention module normalizes the obtained multiple feature maps, so that the basic semantic convolutional segmentation network captures accurate location information and establishes long-term spatial dependency relationships; the multiple feature maps provided in this embodiment use two feature maps;

[0108] These feature maps are then linearly transformed independently in each multi-head attention to generate Query, Key, and Value. For each head, the dot product between Query and Key is used to calculate the attention score, and the GELU activation function and scaling are applied to obtain the calculation result. The calculation result is added to the relative position bias and the Softmax function is applied to obtain the final attention weight.

[0109] The Query, Key and Value can be expressed as:

[0110]

[0111] In the formula, Represents shared parameters;

[0112] F img Represents the multiplication result of the input original image and the normalized feature original image;

[0113] F label Represents the result of multiplying the label map and the normalized feature prediction map;

[0114] I represents the number of self-attention heads;

[0115] D represents the number of channels;

[0116] N represents the window size.

[0117] The process of calculating the corresponding weights of the multi-head attention can be expressed as:

[0118] MultiHead(Q,K,V)=Concat(head1,head2,...,head h )W O

[0119] Each head i ) is calculated as:

[0120]

[0121] The calculation formula for single-head attention is:

[0122]

[0123] In the above formula, Q, K, and V represent Query, Key, and Value respectively;

[0124] W O Represents the learnable parameter matrix during model training;

[0125] Concat means concatenating the calculation results of multiple heads;

[0126] h represents the number of heads;

[0127] d k Indicates the dimension of the Key vector;

[0128] The softmax function represents a normalized exponential function applied to each row.

[0129] The fully connected layer uses a 1×1 convolution and two 3×3 dilated convolutions, and the filling parameters and expansion parameters of the dilated convolution are [6, 6] and [12, 12] respectively, so as to extract feature information of different receptive fields.

[0130] In this embodiment, the semantic segmentation algorithm model training operation in step S4 is:

[0131] Step S41 uses the three-channel image after the channel fusion layer as the input of the basic semantic segmentation network;

[0132] Step S42 initializes the basic semantic segmentation network and retrains the semantic segmentation network;

[0133] Step S43 uses the regional continuity loss combined with the focal loss to update the semantic segmentation algorithm model weights until the network converges.

[0134] The loss calculation in step S43 includes a focus loss calculation function and a regional continuity loss calculation function, which are specifically as follows:

[0135] The focus loss calculation function is based on the binary cross entropy loss. By introducing a balance parameter and a dynamic adjustment parameter, the model's focus on easily distinguishable objects is reduced, and the imbalance in the number of positive and negative samples is suppressed, thereby improving the model's ability to segment small roads and fuzzy boundaries. The calculation formula is as follows:

[0136]

[0137] L f = -α t (1―p t ) γ log(p t )

[0138] In the formula, p t Represents the probability that the model predicts the correct category;

[0139] α t Represents the balance parameter, which is used to balance the ratio between the positive and negative sample losses;

[0140] γ is a dynamic adjustment parameter used to reduce the contribution of easy-to-classify samples and increase the impact of difficult-to-classify samples;

[0141] α t Set to 0.25;

[0142] The gamma value is set to 2.

[0143] The regional continuity loss calculation function first uses morphological opening and closing operations and binary image thinning operations to simplify the model to predict the road structure of the image and detect breakpoints. Define the road breakpoint c as a pixel position that is only connected to one other pixel point. The detection of this position can be obtained by checking the skeleton pixel domain. For each skeleton pixel, if only one of its four domain pixels belongs to the road category, the position is considered to be a road breakpoint c. Then initialize the weight map (the initial weight is set to 1), and calculate the Euclidean distance between each breakpoint in turn. Update the corresponding weight according to the distance between the breakpoints, so that the model pays more attention to the disconnected road area. The weight calculation and update formula is as follows:

[0144] w c =exp(α·D min (c))

[0145] Where D min (c) represents the Euclidean distance between the breakpoints closest to the breakpoint;

[0146] α is a positive adjustment parameter used to control the speed at which the distance affects the weight decay;

[0147] Alpha is set to 50.

[0148] The differentiable soft-IOU loss function is then used for calculation and multiplied by the weight map to finally obtain the regional continuity loss, which is calculated as follows.

[0149]

[0150] In the formula, N is the total number of categories;

[0151] It represents the prediction score of class n at position k;

[0152] is the real sample label.

[0153] The two loss results are added together as the total loss of the model training for back propagation and gradient update until the semantic segmentation network model converges. Fig. 9 As shown, Fig. 9 This is the training flow chart.

[0154] In this embodiment, the specific operation of iterative training in step S5 is:

[0155] Step S51 uses the trained semantic segmentation algorithm model for reasoning;

[0156] Step S52 conditionally masks the predicted image obtained by inference;

[0157] Step S53 repeats the steps S3-S5 and iterates the training for 5 times. The final semantic segmentation result is used as the prediction result, and the matching relationship between the grayscale value and the RGB value of each object category is found. The RGB value of the road category is assigned according to the grayscale value and coloring is performed. The grayscale value and RGB value assigned to the road and background categories in this embodiment are shown in Table 1:

[0158] Table 1

[0159]

[0160]

[0161] like Fig.10 As shown in FIG. 1 , the prediction result coloring diagram after the high-resolution remote sensing image with a size of 512×512 is input into the network in this embodiment is as follows:

[0162] When the object type is road, the RGB values ​​assigned to it are 255, 255, 255 respectively, the gray value assigned to it is 0, and the color is white;

[0163] When the object class target is the background, the RGB values ​​assigned to it are 0, 0, 0 respectively, the gray value assigned to it is 1, and the color is black.

[0164] The Intersection Over Union (IoU) accuracy and mean IoU of the prediction results before and after the present invention are shown in Table 2. The accuracy is defined as the number of correctly predicted positive samples divided by the total number of samples predicted as positive, and the recall is defined as the number of correctly predicted positive samples divided by the total number of samples actually positive. The F1 score is obtained by averaging the scores of all categories. IoU indicates the degree of similarity between the predicted area and the real area in a given set of images.

[0165] Table 2

[0166] method Accuracy Recall F1 score IoU QUR Before use 0.775 0.761 0.769 0.625 0.804 After use 0.787 0.772 0.780 0.636 0.813

[0167] It can be seen from the experimental results and Table 2 that this method effectively improves the segmentation accuracy of the semantic segmentation network for road backbones and can provide rich feature information for urban planning, road construction and road network extraction.

[0168] The system for enhancing road connectivity of remote sensing images based on conditional masks provided in this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The above method is implemented when the processor executes the program.

[0169] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for enhancing road connectivity of remote sensing images based on conditional masks, characterized in that: The following steps are involved: Step S1, select a basic semantic segmentation network, train the network until convergence, and use the inference result of the network as a predicted image; Step S2, conditionally masking the predicted image obtained in step S1 according to different strategies to remove structural information in the predicted image; Step S3, designing a channel fusion layer to fuse the conditional mask prediction image obtained in step S2 with the input original image into a three-channel image with clear road boundary semantic information; Step S4, using the three-channel image obtained in step S3 as the training set input of the same basic semantic segmentation network, using regional continuity loss and focal loss to calculate the loss and back propagate and update the parameters until the network converges to obtain the best semantic segmentation algorithm model; Step S5: Use the semantic segmentation algorithm model trained in step S4 to perform prediction, and use the obtained predicted image as the input of step S2. Repeat steps S2-S5 to obtain a final semantic segmentation result image with excellent continuity.

2. The method for enhancing road connectivity of remote sensing images based on conditional mask according to claim 1, characterized in that: The specific operations of basic semantic segmentation network training in step S1 are as follows: Step S11, randomly selecting training samples and verification samples of the basic semantic segmentation network from the total samples; Step S12, traverse all training samples, randomly select one of the three enhancement methods to process the image and add it to the total training samples; Step S13: Use the loss function to iteratively train the basic semantic segmentation network and update the gradient until the network converges relatively.

3. The method for enhancing road connectivity of remote sensing images based on conditional mask according to claim 1, characterized in that: The specific operations of conditional masking according to different strategies in step S2 are as follows: Step S21, respectively selecting the space domain and the road skeleton domain to conditionally mask the label image; Step S22, using three strategies of random mask, block-by-block mask and chessboard mask to edit the predicted image; Step S23: extract the road skeleton in the predicted image, randomly select a pixel point in the road skeleton and use a matrix block to cover the road corresponding to the pixel point.

4. The method for enhancing road connectivity of remote sensing images based on conditional mask according to claim 3, characterized in that: The specific operations of conditional masking according to different strategies in step S2 are as follows: Step S21 selects a spatial mask strategy to destroy the structural information of the road, prompting the model to learn deep representation from limited information; the road backbone mask is selected to simulate various road damage scenarios in the image; Step S22 selects three strategies: random mask, block mask and chessboard mask. The random mask evenly divides the road label image into regular non-overlapping small blocks according to the pixel size, and then samples each small block and randomly masks the image blocks therein according to different proportions. Block-by-block masking randomly generates rectangular blocks with different aspect ratios and pixel coordinates, and masks the label image based on the coordinates; chessboard masking generates a mask grid covering the entire image based on the determined grid size and area; Step S23 creates a cross convolution kernel to extract the skeleton of each road, then selects the skeleton points in the inner area of ​​the image from the coordinate points, randomly generates a rectangle size, and finally creates a mask on the corresponding coordinates.

5. The method for enhancing road connectivity of remote sensing images based on conditional mask according to claim 1, characterized in that: The specific fusion operation in step S3 is as follows: The channel fusion layer includes an adaptive multi-head attention module and a fully connected layer with dilated convolution; The predicted grayscale image after the conditional mask and the input original RGB image will be used as the two inputs of the channel fusion layer through the convolutional attention mechanism module respectively; It is then converted into a three-channel image through the self-attention mechanism; The multi-layer perceptron with hole convolution adaptively retains the spatial information of the original image and provides clear road boundary semantic information for the basic semantic segmentation network.

6. The method for enhancing road connectivity of remote sensing images based on conditional mask according to claim 1, characterized in that: The semantic segmentation algorithm model training operation in step S4 is as follows: Step S41 uses the three-channel image after the channel fusion layer as the input of the basic semantic segmentation network; Step S42 initializes the basic semantic segmentation network and retrains the semantic segmentation network; Step S43 uses the regional continuity loss combined with the focal loss to update the semantic segmentation algorithm model weights until the network converges.

7. The method for enhancing road connectivity of remote sensing images based on conditional mask according to claim 6, characterized in that: The process of calculating the regional continuity loss in step S43 is as follows: The regional continuity loss first uses morphological opening and closing operations and binary image refinement operations to simplify the model to predict the road structure of the image and detect breakpoints, calculates the Euclidean distance between each breakpoint to update the weight map, so that the model pays attention to the disconnected road areas, and then uses a differentiable soft-IOU loss function for calculation and multiplies it with the weight map to finally obtain the regional continuity loss.

8. The method for enhancing road connectivity of remote sensing images based on conditional mask according to claim 1, characterized in that: The iterative training operation in step S5 is as follows: Step S51 uses the trained semantic segmentation algorithm model for reasoning; Step S52 conditionally masks the predicted image obtained by inference; Step S53 repeats steps S3-S5 described in claim 1, iterates the training 5 times, and obtains the final semantic segmentation result image.

9. The method for enhancing road connectivity of remote sensing images based on conditional masks according to any one of claims 1 to 8, characterized in that: In step S1, a total of 3216 original remote sensing images with a resolution of 512×512 are obtained, including two types of targets.

10. A system for enhancing road connectivity of remote sensing images based on conditional masks, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

Citation Information

Patent Citations

  • Remote sensing image semantic segmentation method based on channel-space attention and DeeplabV3plus

    CN114596500A

  • Remote sensing image road extraction method, device and equipment based on mask image modeling

    CN116740570A

  • Multi-branch high-resolution remote sensing image semantic segmentation method and system based on edge perception

    CN117809029A

  • Semantic segmentation method and system for high-resolution remote sensing image based on random blocks

    US11189034B1

  • Boundary-optimized remote sensing image semantic segmentation method and apparatus, and device and medium

    WO2023077816A1

Cited By

  • Image enhancement method, system and equipment for automatic driving vision task

    CN120563789A

  • Directional connectivity road detection method, system, equipment and medium

    CN121121500A

  • Image segmentation method and device, computer equipment and storage medium

    CN121366294A

  • Image segmentation method and device, computer device and storage medium

    CN121366294B