A remote sensing image road extraction method and system combining semantic segmentation and angle prediction

Abstract

This invention relates to a method and system for road extraction from remote sensing images, combining semantic segmentation and angle prediction, belonging to the field of computer vision technology. This invention calculates road angle information from real road information, explicitly reflecting the geometric feature that road tilt angles have a certain stability, providing the neural network with ground truth road angles for comparison to constrain this geometric feature. The core model incorporates an angle prediction module and a feature fusion section to predict and fuse road angle features, and designs a corresponding loss function to add explicit angle-related constraints, enabling the model to better utilize the various information contained in the remote sensing imagery and improve the accuracy of the final road prediction result. This invention can improve the accuracy of predicting road information in complex real remote sensing images without requiring special equipment and with a relatively fast generation speed.

Description

Technical Field

[0001] This invention relates to a method and system for road extraction from remote sensing images, specifically a method and system for road extraction from remote sensing images that combines semantic segmentation and angle prediction, belonging to the field of computer vision technology. Background Technology

[0002] Remote sensing imagery refers to images of the Earth's surface formed by collecting electromagnetic radiation information from artificial satellites, aircraft, or other spacecraft. Remote sensing imagery contains rich ground information, providing crucial data for analyzing surface conditions and resources. In recent years, with the continuous development of artificial satellite technology, the quantity and quality of artificial satellites available for acquiring remote sensing imagery have significantly improved, and the cost of acquiring high-resolution remote sensing imagery has been further reduced.

[0003] Road extraction is a crucial task in the field of remote sensing image analysis and processing. Its objective is to acquire road location information and mark it in the form of a road map. Road information is essential for people's travel planning and plays a vital role in commercial sectors such as logistics, food delivery, ride-hailing, and vehicle sharing. It also plays an important role in emergency response, traffic management, and urban planning.

[0004] Traditional methods of acquiring road information require significant manpower and resources for on-site surveying, obtaining and recording road vector data, and then importing it into a computer to create a road network map for easy viewing. This process is cumbersome and lengthy, incurring high costs. Furthermore, the on-site surveying process is greatly affected by surface conditions, making it difficult to respond quickly and update in extreme weather or emergency situations.

[0005] Considering that remote sensing imagery is easier, faster, cheaper, and unaffected by ground conditions, and has better adaptability, and that it can usually reflect road information on the ground surface, methods for extracting roads from remote sensing imagery have become a worthy research subject.

[0006] Since the 1970s, the academic community has proposed numerous methods for road extraction from different perspectives. Based on the fundamental principles employed, traditional road extraction methods can be categorized into three types: template matching-based methods, knowledge-based methods, and object analysis-based methods. Template matching is a relatively mature method among early traditional road extraction methods. It effectively combines the radial and geometric features of roads for judgment and allows for human-computer interaction through methods such as manually setting seed points or initial template contours. This gives these methods a certain degree of error correction capability and good application value. Knowledge-based methods generally establish road-related knowledge models, especially hypothesis testing models between road-related knowledge and remote sensing image processing results, to achieve the goal of road extraction using this knowledge. Object analysis methods mainly identify the radial and textural features of remote sensing images, segment the remote sensing images using early segmentation methods, and then extract roads through classification or post-processing methods. These traditional road extraction methods have achieved certain results in some relatively simple road extraction tasks and have been used in some commercial software, demonstrating practical application value. However, when faced with more complex real remote sensing images, the detection performance of these road extraction methods is still easily affected by various factors such as shadows, weather, season, and vegetation.

[0007] In recent years, the rapid development of deep learning technology has significantly improved the road extraction performance based on remote sensing imagery. Deep learning is a new research direction in the field of machine learning. It typically uses deep, nonlinear neural networks to approximate complex functions and trains them with appropriate data and loss functions. By changing the parameters stored in the neural network, it learns statistical patterns from a large amount of known data, thereby making predictions about unknown events. With the continuous emergence of new algorithms and ideas, human intelligence has endowed machines with increasingly higher capabilities to understand image information, and models are gradually becoming able to automatically extract road information from more complex and realistic remote sensing imagery.

[0008] In 2010, Mnih et al. proposed a road extraction method based on Restricted Boltzmann Machines (RBMs) from remote sensing images. This was the first attempt at using deep learning technology in this field and achieved results significantly better than traditional methods. Since then, more road extraction methods have treated this task as a semantic segmentation task, using semantic segmentation models with deep convolutional neural networks (CNNs) at their core for road region extraction. Compared to traditional methods that rely on one or more manually summarized road feature models for road extraction, semantic segmentation models using deep convolutional neural networks can learn the necessary composite features from a large amount of remote sensing imagery and comprehensively utilize these features to improve the accuracy of road extraction.

[0009] However, current semantic segmentation methods still commonly exhibit issues such as road breaks, blurred road boundaries, and abrupt changes in road width when extracting roads from remote sensing images. Therefore, to improve the accuracy and effectiveness of remote sensing image road extraction methods, reduce the cost of acquiring road information, increase the speed and timeliness of road information updates, and better serve people's needs in travel, commerce, management, and planning, there is an urgent need for a remote sensing image road extraction method and system that can automatically extract road information from remote sensing images and maintain good performance even in complex and interference-prone real remote sensing images. Summary of the Invention

[0010] The purpose of this invention is to address the shortcomings and defects of existing technologies and, in order to effectively solve the technical problem of road extraction from remote sensing images, to creatively propose a method and system for road extraction from remote sensing images that combines semantic segmentation and angle prediction. This invention can improve the accuracy of road information extraction from complex and realistic remote sensing images without requiring special equipment and with a relatively fast extraction speed.

[0011] The innovation of this invention lies in dividing the road extraction from remote sensing images based on joint semantic segmentation and angle prediction into a training phase and an application phase.

[0012] During the training phase, firstly, road angles are calculated based on real road information in the dataset, obtaining the angle value corresponding to each road category pixel, thus forming real road angle information. Then, a road extraction algorithm model is used to extract road information from the input remote sensing imagery, generating predicted road information and simultaneously generating predicted road angles. After prediction, the road extraction algorithm model compares the predicted road information and road angles with the corresponding real values, calculates the loss function, and updates the model parameters accordingly. This prediction and comparison process is repeated until a set termination condition is met, at which point the network structure and model parameters are saved.

[0013] During the usage phase, firstly, a network model is created based on the model structure and parameters saved during the training phase. The acquired remote sensing images are then input into the model for calculation, and the extracted and predicted road information is stored. If the remote sensing images used contain corresponding real road information, the predicted road information is compared and evaluated with the real road information to determine the effectiveness of the model's road prediction.

[0014] To achieve the above objectives, the present invention employs the following technical solutions.

[0015] A method for road extraction from remote sensing images that combines semantic segmentation and angle prediction includes the following steps:

[0016] Step 1: Training phase.

[0017] The parameter dictionary of the algorithm model was iteratively modified using a remote sensing image-road information pairing dataset.

[0018] Specifically, step 1 includes the following steps:

[0019] Step 1.1: Calculate the road angles based on the real road information in the dataset (this can be done through methods such as Hough transform) to obtain the angle value corresponding to each road category pixel and thus obtain the real road angle information.

[0020] Preferably, the following method can be used:

[0021] First, morphological methods are used to refine the real road information to obtain road skeleton information. The aim is to reduce the amount of subsequent calculations and thus improve computational efficiency.

[0022] Then, for each pixel in the road skeleton information, take a neighborhood road skeleton information image centered on it, and use Hough transform on the neighborhood image to calculate a straight line that fits the road skeleton of the region. The tilt angle of the straight line is taken as the road angle corresponding to the center pixel.

[0023] Then, based on the angle information of the road skeleton points, the angle information of the remaining points on the road is inferred according to the principle of proximity to obtain the complete road angle information.

[0024] Furthermore, all road angle information is stored for training purposes. For example, it can be stored in a matrix format.

[0025] Step 1.2: The road extraction algorithm model extracts road information from the input remote sensing image, generates prediction results of road information, and simultaneously generates prediction results of road angles during the prediction process.

[0026] Preferably, the following method can be used:

[0027] Several remote sensing images are randomly selected from the training dataset and input into the road extraction algorithm model.

[0028] The road extraction algorithm model outputs a prediction of the road information corresponding to each remote sensing image, and simultaneously generates a prediction result of the road angle during the process.

[0029] The width and height of the predicted road information are consistent with the input remote sensing image. The solution space is 0 or 1, where 0 represents that the pixel is not predicted as a road, and 1 represents that the pixel is predicted as a road. The predicted road angle information is in matrix format, and its solution space contains any valid angle values. For example, the range of angle values ​​can be limited to any real number in the range [-π / 2, π / 2). The dimensions of this matrix are consistent with the input remote sensing image.

[0030] Specifically, the road extraction algorithm model consists of three parts: feature extraction, preliminary prediction, and feature fusion.

[0031] When remote sensing imagery is input into the road extraction algorithm model, it first undergoes a feature extraction section. This section uses a Transformer structure based on a self-attention mechanism to extract image features from the remote sensing imagery. The steps include:

[0032] The feature extraction part first uses tile splitting to divide the input H×W×3 RGB remote sensing image into N×p layers. 2 The ×3 format involves dividing the original remote sensing image, which has a height of H, a width of W, and 3 channels, into N tiles. Each tile has a width and height of p and still has 3 channels.

[0033] These tiles are passed through a linear embedding layer, and the p of each tile is... 2 The 3×3 tensor is projected onto a vector of arbitrary dimension; here, the linear embedding layer is essentially a fully connected layer. The goal is to map each p... 2 The RGB tiles of size ×3 are mapped to linear vectors for use in subsequent structures.

[0034] These linear vectors are input into the Swing block of the self-attention mechanism. In the Swing block, these linear vectors are first regularized and then input into either the windowing multi-head self-attention module W-MSA (Windowing Multi-head Self-Attention) or the shifted windowing multi-head self-attention module SW-MSA (Shifted Windowing Multi-head Self-Attention).

[0035] Among them, the window-based multi-head self-attention module calculates self-attention within each tile of the tile-slicing layer, which significantly improves the computational efficiency of the model, but it limits information exchange between windows. Therefore, the model alternates between the window-based multi-head self-attention module and the shift-window-based multi-head self-attention module in every two consecutive adjacent Swin blocks.

[0036] In the multi-head self-attention module based on shifted windows, the model shifts each window by 1 / 2 window size in both the horizontal and vertical directions, calculates self-attention, and then performs reverse shifting to complete the information exchange between windows.

[0037] Following the self-attention module, the model performs residual connections and continues to process features using regularization layers and multilayer perceptron layers. The multilayer perceptron is a fundamental structure in deep learning, based on fully connected principles. In the Swin structures following the feature extraction part, the model uses patch fusion layers instead of linear embedding layers, downsampling the current feature map at the beginning of each Swin structure.

[0038] After the feature extraction section extracts features from the remote sensing image, the model inputs the obtained features into the preliminary prediction section to make preliminary predictions about the area and angle of the road.

[0039] In the initial prediction part, the model uses two modules: a semantic segmentation module and an angle prediction module, as decoders to decode the feature map.

[0040] The semantic segmentation module aims to perform preliminary road area prediction, while the angle prediction module aims to predict road angles.

[0041] In both decoders, instead of directly using convolutional networks to upsample feature maps to form predictions, a pixel decoder and a standard Transformer decoder are used in parallel to predict two sets of values: a mask set M and a mask-to-class mapping matrix C. The mask set M contains N masks, each with a size of H×W, consistent with the input image. The mapping matrix C has a size of N×(K+1), which associates the N masks with the distribution of the K classes to be predicted (with an additional empty class label) through matrix multiplication with the mask set M. The pixel decoder uses traditional convolutional upsampling, while the Transformer decoder uses multiple attention layers for upsampling. In the angle prediction module, an additional tanh layer is used to constrain the output to -1 to 1, and then multiply by π / 2 to map it to the angle space when subsequently calculating the angle.

[0042] The semantic segmentation module and the angle prediction module obtain preliminary road area predictions and road angle predictions, respectively. These predicted values ​​are compared with the corresponding ground truth values ​​to calculate the loss function.

[0043] After obtaining preliminary road area and road angle predictions through the preliminary prediction section, the feature fusion section is used to fuse the road area prediction features and road angle features, thereby further improving the accuracy of road prediction.

[0044] Specifically, the model concatenates the original remote sensing imagery, preliminary road prediction results, road angle prediction results, and the mask group M generated during the preliminary road prediction process into a multi-channel feature map, which is then input into the final feature fusion module to obtain the final road prediction result. This feature fusion module uses several convolutional layers to upsample the feature map.

[0045] Step 1.3: The road extraction algorithm model compares the predicted road information and road angles with the corresponding true values, calculates the loss function, and updates the parameters in the model accordingly.

[0046] Preferably, the following method can be used:

[0047] To effectively train the model, three sub-loss functions are used at different modules, and the overall loss function is expressed as follows:

[0048]

[0049] in, Represents the overall loss function. This represents the loss function that generates the preliminary road prediction results from the constraint-based preliminary prediction part. This represents the loss function that generates the angle prediction results from the initial constraint prediction part. This represents the loss function that constrains the final road prediction result.

[0050] Specifically, for the semantic segmentation module in the preliminary prediction part, using The prediction results M of the mask group of the module are constrained to the prediction results C of the mapping matrix from the mask to the category of the module.

[0051] For mask group M, a combination of focus loss and dice loss is used for constraint.

[0052] For the mapping matrix C, cross-entropy classification loss is used as a constraint.

[0053] Therefore, the loss function constrains the preliminary road prediction results. Expressed as:

[0054]

[0055] in, Let c and c represent the cross-entropy classification loss function, focus loss function, and dice loss function, respectively; gt Let m and m represent the predicted result and the true value of the mapping matrix C, respectively; gt These represent the predicted result and the true value of mask group M, respectively.

[0056] For the angle prediction module in the preliminary prediction section, this module predicts the angle value corresponding to each road pixel, expressed as a value between -π / 2 and π / 2. Considering this, a loss function is designed to constrain the angle prediction results. It is represented as:

[0057]

[0058] Where x and y represent the horizontal and vertical coordinates of the mask / matrix, respectively; r gt Let t and t represent the truth values ​​of the road region. gt These represent the predicted result and the true value of the road angle matrix, respectively.

[0059] Finally, a loss function is also needed. The final road prediction result is constrained by multiple convolutional layers. The feature fusion and road prediction in the last step can be regarded as a pixel-level semantic segmentation result. Therefore, the final prediction result is constrained by a pixel classification type loss function (such as cross-entropy loss, focus loss, lovász loss, or a combination thereof).

[0060] Step 1.4: Repeat steps 1.2 and 1.3 until the set termination condition is met, and save the network structure and model parameters.

[0061] During training, remote sensing images are repeatedly input into the model, and the parameters in the model are dynamically adjusted according to the loss function until the preset conditions are met (such as: the number of training rounds reaches a certain amount or a certain evaluation index is better than a certain preset value). At this point, the training process stops, and the model parameters are stored.

[0062] Step 2: Usage phase.

[0063] Using the model parameters obtained in step 1 during the training phase, road information is extracted from remote sensing images.

[0064] Furthermore, if there is real road information corresponding to the remote sensing image, the extracted road information will be compared and evaluated with the real road information to determine the quality of the model's road information extraction.

[0065] Specifically, step 2 includes the following steps:

[0066] Step 2.1: Create a network model based on the model structure and parameters saved during the training phase, input the collected remote sensing images into the model for computation, and store the road information extracted and predicted by the model.

[0067] The remote sensing images used in the model are usually in RGB format and are composed of multiple tiles of the same size stitched together.

[0068] If the acquired remote sensing image is not in this format, the complete remote sensing image is first segmented and stored. Then, a network model is created based on the model structure and parameters saved during the training phase, and the remote sensing image tiles are input into the model. The model performs predictions through feature extraction, preliminary prediction, and feature fusion, and automatically saves the final predicted road information.

[0069] The road information is in binary mask or binary matrix format, and its length and width dimensions are consistent with the input remote sensing image tiles.

[0070] Step 2.2: If the remote sensing image used has corresponding real road information, then compare and evaluate the predicted road information with the real road information to determine the effectiveness of the model in predicting road information.

[0071] To objectively assess the accuracy of the model's road predictions, the predicted road information can be directly compared with the real road information based on accuracy metrics such as IoU (Intersection over Union) score and F1 score. The IoU score is a commonly used evaluation metric in semantic segmentation; it is the ratio of the area of ​​intersection to the area of ​​union between the real and predicted regions corresponding to a certain semantic category. Its calculation formula is as follows:

[0072]

[0073] In this context, TP, FP, and FN represent the number of pixels that are true positives, false positives, and false negatives, respectively.

[0074] The F1 score is a statistical metric used to measure the accuracy of binary classification. It is calculated based on precision and recall, and its formula is as follows:

[0075]

[0076] Where P and R represent precision and recall, respectively, and TP, FP, and FN represent the number of true positives, false positives, and false negatives, respectively.

[0077] Furthermore, based on the above method, this invention proposes a remote sensing image road extraction system that combines semantic segmentation and angle prediction. The system includes: a real road information preprocessing subsystem, a remote sensing image preprocessing subsystem, a feature extraction subsystem, a preliminary prediction subsystem, a feature fusion subsystem, a loss function calculation subsystem, and a result evaluation subsystem.

[0078] The real road information preprocessing subsystem is used to calculate real road angle information based on real road information. This facilitates comparison with the predicted road angle information in the loss function calculation subsystem to calculate the loss function, thereby improving model training. It includes a road skeleton extraction module, a road skeleton pixel-by-pixel angle calculation module, and a road non-skeleton point angle inference module.

[0079] The remote sensing image preprocessing subsystem includes a remote sensing image augmentation module and a remote sensing image formatting module. The remote sensing image augmentation module performs random cropping, rotation, and flipping operations on remote sensing images to increase the size of the remote sensing image dataset, making the images as diverse as possible and thus enhancing the generalization ability of the trained model. The remote sensing image formatting module converts remote sensing images stored in bitmap format into tensor format and normalizes them to meet the input data format requirements of the feature extraction subsystem.

[0080] The feature extraction subsystem is used to extract road features from remote sensing images, providing easily usable road feature information for the subsequent preliminary prediction subsystem. It includes a tile segmentation layer, a linear embedding layer, a tile fusion layer, and a Swing block structure.

[0081] The preliminary prediction subsystem includes a semantic segmentation module and an angle prediction module, which are used to make preliminary predictions about the region and angle of the road based on the road features extracted by the feature extraction subsystem. These preliminary prediction results will be temporarily saved so that the subsequent loss function calculation subsystem can calculate the loss function and adjust the model parameters.

[0082] The feature fusion subsystem includes several convolutional layers, which are used to fuse road area prediction features with road angle features, and combine these features to generate the final road prediction result, thereby further improving the accuracy of road prediction.

[0083] The loss function calculation subsystem includes a preliminary road prediction loss function calculation module, a preliminary angle prediction loss function calculation module, and a final road prediction loss function calculation module. The preliminary road prediction loss function calculation module calculates the loss function based on real road information and the preliminary road prediction results provided by the semantic segmentation module in the preliminary prediction subsystem. The preliminary angle prediction loss function calculation module calculates the loss function based on real road angle information and the road angle prediction matrix provided by the angle prediction module in the preliminary prediction subsystem. The final road prediction loss function calculation module calculates the loss function based on real road information and the final road prediction results provided by the feature fusion subsystem. These loss functions are backpropagated to update the parameters of each neural network layer in the feature extraction subsystem, the preliminary prediction subsystem, and the feature fusion subsystem, correcting the final road prediction results of the system.

[0084] The result evaluation subsystem is used to compare the final road prediction results with the actual road information and automatically calculate various road accuracy evaluation indicators such as IoU score and F1 score. As a supplement to the user's subjective experience, it helps the user to quantitatively understand the accuracy of the system's predicted road information.

[0085] The connections between the above-mentioned components are as follows:

[0086] The real road information preprocessing subsystem is connected to the loss function calculation subsystem and the generation result evaluation subsystem, providing real road information and real road angle matrix to the loss function calculation subsystem and the generation result evaluation subsystem;

[0087] The remote sensing image preprocessing subsystem is connected to the feature extraction subsystem and the feature fusion subsystem, providing the feature extraction subsystem and the feature fusion subsystem with processed remote sensing image images that meet the requirements;

[0088] The feature extraction subsystem is connected to the remote sensing image preprocessing subsystem and the preliminary prediction subsystem. It provides the extracted road features to the preliminary prediction subsystem and simultaneously receives the loss values ​​returned by the preliminary prediction subsystem for training itself.

[0089] The preliminary prediction subsystem is connected to the feature extraction subsystem, the feature fusion subsystem, and the loss function calculation subsystem, respectively. It provides the feature fusion subsystem with preliminary road prediction results, preliminary angle prediction results, and features generated during the preliminary road prediction process, and simultaneously receives the loss values ​​returned by the feature fusion subsystem for training itself. In addition, the preliminary road prediction results and preliminary angle prediction results generated by it are provided to the loss function calculation subsystem, and it receives the loss values ​​returned by the loss function calculation subsystem for training itself.

[0090] The feature fusion subsystem is connected to the remote sensing image preprocessing subsystem, the preliminary prediction subsystem, the loss function calculation subsystem, and the generation result evaluation subsystem. It fuses multiple features provided by the remote sensing image preprocessing subsystem and the preliminary prediction subsystem to generate the final road prediction information, and provides the final road prediction information to the loss function calculation subsystem and the generation result evaluation subsystem. At the same time, it receives the loss value calculated by the loss function calculation subsystem to train itself.

[0091] The loss function calculation subsystem is connected to the real road information preprocessing subsystem, the preliminary prediction subsystem, and the feature fusion subsystem. It compares the real road information provided by the real road information preprocessing subsystem with the preliminary road prediction information provided by the preliminary prediction subsystem and the final road prediction information provided by the feature fusion subsystem to calculate the loss value. At the same time, it compares the real angle matrix provided by the real road information preprocessing subsystem with the preliminary angle prediction result provided by the preliminary prediction subsystem to calculate the loss value. The loss values ​​are then fed back to the preliminary prediction subsystem and the feature fusion subsystem respectively for model training.

[0092] The result evaluation subsystem is connected to the real road information preprocessing subsystem and the feature fusion subsystem. It calculates evaluation indicators by comparing the real road information provided by the real road information preprocessing subsystem with the final road prediction information provided by the feature fusion subsystem. This helps users quantitatively understand the accuracy of the system's road prediction information.

[0093] The working process of the above system is as follows:

[0094] Step 1: Input all real road information into the real road information preprocessing subsystem in sequence, calculate the corresponding real road angle matrix, and store them in sequence for easy reuse later;

[0095] Step 2: Input the remote sensing image into the remote sensing image preprocessing subsystem for data augmentation, normalization and formatting. The processed remote sensing image is stored in memory.

[0096] Step 3: The feature extraction subsystem reads remote sensing images from the remote sensing image preprocessing subsystem, processes them through a neural network layer, and extracts road features;

[0097] Step 4: The preliminary prediction subsystem reads the road features and processes them using the semantic segmentation module and the angle prediction module respectively to obtain the preliminary road prediction results and the preliminary angle prediction results. At the same time, the intermediate results generated by the semantic segmentation module and the mask group M are saved.

[0098] Step 5: The feature fusion subsystem reads the remote sensing image, preliminary road prediction results, preliminary angle prediction results and mask group M, fuses these features, and generates the final road prediction result based on the composite features;

[0099] Step 6: The loss function calculation subsystem compares the real road information with the preliminary road prediction information and calculates the loss value, and sends the loss value back to the preliminary prediction subsystem for training; it compares the real angle matrix with the preliminary angle prediction results and calculates the loss value, and sends the loss value back to the preliminary prediction subsystem for training; it compares the real road information with the final road prediction information and calculates the loss value, and sends the loss value back to the feature fusion subsystem for training.

[0100] Step 7: Each neural network layer in the feature fusion subsystem calculates the loss value returned by the subsystem according to the loss function, modifies the parameters, completes one training iteration, and returns the loss value of the first layer to the preliminary prediction subsystem.

[0101] Step 8: The neural network layers in the preliminary prediction subsystem calculate the loss value returned by the subsystem based on the loss function and modify the parameters with the loss value returned by the feature fusion subsystem to complete one training iteration, and then return the loss value of the first layer to the feature extraction subsystem.

[0102] Step 9: Each neural network layer in the feature extraction subsystem modifies its parameters based on the loss value returned by the preliminary prediction subsystem, completing one training iteration;

[0103] Step 10: Repeat steps 2 to 9. After each iteration of all training data, send the real road information and the final road prediction information to the result evaluation subsystem to calculate the index. If the index meets the preset conditions or the total number of iterations reaches the preset upper limit, stop training and proceed to step 11.

[0104] Step 11: Execute steps 2 to 5 to generate the final road prediction information, and send the actual road information and the final road prediction information to the generation result evaluation subsystem to calculate the final index.

[0105] Beneficial effects

[0106] Compared with the prior art, the present invention has the following advantages:

[0107] 1. This invention calculates the road angle pixel by pixel from the original or collected road information in the dataset, which explicitly reflects the geometric feature that the road tilt angle has a certain stability, and provides the neural network with the true value of the road angle for comparison to constrain this geometric feature;

[0108] 2. This invention designs an angle prediction module and a feature fusion part in the model. The angle prediction module is used to predict the road angle features, and the feature fusion part is used to fuse remote sensing images, preliminary predicted road information and features, and predicted road angle features. Based on this, the road is finally predicted, which better utilizes the various information contained in the remote sensing images to improve the accuracy of the final road prediction results.

[0109] 3. This invention designs a loss function for the prediction results based on constraint angles. Explicit constraints are added to the angle prediction module in the above model to enable the model structure to converge effectively. Attached Figure Description

[0110] Figure 1 This is a flowchart of the method of the present invention.

[0111] Figure 2 This is a schematic diagram of the core algorithm model described in the method of this invention.

[0112] Figure 3 This is a detailed structural diagram of the feature extraction part in the core algorithm model of the present invention. The upper part is a structural diagram of the backbone feature extractor, and the lower part is a structural diagram of the internal structure of two consecutive adjacent Swing blocks in the backbone feature extractor.

[0113] Figure 4 This is a detailed structural diagram of the preliminary prediction part in the core algorithm model of the method of the present invention. The upper part is a structural diagram of the semantic segmentation module in the preliminary prediction part, and the lower part is a structural diagram of the angle prediction module in the preliminary prediction part.

[0114] Figure 5 This is a schematic diagram of the system composition of the present invention. Detailed Implementation

[0115] To better illustrate the purpose and advantages of the present invention, the method of the invention will be further described below in conjunction with the accompanying drawings and examples.

[0116] Example

[0117] This embodiment discloses a method for road extraction from remote sensing images that combines semantic segmentation and angle prediction, including a training phase and an application phase.

[0118] During the training phase, road angles are first calculated using methods such as Hough transform based on real road information in the dataset, obtaining the angle value corresponding to each road category pixel and forming real road angle information. Then, a road extraction algorithm model extracts road information from the input remote sensing imagery, generating a road information prediction result and simultaneously generating a road angle prediction result. After prediction, the road extraction algorithm model compares the predicted road information and road angles with the corresponding real values, calculates the loss function, and updates the model parameters accordingly. The second and third steps are repeated until a set termination condition is met, at which point the network structure and model parameters are saved. During the usage phase, a network model is created based on the model structure and parameters saved during training. The acquired remote sensing images are input into the model for computation, and the extracted and predicted road information is stored. If the remote sensing imagery contains corresponding real road information, the predicted road information is compared with the real road information to evaluate the model's performance in predicting road information. The flowchart of this embodiment is as follows: Figure 1 As shown.

[0119] Traditional road information acquisition methods require significant manpower and resources for on-site surveying, acquiring and recording road vector data, and then importing it into computers to create easily viewable road network maps. This process is cumbersome, lengthy, and costly. In recent years, the rapid development of deep learning technology has significantly improved the effectiveness of road extraction based on remote sensing imagery. Most existing deep learning-based road extraction methods treat this task as a semantic segmentation task, using semantic segmentation models with deep convolutional neural networks at their core for road region extraction. Compared to traditional methods that rely on one or more manually summarized road feature models, semantic segmentation models using deep convolutional neural networks can learn the necessary composite features from a large amount of remote sensing imagery and comprehensively utilize these features to improve the accuracy of road extraction. However, current semantic segmentation methods still commonly produce road breaks, blurred road boundaries, and abrupt changes in road width when extracting roads from remote sensing imagery. This embodiment discloses a remote sensing image road extraction method and system based on joint semantic segmentation and angle prediction. It proposes a core algorithm incorporating an angle prediction module and a feature fusion module. By adding prediction, fusion, and explicit constraints on road angle features to the model, the model can better utilize the various information contained in the remote sensing image to improve the accuracy of the final road prediction result. A schematic diagram of the main network model structure of this method and system is shown below. Figure 2 As shown. The detailed structure of the feature extraction part in the model is as follows: Figure 3 As shown, the detailed structure of the preliminary prediction part in the model is as follows: Figure 4 As shown.

[0120] The aforementioned remote sensing image road extraction system combining semantic segmentation and angle prediction includes a real road information preprocessing subsystem, a remote sensing image preprocessing subsystem, a feature extraction subsystem, a preliminary prediction subsystem, a feature fusion subsystem, a loss function calculation subsystem, and a result evaluation subsystem. The structure and connections of these subsystems are as follows: Figure 5 As shown.

[0121] The details of this embodiment are as follows.

[0122] A method for road extraction from remote sensing images combining semantic segmentation and angle prediction, comprising two phases: training and application. It includes the following steps:

[0123] Step 1: During the training phase, using the remote sensing imagery-road information pairing dataset, iteratively modify and store the parameter dictionary of the algorithm model.

[0124] Step 1.1: Based on the real road information in the dataset, calculate the road angles using methods such as Hough transform to obtain the angle value corresponding to each road category pixel, and form the real road angle information.

[0125] First, morphological methods are used to refine the real road information, obtaining road skeleton information. This step aims to reduce the computational load and improve efficiency in subsequent steps. Second, for each pixel in the road skeleton information, a neighborhood road skeleton image centered on that pixel is taken, and a Hough transform is applied to this neighborhood image to calculate a straight line that fits the road skeleton of that region. The inclination angle of this line is taken as the road angle corresponding to the center pixel. Third, based on the angle information of the road skeleton points, the angle information of other points on the road is inferred according to the proximity principle to obtain complete road angle information. Finally, all road angle information is stored in matrix format for training purposes.

[0126] Step 1.2: The road extraction algorithm model extracts road information from the input remote sensing image, generates a prediction result of the road information, and simultaneously generates a prediction result of the road angle during the prediction process.

[0127] Several remote sensing images are randomly selected from the training dataset and input into the road extraction algorithm model. The model outputs a predicted road information for each remote sensing image, and simultaneously generates a predicted road angle. The width and height of the predicted road information are consistent with the input remote sensing images, and the solution space is 0 or 1, where 0 represents that the pixel is not predicted as a road, and 1 represents that the pixel is predicted as a road. The predicted road angle information is in matrix format, and its solution space contains any valid angle values, typically limited to any real number in the range [-π / 2, π / 2). The dimensions of this matrix are consistent with the input remote sensing images.

[0128] The road extraction algorithm model consists of three main parts: feature extraction, preliminary prediction, and feature fusion.

[0129] After the remote sensing image is input into the algorithm model, it first undergoes a feature extraction section. This section uses a Transformer structure based on a self-attention mechanism to extract image features from the remote sensing image. The feature extraction section first uses tile-based layering to divide the input H×W×3 RGB remote sensing image into N×p layers. 2 The original remote sensing image, with a height of H, a width of W, and three channels, is divided into N tiles in a ×3 format. Each tile has a width and height of p and still has 3 channels. Next, these tiles are processed through a linear embedding layer, which further divides the p channels of each tile into N tiles. 2The 3×3 tensor is projected onto a vector of arbitrary dimension; here, the linear embedding layer is essentially a fully connected layer. The significance of this step is to map each p... 2 The RGB tiles, each ×3, are mapped to linear vectors for use in subsequent structures. These vectors are then fed into the Swin block, which employs a self-attention mechanism. Within the Swin block, these vectors are first regularized and then input into either the windowing multi-head self-attention (W-MSA) module or the shifted windowing multi-head self-attention (SW-MSA) module. The windowing-based W-MSA module calculates self-attention within each tile, significantly improving computational efficiency, but it limits cross-window communication. Therefore, the model alternates between the windowing-based and shifted windowing multi-head self-attention modules in every two consecutive adjacent Swin blocks. In the shifted windowing multi-head self-attention module, each window is shifted by half its size horizontally and vertically, self-attention is calculated, and then the shift is reversed to facilitate cross-window communication. Following the self-attention module, the model performs residual connections and continues to process features using regularization layers and multilayer perceptron layers. The multilayer perceptron is a fundamental structure in deep learning, based on fully connected principles. In the later Swin structures of the feature extraction part, the model uses patch fusion layers instead of linear embedding layers, which downsample the current feature map at the beginning of each Swin structure.

[0130] After extracting features from the remote sensing imagery, the model inputs these features into the preliminary prediction section to make initial predictions about the road's region and angle. In the preliminary prediction section, the model uses two modules—a semantic segmentation module and an angle prediction module—as decoders to decode the feature maps. The semantic segmentation module aims to predict the road region, while the angle prediction module aims to predict the road angle. Instead of directly using convolutional networks to upsample the feature maps and form prediction results, the two decoders use a pixel decoder and a standard Transformer decoder in parallel to predict two sets of values: a mask set M and a mask-to-class mapping matrix C. The mask set M contains N masks, each with a size of H×W, consistent with the input image. The mapping matrix C has a size of N×(K+1), which associates the N masks with the distribution of the K classes to be predicted (with an additional empty class label) through matrix multiplication with the mask set M. The pixel decoder uses traditional convolutional upsampling, while the Transformer decoder uses multiple attention layers for upsampling. An additional tanh layer is used in the angle prediction module to constrain the output to -1 to 1, and multiplies it by π / 2 to map it to the angle space when calculating the angle later. The semantic segmentation module and the angle prediction module will obtain preliminary road region predictions and road angle predictions, respectively. These predicted values ​​will be compared with the corresponding ground truth values ​​to calculate the loss function.

[0131] After obtaining preliminary road area and road angle predictions through the initial prediction section, the model uses a feature fusion module to fuse the road area and road angle features, further improving the accuracy of road prediction. Specifically, the model concatenates the original remote sensing image, the preliminary road prediction results, the road angle prediction results, and the mask group M generated during the preliminary road prediction process into a multi-channel feature map, and inputs it into the final feature fusion module to obtain the final road prediction result. This feature fusion module uses multiple convolutional layers to upsample the feature map.

[0132] Step 1.3: The road extraction algorithm model compares the predicted road information and road angles with the corresponding true values, calculates the loss function, and updates the parameters in the model accordingly.

[0133] To effectively train the model, three sub-loss functions are used at different modules. The overall loss function can be expressed as follows:

[0134]

[0135] in Represents the overall loss function. This represents the loss function that generates the preliminary road prediction results from the constraint-based preliminary prediction part. This represents the loss function that generates the angle prediction results from the initial constraint prediction part. This represents the loss function that constrains the final road prediction result.

[0136] Specifically, for the semantic segmentation module in the initial prediction part, using The loss function constrains the prediction results M of the mask group of the module and the prediction results C of the mask-to-class mapping matrix of the module. For the mask group M, a combination of focus loss and dice loss is used for constraint; for the mapping matrix C, the traditional cross-entropy classification loss is used for constraint. Therefore, the loss function constrains the initial road prediction results. It can be expressed as

[0137]

[0138] in These are the cross-entropy classification loss function, focus loss function, and dice loss function, respectively; c and c gt Let m and m represent the predicted result and the true value of the mapping matrix C, respectively; gt These represent the predicted result and the true value of mask group M, respectively.

[0139] For the angle prediction module in the preliminary prediction section, this module predicts the angle value corresponding to each road pixel, expressed as a value between -π / 2 and π / 2. Considering this, a loss function is designed to constrain the angle prediction results. It can be represented as

[0140]

[0141] Where x and y represent the horizontal and vertical coordinates of the mask or matrix, respectively, and r gt Represent the truth values ​​of the road region, t and t gt These represent the predicted result and the true value of the road angle matrix, respectively.

[0142] Finally, a loss function is also needed. The final road prediction result is constrained by multiple convolutional layers. The feature fusion and road prediction in the last step can be regarded as a pixel-level semantic segmentation result. Therefore, a pixel classification type loss function is used to constrain the final prediction result, such as cross-entropy loss, focus loss, lovász loss, or a combination thereof.

[0143] Step 1.4: Repeat steps 1.2 and 1.3 until the set termination condition is met, then save the network structure and model parameters.

[0144] During training, remote sensing images are repeatedly input into the model, and the parameters in the model are dynamically adjusted according to the loss function until a certain preset condition is met, such as the number of training rounds reaching a certain amount or a certain evaluation index being better than a certain preset value. At this point, the training process stops and the model parameters are stored.

[0145] Step 2: Usage Phase. Using the model parameters obtained in Step 1 (training phase), road information is extracted from the remote sensing imagery and stored. Furthermore, if corresponding real road information exists for the remote sensing imagery, the extracted road information can be compared and evaluated with the real road information to determine the effectiveness of the model in extracting road information.

[0146] Step 2.1: Create a network model based on the model structure and parameters saved during the training phase, input the collected remote sensing images into the model for computation, and store the road information extracted and predicted by the model.

[0147] The remote sensing images used by the model are typically in RGB format and are composed of multiple tiles of uniform size stitched together. If the acquired remote sensing image is not in this format, the complete remote sensing image is first segmented and stored. Then, a network model is created based on the model structure and parameters saved during the training phase. The remote sensing image tiles are input into the model, which performs predictions through feature extraction, preliminary prediction, and feature fusion. The model automatically saves the final predicted road information, which will be in binary mask or binary matrix format, with dimensions consistent with the input remote sensing image tiles.

[0148] Step 2.2: If the remote sensing image used has corresponding real road information, then compare and evaluate the predicted road information with the real road information to determine the effectiveness of the model in predicting road information.

[0149] To objectively assess the accuracy of the model's predicted road information, the predicted road information can be directly compared with the real road information based on accuracy metrics such as IoU (Intersection over Union) score and F1 score. The IoU score is a commonly used evaluation metric in semantic segmentation; it is the ratio of the area of ​​intersection to the area of ​​union between the real and predicted regions corresponding to a certain semantic category. Its calculation formula can be expressed as follows:

[0150]

[0151] TP, FP, and FN represent the number of pixels that are true positives, false positives, and false negatives, respectively.

[0152] The F1 score is a statistical metric used to measure the accuracy of binary classification. It is calculated based on precision and recall, and the formula can be expressed as follows:

[0153]

[0154] Where P and R represent precision and recall, respectively, and TP, FP, and FN represent the number of true positives, false positives, and false negatives, respectively.

[0155] Furthermore, to better illustrate the purpose and advantages of the present invention, the inventive system will be further described below in conjunction with the accompanying drawings.

[0156] Example

[0157] This embodiment discloses a remote sensing image road extraction system that combines semantic segmentation and angle prediction, specifically comprising the following subsystems: a real road information preprocessing subsystem, a remote sensing image preprocessing subsystem, a feature extraction subsystem, a preliminary prediction subsystem, a feature fusion subsystem, a loss function calculation subsystem, and a result evaluation subsystem.

[0158] The real road information preprocessing subsystem includes a road skeleton extraction module, a road skeleton pixel-by-pixel angle calculation module, and a road non-skeleton point angle inference module. It is used to calculate real road angle information based on real road information, so that it can be compared with the predicted road angle information in the loss function calculation subsystem to calculate the loss function and better train the model.

[0159] The remote sensing image preprocessing subsystem includes a remote sensing image augmentation module and a remote sensing image formatting module. The remote sensing image augmentation module performs random cropping, rotation, and flipping operations on the remote sensing images to increase the size of the remote sensing image dataset, making the images as diverse as possible, thereby enhancing the generalization ability of the training model. The remote sensing image formatting module converts remote sensing images stored in bitmap format into tensor format and normalizes them to meet the input data format requirements of the feature extraction subsystem.

[0160] The feature extraction subsystem includes a tile segmentation layer, a linear embedding layer, a tile fusion layer, and a Swin block structure, which are used to extract road features from remote sensing images and provide easy-to-use road feature information for the subsequent preliminary prediction subsystem.

[0161] The preliminary prediction subsystem includes a semantic segmentation module and an angle prediction module, which respectively make preliminary predictions about the region and angle of the road based on the road features extracted by the feature extraction subsystem. These preliminary prediction results will be temporarily saved so that the subsequent loss function calculation subsystem can calculate the loss function and adjust the model parameters;

[0162] The feature fusion subsystem includes multiple convolutional layers that fuse road area prediction features with road angle features, and use these features in combination to generate the final road prediction result, thereby further improving the accuracy of road prediction.

[0163] The loss function calculation subsystem includes a preliminary road prediction loss function calculation module, a preliminary angle prediction loss function calculation module, and a final road prediction loss function calculation module. The preliminary road prediction loss function calculation module calculates the loss function based on real road information and the preliminary road prediction results provided by the semantic segmentation module in the preliminary prediction subsystem. The preliminary angle prediction loss function calculation module calculates the loss function based on real road angle information and the road angle prediction matrix provided by the angle prediction module in the preliminary prediction subsystem. The final road prediction loss function calculation module calculates the loss function based on real road information and the final road prediction results provided by the feature fusion subsystem. These loss functions are backpropagated to update the parameters of each neural network layer in the feature extraction subsystem, the preliminary prediction subsystem, and the feature fusion subsystem, correcting the final road prediction results of the system.

[0164] The generated result evaluation subsystem is used to compare the final road prediction result with the actual road information and automatically calculate various road accuracy evaluation indicators such as IoU score and F1 score. As a supplement to the user's subjective feeling, it helps the user to quantitatively understand the accuracy of the system's predicted road information.

[0165] The connections between the above-mentioned components are as follows:

[0166] The real road information preprocessing subsystem is connected to the loss function calculation subsystem and the generation result evaluation subsystem, providing real road information and real road angle matrix to the loss function calculation subsystem and the generation result evaluation subsystem;

[0167] The remote sensing image preprocessing subsystem is connected to the feature extraction subsystem and the feature fusion subsystem, providing the feature extraction subsystem and the feature fusion subsystem with processed remote sensing image images that meet the requirements;

[0168] The feature extraction subsystem is connected to the remote sensing image preprocessing subsystem and the preliminary prediction subsystem. It provides the extracted road features to the preliminary prediction subsystem and simultaneously receives the loss values ​​returned by the preliminary prediction subsystem for training itself.

[0169] The preliminary prediction subsystem is connected to the feature extraction subsystem, the feature fusion subsystem, and the loss function calculation subsystem. It provides the feature fusion subsystem with preliminary road prediction results, preliminary angle prediction results, and features generated during the preliminary road prediction process. At the same time, it receives the loss value returned by the feature fusion subsystem for training itself. In addition, the preliminary road prediction results and preliminary angle prediction results generated by it are provided to the loss function calculation subsystem, and it receives the loss value returned by the loss function calculation subsystem for training itself.

[0170] The feature fusion subsystem is connected to the remote sensing image preprocessing subsystem, the preliminary prediction subsystem, the loss function calculation subsystem, and the generation result evaluation subsystem. It fuses multiple features provided by the remote sensing image preprocessing subsystem and the preliminary prediction subsystem to generate the final road prediction information, and provides the final road prediction information to the loss function calculation subsystem and the generation result evaluation subsystem. At the same time, it receives the loss value calculated by the loss function calculation subsystem to train itself.

[0171] The loss function calculation subsystem is connected to the real road information preprocessing subsystem, the preliminary prediction subsystem, and the feature fusion subsystem. It compares the real road information provided by the real road information preprocessing subsystem with the preliminary road prediction information provided by the preliminary prediction subsystem and the final road prediction information provided by the feature fusion subsystem to calculate the loss value. At the same time, it compares the real angle matrix provided by the real road information preprocessing subsystem with the preliminary angle prediction result provided by the preliminary prediction subsystem to calculate the loss value. The loss values ​​are then fed back to the preliminary prediction subsystem and the feature fusion subsystem respectively for model training.

[0172] The result evaluation subsystem is connected to the real road information preprocessing subsystem and the feature fusion subsystem. It calculates evaluation indicators by comparing the real road information provided by the real road information preprocessing subsystem with the final road prediction information provided by the feature fusion subsystem. This helps users quantitatively understand the accuracy of the system's road prediction information.

[0173] The usage method of the above system is as follows:

[0174] Step 1: Input all real road information into the real road information preprocessing subsystem in sequence, calculate the corresponding real road angle matrix, and store them in sequence for easy reuse later;

[0175] Step 2: Input the remote sensing image into the remote sensing image preprocessing subsystem for data augmentation, normalization and formatting. The processed remote sensing image is stored in memory.

[0176] Step 3: The feature extraction subsystem reads remote sensing images from the remote sensing image preprocessing subsystem, processes them through a neural network layer, and extracts road features;

[0177] Step 4: The preliminary prediction subsystem reads the road features and processes them using the semantic segmentation module and the angle prediction module respectively to obtain the preliminary road prediction results and the preliminary angle prediction results. At the same time, the intermediate results generated by the semantic segmentation module and the mask group M are saved.

[0178] Step 5: The feature fusion subsystem reads the remote sensing image, preliminary road prediction results, preliminary angle prediction results and mask group M, fuses these features, and generates the final road prediction result based on the composite features;

[0179] Step 6: The loss function calculation subsystem compares the real road information with the preliminary road prediction information and calculates the loss value, and sends the loss value back to the preliminary prediction subsystem for training; it compares the real angle matrix with the preliminary angle prediction results and calculates the loss value, and sends the loss value back to the preliminary prediction subsystem for training; it compares the real road information with the final road prediction information and calculates the loss value, and sends the loss value back to the feature fusion subsystem for training.

[0180] Step 7: Each neural network layer in the feature fusion subsystem calculates the loss value returned by the subsystem according to the loss function, modifies the parameters, completes one training iteration, and returns the loss value of the first layer to the preliminary prediction subsystem.

[0181] Step 8: The neural network layers in the preliminary prediction subsystem calculate the loss value returned by the subsystem based on the loss function and modify the parameters with the loss value returned by the feature fusion subsystem to complete one training iteration, and then return the loss value of the first layer to the feature extraction subsystem.

[0182] Step 9: Each neural network layer in the feature extraction subsystem modifies its parameters based on the loss value returned by the preliminary prediction subsystem, completing one training iteration;

[0183] Step 10: Repeat steps 2 to 9. After each iteration of all training data, send the real road information and the final road prediction information to the generation result evaluation subsystem to calculate the index. If the index meets the preset conditions, or the total number of iterations reaches the preset upper limit, stop training and proceed to step 11.

[0184] Step 11: Execute steps 2 to 5 to generate the final road prediction information, and send the actual road information and the final road prediction information to the generation result evaluation subsystem to calculate the final index.

[0185] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for road extraction from remote sensing images that combines semantic segmentation and angle prediction, characterized in that, Includes the training phase and the usage phase; During the training phase, the parameter dictionary of the algorithm model was iteratively modified using a remote sensing image-road information pairing dataset. First, the road angles are calculated based on the real road information in the dataset to obtain the angle value corresponding to each road category pixel, thus forming real road angle information; Then, the road extraction algorithm model is used to extract road information from the input remote sensing image, generate prediction results of road information, and generate prediction results of road angles simultaneously during the prediction process; After making a prediction, the road extraction algorithm model compares the predicted road information and road angle with the corresponding true values, calculates the loss function, and updates the parameters in the model accordingly. Repeat the above prediction and comparison process until the set termination condition is met, then save the network structure and model parameters. During the training phase, the road extraction algorithm model extracts road information from the input remote sensing imagery, generates prediction results for road information, and simultaneously generates prediction results for road angles during the prediction process. The specific method is as follows: First, randomly select several remote sensing images from the training dataset and input them into the road extraction algorithm model; The road extraction algorithm model outputs a prediction of the road information corresponding to each remote sensing image, and simultaneously generates a prediction result of the road angle during the process. Among them, the width and height of the road information prediction information are consistent with the input remote sensing image, and the solution space is 0 or 1, where 0 means that the pixel is not predicted as a road and 1 means that the pixel is predicted as a road; the road angle information predicted by the model is in matrix format, and its solution space is any legal angle value; the length and width of the matrix are consistent with the input remote sensing image. The road extraction algorithm model consists of three parts: feature extraction, preliminary prediction, and feature fusion. When remote sensing images are input into the road extraction algorithm model, they first go through the feature extraction part, which uses a Transformer structure based on a self-attention mechanism to extract image features from the remote sensing images. After the feature extraction part extracts features from the remote sensing image, the model inputs the obtained features into the preliminary prediction part to make a preliminary prediction of the area and angle of the road. After obtaining preliminary road area predictions and road angle predictions through the preliminary prediction part, the feature fusion part is used to fuse the road area prediction features and road angle features. During the usage phase, firstly, a network model is created based on the model structure and parameters saved during the training phase. The collected remote sensing images are then input into the model for calculation, and the road information extracted and predicted by the model is stored. If the remote sensing images used have corresponding real road information, the predicted road information is compared and evaluated with the real road information to determine the effectiveness of the model in predicting road information. The feature extraction part first uses tile splitting to divide the input H×W×3 RGB remote sensing image into N×p layers. 2 The ×3 format is used to divide the original remote sensing image with a height of H, a width of W, and 3 channels into N tiles, each tile having a width and height of p and still having 3 channels; These tiles are passed through a linear embedding layer, and the p of each tile is... 2 The 3×3 tensor is projected onto a vector of arbitrary dimension. Here, the linear embedding layer is essentially a fully connected layer, the purpose of which is to map each p... 2 The RGB tiles of size ×3 are mapped to linear vectors for later use; These linear vectors are input into the Swing block of the self-attention mechanism; in the Swing block, these linear vectors are first regularized and then input into the window-based multi-head self-attention module W-MSA or the shift-window-based multi-head self-attention module SW-MSA. Among them, the window-based multi-head self-attention module takes the tiles segmented by the tile cutting layer as the unit, and the model alternates between the window-based multi-head self-attention module and the shift window-based multi-head self-attention module in every two consecutive adjacent Swin blocks. In the multi-head self-attention module based on shifted windows, the model shifts each window by 1 / 2 window size in the horizontal and vertical directions respectively, calculates self-attention, and then performs reverse shifting to complete the information exchange between windows. After the self-attention module, the model performs residual connections and continues to use regularization layers and multilayer perceptron layers to process features; In the Swin structure segment following the feature extraction part, the model uses a patch fusion layer instead of a linear embedding layer, which downsamples the current feature map at the beginning of each Swin structure segment. In the initial prediction part, the model uses two modules: a semantic segmentation module and an angle prediction module, as decoders to decode the feature map; The semantic segmentation module aims to perform preliminary road area prediction, while the angle prediction module aims to predict road angles. In both decoders, instead of directly using convolutional networks to upsample feature maps to form prediction results, pixel decoders and standard Transformer decoders are used in parallel to predict two sets of values: a mask set M and a mask-to-class mapping matrix C. The mask set M contains N masks, each with a size of H×W, consistent with the input image. The mapping matrix C has a size of N×(K+1), which associates the N masks with the distribution of the K classes to be predicted through matrix multiplication with the mask set M. The pixel decoder uses traditional convolutional upsampling, while the Transformer decoder uses multiple attention layers for upsampling. In the angle prediction module, an additional tanh layer is used to constrain the output to -1 to 1, and then multiplies it by π / 2 to map it to the angle space when calculating the angle later. The semantic segmentation module and the angle prediction module obtain preliminary road area predictions and road angle predictions, respectively. These predicted values ​​are compared with the corresponding ground truth values ​​to calculate the loss function.

2. The remote sensing image road extraction method based on joint semantic segmentation and angle prediction as described in claim 1, characterized in that, During the training phase, road angles are calculated based on real road information in the dataset to obtain the angle value corresponding to each road category pixel, thus obtaining the real road angle information. The implementation method is as follows: First, morphological methods are used to refine the real road information to obtain road skeleton information; Then, for each pixel in the road skeleton information, take a neighborhood road skeleton information image centered on it, and use Hough transform on the neighborhood road skeleton information image to calculate a straight line that fits the road skeleton of the region. The tilt angle of the straight line is taken as the road angle corresponding to the center pixel. Then, based on the angle information of the road skeleton points, the angle information of the remaining points on the road is inferred according to the principle of proximity to obtain the complete road angle information.

3. The remote sensing image road extraction method based on joint semantic segmentation and angle prediction as described in claim 1, characterized in that, The model connects the original remote sensing image, the preliminary road prediction result, the road angle prediction result, and the mask group M generated during the preliminary road prediction process into a multi-channel feature map, and inputs it into the final feature fusion module to obtain the final road prediction result. This feature fusion module uses several convolutional layers to upsample the feature map.

4. The remote sensing image road extraction method based on joint semantic segmentation and angle prediction as described in claim 1, characterized in that, The road extraction algorithm model compares the predicted road information and road angles with the corresponding true values, calculates the loss function, and updates the parameters in the model accordingly. The specific implementation method is as follows: To effectively train the model, three sub-loss functions are used at different modules, and the overall loss function is expressed as follows: in, Represents the overall loss function. This represents the loss function that generates the preliminary road prediction results from the constraint-based preliminary prediction part. This represents the loss function that generates the angle prediction results from the initial constraint prediction part. The loss function represents the constraint on the final road prediction result; For the semantic segmentation module in the preliminary prediction part, use The prediction results M of the mask group of the module are constrained to the prediction results C of the mapping matrix from the mask to the category of the module; For mask group M, a combination of focal loss and dice loss is used for constraint; For the mapping matrix C, cross-entropy classification loss is used as a constraint; Therefore, the loss function constrains the preliminary road prediction results. Expressed as: in, Let c and c represent the cross-entropy classification loss function, focus loss function, and dice loss function, respectively; gt Let m and m represent the predicted result and the true value of the mapping matrix C, respectively; gt These represent the predicted result and the true value of mask group M, respectively. For the angle prediction module in the preliminary prediction section, this module predicts the angle value corresponding to each road pixel, expressed as a value between -π / 2 and π / 2. Considering this, a loss function is designed to constrain the angle prediction results. It is represented as: Where x and y represent the horizontal and vertical coordinates of the mask / matrix, respectively; r gt Represent the truth values ​​of the road region, t and t gt These represent the predicted result and the true value of the road angle matrix, respectively. Finally, a loss function is also needed. The final road prediction result is constrained by multiple convolutional layers. The feature fusion and road prediction in the last step can be regarded as a pixel-level semantic segmentation result. Therefore, a pixel classification type loss function is used to constrain the final prediction result. During training, remote sensing images are repeatedly input into the model, and the parameters in the model are dynamically adjusted according to the loss function until the preset conditions are met. At this point, the training process stops, and the model parameters are stored.

5. The remote sensing image road extraction method based on joint semantic segmentation and angle prediction as described in claim 1, characterized in that, During the usage phase, if the acquired remote sensing image is not in this format, the complete remote sensing image is first cut and stored. Then, a network model is created based on the model structure and parameters saved during the training phase, and the remote sensing image tiles are input into the model. The model performs predictions through the feature extraction part, the preliminary prediction part, and the feature fusion part, and automatically saves the final predicted road information. The road information is in binary mask or binary matrix format, and its length and width dimensions are consistent with the input remote sensing image tiles. If the remote sensing image used has corresponding real road information, the predicted road information is compared and evaluated with the real road information to determine the effectiveness of the model in predicting road information.

6. The remote sensing image road extraction method based on joint semantic segmentation and angle prediction as described in claim 5, characterized in that, To objectively assess the accuracy of the model's road predictions, the predicted road information is directly compared and evaluated with the actual road information based on accuracy metrics, including IoU score and F1 score. The IoU score is the ratio of the area of ​​intersection to the area of ​​union between the ground truth region and the predicted region for a given semantic category. Its calculation formula is as follows: Among them, TP, FP, and FN represent the number of pixels that are true positives, false positives, and false negatives, respectively; The F1 score is calculated based on precision and recall, and its formula is as follows: Where P and R represent precision and recall, respectively, and TP, FP, and FN represent the number of true positives, false positives, and false negatives, respectively.

7. A remote sensing image road extraction system that combines semantic segmentation and angle prediction, characterized in that, It includes a real road information preprocessing subsystem, a remote sensing image preprocessing subsystem, a feature extraction subsystem, a preliminary prediction subsystem, a feature fusion subsystem, a loss function calculation subsystem, and a result evaluation subsystem; The real road information preprocessing subsystem is used to calculate real road angle information based on real road information, so that it can be compared with the predicted road angle information in the loss function calculation subsystem to calculate the loss function and better train the model; it includes a road skeleton extraction module, a road skeleton pixel-by-pixel angle calculation module, and a road non-skeleton point angle inference module. The remote sensing image preprocessing subsystem includes a remote sensing image augmentation module and a remote sensing image formatting module. The remote sensing image augmentation module is used to perform random cropping, rotation, flipping and other operations on remote sensing images to increase the remote sensing image dataset and make the remote sensing images as diverse as possible, thereby enabling the training model to have stronger generalization ability. The remote sensing image formatting module is used to convert remote sensing images stored in bitmap format into tensor format and normalize them to meet the input data format requirements of the feature extraction subsystem. The feature extraction subsystem is used to extract road features from remote sensing images, providing easy-to-use road feature information for the subsequent preliminary prediction subsystem; it includes a tile segmentation layer, a linear embedding layer, a tile fusion layer, and a Swing block structure; The preliminary prediction subsystem includes a semantic segmentation module and an angle prediction module, which are used to make preliminary predictions of the region and angle of the road based on the road features extracted by the feature extraction subsystem. These preliminary prediction results will be temporarily saved so that the subsequent loss function calculation subsystem can calculate the loss function and adjust the model parameters. The feature fusion subsystem includes several convolutional layers, which are used to fuse road area prediction features with road angle features, and combine these features to generate the final road prediction result, thereby further improving the accuracy of road prediction. The loss function calculation subsystem includes a preliminary road prediction loss function calculation module, a preliminary angle prediction loss function calculation module, and a final road prediction loss function calculation module. The preliminary road prediction loss function calculation module calculates the loss function based on real road information and the preliminary road prediction results provided by the semantic segmentation module in the preliminary prediction subsystem. The preliminary angle prediction loss function calculation module calculates the loss function based on real road angle information and the road angle prediction matrix provided by the angle prediction module in the preliminary prediction subsystem. The final road prediction loss function calculation module calculates the loss function based on real road information and the final road prediction results provided by the feature fusion subsystem. These loss functions are backpropagated to update the parameters of each neural network layer in the feature extraction subsystem, the preliminary prediction subsystem, and the feature fusion subsystem, correcting the final road prediction results of the system. The result evaluation subsystem is used to compare the final road prediction results with the actual road information and automatically calculate various road accuracy evaluation indicators such as IoU score and F1 score. As a supplement to the user's subjective feelings, it helps the user to quantitatively understand the accuracy of the system's predicted road information. The connection relationships between the above-mentioned component systems are as follows: The real road information preprocessing subsystem is connected to the loss function calculation subsystem and the generation result evaluation subsystem, providing real road information and real road angle matrix to the loss function calculation subsystem and the generation result evaluation subsystem; The remote sensing image preprocessing subsystem is connected to the feature extraction subsystem and the feature fusion subsystem, providing the feature extraction subsystem and the feature fusion subsystem with processed remote sensing image images that meet the requirements; The feature extraction subsystem is connected to the remote sensing image preprocessing subsystem and the preliminary prediction subsystem. It provides the extracted road features to the preliminary prediction subsystem and simultaneously receives the loss values ​​returned by the preliminary prediction subsystem for training itself. The preliminary prediction subsystem is connected to the feature extraction subsystem, the feature fusion subsystem, and the loss function calculation subsystem, respectively. It provides the feature fusion subsystem with preliminary road prediction results, preliminary angle prediction results, and features generated during the preliminary road prediction process, and simultaneously receives the loss values ​​returned by the feature fusion subsystem for training itself. In addition, the preliminary road prediction results and preliminary angle prediction results generated by it are provided to the loss function calculation subsystem, and it receives the loss values ​​returned by the loss function calculation subsystem for training itself. The feature fusion subsystem is connected to the remote sensing image preprocessing subsystem, the preliminary prediction subsystem, the loss function calculation subsystem, and the generation result evaluation subsystem. It fuses multiple features provided by the remote sensing image preprocessing subsystem and the preliminary prediction subsystem to generate the final road prediction information, and provides the final road prediction information to the loss function calculation subsystem and the generation result evaluation subsystem. At the same time, it receives the loss value calculated by the loss function calculation subsystem to train itself. The loss function calculation subsystem is connected to the real road information preprocessing subsystem, the preliminary prediction subsystem, and the feature fusion subsystem. It compares the real road information provided by the real road information preprocessing subsystem with the preliminary road prediction information provided by the preliminary prediction subsystem and the final road prediction information provided by the feature fusion subsystem to calculate the loss value. At the same time, it compares the real angle matrix provided by the real road information preprocessing subsystem with the preliminary angle prediction result provided by the preliminary prediction subsystem to calculate the loss value. The loss values ​​are then fed back to the preliminary prediction subsystem and the feature fusion subsystem respectively for model training. The result evaluation subsystem is connected to the real road information preprocessing subsystem and the feature fusion subsystem. It calculates evaluation indicators by comparing the real road information provided by the real road information preprocessing subsystem with the final road prediction information provided by the feature fusion subsystem. This helps users quantitatively understand the accuracy of the system's road prediction information.

8. The remote sensing image road extraction system based on joint semantic segmentation and angle prediction as described in claim 7, characterized in that, The system works as follows: Step 1: Input all real road information into the real road information preprocessing subsystem in sequence, calculate the corresponding real road angle matrix, and store them in sequence for easy reuse later; Step 2: Input the remote sensing image into the remote sensing image preprocessing subsystem for data augmentation, normalization and formatting. The processed remote sensing image is then stored in memory. Step 3: The feature extraction subsystem reads remote sensing images from the remote sensing image preprocessing subsystem, processes them through a neural network layer, and extracts road features; Step 4: The preliminary prediction subsystem reads the road features and processes them using the semantic segmentation module and the angle prediction module respectively to obtain the preliminary road prediction results and the preliminary angle prediction results. At the same time, the intermediate results generated by the semantic segmentation module and the mask group M are saved. Step 5: The feature fusion subsystem reads the remote sensing image, preliminary road prediction results, preliminary angle prediction results and mask group M, fuses these features, and generates the final road prediction result based on the composite features; Step 6: The loss function calculation subsystem compares the real road information with the preliminary road prediction information and calculates the loss value, and sends the loss value back to the preliminary prediction subsystem for training; it compares the real angle matrix with the preliminary angle prediction results and calculates the loss value, and sends the loss value back to the preliminary prediction subsystem for training; it compares the real road information with the final road prediction information and calculates the loss value, and sends the loss value back to the feature fusion subsystem for training. Step 7: Each neural network layer in the feature fusion subsystem calculates the loss value returned by the subsystem according to the loss function, modifies the parameters, completes one training iteration, and returns the loss value of the first layer to the preliminary prediction subsystem. Step 8: The neural network layers in the preliminary prediction subsystem calculate the loss value returned by the subsystem based on the loss function and modify the parameters with the loss value returned by the feature fusion subsystem to complete one training iteration, and then return the loss value of the first layer to the feature extraction subsystem. Step 8: Each neural network layer in the feature extraction subsystem modifies its parameters based on the loss value returned by the preliminary prediction subsystem, completing one training iteration; Step 9: Repeat steps 2 to 8. After each iteration of all training data, send the real road information and the final road prediction information to the result evaluation subsystem to calculate the index. If the index meets the preset conditions or the total number of iterations reaches the preset upper limit, stop training and proceed to step 10. Step 10: Execute steps 2 to 5 to generate the final road prediction information, and send the actual road information and the final road prediction information to the generation result evaluation subsystem to calculate the final index.

Images