Remote sensing image road extraction method, device and equipment based on mask image modeling

Abstract

This application relates to a method, apparatus, and device for road extraction from remote sensing images based on masked image modeling. By constructing a road extraction network model with a backbone network, an image reconstruction module, and a semantic reconstruction module, the acquired masked image is input into the backbone network for feature extraction, yielding low-level and high-level features. The low-level features are then processed by the image reconstruction module to obtain image predictions. The high-level features are then processed by the semantic reconstruction module to obtain road predictions. The road extraction network model is then trained in a first-order manner based on a pre-constructed loss function, image predictions, and road predictions to obtain a pre-trained road extraction network model. A labeled remote sensing image is then input into the pre-trained road extraction network model for a second-order training, resulting in a trained road extraction network model. The trained road extraction network model is then used to predict roads in masked areas of the remote sensing image, improving the connectivity of road predictions and extracting complete road information.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus and equipment for road extraction from remote sensing images based on masked image modeling. Background Technology

[0002] Road extraction from remote sensing images is the process of automatically extracting roads from remote sensing imagery, and it is one of the important applications of remote sensing image processing. Because roads are often long and narrow, and roadside objects such as vegetation and buildings can cause occlusion in remote sensing images, these occluded parts can affect road detection, extraction, and classification. Figure 1 As shown, road occlusion is widespread, making research into this problem of great significance. Some existing technologies have addressed the occlusion problem and attempted to solve it through attention mechanisms, centerline detection, or contextual information.

[0003] However, when using attention mechanisms for road extraction, over-focusing on certain local areas can lead to incomplete or inaccurate road extraction. While centerline detection can improve road accuracy, it requires significant computational resources, resulting in high time and computational costs for large-scale remote sensing image data processing. Contextual information can also be affected by occlusion and interference, thus impacting the accuracy of road extraction.

[0004] Therefore, while some of these methods improve the network's feature extraction capabilities and others utilize additional information for processing, none of them consider improving the interaction capabilities of different regions of the network to predict roads in occluded areas, thereby extracting complete roads. Summary of the Invention

[0005] Therefore, it is necessary to provide a remote sensing image road extraction method, apparatus, and equipment based on mask image modeling to enhance the interaction capabilities of different regions, thereby improving road prediction connectivity and performing road prediction in occluded areas, in order to address the above-mentioned technical problems.

[0006] A method for road extraction from remote sensing images based on masked image modeling, the method comprising:

[0007] A road extraction network model is constructed, which includes a backbone network, an image reconstruction module, and a semantic reconstruction module.

[0008] Obtain the mask image, input the mask image into the backbone network for feature extraction, and obtain low-level features and high-level features;

[0009] The low-level features are input into the image reconstruction module for processing to obtain image prediction;

[0010] The high-level features are input into the semantic reconstruction module for processing to obtain road predictions.

[0011] The road extraction network model is trained in first order based on the pre-constructed loss function, the image prediction, and the road prediction to obtain a pre-trained road extraction network model.

[0012] The labeled remote sensing image is input into the pre-trained road extraction network model for second-order training to obtain the trained road extraction network model.

[0013] The road extraction network model, trained beforehand, is used to predict roads in the obscured areas of the remote sensing image, thus completing the road extraction process.

[0014] In one embodiment, obtaining the mask image includes:

[0015] Obtain a training set, which includes labeled remote sensing images and coded images;

[0016] The labeled remote sensing image is linearly encoded to obtain an image linear encoding vector; and

[0017] Perform a masking operation on the encoded image to obtain a linear vector of the masked region and a linear vector of the unmasked region;

[0018] The image linear encoding vector is multiplied by the linear vector of the unmasked region, and then superimposed with the linear vector of the masked region to obtain the masked image.

[0019] In one embodiment, both the image reconstruction module and the semantic reconstruction module are linear layer structures.

[0020] In one embodiment, the pre-constructed loss function includes a label loss function, an image reconstruction loss function, and a semantic reconstruction loss function.

[0021] In one embodiment, the label loss function is expressed as:

[0022]

[0023] Where y represents the road label. Indicates road prediction, l bce Represents the binary cross-entropy loss function; l dice This indicates a loss in dice.

[0024] In one embodiment, the image reconstruction loss function is expressed as:

[0025]

[0026] Where x represents a labeled remote sensing image, denoted as image prediction, where m represents the encoded image.

[0027] In one embodiment, the semantic reconstruction loss function is expressed as:

[0028]

[0029] Where y represents the road label. Indicates road prediction, l bce Represents the binary cross-entropy loss function; l dice denoted as Dice loss, and m represents the encoded image.

[0030] A remote sensing image road extraction device based on masked image modeling, the device comprising:

[0031] A network model construction module is used to construct a road extraction network model, which includes a backbone network, an image reconstruction module, and a semantic reconstruction module.

[0032] The feature extraction module is used to acquire a mask image, input the mask image into the backbone network for feature extraction, and obtain low-level features and high-level features.

[0033] An image reconstruction module is used to input the low-level features into the image reconstruction module for processing to obtain image prediction;

[0034] The semantic reconstruction module is used to input the high-level features into the semantic reconstruction module for processing to obtain road predictions;

[0035] The first-order training module is used to perform first-order training on the road extraction network model based on the pre-constructed loss function, the image prediction, and the road prediction to obtain a pre-trained road extraction network model.

[0036] The second-order training module is used to input labeled remote sensing images into the pre-trained road extraction network model for second-order training, so as to obtain a trained road extraction network model.

[0037] The road extraction module is used to predict roads in the occluded areas of the remote sensing image using a trained road extraction network model, thereby completing road extraction.

[0038] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the remote sensing image road extraction method based on mask image modeling described above.

[0039] The aforementioned remote sensing image road extraction method, apparatus, and equipment based on masked image modeling first construct a road extraction network model with a backbone network, an image reconstruction module, and a semantic reconstruction module. The acquired masked image is input into the backbone network for feature extraction, yielding low-level and high-level features. The low-level features are then processed by the image reconstruction module to obtain image predictions. The high-level features are then processed by the semantic reconstruction module to obtain road predictions. The road extraction network model is then trained in a first-order manner based on a pre-constructed loss function, image predictions, and road predictions, resulting in a pre-trained road extraction network model. Labeled remote sensing images are then input into the pre-trained road extraction network model for a second-order training process, resulting in a trained road extraction network model. Finally, the trained road extraction network model is used to predict roads in the masked areas of the remote sensing image, completing the road extraction process.

[0040] The above scheme enhances the interaction capabilities of both low-level and high-level regions by reconstructing the extracted low-level and high-level features respectively. Simultaneously, masked images are used for model training in the first stage of pre-training to improve the contextual reasoning ability of the road extraction network model. Furthermore, the second stage of training using only labeled remote sensing images further improves the performance of the road extraction network model and strengthens its ability to predict roads in complete images. Since roads are linear, the prediction of occluded areas can be inferred from roads in other areas. However, this scheme improves and enhances the contextual reasoning ability of the road extraction network model, improving the connectivity of road prediction and enabling the prediction of roads in occluded areas, thereby extracting complete road information. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of road occlusion in one embodiment, where (a) represents a schematic diagram of the occluded part of the remote sensing image, and (b) represents the road label of the occluded part;

[0042] Figure 2 This is a schematic diagram of a remote sensing image road extraction method based on mask image modeling in one embodiment;

[0043] Figure 3 This is a schematic diagram of the road extraction network model framework in one embodiment;

[0044] Figure 4The following is a schematic diagram of the experimental verification prediction results in one embodiment, where (a) is a remote sensing image, (b) is a road label that has been marked, (c) is a schematic diagram of the road extraction results using DeepLab v3+, (d) is a schematic diagram of the road extraction results using D-LinkNet, (e) is a schematic diagram of the road extraction results using NL-LinkNet, (f) is a schematic diagram of the road extraction results using DBRANet, (g) is a schematic diagram of the road extraction results using MACU-Net, and (h) is a schematic diagram of the road extraction results using RemainNet proposed in this invention.

[0045] Figure 5 The diagram shows the image reconstruction results, where (a) represents the remote sensing image and (b) represents the reconstructed image.

[0046] Figure 6 This is a schematic diagram of the module structure of a remote sensing image road extraction device based on mask image modeling in one embodiment;

[0047] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] It should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application and in its specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items.

[0050] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly and specifically defined.

[0051] Before beginning the description of the real-time mode of this invention, the following are explanations of some technical terms:

[0052] Road extraction: This refers to assigning a road or non-road label to each pixel in a remote sensing image.

[0053] Masking: In masked image modeling, it is mainly used to cover a portion of the pixels in an image so that the model can be trained and predicted on the uncovered pixel portion.

[0054] Masked Image Modeling (MIM): A generative method that can reconstruct the original image from a masked image.

[0055] Patch partitioning (PP) refers to dividing a large image into many smaller patches of the same size. It is often used to convert large images into smaller patches to facilitate processing and feature extraction.

[0056] Linear embedding (LE) is a technique that maps data to a low-dimensional space. The basic idea is to map the original high-dimensional data to a low-dimensional space through a linear transformation, thereby extracting the key features of the data.

[0057] Swin transformerblock (STB): Primarily used for extracting image features; there is currently no universally accepted Chinese translation for this term.

[0058] Patch merging (PM): This is used to concatenate the embedding vectors of multiple small patches into a large feature vector and then map it back to the original dimension through a linear transformation.

[0059] (Multiply): Primarily used to scale features in order to balance the contributions of features from different levels during feature fusion.

[0060] (Add): Used to add features from different levels to help the model better integrate features from different levels.

[0061] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings provided in this embodiment.

[0062] In one embodiment, such as Figure 2 As shown, a method for road extraction from remote sensing images based on masked image modeling is provided, including the following steps:

[0063] Step 12: Construct a road extraction network model, which includes a backbone network, an image reconstruction module, and a semantic reconstruction module.

[0064] Specifically, the road extraction network model constructed in this invention is named RemainNet. Its structural framework is as follows: Figure 3As shown, STB and PM constitute the backbone network, which is mainly used to extract image features; the image reconstruction module (IH) is used to generate image predictions from low-level features; and the semantic reconstruction module (SH) is used to generate road predictions from high-level features.

[0065] In one embodiment, both the image reconstruction module and the semantic reconstruction module can adopt a linear layer structure. On the one hand, the linear layer structure is simple and can increase the computation speed of the model; on the other hand, using a linear layer will not cause overfitting, so that the model has better generalization ability.

[0066] Step 14: Obtain the mask image, input the mask image into the backbone network for feature extraction, and obtain low-level features and high-level features.

[0067] The reason for masking the images is understandable. When training a model, directly using remote sensing images can introduce interference, leading to incorrect information perception and hindering training accuracy. Therefore, preliminary encoding of the remote sensing images is necessary. Masking involves randomly occluding the images and then reconstructing them to enhance the model's contextual reasoning ability. Figure 3 As shown in the coded image, a value of 1 for the white part indicates that the position is not occluded, while a value of 0 for the black part indicates that the position is occluded.

[0068] In one embodiment, the mask image is obtained primarily through the following method:

[0069] Step 22: Obtain the training set, which includes labeled remote sensing images and coded images.

[0070] Step 24: Perform linear encoding on the labeled remote sensing image to obtain the image linear encoding vector; and perform a masking operation on the encoded image to obtain the linear vector of the masked region and the linear vector of the unmasked region.

[0071] Step 26: Multiply the linear encoded vector of the image with the linear vector of the unmasked region, and then superimpose the features with the linear vector of the masked region to obtain the masked image.

[0072] Specifically, the mask image is calculated using the following formula:

[0073] x m =L(x)·F(m)+v m ·(1-F(m));

[0074] Where L(x) represents the linear encoded vector of the image, F(m) represents the linear vector of the unmasked region, 1-F(m) represents the linear vector of the masked region, and v m These are learnable parameters. It's worth noting that v... mAs training progresses, its value will automatically generate a value that has the least negative impact on the result, which is mainly adjusted automatically through loss backpropagation.

[0075] It should be noted that the training set consists only of a subset of labeled remote sensing images extracted from the dataset. The dataset also includes a validation set and a test set, both of which are labeled remote sensing images. The validation set is used to evaluate the model's performance in each iteration, allowing for the selection of the best-performing network model. The test set is used for comparison with other methods to test the performance of the proposed road extraction network model.

[0076] Furthermore, to improve the model's generalization ability, data augmentation operations such as random cropping, random flipping, random rotation, random affine transformation, random color dithering, random grayscale, and Gaussian blur are employed.

[0077] In one embodiment, f1, f2, and f3 extracted through the backbone network are low-level features, also known as shallow features, while f4 extracted through the backbone network is a high-level feature, also known as a deep feature. The reason for extracting features at different depths is that low-level and high-level features have different characteristics. Shallow features are more sensitive to the local structure of the image but have a weaker representation of global semantic information; while high-level features focus more on the global semantic information of the image. Therefore, combining low-level and high-level features can yield comprehensive and accurate feature representations.

[0078] Step 16: Input the low-level features into the image reconstruction module for processing to obtain image prediction; input the high-level features into the semantic reconstruction module for processing to obtain road prediction.

[0079] It is understandable that, on the one hand, when traditional MIM reconstructs the original image from the masked image, its downstream task also directly reconstructs the image using high-level features, which leads to differences between the pre-trained and trained feature representations. In this embodiment, however, image reconstruction features and semantic segmentation features are separated. Low-level features f1, f2, and f3 are input into the image reconstruction module IH, and then image prediction is generated. The image reconstruction module encourages interaction between low-level regions, thereby improving the consistency between pre-trained and trained feature representations.

[0080] On the other hand, because traditional MIM focuses on self-supervised learning and does not use label information, the model ignores the correlation between labels, thus reducing model performance and even making it difficult to evaluate model performance. Therefore, high-level features f4 are input into the semantic reconstruction module SH to generate road predictions. It can also be called semantic prediction. The semantic reconstruction module improves the interaction of high-level regions, strengthens the high-level semantic interaction between different regions, and improves the performance of the model.

[0081] Therefore, by adding the image reconstruction module IH and the semantic reconstruction module SH to the road extraction network model, the model's performance can be optimized and its generalization performance improved.

[0082] Step 18: Perform first-order training on the road extraction network model based on the pre-constructed loss function, image prediction, and road prediction to obtain the pre-trained road extraction network model.

[0083] Understandably, on the one hand, road prediction needs to be considered. We construct a label loss function based on the difference between the road label y and the unmasked areas.

[0084] On the other hand, f1, f2 and f3 provide different low-level information for image reconstruction. In order to make the image prediction of the unmasked area in the reconstructed image close to the unmasked area in the original remote sensing image, an image reconstruction loss function is constructed.

[0085] On the other hand, since traditional MIM methods are mainly used for unsupervised pre-training, they are primarily used to reconstruct the original image. However, image reconstruction is based on RGB, not semantics. Since the ultimate goal is to enhance road prediction connectivity, and road labels are available, this invention uses semantic reconstruction to increase semantic interaction. Similar to image reconstruction, this embodiment employs a semantic reconstruction module for semantic reconstruction. To enable road prediction... The occluded regions in the encoded image m become more similar to the occluded regions, and a semantic reconstruction loss function is constructed.

[0086] In one embodiment, the label loss function is expressed as:

[0087]

[0088] Where y represents the road label. Indicates road prediction, l bce Represents the binary cross-entropy loss function; l dice This indicates a loss in dice.

[0089] Binary cross-entropy loss function l bce The calculation formula is as follows:

[0090]

[0091] The formula for calculating dice loss is as follows:

[0092]

[0093] In one embodiment, the image reconstruction loss function is expressed as:

[0094]

[0095] Where x represents a labeled remote sensing image, denoted as image prediction, where m represents the encoded image.

[0096] In one embodiment, the semantic reconstruction loss function is expressed as:

[0097]

[0098] Step 20: Input the labeled remote sensing image into the pre-trained road extraction network model for second-order training to obtain the trained road extraction network model.

[0099] The first stage, pre-training, is understood to primarily enhance the model's contextual reasoning ability. It uses masked images and is trained using label loss functions, image reconstruction loss functions, and semantic reconstruction loss functions. However, since the testing stage deals with complete images, which differ somewhat from the segmented images of the pre-training stage, a second-order training phase is needed to improve the model's road prediction capability. This second-order training uses only the label loss function. This approach allows for further model optimization and enhances the model's road prediction performance.

[0100] Step 22: Use the trained road extraction network model to predict roads in the occluded areas of the remote sensing image to complete road extraction.

[0101] In one embodiment, experiments were conducted on the Massachusetts road dataset to verify the effectiveness of the proposed RemainNet.

[0102] The Massachusetts Roads dataset contains 1171 labeled remote sensing images, including 1108 training images, 14 validation images, and 49 test images. The dataset covers an area of ​​approximately 2600 square kilometers with a resolution of 120 cm / pixel. Each image originally measures 1500×1500 pixels. We cropped them to 512×512 pixels, resulting in 9972 training images, 126 validation images, and 441 test images.

[0103] The experiment was conducted using the PyTorch framework on a single NVIDIA RTX 3080 GPU, with AdamW as the parameter optimizer and the learning rate initially set to 2×10⁻⁶. -4 The weight decay of the AdamW optimizer is set to 1×10. -4 The number of pre-training iterations in the first stage is 100, the number of training iterations in the second stage is 100, the batch size is set to 5, and the mask rate and mask size are set to 0.5 and 16×16, respectively.

[0104] In the experiments, precision, recall, F1 score (F1), and intersection of union and intersection (IoU) were used as evaluation metrics. To demonstrate the effectiveness of the proposed RemainNet, several advanced road extraction and semantic segmentation models were used for quantitative and qualitative comparisons, namely DeepLab v3+, D-LinkNet, NL-LinkNet, DBRANet, and MACU-Net.

[0105] Table 1 lists the results of each method on the dataset. Figure 4 The results of partial road extraction for each method are listed. The experimental results show that the proposed RemainNet performs the best overall and has good road continuity.

[0106] Table 1. Experimental results of the Massachusetts road dataset

[0107]

[0108]

[0109] like Figure 5 As shown, the image reconstruction result performed by the image reconstruction module of this invention is similar to the original image region. Since the reconstructed image only uses half of the original image's region information, i.e., the unoccluded information, the reconstruction process is a process of learning the relationship between different regions.

[0110] The aforementioned remote sensing image road extraction method, apparatus, and equipment based on masked image modeling first construct a road extraction network model with a backbone network, an image reconstruction module, and a semantic reconstruction module. The acquired masked image is input into the backbone network for feature extraction, yielding low-level and high-level features. The low-level features are then processed by the image reconstruction module to obtain image predictions. The high-level features are then processed by the semantic reconstruction module to obtain road predictions. The road extraction network model is then trained in a first-order manner based on a pre-constructed loss function, image predictions, and road predictions, resulting in a pre-trained road extraction network model. Labeled remote sensing images are then input into the pre-trained road extraction network model for a second-order training process, resulting in a trained road extraction network model. Finally, the trained road extraction network model is used to predict roads in the masked areas of the remote sensing image, completing the road extraction process.

[0111] The above scheme enhances the interaction capabilities of both low-level and high-level regions by reconstructing the extracted low-level and high-level features respectively. Simultaneously, masked images are used for model training in the first stage of pre-training to enhance the contextual reasoning ability of the road extraction network model. Furthermore, the second stage of training using only labeled remote sensing images further improves the performance of the road extraction network model and strengthens its ability to predict roads in complete images. Since roads are linear, the prediction of occluded areas can be inferred from roads in other areas. However, this scheme improves and enhances the contextual reasoning ability of the road extraction network model, improves the connectivity of road prediction, and enables the prediction of roads in occluded areas, thereby extracting complete road information. It also features a simple structure, fast computation speed, and good robustness.

[0112] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this application, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Furthermore, Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0113] In one embodiment, such as Figure 6 As shown, a remote sensing image road extraction device based on masked image modeling is provided, including: a network model construction module, a feature extraction module, an image reconstruction module, a semantic reconstruction module, a first-order training module, a second-order training module, and a road extraction module, wherein:

[0114] The network model construction module is used to build a road extraction network model, which includes a backbone network, an image reconstruction module, and a semantic reconstruction module.

[0115] The feature extraction module is used to acquire the mask image, input the mask image into the backbone network for feature extraction, and obtain low-level features and high-level features.

[0116] The image reconstruction module is used to process low-level features input into the image reconstruction module to obtain image predictions.

[0117] The semantic reconstruction module is used to process high-level features to obtain road predictions.

[0118] The first-order training module is used to perform first-order training on the road extraction network model based on the pre-constructed loss function, image prediction, and road prediction to obtain the pre-trained road extraction network model.

[0119] The second-order training module is used to input labeled remote sensing images into the pre-trained road extraction network model for second-order training, resulting in a trained road extraction network model.

[0120] The road extraction module is used to predict roads in occluded areas of remote sensing images using a trained road extraction network model, thus completing road extraction.

[0121] Specific limitations regarding the remote sensing image road extraction device based on masked image modeling can be found in the limitations of the remote sensing image road extraction method based on masked image modeling mentioned above, and will not be repeated here. Each module in the aforementioned remote sensing image road extraction device based on masked image modeling can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0122] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores road extraction data from remote sensing images based on masked image modeling. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a road extraction method from remote sensing images based on masked image modeling.

[0123] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0124] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps:

[0125] Step 12: Construct a road extraction network model, which includes a backbone network, an image reconstruction module, and a semantic reconstruction module.

[0126] Step 14: Obtain the mask image, input the mask image into the backbone network for feature extraction, and obtain low-level features and high-level features.

[0127] Step 16: Input the low-level features into the image reconstruction module for processing to obtain image prediction; input the high-level features into the semantic reconstruction module for processing to obtain road prediction.

[0128] Step 18: Perform first-order training on the road extraction network model based on the pre-constructed loss function, image prediction, and road prediction to obtain the pre-trained road extraction network model.

[0129] Step 20: Input the labeled remote sensing image into the pre-trained road extraction network model for second-order training to obtain the trained road extraction network model.

[0130] Step 22: Use the trained road extraction network model to predict roads in the occluded areas of the remote sensing image to complete road extraction.

[0131] In one embodiment, when the processor executes the computer program, it can also implement the steps or sub-steps added to the various embodiments of the remote sensing image road extraction method based on mask image modeling described above.

[0132] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0133] Step 12: Construct a road extraction network model, which includes a backbone network, an image reconstruction module, and a semantic reconstruction module.

[0134] Step 14: Obtain the mask image, input the mask image into the backbone network for feature extraction, and obtain low-level features and high-level features.

[0135] Step 16: Input the low-level features into the image reconstruction module for processing to obtain image prediction; input the high-level features into the semantic reconstruction module for processing to obtain road prediction.

[0136] Step 18: Perform first-order training on the road extraction network model based on the pre-constructed loss function, image prediction, and road prediction to obtain the pre-trained road extraction network model.

[0137] Step 20: Input the labeled remote sensing image into the pre-trained road extraction network model for second-order training to obtain the trained road extraction network model.

[0138] Step 22: Use the trained road extraction network model to predict roads in the occluded areas of the remote sensing image to complete road extraction.

[0139] In one embodiment, when the computer program is executed by a processor, it can also implement the steps or sub-steps added to the various embodiments of the remote sensing image road extraction method based on mask image modeling described above.

[0140] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0141] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for road extraction from remote sensing images based on mask image modeling, characterized in that, The method comprises: a road extraction network model is constructed, the road extraction network model comprising a backbone network, an image reconstruction module and a semantic reconstruction module; a mask image is obtained, and the mask image is input into the backbone network for feature extraction to obtain low-level features and high-level features; the low-level features are input into the image reconstruction module for processing to obtain an image prediction; the high-level features are input into the semantic reconstruction module for processing to obtain a road prediction; the road extraction network model is first-order trained according to a pre-constructed loss function, the image prediction and the road prediction to obtain a pre-trained road extraction network model; a labeled remote sensing image is input into the pre-trained road extraction network model for second-order training to obtain a trained road extraction network model; the trained road extraction network model is used to perform road prediction on the occluded area in the remote sensing image to complete road extraction; the mask image is obtained, comprising: a training set is obtained, the training set comprising a labeled remote sensing image and an encoded image; the labeled remote sensing image is linearly encoded to obtain an image linear encoding vector; and the encoded image is subjected to a mask operation to obtain a mask region linear vector and a non-mask region linear vector; the image linear encoding vector and the non-mask region linear vector are dot multiplied, and then feature superposition is performed on the mask region linear vector to obtain a mask image.

2. The method of claim 1, wherein, The image reconstruction module and the semantic reconstruction module are both linear layer structures.

3. The method of claim 1 or 2, wherein, The pre-constructed loss function comprises a label loss function, an image reconstruction loss function and a semantic reconstruction loss function.

4. The method of claim 3, wherein, The label loss function is represented as: ； wherein, denotes a road label, denotes a road prediction, denotes a binary cross-entropy loss function; denotes a dice loss.

5. The method of claim 3, wherein the mask image is generated by using a road model. The image reconstruction loss function is represented as: ； wherein, represents a labeled remote sensing image, represents image prediction, represents an encoded image.

6. The method of claim 3, wherein the mask image is generated by using a road model. The semantic reconstruction loss function is represented as: ； wherein, denotes a road label, denotes a road prediction, denotes a binary cross-entropy loss function; denotes a dice loss, denotes an encoded image.

7. The device for extracting a road from a remote sensing image based on a mask image modeling, characterized in that, The device comprises: a network model construction module configured to construct a road extraction network model, the road extraction network model comprising a backbone network, an image reconstruction module and a semantic reconstruction module; a feature extraction module configured to obtain a mask image, input the mask image into the backbone network for feature extraction to obtain low-level features and high-level features; an image reconstruction module configured to input the low-level features into the image reconstruction module for processing to obtain an image prediction; a semantic reconstruction module configured to input the high-level features into the semantic reconstruction module for processing to obtain a road prediction; a first-order training module configured to first-order train the road extraction network model according to a pre-constructed loss function, the image prediction and the road prediction to obtain a pre-trained road extraction network model; a second-order training module configured to input a labeled remote sensing image into the pre-trained road extraction network model for second-order training to obtain a trained road extraction network model; a road extraction module configured to perform road prediction on an occluded area in the remote sensing image by using the trained road extraction network model to complete road extraction; in the feature extraction module, the mask image is obtained, comprising: a training set is obtained, the training set comprising a labeled remote sensing image and an encoded image; the labeled remote sensing image is linearly encoded to obtain an image linear encoding vector; and the encoded image is subjected to a mask operation to obtain a mask region linear vector and a non-mask region linear vector; masking the coded image to obtain a masked region linear vector and an unmasked region linear vector; point-multiplying the image linear coded vector and the unmasked region linear vector, and performing feature superposition with the masked region linear vector to obtain a masked image.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor implements the steps of the method in any one of claims 1 to 6 when executing the computer program.

Images