High-resolution remote sensing image road extraction method, device, equipment and storage medium

Abstract

The present application discloses a method, device, equipment and storage medium for extracting roads from high-resolution remote sensing images, and relates to the field of remote sensing technology. The specific implementation scheme includes: obtaining a sample set; constructing a road extraction model, training the road extraction model through the sample set until the loss function converges, and obtaining a target road extraction model; extracting roads from the input high-resolution remote sensing image through the target road extraction model to obtain a road image. The present application solves the problem of low road extraction accuracy for road networks with complex backgrounds in the prior art, and improves the road extraction accuracy for road networks with complex backgrounds.

Description

Technical Field

[0001] The present application relates to the field of remote sensing technology, and in particular to a method, device, equipment and storage medium for extracting roads from high-resolution remote sensing images. Background Art

[0002] Remote sensing technology is a technology that detects and identifies targets by sensing electromagnetic waves, visible light, and infrared rays reflected or radiated by the target from a long distance. Images obtained through remote sensing technology are called remote sensing images. Remote sensing images are used in many fields, and determining road information from remote sensing images is also one of the common scenarios of remote sensing images.

[0003] At present, road extraction from remote sensing images is generally achieved through semantic segmentation neural network models (such as D-LinkNet model and NL-LinkNet model).

[0004] However, due to the characteristics of high-resolution remote sensing images (i.e. high-resolution remote sensing images) such as multi-scale features, imbalanced positive and negative samples (the proportion of pixels occupied by road targets is small), and complex image background, the existing semantic segmentation neural network models have low road extraction accuracy for road networks with complex backgrounds (for example, there are water bodies, buildings, trees and other objects near the road). Summary of the invention

[0005] The embodiments of the present application solve the problem of low road extraction accuracy for road networks with complex backgrounds in the prior art by providing a method, device, equipment and storage medium for road extraction from high-resolution remote sensing images, thereby improving the road extraction accuracy for road networks with complex backgrounds.

[0006] In a first aspect, an embodiment of the present application provides a method for extracting roads from high-resolution remote sensing images, comprising:

[0007] A sample set is obtained, where the sample set includes multiple samples, each sample includes a high-resolution remote sensing image and a corresponding label file, where the label file is used to indicate roads in the high-resolution remote sensing image; a road extraction model is constructed, and the road extraction model is trained using the sample set until the loss function converges to obtain a target road extraction model; and the target road extraction model is used to extract roads from the input high-resolution remote sensing image to obtain a road image.

[0008] Among them, the road extraction model includes an encoding network, a hole convolution network and a decoding network; the encoding network includes a convolution module and a residual convolution module; the residual convolution module includes a first extraction module and a second extraction module; the high-resolution remote sensing image is input into the convolution module for feature extraction to obtain convolution features; the convolution features are input into the first extraction module for initial feature extraction to obtain the first intermediate output features; the first intermediate output features are input into the second extraction module for feature extraction to obtain the terminal output features, so as to learn different feature representations in different spaces and improve the model performance; the terminal output features are input into the hole convolution network for hole convolution feature extraction to obtain hole convolution features; the first intermediate output features and the hole convolution features are input into the decoding network for road extraction to obtain the road image in the input high-resolution remote sensing image.

[0009] Furthermore, the first extraction module includes at least one dual residual module, the second extraction module includes a residual extraction unit, the residual extraction unit includes a downsampling dual residual convolution attention module and at least one dual residual convolution attention module; the downsampling dual residual convolution attention module and the dual residual convolution attention module both contain convolution attention modules; the convolution attention module is used to reduce the amount of calculation while learning different feature representations in different spaces to improve model efficiency; the convolution attention module is used to perform three grouped convolutions and reshape the features of the input convolution attention module to obtain the first intermediate feature, the second intermediate feature and the third intermediate feature respectively; the first intermediate feature and the second intermediate feature are cross-multiplied and activated by an activation function to obtain a fourth intermediate feature; the fourth intermediate feature is cross-multiplied and reshaped with the third intermediate feature and then convolved to obtain a fifth intermediate feature; the fifth intermediate feature is added to the features of the input convolution attention module, and the output features of the convolution attention module are obtained and output through batch normalization.

[0010] Furthermore, the downsampling dual residual convolutional attention module and the dual residual convolutional attention module also include a deep convolutional layer, which is used to provide the required input features for the convolutional attention module to maintain model performance while significantly reducing the amount of computation.

[0011] Furthermore, the first intermediate output feature is input into the second extraction module for feature extraction to obtain the terminal output feature, including:

[0012] The first intermediate output feature is input into the second extraction module for feature extraction to obtain the second intermediate output feature and the terminal output feature; wherein the second extraction module includes M residual extraction units connected in sequence, M is a positive integer; the second intermediate output feature includes the output features of the first M-1 residual extraction units.

[0013] Furthermore, the first intermediate output feature and the hole convolution feature are input into the decoding network for road extraction to obtain a road image in the input high-resolution remote sensing image, including:

[0014] The first intermediate output feature, the second intermediate output feature and the hole convolution feature are input into the decoding network for road extraction to obtain a road image in the input high-resolution remote sensing image.

[0015] Furthermore, the first extraction module includes three dual residual modules, and the second extraction module includes three residual extraction units.

[0016] Furthermore, the loss function is shown in formula (1):

[0017] (1)

[0018] In formula (1), represents the binary cross entropy loss function, represents the Dice similarity coefficient loss function, represents the connectivity loss weight, represents the connectivity loss function; where, , represents the current pixel position, represents the true label value, represents the predicted value, represents the distance transform of the true label, represents the distance transform of the prediction results, , represents the set of all foreground points, Represents pixel To the previous attraction The Euclidean distance of , N represents the number of samples.

[0019] Furthermore, after obtaining the sample set, the method further includes:

[0020] The high-resolution remote sensing images of the samples in the sample set are normalized and randomly enhanced; wherein the image enhancement processing includes: randomly adjusting at least one of the hue, saturation, and brightness of the high-resolution remote sensing image; and / or, transforming the high-resolution remote sensing image and the mask image in the label file corresponding to the high-resolution remote sensing image, wherein the transformation processing includes at least one of translation, scaling, rotation, aspect ratio transformation, horizontal flipping, and vertical flipping.

[0021] Furthermore, after obtaining the sample set, the method further includes:

[0022] According to the preset screening threshold, the samples in the sample set are screened to eliminate the samples containing high-resolution remote sensing images in which the road proportion is less than 0.2% of the pixels in the whole image.

[0023] In a second aspect, an embodiment of the present application provides a high-resolution remote sensing image road extraction device, including: an acquisition module, a model building and training module, and a road extraction module.

[0024] The acquisition module is used to acquire a sample set, the sample set includes multiple samples, each sample includes a high-resolution remote sensing image and a corresponding label file, and the label file is used to indicate the road in the high-resolution remote sensing image.

[0025] The model building and training module is used to build a road extraction model. The road extraction model is trained through a sample set until the loss function converges to obtain a target road extraction model.

[0026] The road extraction module is used to extract roads from the input high-resolution remote sensing image through the target road extraction model to obtain a road image.

[0027] Among them, the road extraction model includes an encoding network, a hole convolution network and a decoding network; the encoding network includes a convolution module and a residual convolution module; the residual convolution module includes a first extraction module and a second extraction module; the high-resolution remote sensing image is input into the convolution module for feature extraction to obtain convolution features; the convolution features are input into the first extraction module for initial feature extraction to obtain the first intermediate output features; the first intermediate output features are input into the second extraction module for feature extraction to obtain the terminal output features, so as to learn different feature representations in different spaces and improve the model performance; the terminal output features are input into the hole convolution network for hole convolution feature extraction to obtain hole convolution features; the first intermediate output features and the hole convolution features are input into the decoding network for road extraction to obtain the road image in the input high-resolution remote sensing image.

[0028] In a third aspect, an embodiment of the present application provides a device, comprising: a processor; a memory for storing processor executable instructions; and a method for implementing the first aspect or any possible implementation of the first aspect when the processor executes the executable instructions.

[0029] In a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium, which includes a device for storing a computer program or instruction. When the computer program or instruction is executed, the method of the first aspect or any possible implementation method of the first aspect is implemented.

[0030] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0031] A sample set is obtained, and a road extraction model is constructed that can learn different feature representations in different spaces. The road extraction model is trained through the sample set until the loss function converges to obtain a target road extraction model. The target road extraction model is used to extract roads from the input high-resolution remote sensing image to obtain a road image. Different feature representations can be learned in different spaces to improve model performance, thereby improving the road extraction accuracy under road networks with complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 A schematic diagram of the process of extracting roads from high-resolution remote sensing images provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the structure of a road extraction model provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of the structure of a dual residual module provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the structure of the down-sampling double residual convolution attention module provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of the structure of a dual residual convolution attention module provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of the structure of the convolutional attention module provided in the embodiment of the present application;

[0039] Figure 7 A schematic diagram of a high-resolution remote sensing image;

[0040] Figure 8 This is another schematic diagram of a high-resolution remote sensing image;

[0041] Fig. 9 To use the NL-LinkNet model Figure 7 A road image obtained by extracting roads from a high-resolution remote sensing image;

[0042] Fig.10 To use the NL-LinkNet model Figure 8 A road image obtained by extracting roads from a high-resolution remote sensing image;

[0043] Fig.11 In order to use the road extraction model of this embodiment Figure 7 A road image obtained by extracting roads from a high-resolution remote sensing image;

[0044] Fig.12 In order to use the road extraction model of this embodiment Figure 8 A road image obtained by extracting roads from a high-resolution remote sensing image;

[0045] Fig.13 This is the road extraction model used in the training process of this embodiment. Figure 8 The corresponding label image;

[0046] Fig.14 A schematic diagram of the composition of a high-resolution remote sensing image road extraction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The following describes some of the techniques involved in the embodiments of the present application to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted in the following description.

[0049] Remote sensing technology is a technology that detects and identifies targets by sensing electromagnetic waves, visible light, and infrared rays reflected or radiated by the target from a long distance. Images obtained through remote sensing technology are called remote sensing images. Remote sensing images are used in many fields, and determining road information from remote sensing images is also one of the common scenarios of remote sensing images.

[0050] At present, road extraction from remote sensing images is generally achieved through semantic segmentation neural network models (such as D-LinkNet model and NL-LinkNet model).

[0051] However, due to the characteristics of high-resolution remote sensing images such as multi-scale features, imbalanced positive and negative samples (road targets occupy a small proportion of pixels), and complex image background, the existing semantic segmentation neural network models have low road extraction accuracy for road networks with complex backgrounds (for example, water bodies, buildings, trees and other objects near the road).

[0052] Against this background, the present invention provides a method for extracting roads from high-resolution remote sensing images, which can improve the accuracy of road extraction in road networks with complex backgrounds.

[0053] The execution subject of the high-resolution remote sensing image road extraction method provided by the embodiment of the present disclosure may be a computer or a server, or may also be other electronic devices with data processing capabilities; or, the execution subject of the method may also be a processor (such as a central processing unit (CPU)) in the above electronic device; or, the execution subject of the method may also be an application (application, APP) installed in the above electronic device that can implement the function of the method; or, the execution subject of the method may also be a functional module or unit in the above electronic device that has the function of the method, etc. There is no limitation on the execution subject of the method herein.

[0054] The high-resolution remote sensing image road extraction method is exemplarily described below with reference to the accompanying drawings.

[0055] Figure 1 : is a flow chart of a method for extracting roads from high-resolution remote sensing images provided in an embodiment of the present application. Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of the high-resolution remote sensing image road extraction method. If the final result can be achieved, Figure 1 The steps shown can be performed in parallel or in reverse. Figure 1 As shown, the method may include S101 to S103.

[0056] S101: Obtain a sample set.

[0057] The sample set includes multiple samples, each sample includes a high-resolution remote sensing image and a corresponding label file, and the label file is used to indicate the roads in the high-resolution remote sensing image.

[0058] In some possible implementations, after obtaining the sample set, the method further includes: normalizing the high-resolution remote sensing images of the samples in the sample set, and randomly performing image enhancement processing; wherein the image enhancement processing includes: randomly adjusting at least one of the hue, saturation, and brightness of the high-resolution remote sensing images; and / or, transforming the high-resolution remote sensing images and the mask images in the label files corresponding to the high-resolution remote sensing images, and the transformation processing includes at least one of translation, scaling, rotation, aspect ratio transformation, horizontal flipping, and vertical flipping. In this way, the model training speed can be accelerated, the model robustness can be enhanced, and the model overfitting can be prevented.

[0059] In some possible implementations, after obtaining the sample set, the method further includes: screening the samples in the sample set according to a preset screening threshold, and removing samples containing high-resolution remote sensing images whose road proportion is less than 0.2% of the pixels of the whole image. It is understandable that samples containing high-resolution remote sensing images whose road proportion is equal to 0.2% of the pixels of the whole image may be removed or not. In this way, excessive imbalance of positive and negative samples can be prevented from causing a decrease in model accuracy.

[0060] S102, constructing a road extraction model, and training the road extraction model through a sample set until the loss function converges, thereby obtaining a target road extraction model.

[0061] Among them, the road extraction model includes an encoding network, a hole convolution network and a decoding network; the encoding network includes a convolution module and a residual convolution module; the residual convolution module includes a first extraction module and a second extraction module; the high-resolution remote sensing image is input into the convolution module for feature extraction to obtain convolution features; the convolution features are input into the first extraction module for initial feature extraction to obtain the first intermediate output features; the first intermediate output features are input into the second extraction module for feature extraction to obtain the terminal output features, so as to learn different feature representations in different spaces and improve the model performance; the terminal output features are input into the hole convolution network for hole convolution feature extraction to obtain hole convolution features; the first intermediate output features and the hole convolution features are input into the decoding network for road extraction to obtain the road image in the input high-resolution remote sensing image.

[0062] Exemplarily, the specific structures of the dilated convolutional network and the decoding network of the road extraction model can refer to the structures of the dilated convolutional network and the decoding network in the existing semantic segmentation neural network model for road extraction (such as the D-LinkNet model) and be adjusted according to actual needs, which will not be elaborated herein.

[0063] Exemplarily, the convolution module may include a plurality of convolutional layers, batch normalization layers, and activation functions of the same number, for performing downsampling and dimension-upgrading feature extraction on the input high-resolution remote sensing image.

[0064] Specifically, the first extraction module includes at least one dual residual module, the second extraction module includes a residual extraction unit, the residual extraction unit includes a downsampling dual residual convolution attention module and at least one dual residual convolution attention module; the downsampling dual residual convolution attention module and the dual residual convolution attention module both include a convolution attention module; the convolution attention module is used to reduce the amount of computation while learning different feature representations in different spaces to improve model efficiency.

[0065] The convolutional attention module is used to perform three grouped convolutions and reshape the features of the input convolutional attention module, respectively, to obtain the first intermediate feature, the second intermediate feature and the third intermediate feature respectively; the first intermediate feature and the second intermediate feature are cross-multiplied and activated through an activation function to obtain a fourth intermediate feature; the fourth intermediate feature and the third intermediate feature are cross-multiplied and reshaped and then convolved to obtain a fifth intermediate feature; the fifth intermediate feature is added to the features of the input convolutional attention module, and the output features of the convolutional attention module are obtained through batch normalization and output.

[0066] Furthermore, the downsampling dual residual convolutional attention module and the dual residual convolutional attention module also include a deep convolutional layer, which is used to provide the required input features for the convolutional attention module to maintain model performance while significantly reducing the amount of computation.

[0067] Furthermore, the first intermediate output feature is input into the second extraction module for feature extraction to obtain the terminal output feature, including:

[0068] The first intermediate output feature is input into the second extraction module for feature extraction to obtain the second intermediate output feature and the terminal output feature; wherein the second extraction module includes M residual extraction units connected in sequence, M is a positive integer; the second intermediate output feature includes the output features of the first M-1 residual extraction units.

[0069] Furthermore, the first intermediate output feature and the hole convolution feature are input into the decoding network for road extraction to obtain a road image in the input high-resolution remote sensing image, including:

[0070] The first intermediate output feature, the second intermediate output feature and the hole convolution feature are input into the decoding network for road extraction to obtain a road image in the input high-resolution remote sensing image.

[0071] Furthermore, the first extraction module includes three dual residual modules, and the second extraction module includes three residual extraction units.

[0072] In this way, computing resources can be saved while achieving high model accuracy.

[0073] It should be noted that the number of dual residual convolution attention modules in different residual extraction units can be the same or different.

[0074] Exemplarily, the AdamW optimizer can be used to train the model using the gradient descent method and the cosine annealing learning rate adjustment algorithm. The AdamW optimizer can dynamically adjust the learning rate at different stages of training by adaptively adjusting the learning rate to adapt to changes in different parameters. AdamW also introduces a weight moment update mechanism, which can smooth the update amount of different parameters during the gradient descent process, effectively reducing the impact of gradient noise, thereby converging to the optimal solution more smoothly. The cosine annealing learning rate adjustment algorithm can gradually reduce the learning rate during training.

[0075] Exemplarily, a function obtained by adding a binary cross entropy loss function and a Dice loss function can be used as the loss function, and there is no restriction on the loss function.

[0076] The following is a specific example to further illustrate the model structure of the road extraction model in the high-resolution remote sensing image road extraction method.

[0077] Figure 2 This is a schematic diagram of the structure of the road extraction model provided in the embodiment of the present application. Figure 2 As shown, the convolutional module of the encoding network in the road extraction model ( Figure 2 (not directly shown in the figure), which contains three convolutional layers (i.e., Conv2d), three batch normalization layers (i.e., BatchNorm2d), and three GELU activation functions (i.e., GELU). The convolution kernel of the first convolutional layer is 1×1 (i.e., kernel=1×1) with a stride of 1 (i.e., stride=1); the convolution kernel of the second convolutional layer is 3×3 (i.e., kernel=3×3) with a stride of 2 (i.e., stride=2); the convolution kernel of the third convolutional layer is 1×1 (i.e., kernel=1×1) with a stride of 1 (i.e., stride=1). For the input high-resolution remote sensing image of size 1024×1024×3, after passing through the first convolutional layer, the first batch normalization layer, and the first GELU activation function, a feature of size 512×512×64 is obtained. After passing through the second convolutional layer, the second batch normalization layer, and the second GELU activation function, the feature of size 512×512×64 is obtained. After the features with a size of 256×256×64 pass through the third convolution layer, the third batch normalization layer, and the third GELU activation function, the features with a size of 256×256×256 are obtained.

[0078] The residual convolution module of the encoding network in the road extraction model ( Figure 2 ), including a first extraction module ( Figure 2 not directly shown) and a second extraction module ( Figure 2 ). The first extraction module includes three stacked GroupB modules (i.e., double residual modules); the second extraction module includes three residual extraction units, the first residual extraction unit includes a GroupD module (i.e., downsampling double residual convolution attention module) and three stacked Group modules (i.e., double residual convolution attention module); the second residual extraction unit includes a GroupD module and thirteen stacked Group modules; the third residual extraction unit includes a GroupD module and two stacked Group modules.

[0079] Specifically, Figure 3 This is a schematic diagram of the structure of the dual residual module provided in the embodiment of the present application. Figure 3 As shown in the figure, the double residual module (i.e., GroupB module) is used to process the input channel 4C (i.e., input channel=4C in the figure) feature through a convolution layer with a convolution kernel of 1×1 (i.e., 1×1Conv) and a channel of C (i.e., channel=C), a batch normalization layer (i.e., BatchNorm2d in the figure), a convolution layer with a convolution kernel of 3×3 (i.e., 3×3Conv) and a channel of C (i.e., channel=C), a batch normalization layer, a GELU activation function, and a convolution layer with a convolution kernel of 1×1 (i.e., 1×1Conv) and a channel of 4C (i.e., channel=4C) to obtain the sixth intermediate feature; after adding the sixth intermediate feature to the feature of the input double residual module, it is processed through a batch normalization layer and a GELU activation function to obtain the output feature of the double residual module and output it.

[0080] Specifically, Figure 4 This is a schematic diagram of the structure of the down-sampling double residual convolution attention module provided in the embodiment of the present application. Figure 4As shown in the figure, the down-sampled double residual convolution attention module (i.e., GroupD module) is used to transform the input channel 2C (i.e., input channel=2C in the figure) features, which are sequentially passed through a deep convolution layer with a convolution kernel of 3×3 (i.e., 3×3Conv), a channel of 4C (i.e., channel=4C), a group of 2C (i.e., Groups=2C), and a stride of 2 (i.e., stride=2), a batch normalization layer (i.e., BatchNorm2d in the figure), a GELU activation function (i.e., GELU in the figure), a convolution kernel of 1×1 (i.e., 1×1Conv), a channel of C (i.e., channel=C), a batch normalization layer, and a convolution attention module (i.e., Attenti in the figure). The seventh intermediate feature is obtained by processing the input channel 2C feature through a convolution layer with a convolution kernel of 1×1 (i.e. 1×1Conv), a channel of 4C (i.e. channel=4C), and a stride of 2 (i.e. stride=2) to obtain the eighth intermediate feature; after adding the seventh intermediate feature to the eighth intermediate feature, the batch normalization layer and the GELU activation function are used to process the output feature of the downsampled double residual convolution attention module and output it.

[0081] Specifically, Figure 5 This is a schematic diagram of the structure of the dual residual convolution attention module provided in the embodiment of the present application. Figure 5 As shown in the figure, the double residual convolution attention module (i.e., Group module) is used to process the input channel 2C (i.e., inputchannel=2C in the figure) feature through a deep convolution layer with a convolution kernel of 3×3 (i.e., 3×3Conv), a channel of 4C (i.e., channel=4C), and a group number of grouped convolutions of 4C (i.e., Groups=4C), a batch normalization layer (i.e., BatchNorm2d in the figure), a GELU activation function (i.e., GELU in the figure), a convolution layer with a convolution kernel of 1×1 (i.e., 1×1Conv), a channel of C (i.e., channel=C), a batch normalization layer, a convolution attention module (i.e., Attention module in the figure), and a convolution layer with a convolution kernel of 1×1 (i.e., 1×1Conv) and a channel of 4C (i.e., channel=4C) to obtain the ninth intermediate feature; after adding the ninth intermediate feature to the feature of the input double residual convolution attention module, it is processed through the batch normalization layer and the GELU activation function to obtain the output feature of the double residual convolution attention module and output it.

[0082] Specifically, Figure 6 This is a schematic diagram of the structure of the convolutional attention module provided in the embodiment of the present application. Figure 6As shown in the figure, the convolutional attention module (i.e., Attention module) is used to obtain the first intermediate feature, the second intermediate feature, and the third intermediate feature respectively through three convolutional layers with convolution kernels of 1×1 (i.e., 1×1Conv) and the number of groups of grouped convolutions is the number of heads (i.e., groups=number_heads) for the features of the input convolutional attention module; the first intermediate feature and the second intermediate feature are cross-multiplied and then passed through the softmax activation function to obtain the fourth intermediate feature; the fourth intermediate feature is cross-multiplied with the third intermediate feature and reshaped, and then passed through a convolution layer with a convolution kernel of 1×1 to obtain the fifth intermediate feature; the fifth intermediate feature is added to the features of the input convolutional attention module, and the output features of the convolutional attention module are obtained and output through a batch normalization layer (i.e., BatchNorm2d in the figure).

[0083] like Figure 2 As shown in the figure, for the feature with a size of 256×256×256 obtained by the convolution module, the feature size obtained by the first extraction module is still 256×256×256; the output feature of the first extraction module (i.e., the first intermediate output feature) is input into the first residual extraction unit, and the number of heads of the first residual extraction unit is set to 2 (i.e., number_heads=2), and a feature with a size of 128×128×512 is obtained; the output feature of the first residual extraction unit is input into the second residual extraction unit, and the number of heads of the second residual extraction unit is set to 4 (i.e., number_heads=4), and a feature with a size of 64×64×1024 is obtained; the output feature of the second residual extraction unit is input into the third residual extraction unit, and the number of heads of the third residual extraction unit is set to 8 (i.e., number_heads=8), and a feature with a size of 32×32×2048 is obtained.

[0084] Continue to refer Figure 2 The decoding network includes a first decoder module, a second decoder module, a third decoder module, a fourth decoder module, a transposed convolution module, and two convolution layers with a convolution kernel of 3×3 (i.e., 3×3Conv) and a stride of 1 (i.e., stride=1), which are connected in sequence.

[0085] like Figure 2As shown in the figure, the output of the third residual extraction unit is used as the input of the dilated convolution module, and the output of the dilated convolution module is used as the input of the first decoder module; the sum of the output of the first decoder module and the output of the second residual extraction unit is used as the input of the second decoder module; the sum of the output of the second decoder module and the output of the first residual extraction unit is used as the input of the third decoder module; the sum of the output of the third decoder module and the output of the first extraction module is used as the input of the fourth decoder module; the output of the fourth decoder module passes through the transposed convolution module and two convolution layers in sequence to obtain a road image with a size of 1024×1024×1, thereby realizing the extraction of road images in high-resolution remote sensing images.

[0086] It should be noted that the number of decoder modules is related to the number of residual extraction units, and the specific number is the number of residual extraction units plus one.

[0087] For example, when the number of residual extraction units is 3 and the number of decoder modules is 4 (eg Figure 2 As shown); when the number of residual extraction units is 2, the number of decoder modules is 3; when the number of residual extraction units is 4, the number of decoder modules is 5.

[0088] It can be understood that calculating spatial attention in the process of multi-scale feature extraction can capture more feature relationships over long distances and improve the connectivity and completeness of the extracted roads.

[0089] The traditional Spatial Attention Module (SAM) uses global average pooling and global maximum pooling to reduce the channel of the feature map to 1, which loses a lot of detail information in the channel dimension.

[0090] The convolutional attention module constructed in this embodiment can retain the detailed information of all channels while capturing the relationship between different spatial positions on the feature map, compared to the traditional spatial attention module. For ease of understanding, it can be compared to the Transformer architecture of natural language processing: the channel dimension is regarded as a word embedding vector, and the feature map of each channel is regarded as a sequence of length H×W, and global attention is calculated for this sequence. When calculating Q, K, and V, the grouped convolution used by the convolutional attention module can realize the multi-head attention mechanism (Multi-Head Attention) while reducing the amount of calculation, allowing the model to learn different feature representations in different subspaces. Each attention head can focus on different parts of the input, thereby capturing more diverse contextual information. And because the position of the feature map is fixed in the convolution operation, the effect of position encoding is achieved without position encoding.

[0091] At the same time, this embodiment constructs a double residual convolution attention module (i.e., Group module) and a downsampling double residual convolution attention module (i.e., GroupD module) for downsampling based on the convolution attention module. Both the Group module and the GroupD module use depthwise convolution and GELU activation function.

[0092] In the calculation of deep convolution, each channel corresponds to a separate convolution kernel, which is equivalent to performing separate feature extraction on the feature map of each channel. The convolution result obtained can be regarded as a feature vector in the channel dimension, which just meets the input type required by the convolutional attention module constructed in this embodiment, and can maintain model performance while greatly reducing the amount of calculation.

[0093] Compared with the Relu activation function, the GELU activation function outputs a non-zero value on the negative semi-axis and changes smoothly at the zero point, which can provide better nonlinearity for the model. In the double residual convolutional attention module, the long-distance residual link is nested with the residual link of the convolutional attention module. This double residual structure can promote feature reuse, enhance information flow, reduce the risk of overfitting and improve model robustness.

[0094] Calculating global attention on a larger feature map will greatly increase the amount of calculation. In order to reduce the amount of calculation of the model, this embodiment uses a dual residual module (i.e., GroupB module) in the shallow layer of the model. Compared with the Group module, the GroupB module deletes the attention calculation part and uses ordinary convolution instead.

[0095] Through the first extraction module and the second extraction module of this embodiment, the features in the high-resolution remote sensing image can be extracted more accurately on the basis of reducing the amount of calculation, thereby improving the model accuracy.

[0096] For example, Figure 2 When training the road extraction model shown, the total training epochs can be set to 300, the initial learning rate can be set to 1e-4, the weight decay can be set to 5e-6, the cosine annealing minimum learning rate can be set to 1e-8, and the batch size can be set to 4; if the model loss does not decrease for 6 consecutive epochs, the training is stopped early.

[0097] Specifically, the loss function is shown in formula (1):

[0098] (1)

[0099] In formula (1), represents the binary cross entropy loss function, represents the Dice similarity coefficient loss function, represents the connectivity loss weight, represents the connectivity loss function; where, , represents the current pixel position, represents the true label value, represents the predicted value, represents the distance transform of the true label, represents the distance transform of the prediction results, , represents the set of all foreground points, Represents pixel To the previous attraction The Euclidean distance is , and N represents the number of samples. In this embodiment, the model accuracy can be improved by using the loss function and setting corresponding weights for the connectivity loss function.

[0100] S103, performing road extraction on the input high-resolution remote sensing image using a target road extraction model to obtain a road image.

[0101] Exemplarily, after obtaining the target road extraction model, the high-resolution remote sensing image that requires road extraction can be input into the target road extraction model. The target road extraction model identifies and extracts the roads in the high-resolution remote sensing image to obtain a road image corresponding to the high-resolution remote sensing image.

[0102] The embodiment of the present application obtains a sample set, constructs a road extraction model including a convolutional attention module for learning different feature representations in different subspaces to improve model performance while reducing the amount of computation, trains the road extraction model to obtain a target road extraction model, and uses the target road extraction model to perform road extraction on an input high-resolution remote sensing image to obtain a road image, thereby improving the accuracy of road extraction in road networks with complex backgrounds.

[0103] It should be noted that in the road extraction task, the model will predict each pixel in the high-resolution remote sensing image input to the model and mark it as road / background, which requires retaining a large amount of detail information when performing feature extraction; and because roads are usually manifested as long and thin linear structures with good continuity in remote sensing images, due to this geometric characteristic, the model needs to be able to capture long-distance linear features during extraction, rather than focusing only on local features. The dual residual convolution attention module constructed in this application can meet both requirements at the same time, improving the connectivity and integrity problems of traditional models when extracting roads. This application improves the overall performance of the network without increasing the amount of computation by using deep convolution and designing a reasonable structure, thereby improving the accuracy of road extraction.

[0104] Effect verification 1

[0105] The neural network models used for road extraction in the prior art (including the Unet model, the D-LinkNet model, and the NL-LinkNet model) and the road extraction model of this embodiment are tested on the DeepGlobe dataset, and the test results of different models on the DeepGlobe dataset are obtained as shown in Table 1.

[0106] Table 1

[0107]

[0108] Referring to Table 1, it can be seen that compared with other models, the road extraction model of this embodiment improves the F1 Score by 0.78%~2.32% and the IOU by 1.08%~3.15%.

[0109] Among them, IOU and F1 Score are commonly used evaluation indicators of neural network models in this field.

[0110] IOU measures the ratio of the intersection and union of the predicted result and the true value. The higher the IOU, the greater the overlap between the model prediction and the true value, indicating better performance. For road extraction scenarios, IOU can reflect the comprehensive prediction quality of the model for the entire area.

[0111] F1 Score is the harmonic mean of precision and recall. F1 Score takes precision and recall into consideration and is suitable for tasks with unbalanced data such as road extraction, where the proportion of background is much larger than that of road. The larger the F1 Score, the more accurate the model's prediction of the positive class and the lower the missed detection rate.

[0112] Effect Verification 2

[0113] The following is an analysis of the NL-LinkNet model used for road extraction in the prior art and the road extraction model of this embodiment. Figure 7 , Figure 8 Road extraction is performed on the high-resolution remote sensing image shown.

[0114] Through the NL-LinkNet model, Figure 7 , Figure 8 The high-resolution remote sensing image shown in the figure is used to extract roads, and the road images obtained are as follows Fig. 9 , Fig.10 shown.

[0115] The road extraction model of this embodiment is used to respectively Figure 7 , Figure 8 The high-resolution remote sensing image shown in the figure is used to extract roads, and the road images obtained are as follows Fig.11 , Fig.12shown.

[0116] refer to Fig. 9 and Fig.11 It can be seen that compared with the NL-LinkNet model with the best effect in effect verification 1, the road extraction model of this embodiment extracts a more complete road, effectively identifies the green belt part in the center of the road, and correctly divides the road into two-way lanes.

[0117] refer to Fig.10 and Fig.12 It can be seen that compared with the NL-LinkNet model with the best effect in effect verification 1, the recognition results of the road extraction model in this embodiment are more complete and continuous.

[0118] Fig.13 This is the road extraction model used in the training process of this embodiment. Figure 8 Corresponding label image. Reference Fig.13 and Fig.12 It can be seen that the road extraction model of this embodiment recognizes some roads that were missed when the images were manually annotated, which will slightly reduce the evaluation index. This shows that the actual performance improvement of this embodiment should be greater than the improvement reflected in the evaluation index in effect verification 1.

[0119] Although the present application provides method operation steps such as embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in this embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in this embodiment or the accompanying drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0120] like Fig.14 As shown, the embodiment of the present application also provides a high-resolution remote sensing image road extraction device. The device includes: an acquisition module 1401, a model building and training module 1402 and a road extraction module 1403.

[0121] The acquisition module 1401 is used to acquire a sample set, where the sample set includes multiple samples, each sample includes a high-resolution remote sensing image and a corresponding label file, and the label file is used to indicate a road in the high-resolution remote sensing image.

[0122] The model building and training module 1402 is used to build a road extraction model, and train the road extraction model through a sample set until the loss function converges to obtain a target road extraction model.

[0123] The road extraction module 1403 is used to perform road extraction on the input high-resolution remote sensing image through a target road extraction model to obtain a road image.

[0124] Among them, the road extraction model includes an encoding network, a hole convolution network and a decoding network; the encoding network includes a convolution module and a residual convolution module; the residual convolution module includes a first extraction module and a second extraction module; the high-resolution remote sensing image is input into the convolution module for feature extraction to obtain convolution features; the convolution features are input into the first extraction module for initial feature extraction to obtain the first intermediate output features; the first intermediate output features are input into the second extraction module for feature extraction to obtain the terminal output features, so as to learn different feature representations in different spaces and improve the model performance; the terminal output features are input into the hole convolution network for hole convolution feature extraction to obtain hole convolution features; the first intermediate output features and the hole convolution features are input into the decoding network for road extraction to obtain the road image in the input high-resolution remote sensing image.

[0125] The beneficial effects and specific implementation methods of the device embodiment can be referred to the aforementioned method embodiment, and will not be described in detail here.

[0126] Some modules in the apparatus described in the present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0127] The devices or modules described in the above application embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described in various modules according to their functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, the module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0128] The methods, devices or modules described in this application can be implemented in the form of computer-readable program codes. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program codes (such as software or firmware) that can be executed by the (micro)processor, logic gates, switches, application-specific integrated circuits (English: Application Specific Integrated Circuit; Abbreviation: ASIC), programmable logic controllers and embedded microcontrollers. Examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program codes, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included in it for implementing various functions can also be regarded as structures within the hardware component. Or even, the means for realizing various functions may be regarded as both a software module for realizing the method and a structure within a hardware component.

[0129] An embodiment of the present application further provides a device, comprising: a processor; a memory for storing processor executable instructions; when the processor executes the executable instructions, the method described in the embodiment of the present application is implemented.

[0130] The embodiments of the present application also provide a non-volatile computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed, the method described in the embodiments of the present application is implemented.

[0131] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist independently, or two or more modules may be integrated into one module.

[0132] The above storage media include but are not limited to random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD) or memory card. The memory can be used to store computer program instructions.

[0133] It can be seen from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of the present application can be essentially or partly reflected in the prior art in the form of a software product, or it can be reflected in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0134] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0135] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A method for extracting roads from high-resolution remote sensing images, characterized in that: include: Acquire a sample set, the sample set comprising a plurality of samples, each sample comprising a high-resolution remote sensing image and a corresponding label file, the label file being used to indicate a road in the high-resolution remote sensing image; Constructing a road extraction model, and training the road extraction model using the sample set until the loss function converges, thereby obtaining a target road extraction model; Performing road extraction on the input high-resolution remote sensing image by using the target road extraction model to obtain a road image; The road extraction model includes an encoding network, a hole convolution network and a decoding network; the encoding network includes a convolution module and a residual convolution module; the residual convolution module includes a first extraction module and a second extraction module; The first extraction module includes at least one dual residual module, the second extraction module includes a residual extraction unit, and the residual extraction unit includes a downsampled dual residual convolution attention module and at least one dual residual convolution attention module; The down-sampled dual residual convolutional attention module and the dual residual convolutional attention module both include a convolutional attention module; the convolutional attention module is used to reduce the amount of calculation while learning different feature representations in different spaces to improve model efficiency; The convolutional attention module is used to perform three grouped convolutions and reshape the features input to the convolutional attention module, respectively, to obtain a first intermediate feature, a second intermediate feature and a third intermediate feature, respectively; perform cross-product of the first intermediate feature and the second intermediate feature and activate them through an activation function to obtain a fourth intermediate feature; perform cross-product of the fourth intermediate feature and the third intermediate feature and reshape them and then perform convolution to obtain a fifth intermediate feature; add the fifth intermediate feature to the features input to the convolutional attention module, obtain the output features of the convolutional attention module through batch normalization and output them; Inputting the high-resolution remote sensing image into the convolution module for feature extraction to obtain convolution features; Inputting the convolution feature into the first extraction module for initial feature extraction to obtain a first intermediate output feature; Inputting the first intermediate output feature into the second extraction module for feature extraction to obtain the terminal output feature, so as to learn different feature representations in different spaces and improve model performance; Inputting the terminal output feature into the dilated convolutional network to extract dilated convolutional features, thereby obtaining dilated convolutional features; The first intermediate output feature and the hole convolution feature are input into a decoding network to extract roads, so as to obtain a road image in the input high-resolution remote sensing image.

2. The method according to claim 1, characterized in that The down-sampled dual residual convolution attention module and the dual residual convolution attention module also include a deep convolution layer, which is used to provide the required input features for the convolution attention module to maintain model performance while significantly reducing the amount of computation.

3. The method according to claim 2, characterized in that The step of inputting the first intermediate output feature into the second extraction module for feature extraction to obtain the terminal output feature comprises: The first intermediate output feature is input into the second extraction module for feature extraction to obtain the second intermediate output feature and the terminal output feature; wherein the second extraction module includes M residual extraction units connected in sequence, M is a positive integer; the second intermediate output feature includes the output features of the first M-1 residual extraction units.

4. The method according to claim 3, characterized in that The step of inputting the first intermediate output feature and the hole convolution feature into a decoding network to extract a road, and obtaining a road image in the input high-resolution remote sensing image, comprises: The first intermediate output feature, the second intermediate output feature and the hole convolution feature are input into a decoding network to extract roads, so as to obtain a road image in the input high-resolution remote sensing image.

5. The method according to claim 3, characterized in that: The first extraction module includes three dual residual modules, and the second extraction module includes three residual extraction units.

6. The method according to claim 1, characterized in that The loss function is shown in formula (1): （1） In formula (1), represents the binary cross entropy loss function, represents the Dice similarity coefficient loss function, represents the connectivity loss weight, represents the connectivity loss function; in, , represents the current pixel position, represents the true label value, represents the predicted value, represents the distance transform of the true label, represents the distance transform of the prediction results, , represents the set of all foreground points, Represents pixel To the previous attraction The Euclidean distance of , N represents the number of samples.

7. The method according to claim 1, characterized in that After obtaining the sample set, the method further includes: The high-resolution remote sensing images of the samples in the sample set are normalized and randomly enhanced; wherein the image enhancement includes: randomly adjusting at least one of the hue, saturation, and brightness of the high-resolution remote sensing images; and / or transforming the high-resolution remote sensing images and the mask images in the label files corresponding to the high-resolution remote sensing images, wherein the transformation includes at least one of translation, scaling, rotation, aspect ratio transformation, horizontal flipping, and vertical flipping.

8. The method according to claim 1, characterized in that After obtaining the sample set, the method further includes: According to the preset screening threshold, the samples in the sample set are screened to eliminate the samples containing high-resolution remote sensing images in which the road proportion is less than 0.2% of the pixels in the whole image.

9. A high-resolution remote sensing image road extraction device, characterized in that: include: An acquisition module, used for acquiring a sample set, wherein the sample set includes a plurality of samples, each sample includes a high-resolution remote sensing image and a corresponding label file, and the label file is used for indicating a road in the high-resolution remote sensing image; A model building and training module, used to build a road extraction model, and train the road extraction model through the sample set until the loss function converges to obtain a target road extraction model; A road extraction module, used to extract roads from the input high-resolution remote sensing image using the target road extraction model to obtain a road image; Among them, the road extraction model includes an encoding network, a hole convolution network and a decoding network; the encoding network includes a convolution module and a residual convolution module; the residual convolution module includes a first extraction module and a second extraction module; the first extraction module includes at least one double residual module, the second extraction module includes a residual extraction unit, and the residual extraction unit includes a downsampled double residual convolution attention module and at least one double residual convolution attention module; the downsampled double residual convolution attention module and the double residual convolution attention module both contain a convolution attention module; the convolution attention module is used to reduce the amount of calculation while learning different feature representations in different spaces to improve the model efficiency; the convolution attention module is used to perform three grouped convolutions and reshape the features input to the convolution attention module to obtain the first intermediate feature, the second intermediate feature and the third intermediate feature respectively; the first intermediate feature and the second intermediate feature are cross-multiplied Activate through the activation function to obtain the fourth intermediate feature; cross-multiply the fourth intermediate feature with the third intermediate feature and reshape it before convolution to obtain the fifth intermediate feature; add the fifth intermediate feature to the feature input to the convolution attention module, obtain the output feature of the convolution attention module through batch normalization and output it; input the high-resolution remote sensing image into the convolution module for feature extraction to obtain the convolution feature; input the convolution feature into the first extraction module for initial feature extraction to obtain the first intermediate output feature; input the first intermediate output feature into the second extraction module for feature extraction to obtain the terminal output feature, so as to learn different feature representations in different spaces and improve model performance; input the terminal output feature into the void convolution network for void convolution feature extraction to obtain the void convolution feature; input the first intermediate output feature and the void convolution feature into the decoding network for road extraction to obtain the road image in the input high-resolution remote sensing image.

Images