Attention-fused remote sensing image road information extraction method and device

Abstract

The invention provides an attention-fused remote sensing image road information extraction method and device, and the method comprises the steps: obtaining an original remote sensing image, carrying out the multi-scale feature extraction and global pooling of the original remote sensing image, obtaining the multi-scale local features and global pooling features, and carrying out the global pooling of the multi-scale local features and the global pooling features; fusing the multi-scale local features and the global pooling features by adopting an attention mechanism module to form shallow attention fusion features; different weights are given to the shallow attention fusion features, layered dynamic weighted stacking is carried out, deep attention fusion features in a dynamic weighted stacking result are extracted by using a convolution attention module, and the shallow attention fusion features and the deep attention fusion features are subjected to jump connection to obtain remote sensing image road information. Wherein the convolution attention module calculates a dynamic weighted combination loss value by using a BCE loss value and a Dice loss value, and adjusts model parameters based on the dynamic weighted combination loss value. According to the scheme, the feature information extraction efficiency and extraction precision are improved.

Description

Technical Field

[0001] The invention belongs to the field of remote sensing image recognition, and in particular relates to a method and device for extracting road information from remote sensing images by using attention fusion. Background Art

[0002] As part of basic geographic information, roads are also an important part of the transportation system, playing an important role in urban planning, traffic navigation, and digital map updates. With the development of science and technology, efficient and high-precision road extraction has become particularly important in geographic surveying and mapping. However, traditional road area annotation methods mainly use manual methods or GPS-based methods. The former is very cumbersome, while the latter often fails to capture important information such as road details, such as width and edges.

[0003] Remote sensing feature extraction models based on deep learning have performed well in remote sensing monitoring tasks. However, for feature types that account for a relatively small proportion of images (such as roads, rivers, etc.), traditional deep learning models have some feature redundancy, and the more complex network structure will waste hardware resources. At present, in terms of road information extraction, the road network extraction methods based on remote sensing technology mainly involve shallow feature observation methods and deep feature mining methods. However, the shallow feature observation method has problems such as poor stability and adaptability to complex information, gradient loss, and overfitting of results, while the deep feature mining method brings higher hardware requirements. Summary of the invention

[0004] The purpose of the present invention is to provide a method and device for extracting road information from remote sensing images with attention fusion, aiming to achieve efficient, intelligent and high-precision road information extraction with smaller parameters.

[0005] According to a first aspect of the present invention, a method for extracting road information from remote sensing images using attention fusion is provided, comprising:

[0006] Acquire an original remote sensing image, perform multi-scale feature extraction and global pooling on the original remote sensing image to obtain multi-scale local features and global pooling features, and use an attention mechanism module to fuse the multi-scale local features and global pooling features to form a shallow attention fusion feature;

[0007] Different weights are assigned to the shallow attention fusion features, and hierarchical dynamic weighted superposition is performed, the deep attention fusion features in the dynamic weighted superposition result are extracted using a convolutional attention module, the shallow attention fusion features and the deep attention fusion features are jump-connected to obtain remote sensing image road information, wherein the convolutional attention module calculates a dynamic weighted combination loss value using a BCE loss value and a Dice loss value, and adjusts model parameters based on the dynamic weighted combination loss value.

[0008] Preferably, the method of using an attention mechanism module to fuse the multi-scale local features and the global pooling features to form a shallow attention fusion feature further includes:

[0009] The convolutional attention module CBAM is used to replace the multi-scale channel attention module MS-CAM of the second layer of the attention fusion module IAFF, and dynamic weighting is used to give initial feature weights to the multi-scale local features when the initial features are superimposed, and the convolutional attention module CBAM is used to perform weighted superposition on the multi-scale local features;

[0010] Dynamic weighting is used to give initial feature weights to the weighted superposition result and the global pooling feature, and the convolutional attention module CBAM is used to perform weighted superposition again to obtain the fused shallow attention fusion feature;

[0011] The initial feature weights are adjusted according to the difference between the training results of the model and the actual surface distribution.

[0012] Preferably, the extracting the deep attention fusion features in the dynamic weighted superposition result by using a convolutional attention module further comprises:

[0013] The different feature layers of the shallow attention fusion feature are compressed through the maximum pooling operation, the DSC operation is performed on the compressed different feature layers, and the operation results are dynamically weighted and superimposed to form the deep attention fusion feature.

[0014] Preferably, the step of calculating the dynamic weighted combined loss value by using the BCE loss value and the Dice loss value further includes:

[0015] Calculate the dynamic weighted combination loss value RWL:

[0016] RWL=R BCE ×W BCE ×BCE Loss+R Dice ×W Dice ×Dice Los;

[0017] in W Dice =1-W BCE ;

[0018] n is the number of current loss values, R BCE is the rate of change of BCE loss value BCELoss, W BCE is the proportion of BCELoss, R Dice is the rate of change of Dice loss value DiceLoss, W Dice The proportion of DiceLoss.

[0019] According to a second aspect of the present invention, there is provided a remote sensing image road information extraction device with attention fusion, comprising:

[0020] A shallow feature extraction module is used to obtain an original remote sensing image, perform multi-scale feature extraction and global pooling on the original remote sensing image to obtain multi-scale local features and global pooling features, and use an attention mechanism module to fuse the multi-scale local features and global pooling features to form a shallow attention fusion feature;

[0021] A deep feature fusion module is used to assign different weights to the shallow attention fusion features and perform hierarchical dynamic weighted superposition, use a convolutional attention module to extract the deep attention fusion features in the dynamic weighted superposition results, jump-connect the shallow attention fusion features and the deep attention fusion features to obtain remote sensing image road information, wherein the convolutional attention module uses the BCE loss value and the Dice loss value to calculate the dynamic weighted combination loss value, and adjust the model parameters based on the dynamic weighted combination loss value.

[0022] The shallow feature extraction module is further used for:

[0023] The convolutional attention module CBAM is used to replace the multi-scale channel attention module MS-CAM of the second layer of the attention fusion module IAFF, and dynamic weighting is used to give initial feature weights to the multi-scale local features when the initial features are superimposed, and the convolutional attention module CBAM is used to perform weighted superposition on the multi-scale local features;

[0024] Dynamic weighting is used to give initial feature weights to the weighted superposition result and the global pooling feature, and the convolutional attention module CBAM is used to perform weighted superposition again to obtain the fused shallow attention fusion feature;

[0025] The initial feature weights are adjusted according to the difference between the training results of the model and the actual surface distribution.

[0026] The deep feature fusion module is further used to:

[0027] The different feature layers of the shallow attention fusion feature are compressed through the maximum pooling operation, the DSC operation is performed on the compressed different feature layers, and the operation results are dynamically weighted and superimposed to form the deep attention fusion feature.

[0028] The deep feature fusion module is further used to:

[0029] Calculate the dynamic weighted combination loss value RWL:

[0030] RWL=R BCE ×WBCE ×BCE Loss+R Dice ×W Dice ×Dice Loss;

[0031] in W Dice =1-W BCE ;

[0032] n is the number of current loss values, R BCE is the rate of change of BCE loss value BCELoss, W BCE is the proportion of BCELoss, R Dice is the rate of change of Dice loss value DiceLoss, W Dice The proportion of DiceLoss.

[0033] Compared with the related art, the technical solution of the present invention has the following advantages:

[0034] By adopting a lightweight strategy and making full use of remote sensing image features, high-precision ground object information can be quickly and efficiently obtained with limited hardware to achieve real-time road extraction. For shallow low-level features, multi-scale feature extraction is combined with the attention mechanism to better focus on useful feature information during training. For deep features, different weights are given to the feature information generated by the deep network to reduce the difficulty of feature selection, enhance the efficiency of feature information extraction, and improve the accuracy of model road extraction.

[0035] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures and processes indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. It is obvious that the drawings described below are certain embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 is a flow chart of a method for extracting road information from remote sensing images by using attention fusion according to the present invention;

[0038] Figure 2 is a diagram of a lightweight weighted attention fusion network structure according to the present invention;

[0039] Figure 3is a diagram of a shallow feature extraction module according to the dynamic weighted attention fusion of the present invention;

[0040] Figure 4 is a diagram of a deep attention feature fusion and output module according to the present invention;

[0041] Figure 5 is a comparison diagram of road extraction results of Massachusetts city images of multiple models including the present invention;

[0042] Figure 6 It is a comparison diagram of road extraction results of Massachusetts suburban images including the multi-model of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The method provided by the present invention can be implemented in the following device environment, and the device may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.

[0045] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire device, and executes various functions of the device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.

[0046] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instructions.

[0047] The display screen is used to display the user interface of each application.

[0048] In addition, those skilled in the art will appreciate that the structure of the above device does not limit the device, and the device may include more or fewer components, or a combination of certain components, or different component arrangements. For example, the device also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.

[0049] Based on the above analysis, the present invention proposes a road information extraction model based on a lightweight attention fusion dynamic overlay network (hereinafter referred to as LWAFNet) based on an encoder-decoder framework. It adopts a lightweight strategy, makes full use of remote sensing image features, and quickly and efficiently obtains high-precision object information with limited hardware to achieve real-time road extraction. LWAFNet is based on the Unet network structure and uses deep separable convolution as the main convolution process to achieve the goal of model lightweighting. For shallow low-level features, combined with the characteristics of linear presentation, complex structure, diverse forms and low proportion of object pixels in remote sensing images, the model combines multi-scale feature extraction with attention mechanism to better focus on useful feature information during training. For deep features, the attention mechanism is combined with a dynamic weighting method. By assigning different weights to the feature information generated by the deep network, the difficulty of feature selection is reduced and the efficiency of feature information extraction is enhanced. In view of the sample imbalance problem caused by the small proportion of road samples in remote sensing images, the Dice similarity coefficient is introduced to strengthen the foreground information and a dynamic weighted combination loss function is designed to improve the road extraction accuracy of the model.

[0050] See also Figure 1 The method for extracting road information from remote sensing images by using attention fusion provided by the present invention comprises:

[0051] Step 101: Acquire an original remote sensing image, perform multi-scale feature extraction and global pooling on the original remote sensing image to obtain multi-scale local features and global pooling features, and use an attention mechanism module to fuse the multi-scale local features and global pooling features to form a shallow attention fusion feature.

[0052] Figure 2 is a diagram of the lightweight weighted attention fusion network structure according to the present invention. Figure 3As shown, in the shallow feature extraction module of weighted attention fusion, for the feature fusion method based on the attention mechanism in the module, the present invention proposes a dynamic weighted attention fusion module DWAF, which draws on the structure of the iterative attention fusion module IAFF to ensure that multi-layer features can be effectively integrated. In the fusion process, the convolutional attention module CBAM is used to replace the multi-scale channel attention module MS-CAM of the second layer of IAFF, and the initial feature weight is given to the multi-scale local features by dynamic weighting when the initial features are superimposed. The multi-scale local features are weighted and superimposed by the convolutional attention module CBAM. By means of dynamic weighting, the initial feature weight can be adjusted according to the difference between the training results of the model and the real surface distribution, so that the model can approach the target result more quickly. First, global pooling and 1×1 convolution are used respectively to extract the global and local features of the input data; then 3×3 and 5×5 convolutions are further used for the local feature layers to extract feature layers at different scales; the above-mentioned DWAF is used to merge the 3×3, 5×5 and global pooling feature layers to obtain shallow attention fusion features; finally, the results are integrated and output by 1×1 convolution. The use of 1×1 convolution can retain feature information with fewer parameters and realize the dimension increase or decrease of features to meet the needs of the model. Figure 3 The plus sign in represents element-by-element addition, and the multiplication sign represents matrix multiplication.

[0053] Step 102: Assign different weights to the shallow attention fusion features, and perform hierarchical dynamic weighted superposition, use the convolutional attention module to extract the deep attention fusion features in the dynamic weighted superposition result, jump-connect the shallow attention fusion features and the deep attention fusion features to obtain the remote sensing image road information, wherein the convolutional attention module uses the BCE loss value and the Dice loss value to calculate the dynamic weighted combination loss value, and adjust the model parameters based on the dynamic weighted combination loss value.

[0054] like Figure 4 As shown, in the deep attention fusion and output module, the present invention constructs a 6-layer deep neural network module. In the encoder stage, the network layers are combined in pairs and divided into three hierarchical blocks. The feature layers are first dynamically weighted and superimposed between each hierarchical block, and then the convolutional attention module CBAM is used to extract the attention feature layer. In order to reduce the amount of model parameters and improve model efficiency, this module uses 2×2 maximum pooling to compress the feature layer between each hierarchical block, performs DSC operation on the compressed different feature layers, and dynamically weighted superimposes the operation results to form the deep attention fusion feature. In the decoder stage, the attention feature layer and the result layer are jump-connected. By reducing the number of jump connections, the model parameters and calculation amount can be further reduced, the model operation speed can be improved, and the hardware requirements can be reduced.

[0055] The present invention uses DSC (depthwise separable convolution) for convolution, which contains 5 layers, namely 1×1 convolution of feature integration layer, 3×3 channel-by-channel convolution of feature extraction layer, point-by-point convolution of feature output layer, data normalization layer and activation layer. In this module, only the convolution output result is standardized and mapped using activation function.

[0056] In channel-by-channel convolution, one convolution kernel is responsible for one channel, and one channel is convolved by only one convolution kernel. Therefore, the parameter amount in the channel-by-channel convolution process is independent of the output channel, and the parameter amount calculation formula of the convolution process is:

[0057] Nchannel=C in *H k *W k

[0058] Where: C in is the number of input data channels, H k and W k Represent the height and width of the convolution kernel respectively.

[0059] Channel-by-channel convolution can reduce model parameters, but this feature extraction method extracts each channel independently and cannot effectively utilize the feature information of different channels at the same spatial position. To make up for this defect, DSC introduces point-by-point convolution. The calculation formula for point-by-point convolution parameters is:

[0060] Npoint=1*1*C out

[0061] Among them: 1*1 is the convolution kernel size, that is, a 1×1 convolution kernel is used for point-by-point convolution, C out is the number of channels of the output feature.

[0062] The overall parameter of DSC is Nchannel+Npoint, and its calculation formula is:

[0063] NDSC=C in *H k *W k +C out .

[0064] In the response weighted loss function module, since Dice Loss performs well when the positive and negative samples are severely unbalanced, the loss calculation is often unstable when processing small target objects during training, which easily leads to drastic changes in gradients. In order to solve this problem, the present invention proposes a response weighted loss (Response Weighted Loss, RWL) method, which determines the weights according to the current BCE Loss and Dice Loss values ​​and the rate of change. By comprehensively considering the weight and change of each loss value, a higher weight is given to the loss value with a larger response, and a lower weight is given to the function with a smaller response. By increasing the proportion of the current higher phase response loss, the model speeds up the adjustment of parameters to reduce the loss value. At the same time, the low response function is used to correct the parameter adjustment direction of the model to maintain the stability of the model, ultimately achieving the purpose of accelerating model convergence and improving accuracy.

[0065]

[0066] WD ice =1-W BCE

[0067] RWL=R BCE ×W BCE ×BCE Loss+R Dice ×W Dice ×Dice Loss

[0068] Where: n is the number of current loss values, R BCE is the rate of change of BCE loss value BCELoss, W BCE is the proportion of BCELoss, R Dice is the rate of change of Dice loss value DiceLoss, W Dice The proportion of DiceLoss.

[0069] The datasets used in the present invention include: (1) an open source Massachusetts road dataset, which contains 1,171 aerial images with a size of 1,500 × 1,500 pixels, covering various types of geographical environments such as cities, suburbs, and rural areas; and (2) a typical rural dataset constructed based on the DeepGlobe land cover classification challenge dataset. The DeepGlobe data was screened to generate a typical rural road dataset with 600 rural road images, including unpaved roads, paved roads, and dirt roads.

[0070] The image information extraction evaluation indicators selected by the present invention include:

[0071] Precision: It indicates the ratio of the number of correctly classified pixels of each type of ground object in the ground object extraction results to the total number of pixels of that type;

[0072] Recall: It indicates the ratio of the number of pixels predicted as correctly classified in each category of the object extraction results to the total number of pixels of that category of objects in the label image;

[0073] Overall Accuracy (OA): indicates the percentage of all correctly classified results in the extraction results;

[0074] F1 Score: The harmonic mean of Precision and Recall, which can comprehensively evaluate the accuracy and comprehensiveness of the segmentation results;

[0075] The intersection over union (IoU) indicates the degree of overlap between the pixel area occupied by the target object in the prediction result and the area occupied by the object in the label.

[0076] The higher the values ​​of these indicators, the higher the extraction accuracy and the better the effect. In addition to the above performance evaluation methods, the model calculation amount (FLOPs) is also introduced to measure the complexity of the model; the model parameter amount (Params) indicates the total number of parameters that need to be trained in model training. The lower the values ​​of these indicators, the smaller the model, the lower the complexity, and the faster the inference speed.

[0077] The loss function ablation results are shown in Table 1. The BCE loss function, Dice Loss loss function, BCE Loss+Dice Loss loss function and RWL loss function are combined with LWAFNet for road extraction. It can be seen that the combination of the self-built loss function RWL and LWAFNet can greatly improve the road extraction effect. The single loss functions are not much different in F1 score and IoU, but the combined loss functions have obvious improvements, among which RWL has the best performance results. Compared with the single loss function BCE loss function and Dice Loss loss function, the F1 score increased by 1.21% and 1.14% respectively, and the IoU increased by 1.46% and 1.37% respectively. Compared with the combined loss function BCE Loss+Dice Loss, the F1 score increased by 0.77% and the IoU increased by 1.03%.

[0078] Table 1

[0079]

[0080] In order to comprehensively evaluate the applicability of LWAFNet for road extraction in different scenarios, the present invention selected urban area images and suburban images of Massachusetts for model performance evaluation experiments.

[0081] Figure 5 The results of 10 deep neural networks in urban area road extraction tasks are shown. It can be seen that all models have good resistance to interference from buildings in urban road extraction, and the urban road network is well extracted in the road extraction task in the urban area. However, it can be found that there are still some shortcomings in the use of deep neural networks for road extraction. There are certain omissions in all road extraction models. This is mainly due to the relatively complex urban road environment, the existence of tree occlusion and unclear road features in some areas, and the relatively small number of samples of these features, which leads to the inability of the model to generate a mapping relationship during the extraction process, resulting in omissions. In response to the above problems, the present invention selects 4 key areas in the image for analysis. For area 1, it can be found that when the surface roads are manually marked, the small road next to the main road in 1 is not marked, and the small road is extracted in the extraction results of FCN-8s, Unet++, UNet+++, DeeplabV3+, HRNet, LMFFNet and LWAFNet. Although it will affect the model accuracy in the final performance judgment, it shows that it is feasible to use deep learning methods to establish a model to realize intelligent road extraction. For the images of area 2, area 3, and area 4, it can be found that there are some omissions. This is because the roads that are not extracted in the image are highly consistent with the surrounding environment, and such roads account for a small proportion in urban areas, so deep mapping is not well established. Overall, UNet+++, LMFFNet, and LWAFNet show better performance than other models in road extraction. Although there are omissions and misclassifications, they can better ensure the integrity of the road network compared to other models.

[0082] Figure 6This is the result of suburban road extraction. Compared with the results of urban area road extraction, the performance of all models in the suburban road extraction task is slightly insufficient. As can be seen from the figure, the extraction results of all models have certain defects, and the road network disconnection and misclassification are more obvious. This is because rural roads are more diverse, including multiple surface categories such as unpaved roads, paved roads and dirt roads, and the ground features are more complex. In order to further verify the actual performance of the model, the present invention marks 4 key areas in the suburban image for analysis. For the road extraction in area 1, it can be found that FCN 8s, Unet, Unet++, UNet+++, DeeplabV3+, HRNet, LMFFNet and LWAFNet all identify roads that are not marked in the real road labels, which further shows that deep learning neural networks can better realize the intelligent road extraction task by automatically learning target features. According to the images of area 2, it can be seen that the roads in area 2 are seriously affected by the surrounding environment, and it is difficult to extract them even with visual interpretation. However, using the deep learning model, it can be found that all models except MACUNet can extract the marked roads in area 2 well. This is because the deep neural network model is automatically modeled by given data and its true distribution labels. By building a deep network, high-dimensional features that cannot be distinguished by humans can be extracted. Therefore, it is possible to achieve results that exceed visual interpretation when using standard data modeling. The extraction results of areas 3 and 4 reflect the shortcomings of the model. It can be found that except for DeeplabV3+ and HRNet, the other models all performed generally and missed the extraction. It can be seen from the images that the roads in the missed areas are mostly areas that are greatly affected by the surroundings and the road features are not obvious.

[0083] Table 2 shows the comparison of road extraction results of 10 models on the Massachusetts road dataset. It is found that the Precision, F1-score and IoU of the LWAFNet model are lower than those of LMFFNet, but better than the other 8 models, indicating that the LWAFNet model does not show the best results in extraction accuracy. However, the model parameters are compared and analyzed from the three aspects of FLOPs, Params and Model Size, and it is found that LWAFNet has great advantages over other models in terms of lightweight. The FLOPs, Params and Model Size of LWAFNet are the minimum values ​​of model parameters among all models, indicating that LWAFNet can use very few model parameters to build a smaller road extraction model, which can not only obtain better road extraction results, but also greatly improve the inference speed of the model. In summary, LWAFNet reduces at least 49% of FLOPs with less than one-tenth of the Params and Model Size of the LMFFNet model, and has a higher road extraction accuracy; LWAFNet can not only greatly improve the applicability of deep learning in remote sensing road extraction tasks, but also can better complete road information extraction.

[0084] Table 2

[0085]

[0086] Through experiments on Massachusetts road images, it was found that the model has certain deficiencies in extracting rural roads with irregular network structures, diverse road forms, and greater influence of the surrounding environment. Therefore, the present invention uses the DeepGlobe land cover classification challenge dataset to establish a typical rural road remote sensing image dataset. Table 3 shows the comparison of road accuracy extracted from images by 10 different models. Compared with other models, the model of the present invention still shows good performance in the road extraction task. Precision, Recall, F1-score and IoU reached 68.37%, 75.39%, 71.71% and 56.25% respectively. Although it did not reach the maximum value among all the compared models, it still showed good performance as a whole, second only to the best model with a small difference in accuracy. Combined with the lightweight comparative analysis of the previous model, the results show that LWAFNet can better balance the relationship between model accuracy and model parameters, and has strong practicality in road extraction tasks.

[0087] Table 3

[0088]

[0089] It can be seen that the attention-fused remote sensing image road information extraction method proposed in the present invention has the following advantages over related technologies: it adopts a lightweight strategy, makes full use of remote sensing image features, quickly and efficiently obtains high-precision ground object information with limited hardware, and realizes real-time road extraction. For shallow low-level features, multi-scale feature extraction is combined with the attention mechanism to better focus on useful feature information during training. For deep features, by assigning different weights to the feature information generated by the deep network, the difficulty of feature selection is reduced, the efficiency of feature information extraction is enhanced, and the accuracy of model road extraction is improved.

[0090] Accordingly, the present invention provides, in a second aspect, a remote sensing image road information extraction device using attention fusion, comprising:

[0091] A shallow feature extraction module is used to obtain an original remote sensing image, perform multi-scale feature extraction and global pooling on the original remote sensing image to obtain multi-scale local features and global pooling features, and use an attention mechanism module to fuse the multi-scale local features and global pooling features to form a shallow attention fusion feature;

[0092] A deep feature fusion module is used to assign different weights to the shallow attention fusion features and perform hierarchical dynamic weighted superposition, use a convolutional attention module to extract the deep attention fusion features in the dynamic weighted superposition results, jump-connect the shallow attention fusion features and the deep attention fusion features to obtain remote sensing image road information, wherein the convolutional attention module uses the BCE loss value and the Dice loss value to calculate the dynamic weighted combination loss value, and adjust the model parameters based on the dynamic weighted combination loss value.

[0093] The above-mentioned device can be implemented by the attention fusion remote sensing image road information extraction method provided by the embodiment of the first aspect above. The specific implementation method can be found in the description of the embodiment of the first aspect, and will not be repeated here.

[0094] It is understood that the model structures and parameters described in the above embodiments are only examples. Those skilled in the art can also easily combine and adjust the structural features of the above embodiments according to the use requirements, and should not limit the concept of the present invention to the specific details of the above examples.

[0095] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute any one of the methods in the foregoing embodiment one. The processor and the memory can be connected via a bus or other means, taking the bus connection as an example. The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various chips.

[0096] As a non-transient storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the attention-fused remote sensing image road information extraction method in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the attention-fused remote sensing image road information extraction method in the above method embodiment is realized.

[0097] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0098] The present invention also provides a storage medium, wherein the storage medium stores a plurality of instructions, wherein the instructions can be loaded and executed by a processor, so that the processor can execute any one of the methods in Embodiment 1. The storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0099] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting road information from remote sensing images using attention fusion, characterized in that: include: Acquire an original remote sensing image, perform multi-scale feature extraction and global pooling on the original remote sensing image to obtain multi-scale local features and global pooling features, and use an attention mechanism module to fuse the multi-scale local features and global pooling features to form a shallow attention fusion feature; Different weights are assigned to the shallow attention fusion features, and hierarchical dynamic weighted superposition is performed, the deep attention fusion features in the dynamic weighted superposition result are extracted using a convolutional attention module, the shallow attention fusion features and the deep attention fusion features are jump-connected to obtain remote sensing image road information, wherein the convolutional attention module calculates a dynamic weighted combination loss value using a BCE loss value and a Dice loss value, and adjusts model parameters based on the dynamic weighted combination loss value.

2. The method for extracting road information from remote sensing images by attention fusion according to claim 1, characterized in that: The adopting of the attention mechanism module to fuse the multi-scale local features and the global pooling features to form a shallow attention fusion feature further includes: The convolutional attention module CBAM is used to replace the multi-scale channel attention module MS-CAM of the second layer of the attention fusion module IAFF, and dynamic weighting is used to give initial feature weights to the multi-scale local features when the initial features are superimposed, and the convolutional attention module CBAM is used to perform weighted superposition on the multi-scale local features; Dynamic weighting is used to give initial feature weights to the weighted superposition result and the global pooling feature, and the convolutional attention module CBAM is used to perform weighted superposition again to obtain the fused shallow attention fusion feature; The initial feature weights are adjusted according to the difference between the training results of the model and the actual surface distribution.

3. The method for extracting road information from remote sensing images by attention fusion according to claim 1, characterized in that: The extracting the deep attention fusion features in the dynamic weighted superposition result by using the convolutional attention module further includes: The different feature layers of the shallow attention fusion feature are compressed through the maximum pooling operation, the DSC operation is performed on the compressed different feature layers, and the operation results are dynamically weighted and superimposed to form the deep attention fusion feature.

4. The method for extracting road information from remote sensing images by attention fusion according to claim 3, characterized in that: The method of calculating the dynamic weighted combined loss value by using the BCE loss value and the Dice loss value further includes: Calculate the dynamic weighted combination loss value RWL: RWL=R BCE ×W BCE ×BCE Loss+R Dice ×W Dice ×Dice Loss; in: W Dice =1-W BCE ; n is the number of current loss values, R BCE is the rate of change of BCE loss value BCELoss, W BCE is the proportion of BCELoss, R Dice is the rate of change of Dice loss value DiceLoss, W Dice The proportion of DiceLoss.

5. A remote sensing image road information extraction device with attention fusion, characterized in that: include: A shallow feature extraction module is used to obtain an original remote sensing image, perform multi-scale feature extraction and global pooling on the original remote sensing image to obtain multi-scale local features and global pooling features, and use an attention mechanism module to fuse the multi-scale local features and global pooling features to form a shallow attention fusion feature; A deep feature fusion module is used to assign different weights to the shallow attention fusion features and perform hierarchical dynamic weighted superposition, use a convolutional attention module to extract the deep attention fusion features in the dynamic weighted superposition results, jump-connect the shallow attention fusion features and the deep attention fusion features to obtain remote sensing image road information, wherein the convolutional attention module uses the BCE loss value and the Dice loss value to calculate the dynamic weighted combination loss value, and adjust the model parameters based on the dynamic weighted combination loss value.

6. The attention fusion remote sensing image road information extraction device according to claim 5, characterized in that: The shallow feature extraction module is further used for: The convolutional attention module CBAM is used to replace the multi-scale channel attention module MS-CAM of the second layer of the attention fusion module IAFF, and dynamic weighting is used to give initial feature weights to the multi-scale local features when the initial features are superimposed, and the convolutional attention module CBAM is used to perform weighted superposition on the multi-scale local features; Dynamic weighting is used to give initial feature weights to the weighted superposition result and the global pooling feature, and the convolutional attention module CBAM is used to perform weighted superposition again to obtain the fused shallow attention fusion feature; The initial feature weights are adjusted according to the difference between the training results of the model and the actual surface distribution.

7. The attention fusion remote sensing image road information extraction device according to claim 5, characterized in that: The deep feature fusion module is further used to: The different feature layers of the shallow attention fusion feature are compressed through the maximum pooling operation, the DSC operation is performed on the compressed different feature layers, and the operation results are dynamically weighted and superimposed to form the deep attention fusion feature.

8. The attention fusion remote sensing image road information extraction device according to claim 7, characterized in that: The deep feature fusion module is further used to: Calculate the dynamic weighted combination loss value RWL: RWL=R BCE ×W BCE ×BCE Loss+R Dice ×W Dice ×Dice Loss; in: W Dice =1-W BCE ; n is the number of current loss values, R BCE is the rate of change of BCE loss value BCELoss, W BCE is the proportion of BCELoss, R Dice is the rate of change of Dice loss value DiceLoss, W Dice The proportion of DiceLoss.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute the attention fusion remote sensing image road information extraction method as described in any one of claims 1 to 4.

10. A storage medium, characterized in that: The storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor and executed by the method for extracting road information from remote sensing images by attention fusion as described in any one of claims 1 to 4.

Images