A road extraction method and device based on dual-temporal medium-resolution data

Abstract

The present application discloses a road extraction method and device based on dual-phase medium-resolution data, which solves the problems of high cost and small coverage of road extraction in the prior art. The road extraction method based on dual-phase medium-resolution data includes: determining a dual-phase road data set; the dual-phase road data set contains RGB band data of two phases; training a dual-phase branch road extraction model through the dual-phase road data set for identifying roads in the image; determining the RGB band data of two phases in the target area; inputting the RGB band data of two phases into the dual-phase branch road extraction model to determine the inferred results of the road data in the target area. The present application realizes large-scale field road network extraction through medium-resolution data. The road network is completely extracted through dual-phase data. It overcomes the problem that the existing method uses high-resolution data to extract road networks, which is difficult to apply to large-scale road network extraction.

Description

Technical Field

[0001] The present application relates to the field of remote sensing technology, and in particular to a road extraction method and device based on dual-temporal medium-resolution data. Background Art

[0002] Field roads are the key to agricultural production. The density and connectivity of field roads directly affect the efficiency of agricultural product transportation, the convenience of mechanized operations, and the cost of agricultural production. Comprehensive evaluation of regional field roads can reflect the level and potential of agricultural production and provide guidance for the optimization of agricultural resources. At the same time, scientific and reasonable planning and maintenance of farmland roads can promote the sustainable development of agriculture, improve the rural ecological environment, and enhance the quality of life of farmers. Therefore, large-scale extraction of farmland roads is not only an important indicator for evaluating the level of farmland construction, but also an important support and guarantee for promoting high-quality agricultural development.

[0003] Remote sensing technology provides tools for large-scale extraction of field roads, but current research mainly focuses on the extraction of highways, urban roads, etc. In terms of road extraction, research has shifted from machine learning methods to deep learning.

[0004] The main problem at present is that road extraction is overly dependent on sub-meter high-resolution remote sensing images, which are expensive to obtain, have a small coverage area, and poor real-time performance, making it difficult to meet the needs of large-scale road extraction. Medium-resolution data can meet the identification needs of most field roads and has the characteristics of large range and strong real-time performance, but there is a lack of corresponding research; in addition, field roads may be blocked during the plant growth cycle, making them undetectable, and the frequency of use and management differences in different seasons and agricultural production cycles significantly affect road characteristics, making it difficult to fully extract roads with single-phase data; in response to these problems, a method is needed that can achieve large-scale field road extraction and improve extraction accuracy by utilizing dual-phase data and multiple modules, thereby overcoming the limitations of previous methods. Summary of the invention

[0005] The embodiments of the present application provide a road extraction method and device based on dual-temporal medium-resolution data, which solves the problems of high cost and small coverage of road extraction in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a road extraction method based on bi-temporal medium-resolution data, comprising:

[0007] Determine a dual-phase road data set; the dual-phase road data set includes RGB band data of two phases;

[0008] The bi-temporal road dataset is used to train a bi-temporal branch road extraction model to identify roads in images.

[0009] Determine the RGB band data of two phases in the target area;

[0010] The RGB band data of two time phases are input into the dual-time phase branch road extraction model to determine the inference result of the road data in the target area.

[0011] In one of the embodiments, the dual-time-phase branch road extraction model includes a U-type network encoder group;

[0012] The RGB band data of the two phases are respectively passed through a U-shaped network encoder group; the U-shaped network encoder group includes a multi-layer encoder; the encoder is used to extract the characteristics of the road in the target area;

[0013] The two images at different phases are fused into the first feature data through the cross attention module after being output from the last encoder layer;

[0014] The point convolution data of two images at different phases are calculated at the same layer of other encoders, and the point convolution data at different layers constitute the second feature data input to each level of the cascade decoder;

[0015] The first feature data and the second feature data are fused by a decoder and then an inference result of the road data in the target area is output.

[0016] In one embodiment, the encoder uses snake convolution.

[0017] In one embodiment, two images of different time phases are fused into first feature data through a cross attention module after being output from the last encoder layer, specifically comprising the steps of:

[0018] Obtain features of two branches of the U-shaped network encoder group: phase 1 features and phase 2 features;

[0019] Multiply the phase 1 feature or the phase 2 feature with the three weight matrices to obtain three matrices: key K, query Q and value V;

[0020] The K and Q matrices are compressed and merged into one dimension, and the transposed query matrix Q and key matrix K are multiplied and then multiplied by the Softmax function to obtain the attention matrix;

[0021] The first attention matrix and the second attention matrix obtained by the phase 1 feature and the phase 2 feature are multiplied by the value V generated by the two phase features respectively, and then the results of the same phase are combined in the channel dimension to obtain the first cross attention matrix and the second cross attention matrix;

[0022] The first cross attention matrix and the second cross attention matrix are input into the point convolution respectively and then the channels are merged;

[0023] The result of channel merging is multiplied by the learnable parameter and then connected with the residual of the phase feature before the input to output the first feature data after phase fusion.

[0024] In one embodiment, after the first feature data and the second feature data are fused through a cascade decoder, the method further comprises the steps of:

[0025] Obtaining first output data and second output data; the first output data is the output result of the last decoder; the second output data is the output data of the first layer of the network after merging the data of the two phases and then processing;

[0026] The first output data and the second output data are combined through a residual connection to form input data of a U-shaped structure optimization unit;

[0027] The first output data and the second output data losses are added together to adjust the parameters of the dual-time phase branch road extraction model through back propagation.

[0028] In one embodiment, at least one of the following is used as an evaluation indicator for the inference result of the road data:

[0029] Calculate the intersection-and-joint ratio of the center lines of the inferred result of the road data and the true value as the evaluation index;

[0030] The ratio of the centerline length of the inferred result of the road data in the actual road buffer to the centerline length of the actual road label is calculated as an evaluation indicator;

[0031] The harmonic mean of precision and recall is used as the evaluation metric.

[0032] In one embodiment, the encoders other than the last layer use point convolution before entering the decoder of the corresponding layer;

[0033] The fused result of the encoder output of the last layer enters the decoder corresponding to the previous layer;

[0034] The result of each layer of decoders enters the previous layer of decoders, and so on, until the first layer of decoders.

[0035] In the second aspect, the embodiment of the present application also provides a road extraction device based on dual-phase medium-resolution data, which is used to implement the method described in any one of the embodiments of the first aspect, and includes: a dual-phase branch road extraction model. The dual-phase branch road extraction model includes: a U-shaped network encoder group, a cross-attention module, a point convolution module and a decoder module. The U-shaped network encoder group includes multiple layers of encoders for extracting road data of RGB band data. The cross-attention module is used to fuse the underlying features of the encoders at the last layer of different time images. The point convolution module is used to perform point convolution on the output results of each layer of the encoder respectively. The decoder module is used to cascade the point convolution results; it is also used to fuse the output data of the cross-attention module and the cascade data of the point convolution.

[0036] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any one of the embodiments of the first aspect.

[0037] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any one of the embodiments of the first aspect when executing the computer program.

[0038] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0039] This application uses medium-resolution data to achieve large-scale field road network extraction. The road network is completely extracted through dual-phase data. This overcomes the problem that existing methods use high-resolution data to extract road networks, which is difficult to apply to large-scale road network extraction. Due to the large difference between roads and backgrounds in medium-resolution data, it can also be used for the identification and extraction of most roads. At the same time, there is a lack of consideration for the characteristics of field roads that may change over time. This application uses dual-phase data to extract road features in two periods and then extracts them, which can obtain more accurate and complete road network results. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 This is a flowchart of a large-scale road network extraction based on a dual-phase branch network model according to an embodiment of the present application;

[0042] Figure 2 This is a flow chart of a road extraction method based on dual-temporal medium-resolution data according to an embodiment of the present application;

[0043] Figure 3 The process of constructing a dual-phase road dataset in the embodiment of the present application;

[0044] Figure 4 This is a schematic diagram of the structure of a road extraction device based on dual-temporal medium-resolution data in this application;

[0045] Figure 5 This is a schematic diagram of the serpentine convolution structure of an embodiment of the present application;

[0046] Figure 6 This is a schematic diagram of the structure of the cross-attention module in an embodiment of the present application;

[0047] Figure 7 This is a schematic diagram of the U-shaped structure optimization unit structure of an embodiment of the present application;

[0048] Figure 8-1 This is a schematic diagram of the field road extraction results in Youyi County of this application embodiment;

[0049] Figure 8-2 This is a schematic diagram of the field road extraction results in Suihua City in the embodiment of the present application;

[0050] Fig. 9 A schematic diagram of the results of the model of the embodiment of the present application on a dual-phase road test set;

[0051] Fig.10 This is a structural diagram of a road extraction device based on dual-temporal medium-resolution data according to an embodiment of the present application;

[0052] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0054] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0055] The overall technical process is as follows Figure 1 shown. Figure 1This is a flowchart of a large-scale road network extraction based on a dual-phase branch network model for an embodiment of the present application. Based on the high-resolution image data from the Sentinel-2 satellite, synthetic images from April to June and August to October are collected to construct a dual-phase field road dataset. The field road network in some areas of the image is manually outlined to obtain road network data labels. A dual-phase branch road extraction network model is designed, and the model is trained using the constructed dataset. Finally, the model is applied to other areas to achieve large-scale field road prediction, and its accuracy is verified using a test set. The following further describes the various embodiments of the present application.

[0056] Figure 2 This is a flow chart of a road extraction method based on dual-temporal medium-resolution data according to an embodiment of the present application, including steps 110 to 130.

[0057] Step 110: Determine a dual-phase road data set; the dual-phase road data set includes RGB band data of two phases;

[0058] Step 120: training a bi-temporal branch road extraction model using a bi-temporal road dataset to identify roads in an image;

[0059] In one of the embodiments, in step 120, the dual-time-phase branch road extraction model includes a U-type network encoder group;

[0060] The dual-phase branch road extraction model includes two U-shaped network encoder groups; one of the U-shaped network encoder groups includes a multi-layer encoder;

[0061] Inputting the RGB band data of the two time phases into the dual-time phase branch road extraction model, respectively passing through the U-shaped network encoder group;

[0062] Considering that farmland roads have different characteristics at different time nodes, it is difficult to completely extract the roads only through single-phase features. Therefore, a dual-phase branch network is designed to extract farmland roads.

[0063] A dual-temporal field road dataset was created using Sentinel-2 satellite data for model training and testing.

[0064] The data of this dataset are mainly concentrated in Northeast China. Sentinel-2 data can be used free of charge, has a short revisit period, and its resolution can meet the needs of farmland road extraction. This data is selected as the image source of the dataset.

[0065] For example, Figure 3 As shown, the Sentinel-2 RGB composite images were acquired during April to June and August to October, and the two stages represent the lowest and highest stages of vegetation cover, respectively.

[0066] The training data consists of RGB band data from two phases. The data of the two bands are input into the two branches of the model respectively. The road label data is a manually made binary image, with the road surface area marked as 1 and the non-road area marked as 0. The training data and road label data are fed into the model for training, so that the model can acquire the ability to extract road networks.

[0067] It should be noted that the synthesis effect is to facilitate the integration of data into a tiff image. In fact, the data of two phases are separately input into the following dual-phase branch road extraction model.

[0068] For example, the basic framework is to use two U-shaped network encoder parts to extract features from images at two different times.

[0069] The RGB band data of the two phases are respectively passed through a U-shaped network encoder group; the U-shaped network encoder group includes a multi-layer encoder; the encoder is used to extract the characteristics of the road in the target area;

[0070] A dynamic snake convolution encoder is added to the encoder, and after the last layer output of the encoder part, a cross attention module is designed to fuse the bottom-level features of the last layer encoder of different time images, and the remaining encoders use cascaded point convolution to fuse features. At the end of the decoder part, a U-shaped structure optimization unit (UOM) is added to improve the preliminary results, and then the final road inference result is output, and the loss of the two results is calculated to realize the back propagation of the network. The model is as follows Figure 4 shown.

[0071] The encoder may use ordinary convolution, snake convolution, or other types of convolution, but Figure 5 As shown, compared with ordinary convolution, the linear structure of (a) and (b) in the figure is more consistent with the geometric shape of the road and more suitable for capturing the characteristics of the road. Therefore, preferably, the encoder introduces the snake convolution.

[0072] The snake convolution is introduced in the encoder, which is a deformable banded convolution. The snake convolution can learn deformation based on the input feature map and adaptively focus on slender and curved local features.

[0073] The specific encoder unit is a snake convolution and a normal convolution that processes the input features simultaneously, integrates the features through dimensional connection, and then achieves dimensionality reduction through a convolution layer with a normalization layer. These components constitute the encoder structure of the model.

[0074] The two images at different phases are fused into the first feature data through the cross attention module after being output from the last encoder layer;

[0075] In the deep layer of the encoder (encoder 5), a cross-attention module is used to fuse the underlying features of different branches as the first feature data.

[0076] For example, this application designs two modules to fuse data from different phases: one method is cascaded point convolution (generating second feature data), and the other is a cross-attention module (generating first feature data).

[0077] Since the network structure has only one decoder part, the outputs of the two encoders in the same layer need to be fused before entering the decoder.

[0078] In one embodiment, in step 120, two images of different time phases are fused into first feature data through a cross attention module after being output from the last encoder layer, specifically comprising the steps of:

[0079] Step 120-1, obtaining features of two U-shaped network encoder group branches: phase 1 features and phase 2 features;

[0080] Step 120-2, multiplying the phase 1 feature or the phase 2 feature with the three weight matrices to obtain three matrices: key K, query Q and value V;

[0081] After the last encoder layer, we have two encoder branches ( Figure 4 The phase 1 snake convolution encoder 5 and the phase 2 snake convolution encoder 5) are used to obtain the output phase 1 features and phase 2 features.

[0082] For example, Figure 6 As shown, taking the phase 1 feature as an example, the original input feature is multiplied by the three weight matrices (WK, WQ, WV) to obtain three new matrices key K, query Q, value V.

[0083] Step 120-3, compress and merge the K and Q matrices into one dimension, multiply the transposed query matrix Q and the key matrix K and then multiply them by the Softmax function to obtain the attention matrix;

[0084] Step 120-4, multiply the first attention matrix and the second attention matrix obtained by the phase 1 feature and the phase 2 feature by the value V generated by the two phase features respectively, and then merge the results in the channel dimension to obtain the first cross attention matrix and the second cross attention matrix; input the first cross attention matrix and the second cross attention matrix into the point convolution respectively and then merge the channels; multiply the result of the channel merger with the learnable parameter and then link it with the phase feature residual before the input to output the first feature data after the phase fusion.

[0085] The attention matrix values ​​generated by the feature maps of the two encoders are multiplied by V generated by the two phase features respectively, and then merged in the channel dimension to obtain the cross attention matrix. The cross attention matrix is ​​input into the point convolution for processing and merged with the cross attention matrix obtained by the other phase branch for channel merging, multiplied by a learnable parameter, and then residually connected with the phase features before the input to output the phase fusion features.

[0086] The point convolution data of two images at different phases are calculated at the same layer of other encoders, and the point convolution data at different layers constitute the second feature data input to each level of the cascade decoder;

[0087] In the shallow layers of the network (encoders 1, 2, 3, and 4), a point convolution module is used to process shallow sub-features to obtain the second feature data;

[0088] The first feature data and the second feature data are fused by a decoder and then an inference result of the road data in the target area is output.

[0089] Step 130, determining the RGB band data of the two time phases in the target area; inputting the RGB band data of the two time phases into the dual-time phase branch road extraction model to determine the inference result of the road data in the target area.

[0090] The first feature data and the second feature data are fused through a cascade decoder to output a road estimation result of the target area.

[0091] In one embodiment, the encoders other than the last layer are fused into the decoders of the corresponding layers using cascaded point convolution. The fused result of the encoder output of the last layer is fed into the decoder corresponding to the previous layer. The result of each layer of decoders is fed into the decoder of the previous layer, and so on, until the first layer of decoders.

[0092] For example, in one embodiment, after the first feature data and the second feature data are fused through a cascade decoder in step 130, the step further includes the following steps:

[0093] Step 140, obtaining first output data and second output data; the first output data is the output result of the last decoder; the second output data is the output data after the first layer of the network merges and processes the data of the two phases;

[0094] A U-shaped structure optimization unit is designed at the end of the dual-phase branch road extraction model to improve the road inference result at the decoder end.

[0095] Since the last layer of the network decoder outputs preliminary road inference results, the road continuity is usually poor and does not conform to the geometric characteristics of normal roads.

[0096] The last decoder refers to Figure 4 In decoder 1, the features pass through decoders 4, 3, 2, and 1 in order to finally output the inference result.

[0097] It should be noted that, for the last decoder, i.e. the first layer decoder mentioned above, the order in which the feature data passes is opposite to that of the layer where the decoder is located.

[0098] The first layer of the network refers to the layer before encoder 1. The function of this layer is to merge the data of two phases through the convolution layer and input them together into the U-shaped structure optimization unit at the end of the model.

[0099] Therefore, the U-shaped structure optimization unit is used to Figure 4 The result in decoder 1 is corrected to make the final output more consistent with the geometric characteristics of the road.

[0100] For example, the structure of the U-shaped structure optimization unit is a small U-shaped network including a four-layer encoder and a four-layer decoder, such as Figure 7 shown.

[0101] The input data consists of two parts: the first part is the preliminary result output by the last decoder, and the second part is the output result of the first layer of the network after merging and processing the data of the two phases.

[0102] Step 150: Combine the first output data and the second output data through residual connection to form input data of the U-shaped structure optimization unit;

[0103] These two parts are combined via a residual connection to form the input.

[0104] Step 160: Add the losses of the first output data and the second output data to adjust the parameters of the dual-time-phase branch road extraction model through back propagation.

[0105] The loss is calculated for both the preliminary results and the optimization results by adding the two parts of the loss and adjusting the model parameters through back-propagation.

[0106] In summary, the road network extraction result of the target area is determined by the RGB band data of two time phases and the road estimation result.

[0107] Furthermore, according to the dual-phase branch road extraction network model obtained through model training, the extraction range of the road network can be expanded, and the road network extraction can be uniformly performed on the target area not larger than a certain threshold.

[0108] By collecting Sentinel-2 data from two time phases of the target area, for example, April-June and August-October, and inputting them into the network, road network extraction can be achieved for the image.

[0109] For example, Figure 8-1 and 8-2 As shown in Figure 2, the method and model described in this application are applied to a city and a county in China. In the task of predicting road networks over a large area with low data cost and high efficiency, the method is feasible and effective. Figure 8-1 and Figure 8-2 shown.

[0110] in, Figure 8-1 The results of farmland road inference in a certain county (46°28' to 46°59'N, 131°28' to 132°15'E) are shown, covering 1,700 square kilometers.

[0111] Figure 8-2 The results of farmland road inference in a city (45°3' to 48°02'N, 124°13' to 128°30'E) are shown, covering 35,000 square kilometers. Both images of the two areas are 2022 Sentinel-2 synthetic images.

[0112] Sentinel-2 data is free and open source, with a small amount of data and easy collection, making it suitable for extracting road network data on a large scale.

[0113] Furthermore, to verify the effectiveness of the model, accuracy verification was carried out on a dual-phase dataset.

[0114] In these experiments, the designed model was compared with five road extraction methods, including basic general networks: U-net, Deeplabv3, and mainstream road extraction networks: D-linknet, SGCnet, and NL-linknet. U-net is a basic algorithm in the field of image feature extraction, and the architecture of most road extraction models is developed from this network. Deeplabv3 processes feature information of different scales by designing spatial pyramid pooling in the network. D-linknet is a network model that expands the receptive field through void convolution to collect more comprehensive road information. SGCnet uses separable convolution to divide road features into spatial and channel features, and uses graph convolutional neural networks to deeply extract these two types of features. NL-Linknet designs a non-local attention module and integrates the attention mechanism into the network to extract roads, achieving better results than D-Linknet with fewer parameters. The road extraction results of the local test set are shown in the figure. Fig. 9 As shown, (a) original image, (b) true label, (c) U-net, (d) Deeplab V3, (e) D-linknet, (f) SGCnet, (g) NL-linknet, and (h) dual-phase branch model.

[0115] The test set accuracy results are shown in Table 1. The dual-phase branch network performs well in field road extraction compared with these models.

[0116]

[0117] Table 1 Field road recognition accuracy of different road extraction models

[0118] Considering that the characteristics of roads in paddy fields and dry land change inconsistently over time, the experimental area was divided into two categories. At the same time, test sets with different time ranges (April to June, June to August, August to October, and October to December) were produced in the two types of farmland. By conducting road extraction experiments on data from each time phase, as shown in Table 2, it was found that the recognition accuracy between April and June and August to October in the single-phase data was higher, but much lower than the accuracy that can be achieved with the dual-phase data.

[0119]

[0120] Table 2 Road extraction accuracy of data in different time phases

[0121] In one embodiment, at least one of the following is used as an evaluation indicator for the road estimation result:

[0122] For the extraction of linear features, we designed indicators that are more suitable for evaluating road results: road buffer intersection ratio, road length accuracy, and F1 index.

[0123] Calculate the intersection-and-joint ratio of the center lines of the road inference result and the true value as the evaluation index;

[0124] Since the labels of the training set are generated from binary images based on the centerline of the road, the road width is only one pixel. Using the intersection-and-union ratio directly as an evaluation indicator is too low to be effective. Therefore, the centerline of the actual inference result and the true value is extended by three pixels, and then the intersection-and-union ratio of this part is calculated as the evaluation indicator, and this indicator is named the buffer intersection-and-union ratio.

[0125] Formula 1

[0126] Buffer is a buffer function, which represents the expansion of some pixels in the value area of ​​the image. Represents the intersection of the inferred result and the actual road label, Represents a union.

[0127] The length of the road is a key attribute. Most road extraction studies ignore this metric. The experiment introduces a metric called length accuracy to evaluate the inference results. This metric is the ratio of the centerline length of the predicted road in the actual road buffer to the centerline length of the actual road label. In fact, considering the offset of the inference result, a 1-pixel buffer of the extraction result is selected to calculate this metric.

[0128] Formula 2

[0129] Where Length represents the function of the statistical center line length.

[0130] The F1 index is a measure of model accuracy, especially when dealing with imbalanced dataset distribution, and is therefore a suitable evaluation parameter for road extraction tasks. It represents the harmonic mean of precision and recall.

[0131] Formula 3

[0132] Where TP represents the number of correctly predicted road pixels, FP represents the number of incorrectly predicted road pixels, and FN represents the number of incorrectly predicted non-road pixels.

[0133] The ratio of the centerline length of the road inference result in the actual road buffer to the centerline length of the actual road label is calculated as an evaluation indicator;

[0134] The harmonic mean of precision and recall is used as the evaluation metric.

[0135] Figure 4 This is a schematic diagram of the structure of a road extraction device based on dual-temporal medium-resolution data according to an embodiment of the present application, which is used to implement the method described in any one of the embodiments of the first aspect, comprising:

[0136] Dual-temporal branch road extraction model 1.

[0137] The dual-phase branch road extraction model includes: a U-shaped network encoder group 11, a cross attention module 12, a point convolution module 13 and a decoder module 14.

[0138] The U-shaped network encoder group includes multiple layers of encoders for extracting road data from RGB band data.

[0139] The cross attention module is used to fuse the bottom-level features of different time images located in the last layer encoder.

[0140] After the last encoder layer, we get the output phase 1 features and phase 2 features from the two encoder branches.

[0141] Taking the phase 1 feature as an example, the original input feature is multiplied by three weight matrices (WK, WQ, WV) to obtain three new matrices key K, query Q, and value V. The K and Q matrices are compressed and merged into one dimension, and the transposed query matrix Q and key matrix K are multiplied and then multiplied by the Softmax function to obtain the attention matrix. The attention matrix values ​​generated by the feature maps of the two encoders are multiplied by V generated by the two phase features respectively, and then merged in the channel dimension to obtain the cross attention matrix. The cross attention matrix is ​​input into the point convolution for processing and channel-merged with the cross attention matrix obtained from the other phase branch, multiplied by a learnable parameter, and then residually connected with the phase feature before the input to output the phase fusion feature.

[0142] The point convolution module is used to perform point convolution on the output results of the encoder at each layer.

[0143] The decoder module is used to cascade the point convolution results; it is also used to fuse the output data of the cross attention module and the cascade data of the point convolution.

[0144] Furthermore, the dual-phase branch road extraction model further comprises a U-shaped structure optimization module 15, and the U-shaped structure optimization module is used to improve the road prediction result at the decoder end.

[0145] The last layer of the network decoder outputs preliminary road prediction results, where the road continuity is usually poor and does not conform to the geometric characteristics of normal roads. Figure 3 The result in decoder 1 is corrected to make the final output more consistent with the geometric characteristics of the road. Its structure is a small U-shaped network, including four layers of encoders and four layers of decoders, such as Figure 7 As shown in Figure 1. The input data consists of two parts: the first part is the preliminary result output by the last decoder, and the second part is the output result of the first layer of the network after merging the data of the two phases and then processing them. These two parts are combined through residual connection to form the input of the U-shaped structure optimization unit. The loss is calculated for both the preliminary result and the optimization result, and the model parameters are adjusted by adding the two parts of the loss and back propagating.

[0146] Fig.10 This is a structural diagram of a road extraction device based on dual-temporal medium-resolution data in an embodiment of the present application, which is used to implement the method described in any one of the embodiments of the first aspect, comprising:

[0147] The determination module 210 is used to determine the RGB band data of two phases in the target area.

[0148] The construction module 220 is used to construct a dual-time phase branch road extraction model.

[0149] The calculation module 230 is used to input the RGB band data of two phases into the dual-phase branch road extraction model, respectively through different U-shaped network encoder groups; the images of two different times are fused into the first feature data after being output by the last layer of encoder; the encoders other than the last layer use cascaded point convolution to fuse into the second feature data; the first feature data and the second feature data are fused through a cascaded decoder to output the road inference result of the target area.

[0150] Furthermore, the determination module described in the present application includes a first determination unit, which is used to determine RGB band data of two phases in the target area.

[0151] Furthermore, the calculation module described in the embodiment of the present application includes a first calculation unit, which is used to input the RGB band data of two phases into the dual-phase branch road extraction model, respectively through different U-type network encoder groups; the images at two different times are fused into first feature data after being output by the last layer of encoder; the encoders other than the last layer use cascaded point convolution to fuse into second feature data; the first feature data and the u second feature data are fused through a cascaded decoder to output the road inference result of the target area.

[0152] In one of the embodiments, the computing module described in the present application further includes an encoder unit, and the encoder unit adopts snake convolution.

[0153] In one of the embodiments, the determination module of the present application further comprises a second determination unit for obtaining features of two branches of the U-shaped network encoder group: a phase 1 feature and a phase 2 feature;

[0154] The calculation module of the present application also includes a cross-attention unit, which is also used to multiply the phase 1 feature or the phase 2 feature with three weight matrices to obtain three matrices: key K, query Q and value V; it is also used to compress and merge the K and Q matrices into one dimension, multiply the transposed query matrix Q and the key matrix K and then multiply them by the Softmax function to obtain the attention matrix; it is also used to multiply the first attention matrix and the second attention matrix obtained by the phase 1 feature and the phase 2 feature by the value V generated by the two phase features, and then merge the results in the channel dimension to obtain the first cross-attention matrix and the second cross-attention matrix; it is also used to input the first cross-attention matrix and the second cross-attention matrix into point convolution respectively and then perform channel merging; it is also used to multiply the result of channel merging with the learnable parameters and then link it with the phase feature residual before input to output the first feature data after phase fusion.

[0155] The second determining unit and the second calculating unit are used to implement all technical features of steps 120 - 1 to 120 - 4 .

[0156] In one of the embodiments, the determination module described in the present application further includes a third determination unit, which is used to obtain the first output data and the second output data;

[0157] The calculation module described in the present application also includes a U-shaped structure optimization unit, which is used to combine the first output data and the second output data through a residual connection to form input data of the U-shaped structure optimization unit; and is also used to add the losses of the first output data and the second output data to adjust the parameters of the dual-phase branch road extraction model through back propagation.

[0158] The third determination unit and the U-shaped structure optimization unit are used to implement all technical features of steps 140 to 160.

[0159] In one of the embodiments, the calculation module further includes a second calculation unit for determining a large-scale road network extraction result through the RGB band data of two time phases and the road inference result.

[0160] The second computing unit is used to implement Figure 8-1 and Figure 8-2 All technical features shown.

[0161] In one of the embodiments, an evaluation module is also included, which is used to calculate the intersection ratio of the center lines of the road speculation result and the true value as an evaluation indicator; it is also used to calculate the ratio of the center line length of the road speculation result in the actual road buffer to the center line length of the actual road label as an evaluation indicator; and the harmonic mean of the precision and recall rate is used as an evaluation indicator.

[0162] The evaluation module is used to implement all the technical features shown in Formula 1, Formula 2 and Formula 3.

[0163] In one embodiment, the computing module further includes a decoder unit, wherein the decoder unit includes a plurality of decoders, wherein the encoders other than the last layer are fused into the decoders of the corresponding layers using cascaded point convolution. The fused result after the output of the last layer encoder enters the decoder corresponding to the previous layer. The result of each layer decoder enters the decoder of the previous layer, and so on, until the first layer decoder.

[0164] It should be noted that the decoder unit and the encoder unit may be components of any unit in the computing module, and no further limitation is made here.

[0165] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0166] Therefore, the present application also proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any embodiment of the present application is implemented.

[0167] Furthermore, the present application also proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any embodiment of the present application when executing the computer program.

[0168] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0171] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0172] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present application. It includes: one or more processors 620; a storage device 610, which is used to store one or more programs. When the one or more programs are run by the one or more processors 620, the one or more processors 620 implement the road extraction method based on dual-phase medium-resolution data provided in the embodiment of the present application. The method includes:

[0173] Determine the RGB band data of two phases in the target area;

[0174] Constructing a dual-phase branch road extraction model; the dual-phase branch road extraction model comprises two U-shaped network encoder groups; one of the U-shaped network encoder groups comprises a multi-layer encoder;

[0175] Inputting the RGB band data of the two time phases into the dual-time phase branch road extraction model, respectively passing through different U-shaped network encoder groups;

[0176] The images at two different times are fused into the first feature data after being output by the last encoder layer;

[0177] The encoders outside the last layer use cascaded point convolution to fuse the second feature data;

[0178] The first feature data and the second feature data are fused through a cascade decoder and then a road estimation result of the target area is output. The electronic device 600 also includes an input device 630 and an output device 640; the processor 620, the storage device 610, the input device 630 and the output device 640 in the electronic device can be connected via a bus or other methods, and the figure takes the connection via a bus 650 as an example.

[0179] The storage device 610, as a computer-readable storage medium, can be used to store software programs, computer executable programs, and module units, such as program instructions corresponding to the method in the embodiment of the present application. The storage device 610 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to the use of the terminal, etc. In addition, the storage device 610 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 610 may further include a memory remotely arranged relative to the processor 620, and these remote memories may be connected via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0180] The input device 630 may be used to receive input numbers, character information or voice information, and generate key signal input related to user settings and function control of the electronic device. The output device 640 may include electronic devices such as a display screen and a speaker.

[0181] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0182] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A road extraction method based on dual-temporal medium-resolution data, characterized in that: include: Determine a dual-phase road data set; the dual-phase road data set includes RGB band data of two phases; A dual-phase branch road extraction model is trained by using a dual-phase road dataset to identify roads in an image; the dual-phase branch road extraction model comprises a U-shaped network encoder group; Determine the RGB band data of two phases in the target area; input the RGB band data of the two phases into the dual-phase branch road extraction model; the RGB band data of the two phases pass through a U-shaped network encoder group respectively; the U-shaped network encoder group includes a multi-layer encoder; the encoder is used to extract the characteristics of the road in the target area; Get the features of two U-shaped network encoder group branches: phase 1 features and phase 2 features; multiply the phase 1 features or the phase 2 features with three weight matrices to obtain three matrices: key K, query Q and value V; compress and merge the K and Q matrices into one dimension, multiply the transposed query matrix Q and the key matrix K and then multiply them by the Softmax function to obtain the attention matrix; multiply the first attention matrix and the second attention matrix obtained by the phase 1 features and the phase 2 features by the value V generated by the two phase features respectively, and then merge the results in the channel dimension to obtain the first cross attention matrix and the second cross attention matrix; input the first cross attention matrix and the second cross attention matrix into the point convolution respectively and then merge the channels; multiply the result of the channel merging with the learnable parameters and then link it with the phase feature residual before the input to output the first feature data after phase fusion; The point convolution data of two images at different phases are calculated at the same layer of other encoders, and the point convolution data at different layers constitute the second feature data input to each level of the cascade decoder; The first feature data and the second feature data are fused through a cascade decoder to output an inference result of the road data in the target area.

2. The road extraction method based on dual-temporal medium-resolution data according to claim 1 is characterized in that: The encoder uses snake convolution.

3. The road extraction method based on dual-temporal medium-resolution data according to claim 1 is characterized in that: After the first feature data and the second feature data are fused through a cascade decoder, the method further comprises the steps of: Obtaining first output data and second output data; the first output data is the output result of the last decoder; the second output data is the output data of the first layer of the network after merging the data of the two phases and then processing; The first output data and the second output data are combined through a residual connection to form input data of a U-shaped structure optimization unit; The first output data and the second output data losses are added together to adjust the parameters of the dual-time phase branch road extraction model through back propagation.

4. The road extraction method based on dual-temporal medium-resolution data according to claim 1 is characterized in that: For the inference results of road data, at least one of the following is used as an evaluation indicator: Calculate the intersection-and-joint ratio of the center lines of the inferred result of the road data and the true value as the evaluation index; The ratio of the centerline length of the inferred result of the road data in the actual road buffer to the centerline length of the actual road label is calculated as an evaluation indicator; The harmonic mean of precision and recall is used as the evaluation metric.

5. The road extraction method based on dual-temporal medium-resolution data according to claim 2 is characterized in that: The encoders other than the last layer use point convolution and then enter the decoder of the corresponding layer; The fused result of the encoder output of the last layer enters the decoder corresponding to the previous layer; The result of each layer of decoders enters the previous layer of decoders, and so on, until the first layer of decoders.

6. A road extraction device based on dual-temporal medium-resolution data, characterized in that: Used to implement the method described in any one of claims 1 to 5, comprising: a dual-time phase branch road extraction model; The dual-phase branch road extraction model comprises: a U-shaped network encoder group, a cross attention module, a point convolution module and a decoder module; The U-shaped network encoder group includes a multi-layer encoder for extracting road data from RGB band data; The cross attention module is used to fuse the bottom-level features of different time images at the last encoder layer; The point convolution module is used to perform point convolution on the output results of the encoder at each layer respectively; The decoder module is used to cascade the point convolution results; it is also used to fuse the cross-attention module output data and the cascaded data of the point convolution.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

Images