Method and related device for road change detection based on hybrid bar convolution

Abstract

The application provides a road change detection method based on mixed bar convolution and related equipment, and the method comprises the following steps: acquiring two images to be detected for the same position; extracting road linear features in each of the images to be detected through a feature enhancement module; wherein the feature enhancement module is obtained based on a multi-scale bar convolution unit; performing fusion processing on the road linear features in each of the images to be detected through a cross fusion module to obtain fusion features; and performing road change detection based on the fusion features. The application captures the long-distance dependency relationship of road features through the feature enhancement module, and interacts and transmits the spatial information and the time information of two road linear features through the cross fusion module, thereby improving the accuracy of road change detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a road change detection method based on mixed bar convolution and related equipment. BACKGROUND

[0002] The purpose of remote sensing change detection (RSCD) is to locate and segment changes in the ground surface from a pair of co-registered images taken at different times of the same area. It is an important problem in the task of remote sensing understanding and a key step for many real-world tasks, such as global resource monitoring, land use change detection, damage assessment, and urban management.

[0003] Road change detection based on dual-phase remote sensing images is to identify newly built, disappeared, damaged or reconstructed roads according to images collected at different times. Existing road change detection methods based on deep learning are mainly based on general change detection neural networks. General change detection networks can effectively detect land cover changes, however, complex post-processing steps are needed to find changes related to roads. Since roads in remote sensing images are usually distributed throughout the image, they have the characteristics of large span and narrow shape, and the general change detection method uses a square convolution kernel, which cannot capture the linear features of the road well. Moreover, since roads are easily affected by the environment, they are easily shaded by trees and blocked by buildings, resulting in different appearance features of the same road in dual-phase images. SUMMARY

[0004] The present application provides a road change detection method based on mixed bar convolution and related equipment to solve the problem of road discontinuity, incompleteness and easy to be blocked in the existing road change detection of remote sensing images, which leads to low detection accuracy. The present application realizes to improve the accuracy of road change detection.

[0005] The present application provides a road change detection method based on mixed bar convolution, comprising:

[0006] Obtaining two images to be detected for the same position;

[0007] Extracting road linear features in each of the images to be detected through a feature enhancement module, wherein the feature enhancement module is obtained based on a multi-scale bar convolution unit;

[0008] Fusing the road linear features in each of the images to be detected through a cross-fusion module to obtain fused features;

[0009] Performing road change detection based on the fused features.

[0010] According to the road change detection method based on the mixed bar convolution provided by the application, the multi-scale bar convolution unit includes a plurality of bar convolution units for different directions; the road linear features in each of the to-be-detected images are extracted by the feature enhancement module, including:

[0011] The first features in the to-be-detected images are extracted by each of the bar convolution units respectively;

[0012] Based on the first features corresponding to each of the to-be-detected images, the road linear features of each of the to-be-detected images are obtained.

[0013] According to the road change detection method based on the mixed bar convolution provided by the application, the multi-scale bar convolution unit includes bar convolution units for horizontal direction, vertical direction, left diagonal direction and right diagonal direction respectively.

[0014] According to the road change detection method based on the mixed bar convolution provided by the application, the feature enhancement module further includes a multi-scale hollow convolution unit, and the road linear features in each of the to-be-detected images are extracted by the feature enhancement module, including:

[0015] The second features of each of the to-be-detected images are extracted by the multi-scale bar convolution unit, and the third features of each of the to-be-detected images are extracted by the multi-scale hollow convolution unit;

[0016] The second features and the third features of each of the to-be-detected images are connected to obtain the road linear features of each of the to-be-detected images.

[0017] According to the road change detection method based on the mixed bar convolution provided by the application, the cross fusion module includes a self-attention mechanism and a depth separable convolution; the road linear features in each of the to-be-detected images are fused and processed by the cross fusion module to obtain a fusion feature, including:

[0018] The road linear features are spliced to obtain road linear splicing features;

[0019] The road linear splicing features are respectively input into the self-attention mechanism and the depth separable convolution for processing, and channel interaction and spatial interaction are performed in the processing process to obtain output features corresponding to the self-attention mechanism and the depth separable convolution respectively;

[0020] Based on each of the output features, the fusion feature is obtained by fusion processing.

[0021] According to the method for detecting road change based on mixed strip convolution provided by the application, each output feature is summed based on pixels, and the summed result is mixed by convolution to obtain a mixed feature.

[0022] Based on pixels, each output feature is summed and processed, and the result after the summation processing is mixed by convolution to obtain a mixed feature.

[0023] Based on pixels, the mixed feature is processed by difference to obtain the fusion feature.

[0024] According to the method for detecting road change based on mixed strip convolution provided by the application, each output feature is summed based on pixels, and the summed result is mixed by convolution to obtain a mixed feature.

[0025] The fusion feature is up-sampled and spliced, and a road change map is obtained based on the fusion feature after the up-sampling and splicing.

[0026] The application also provides a device for detecting road change based on mixed strip convolution, comprising:

[0027] The acquisition module is configured to acquire two images to be detected for the same position.

[0028] The extraction module is configured to extract road linear features in each image to be detected by a feature enhancement module, wherein the feature enhancement module is obtained based on a multi-scale strip convolution unit.

[0029] The fusion processing module is configured to perform fusion processing on the road linear features in each image to be detected by a cross-fusion module to obtain a fusion feature.

[0030] The detection module is configured to detect road change based on the fusion feature.

[0031] The application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for detecting road change based on mixed strip convolution as described above.

[0032] The application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method for detecting road change based on mixed strip convolution as described above.

[0033] The application also provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the method for detecting road change based on mixed strip convolution as described above.

[0034] The application provides a road change detection method based on mixed bar convolution and related equipment. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0036] Figure 1 Fig. 1 is one of the flow diagrams of the road change detection method based on mixed bar convolution provided by the application.

[0037] Figure 2 Fig. 2 is a structural diagram of the road change detection method based on mixed bar convolution provided by the application.

[0038] Figure 3 Fig. 3 is one of the structural diagrams of the feature enhancement module provided by the application.

[0039] Figure 4 Fig. 4 is another structural diagram of the feature enhancement module provided by the application.

[0040] Figure 5 Fig. 5 is a structural diagram of the cross-fusion module provided by the application.

[0041] Figure 6 Fig. 6 is an experimental comparison diagram provided by the application.

[0042] Figure 7 Fig. 7 is a structural diagram of the road change detection device based on mixed bar convolution provided by the application.

[0043] Figure 8 Fig. 8 is a structural diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0044] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the accompanying drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0045] Early traditional RSCD methods mainly use artificially designed features to obtain change results, such as transform-based and image algebra-based RSCD methods to change detection on these images. For the image algebra-based method, image difference and image regression are usually used to generate difference images (DIs), in which a suitable threshold needs to be selected to determine the change area. The transform-based method uses transform techniques such as Principal Components Analysis (PCA) and Tasseled Cap transform to transform remote sensing images to other spaces, and enhance the change information of the dual temporal images. The above methods mainly obtain change clues based on artificially designed features. However, artificially designed features have low robustness and insufficient semantics, and are only suitable for low-resolution or medium-resolution RSCD tasks.

[0046] In recent years, deep convolutional neural networks (CNN) have become the mainstream solution for RSCD tasks due to their powerful feature learning capabilities. Compared with traditional methods, deep learning-based methods require less human intervention and can automatically learn features from annotated data. They mostly treat RSCD as a binary segmentation task, detecting changes by classifying extracted dual-temporal features into different categories. Currently, most deep learning-based RSCD backbone networks adopt encoder-decoder structures such as Fully Convolutional Network (FCN) and U-Net. Due to the scale variation and complex background of remote sensing images, various multi-scale feature fusion modules and attention mechanisms have been introduced into deep neural networks for remote sensing image CD. In the prior art, there are schemes that impose constraints on different feature maps recorded at each stage of the encoder to produce a more refined representation. The prior art proposes a dense attention refinement network based on a U-shaped encoder-decoder architecture that supports progressive adjustment of the predicted change map at each stage of the decoding process. DARNe uses dense skip connections and hybrid attention to aggregate multi-level feature representations, progressively refining the predicted change map from high to low levels. The emergence of ViT provides another way for the CD task. Specifically, ViT divides the image into a series of patches, treats all patches as a sequence, and captures the global dependencies between patches in the image using attention mechanisms. In the prior art, SwinTransformer is also used to construct the encoder and decoder, and a pure transformer network with a connected Ushaped structure is designed. ChangeFormer is a change detection model based on the Segformer architecture. In the basic module of ChangeFormer, spatial downsampling is performed on the query, key, and value to reduce the computational load. On the other hand, some studies use parallel CNN and Transformer structures. WNet (Windows Net) is proposed, which uses connected CNN and connected Transformer as the encoder, and fuses all features in the decoder. Similarly, ICIF-Net (Integrated Column and Instance Fusion Network) is also proposed, which builds a dual-branch structure of CNN and Transformer, respectively capturing multi-scale local and global features, and fusing them using cross-attention.

[0047] Restricted to the road change detection dataset, the existing deep learning-based road change detection method is mainly based on a general change detection neural network. A convolutional Siamese network is proposed to identify the change area using time series images, a contrast loss function is used to minimize the distance between two feature vectors in the unchanged area, and deliberately trained samples are used to overcome the noise unrelated to the road in the image. The general change detection network needs complex post-processing steps to find the road-related changes, and the post-processing steps are complex and lack robustness. In addition, the general change detection network mainly focuses on spectral changes and lacks geometric constraints of boundaries. Therefore, in order to improve the accuracy of road change detection, the application provides a road change detection method based on mixed strip convolution and related equipment.

[0048] The application will be described below in conjunction with Figures 1-8 The application provides a road change detection method based on mixed strip convolution and related equipment.

[0049] Figure 1 It is a flowchart of a road change detection method based on mixed strip convolution according to an exemplary embodiment. As Figure 1 shown, in an exemplary embodiment, the road change detection method based on mixed strip convolution includes steps 110 to 140, which are described in detail as follows.

[0050] Step 110, acquiring two images to be detected for the same position.

[0051] In the embodiment of the application, the two images to be detected are images taken at different times for the same position.

[0052] Step 120, extracting road linear features in each of the images to be detected by a feature enhancement module; wherein the feature enhancement module is obtained based on a multi-scale strip convolution unit.

[0053] In the embodiment of the application, the feature enhancement module is constructed by a multi-scale strip convolution unit, which can also be called a mixed strip convolution feature enhancement module (MSCFE). The feature enhancement module uses a multi-scale strip convolution unit to capture the long-distance dependence of road features, avoiding the blurring of the boundaries of traditional square convolution on narrow objects. The application uses a multi-scale strip convolution unit to better adapt to the distribution characteristics of the road, reduce the blurring of the road edge caused by irrelevant areas, and improve the completeness and continuity of road extraction.

[0054] Step 130, performing fusion processing on the road linear features in each of the images to be detected by a cross-fusion module to obtain fusion features.

[0055] In the embodiment of the present application, the cross fusion module performs fusion processing on two road linear features, and in the fusion processing process, the spatial information and the temporal information of each other are interactively propagated, the network is guided to pay attention to the change area related to the road, and the influence of the distribution difference in the two images to be detected is suppressed.

[0056] In step 140, road change detection is performed based on the fusion features.

[0057] In the embodiment of the present application, the road change detection with fine details is performed based on the fusion features.

[0058] In the embodiment of the present application, as shown in Figure 2 The feature enhancement module uses the MobileNetV2 network fused with the multi-scale strip convolution unit to extract the road linear features. Then, the cross fusion module (CrossFusion) is used to perform fusion processing on the road linear features, so as to realize the transmission of spatial information and temporal information between the two road linear features. Then, the change features are obtained by using the feature difference method, and the multi-level change features are fused from top to bottom through the decoder to predict the road change map with fine details.

[0059] The MobileNetV2 network used discards the AvgPool layer and the full connection layer therein to adapt to the change detection task, and the output of the feature enhancement module is H / 32×W / 32×320, H and W being the size of the input image. Figure 2 Element-wise Sub & Abs in the above formula represents element-wise subtraction and element-wise absolute value, Skip Connection represents skip connection, and Upsample represents up sampling.

[0060] In an exemplary embodiment of the present application, the multi-scale strip convolution unit includes a plurality of strip convolution units for different directions; and the feature enhancement module extracts the road linear features in each of the images to be detected, including the following steps, which are described in detail as follows.

[0061] The first features in the images to be detected are extracted through each of the strip convolution units.

[0062] Based on the first features corresponding to each of the images to be detected, the road linear features of each of the images to be detected are obtained.

[0063] In the road change detection in the embodiment of the application, the road shape has the characteristics of large span and narrowness, but the square convolution kernel will additionally calculate the non-road area when extracting features, and cannot well capture the linear features of the road. At the same time, the square convolution kernel is not sensitive to the road direction, and cannot effectively extract the oblique road features. Therefore, aiming at the inherent characteristics of the road, the application proposes a feature enhancement module based on a multi-scale strip convolution unit, which uses strip convolution units in different directions to adapt to roads with different direction distributions. The strip convolution unit is not only conducive to extracting linear features, but also can reduce the amount of calculation.

[0064] In the embodiment of the application, each strip convolution unit extracts first features in the to-be-detected image, mixes all the first features of each to-be-detected image, and then obtains road linear features of each to-be-detected image.

[0065] In an exemplary embodiment of the application, the multi-scale strip convolution unit includes strip convolution units respectively for the horizontal direction, the vertical direction, the left diagonal direction and the right diagonal direction.

[0066] In the embodiment of the application, as shown in Figure 3 , the strip convolution units based on the horizontal direction, the vertical direction, the left diagonal direction and the right diagonal direction construct a feature enhancement module, and the feature dimension of the to-be-detected image is , and D is the number of channels. First, strip average pooling (Strip Pool) is performed along the horizontal dimension and the vertical dimension to obtain features and , the global average pooling operation of the horizontal dimension and the vertical dimension is used to obtain the long-range context relationship in the spatial dimension, and then the local information is reserved through local convolution (1D Conv) with a convolution kernel size of Hx1 and 1xW. Then, the channel dimension is aggregated through global average pooling (Channel Avg) and global maximum pooling (Channel Max), and left diagonal convolution (Left Diagonal Conv) and right diagonal convolution (Right Diagonal Conv) are used to adapt to the road distribution in different directions. Finally, the output features of the four directions are mixed, that is, the first features. Then, the weight map of the feature mapping is obtained through 1x1 convolution (1x1 Conv) and sigmoid activation function, the input is weighted using the weight map to obtain the feature map after strip convolution enhancement . Figure 3 Element-wise Add in the above formula represents pixel-by-pixel addition, and Matrix Multiplication represents matrix multiplication.

[0067] In an example embodiment of the present application, the feature enhancement module further comprises a multi-scale bar convolution unit, and the road linear features in each of the to-be-detected images are extracted by the feature enhancement module, including the following steps, which are described in detail as follows.

[0068] The second features of each of the to-be-detected images are extracted by the multi-scale bar convolution unit, and the third features of each of the to-be-detected images are extracted by the multi-scale bar convolution unit.

[0069] The second features and the third features of each of the to-be-detected images are connected to obtain the road linear features of each of the to-be-detected images.

[0070] In the embodiment of the present application, in order to effectively learn different scale road features, a multi-scale bar convolution unit is also added in the feature enhancement module. The dilated convolution can make the model observe the context information of a larger area through its expanded receptive field, which is particularly important for understanding the overall layout of the road. Using a higher dilation rate in the convolution layer can help the model capture the long-distance semantic relationship of the road, even in the case of visual discontinuity of the road (such as being blocked by trees or crossing with other roads). Specifically, as shown in FIG. 2, the multi-scale bar convolution unit uses bar space convolution with dilation rates of 3, 5, and 7 to capture features of different scales, and then connects the third features captured by the bar space convolution and the second features captured by the multi-scale bar convolution unit, and uses 1x1 convolution to map them to a new channel space to obtain road linear features. In this embodiment, the second features are consistent with the first features described above. Figure 4 Figure 4 In FIG. 2, Concatenation represents connection, and Dilation Conv represents dilated convolution.

[0071] In an example embodiment of the present application, the cross-fusion module comprises a self-attention mechanism and a depth separable convolution; and the road linear features in each of the to-be-detected images are fused by the cross-fusion module to obtain fusion features, including the following steps, which are described in detail as follows.

[0072] The road linear features are spliced to obtain road linear spliced features.

[0073] The road linear spliced features are input into the self-attention mechanism and the depth separable convolution respectively for processing, and channel interaction and spatial interaction are performed in the processing process to obtain output features corresponding to the self-attention mechanism and the depth separable convolution respectively.

[0074] The output features are fused to obtain the fusion features.

[0075] ​In the embodiment of the present application, the road linear features have spatiotemporal dependence, and the self-attention mechanism and the time attention mechanism can be used to generate attention weight maps in the spatial dimension and the channel dimension to capture the spatiotemporal relationship between the road linear features. The self-attention mechanism dynamically calculates the weight in the spatial dimension, which can well capture the spatial relationship between pixels, but has weak modeling ability in the channel dimension. The depthwise separable convolution calculates the weight in the channel dimension, which can well capture the spatial relationship between features, but has weak modeling ability in the spatial dimension. In order to explore the spatial correlation and temporal correlation between road linear features, the embodiment proposes a cross fusion module composed of self-attention mechanism and depthwise separable convolution, and the module structure is as shown in Figure 5 .

[0076] As shown in Figure 2 and Figure 5 , two road linear features are concatenated (Concat) to obtain road linear concatenated features, and the road linear concatenated features are respectively sent to parallel self-attention mechanism (Self-Attention) and depthwise separable convolution (Depthwise Conv) to model the spatial dependence and temporal dependence between the two road linear features. Channel interaction (Channel Interaction) is used to enhance the channel modeling ability of the self-attention mechanism, and spatial interaction (Spatial Interaction) is used to provide spatial clues for the depthwise separable convolution. Then, based on the output features and of the self-attention mechanism and the depthwise separable convolution, a fusion feature is obtained.

[0077] In the channel interaction process, the features are aggregated using global average pooling, and then two convolution kernels with a size of (1, 1) are used for channel interaction. The channel attention map is obtained using the sigmoid activation function, and the channel attention map is applied to the value to obtain the output feature of the self-attention mechanism. In the spatial interaction process, the channel number is reduced to 1 by using two convolution kernels with a size of , and finally the spatial attention map is obtained by using the sigmoid activation function. The spatial attention map is applied to the depthwise separable convolution to obtain the output feature of the depthwise separable convolution.

[0078] In an exemplary embodiment of the present application, the fusion processing based on the output features to obtain the fusion feature includes the following steps, which are described in detail as follows.

[0079] The two output features are added pixel by pixel to obtain a feature

[0080] The mixed feature is processed by pixel difference to obtain the fusion feature.

[0081] In the embodiment of the present application, as shown in Figure 5 two output features are added pixel by pixel to obtain a feature , and the mixed feature is obtained by convolution, and and , finally and are subtracted pixel by pixel to obtain the fusion feature .

[0082] In an exemplary embodiment of the present application, the road change detection based on the fusion feature comprises the following steps, which are described in detail as follows.

[0083] The fusion feature is up-sampled and spliced, and the road change map is obtained based on the up-sampled and spliced fusion feature.

[0084] In the embodiment of the present application, as shown in Figure 2 , the different levels of change features are up-sampled (D5-D1) and spliced, the low-level features have fine-grained detailed information, but also contain a lot of noise, and the high-level features have more abstract semantic information, which is helpful for positioning the road and suppressing noise. The difference feature aggregation method from high to low is adopted, and the deep supervision is used to gradually obtain different levels of change features to predict the road change map with fine details.

[0085] In the embodiment of the present application, RSCD can be regarded as a pixel-level classification task in nature, and the network weight is usually optimized using binary cross-entropy loss. However, in the road change detection task, there is a class imbalance problem, that is, the proportion of change regions is much smaller than that of unchanged regions. Binary cross-entropy loss treats each pixel equally, so when the number of positive samples is small, only using binary cross-entropy loss will cause the network to be dominated by negative samples, resulting in deviation in training and failure to identify positive samples. Due to the complex background of high-resolution remote sensing images, many roads cannot be successfully identified, that is, the number of FNs (False Negatives) far exceeds the number of false positives (False Positives, FPs). In this case, in order to improve the recall rate, FNs need to be more punished than FPs. HEM loss (Hybrid Embedding Loss) controls the balance between FNs and FPs through hyperparameters α and β. At the same time, the distribution of VHR (Very High Resolution) remote sensing images is complex, and interference such as occlusion and shadow inevitably leads to difficulty in identifying some roads. In order to alleviate this problem, an index γ is used in HEM loss to pay more attention to difficult samples detected with a lower probability. Therefore, the present application combines binary cross-entropy loss and HEM loss , which can be described as:

[0086] ;

[0087] ;

[0088] wherein, represents dot product operation, represents L1 norm, P is the predicted road change map, Y is the corresponding true label, W and H are the width and height of the image to be detected respectively. The values of α, β and γ are 0.3, 0.7 and 0.75 respectively.

[0089] The total loss of road change detection can be expressed as: .

[0090] In an exemplary embodiment of the present application, experiments are carried out on the road change detection dataset WRCD (Wuhan Road Change Detection) containing only roads and the general change detection dataset SYSU (South China Normal University) containing various change scenes of roads.

[0091] The WRCD is a 17-level remote sensing image downloaded from Google Earth, and the data set only contains road changes. The image resolution of each phase is 10884*13655, and the image is cropped into a patch with a size of 512*512 with an overlap rate of 0.25. The first 48 columns of each phase are taken as test data, and the rest are taken as training data, so as to obtain 1995 pairs of training images and 980 pairs of test images.

[0092] The SYSU data set is composed of 20,000 pairs of spatial size 256*256, spatial resolution 0.5m, and double-phase remote sensing image blocks. The training, verification and test ratio is formally set to 6:2:2. The data set contains various types of complex change scenes, including road expansion, new urban buildings, vegetation changes, suburban expansion and pre-construction groundwork.

[0093] The application uses MobileNetV2 as the backbone, uses random cropping and random flipping for data enhancement of the input image, uses the Adam optimizer, the momentum is 0.9, 0.9, 0.99, the initial learning rate lr is 5e-4, and the poly strategy is used to decay the learning rate, and the learning rate is adjusted to as the training progresses, the total training rounds are set to 100, the power value is 0.9, and the batch size is set to 10. All codes are implemented based on the PyTorch framework, and experiments are carried out on a server configured with a GeForce RTX 4090, with a device memory of 24GB, an operating system version of Ubuntu22.04, and a cuda version of 2.1.2.

[0094] The Intersection over Union (IoU), F1-score (F1), Recall (Rec) and Precision (Pre) are used to evaluate the performance of the mixed bar convolution-based road change detection method provided by the application, wherein Pre reflects the proportion of correct prediction in the positive sample of the prediction result, and measures the precision of the model; Rec reflects the proportion of actual positive samples predicted as positive samples, and measures the recall of the model; F1 is the weighted average of the two; OA reflects the proportion of all correctly classified pixel points to the total pixels. The specific definitions of the above evaluation indexes are as follows:

[0095] ;

[0096] ;

[0097] ;

[0098] ;

[0099] where Tp, Fp, Tn and Fn represent the number of true positive, false positive, true negative and false negative respectively.

[0100] The road change detection method based on mixed bar convolution provided by the application is compared with five most advanced RSCD methods, including four CNN-based methods: TinyCD, USSFCNet, DMINet, A2Net, and a Transformer-based method: BiT. In order to verify the robustness of the proposed method, experiments are respectively carried out on the road change detection dataset WRCD and the general change detection dataset SYSU.

[0101] The change detection result visualization on the WRCD dataset is as shown in Figure 6 , and the quantitative result is as shown in Table 1. In order to make the visualization effect more intuitive, four kinds of white are used to represent the true positive road change detection result.

[0102] Table 1

[0103]

[0104] From the visualization Figure 6 , it can be seen that the application performs better in the aspects of road detection accuracy, road continuity and distinguishing pseudo changes. In terms of detection accuracy, compared with other methods, the road change detection accuracy of the application is higher, and the change target can be more accurately positioned. From the first and second rows of Figure 6 , BiT, TinyCD, USSFCNet and DMINet almost cannot identify road changes, compared with the application which can more accurately identify the change area. In terms of road continuity, the road of the remote sensing image is narrow, which leads to the road change being unable to be completely identified. Compared with other methods, the application has more advantages in road continuity and integrity. In the fifth and sixth rows of Figure 6 , the continuity of the road change area identified by many methods is poor. On the contrary, the application can well identify the road change edge. In terms of distinguishing pseudo changes, the road change detection of the remote sensing image is prone to pseudo changes due to the interference of factors such as season, background and surrounding buildings. For example, in the fourth and ninth rows of Figure 6 , USSFCNet and A2Net cannot distinguish the pseudo changes caused by buildings. Compared with this, the application can better identify pseudo changes.

[0105] From Table 1, it can be known that the best recall (0.7395), IoU (0.5742) and F1 score (0.7295) are achieved on the WRCD road dataset. Compared with BiT, the recall of the application increases by 13.37%, and compared with A2Net, the recall increases by 8.57%, indicating that our model can detect more complete road changes. Compared with BiT, our IoU increases by 9.24%, indicating that the road changes detected by the application are more in line with the true situation.

[0106] In order to further verify the effectiveness of the model in the general change detection task, experiments are carried out on the general change detection dataset SYSU, and the quantitative results are shown in Table 2 as follows,

[0107] Table 2

[0108]

[0109] After comparison, the application also has better effect on the SYSU dataset, and achieves the best recall (0.8484), IoU (0.7200) and F1 score (0.8372). Compared with BiT, the recall of the application increases by 8.16%, and the IoU increases by 6.16%. Compared with A2Net, the recall increases by 4.73%, and the IoU increases by 0.62%, indicating the effectiveness of the application in the general change detection task.

[0110] In order to verify the effectiveness of MSCFE and CrossFusion, a series of ablation experiments are carried out on the WRCD and SYSU datasets, and the quantitative results are shown in Table 3 as follows. The application using MSCFE and CrossFusion obtains the best IoU, F1 and recall on the WRCD and SYSU datasets, proving that the application can detect complete road changes and is also competitive in general change detection.

[0111] Table 3

[0112]

[0113] After removing MSCFE and CrossFusion, the F1 score and recall score on the two datasets decrease significantly, indicating the effectiveness of the application. After removing MSCFE, the recall on the WRCD dataset decreases from 73.95% to 70.87%, indicating that MSCFE is useful for road change detection. On the SYSU dataset, the recall decreases from 84.84% to 82.50%, indicating that MSCFE is useful for general change detection.

[0114] After deleting CrossFusion, the IoU and F1 scores of the model on the WRCD dataset decreased by 2.06% and 1.69% respectively, and the IoU and F1 scores on the SYSU dataset decreased by 0.73% and 0.50% respectively, which indicates that CrossFusion improves the change detection results of the model.

[0115] The road change detection device based on hybrid strip convolution provided by the present application is described below. The road change detection device based on hybrid strip convolution described below can be referred to each other corresponding to the road change detection method based on hybrid strip convolution described above. It should be noted that the device provided by the embodiment described below belongs to the same concept as the method provided by the above embodiment, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment, which will not be described here.

[0116] In an exemplary embodiment of the present application, please refer to Figure 7 , Figure 7 A road change detection device based on hybrid strip convolution is shown according to an exemplary embodiment, comprising the following modules.

[0117] The acquisition module 710 is configured to acquire two images to be detected for the same position;

[0118] The extraction module 720 is configured to extract road linear features in each of the images to be detected by a feature enhancement module; wherein the feature enhancement module is based on a multi-scale strip convolution unit;

[0119] The fusion processing module 730 is configured to perform fusion processing on the road linear features in each of the images to be detected by a cross-fusion module to obtain fusion features;

[0120] The detection module 740 is configured to perform road change detection based on the fusion features.

[0121] In an exemplary embodiment of the present application, the multi-scale strip convolution unit includes a plurality of strip convolution units for different directions; the extraction module 720 includes:

[0122] The first extraction submodule is configured to extract first features in the images to be detected by each of the strip convolution units respectively;

[0123] The feature submodule is configured to obtain the road linear features of each of the images to be detected based on the first features corresponding to each of the images to be detected.

[0124] In an example embodiment of the present application, the multi-scale bar convolution unit includes bar convolution units for horizontal direction, vertical direction, left diagonal direction and right diagonal direction respectively.

[0125] In an example embodiment of the present application, the feature enhancement module further includes a multi-scale hole convolution unit, and the extraction module 720 includes:

[0126] A second extraction submodule is configured to extract second features of each of the to-be-detected images through the multi-scale bar convolution unit, and extract third features of each of the to-be-detected images through the multi-scale hole convolution unit;

[0127] A connection submodule is configured to connect the second features and the third features of each of the to-be-detected images to obtain the road linear features of each of the to-be-detected images.

[0128] In an example embodiment of the present application, the cross-fusion module includes a self-attention mechanism and a depth separable convolution, and the fusion processing module 730 includes:

[0129] A splicing processing submodule is configured to perform splicing processing on the road linear features to obtain road linear splicing features;

[0130] An input submodule is configured to input the road linear splicing features into the self-attention mechanism and the depth separable convolution respectively for processing, and perform channel interaction and spatial interaction in the processing process to obtain output features corresponding to the self-attention mechanism and the depth separable convolution respectively;

[0131] A fusion feature submodule is configured to perform fusion processing based on the output features to obtain the fusion features.

[0132] In an example embodiment of the present application, the fusion feature submodule includes:

[0133] A summation processing unit is configured to perform summation processing on the output features based on pixels, and perform convolution mixing on the results of the summation processing to obtain mixed features;

[0134] A difference processing unit is configured to perform difference processing on the mixed features based on pixels to obtain the fusion features.

[0135] In an example embodiment of the present application, the detection module 740 includes:

[0136] A processing submodule is configured to perform up-sampling and splicing processing on the fusion features, and obtain a road change map based on the fusion features after the up-sampling and splicing processing.

[0137] Figure 8 An example of a schematic diagram of a physical structure of an electronic device is shown in Figure 8 As shown, the electronic device can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 can communicate with each other through the communications bus 840. The processor 810 can invoke a logic instruction in the memory 830 to execute a road change detection method based on hybrid bar convolution, which includes: obtaining two images to be detected for the same location;

[0138] extracting road linear features in each of the images to be detected through a feature enhancement module, wherein the feature enhancement module is based on a multi-scale bar convolution unit;

[0139] performing fusion processing on the road linear features in each of the images to be detected through a cross-fusion module to obtain fusion features;

[0140] performing road change detection based on the fusion features.

[0141] In addition, the logic instruction in the memory 830 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0142] On the other hand, the present application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the road change detection method based on hybrid bar convolution provided by each of the methods, which includes: obtaining two images to be detected for the same location;

[0143] extracting road linear features in each of the images to be detected through a feature enhancement module, wherein the feature enhancement module is based on a multi-scale bar convolution unit;

[0144] The cross fusion module is used for fusing the road linear features in each of the images to be detected to obtain fused features.

[0145] The fused features are used for road change detection.

[0146] In another aspect, the application further provides a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the road change detection method based on mixed bar convolution provided by each of the above methods, the method comprising: acquiring two images to be detected for the same location;

[0147] The feature enhancement module is used for extracting road linear features in each of the images to be detected, wherein the feature enhancement module is based on a multi-scale bar convolution unit.

[0148] The cross fusion module is used for fusing the road linear features in each of the images to be detected to obtain fused features.

[0149] The fused features are used for road change detection.

[0150] The above-described device embodiments are only illustrative, wherein the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0151] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course, it can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in terms of contribution to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0152] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting road changes based on hybrid bar convolution, characterized in that, The method comprises: acquiring two images to be detected for the same position; extracting road linear features in each of the images to be detected by a feature enhancement module, wherein the feature enhancement module is based on a multi-scale strip convolution unit; fusing the road linear features in each of the images to be detected by a cross-fusion module to obtain fused features; performing road change detection based on the fused features; the multi-scale strip convolution unit comprises a plurality of strip convolution units for different directions; the feature enhancement module extracts road linear features in each of the images to be detected, comprising: extracting first features in the images to be detected by each of the strip convolution units; obtaining the road linear features of each of the images to be detected based on the first features corresponding to each of the images to be detected; the multi-scale strip convolution unit comprises strip convolution units for horizontal direction, vertical direction, left diagonal direction and right diagonal direction; the feature enhancement module extracts first features in the images to be detected by each of the strip convolution units, comprising: based on the horizontal direction and vertical direction strip convolution units, performing strip average pooling on the images to be detected along the horizontal and vertical dimensions respectively to obtain the long-range context relationship in the spatial dimension, and performing local convolution to retain local information to obtain the first features in the horizontal and vertical directions; based on the left diagonal direction and right diagonal direction strip convolution units, performing global average pooling and global maximum pooling on the channel dimension of the images to be detected, and using left diagonal convolution and right diagonal convolution respectively for processing to obtain the first features in the left diagonal direction and right diagonal direction; the cross-fusion module comprises a self-attention mechanism and a depth separable convolution; the cross-fusion module fuses the road linear features in each of the images to be detected to obtain fused features, comprising: splicing each of the road linear features to obtain road linear splicing features; inputting the road linear splicing features into the self-attention mechanism and the depth separable convolution respectively for processing, and performing channel interaction and spatial interaction in the processing process to obtain output features corresponding to the self-attention mechanism and the depth separable convolution respectively; fusing each of the output features to obtain the fused features.

2. The hybrid-streak-convolution-based road change detection method of claim 1, wherein, The feature enhancement module further comprises a multi-scale hollow convolution unit; the feature enhancement module extracts road linear features in each of the images to be detected, comprising: extracting second features of each of the images to be detected by the multi-scale strip convolution unit, and extracting third features of each of the images to be detected by the multi-scale hollow convolution unit; connecting the second features and the third features of each of the images to be detected to obtain the road linear features of each of the images to be detected.

3. The hybrid-streak-convolution-based road change detection method of claim 1, wherein, The cross-fusion module fuses each of the output features to obtain the fused features, comprising: based on pixels, summing each of the output features, and performing convolution mixing on the summed results to obtain mixed features; Difference processing is performed on the mixed features based on pixels to obtain the fusion features.

4. The hybrid-streak-convolution-based road change detection method according to any one of claims 1 to 3, characterized in that, The road change detection based on the fusion features comprises: The fusion features are up-sampled and spliced, and a road change map is obtained based on the up-sampled and spliced fusion features.

5. A hybrid-streak-convolution-based road change detection apparatus characterized by comprising: Comprise: The acquisition module is configured to acquire two images to be detected for the same position; The extraction module is configured to extract road linear features in each of the images to be detected through a feature enhancement module, wherein the feature enhancement module is based on a multi-scale strip convolution unit; The fusion processing module is configured to perform fusion processing on the road linear features in each of the images to be detected through a cross-fusion module to obtain fusion features; The detection module is configured to perform road change detection based on the fusion features; The multi-scale strip convolution unit comprises a plurality of strip convolution units for different directions; the extraction module comprises: The first extraction submodule is configured to extract first features in the images to be detected through each of the strip convolution units; The feature submodule is configured to obtain the road linear features of each of the images to be detected based on the first features corresponding to each of the images to be detected; The multi-scale strip convolution unit comprises strip convolution units for horizontal, vertical, left diagonal and right diagonal directions; the first extraction submodule comprises: Based on the horizontal and vertical strip convolution units, the strip average pooling is performed on the images to be detected along the horizontal and vertical dimensions to obtain the long-range context relationship in the spatial dimension, and the local information is retained through local convolution to obtain the first features in the horizontal and vertical directions; Based on the left diagonal and right diagonal strip convolution units, the channel dimension of the images to be detected is aggregated through global average pooling and global maximum pooling, and left diagonal convolution and right diagonal convolution are used for processing respectively to obtain the first features in the left diagonal and right diagonal directions; The cross-fusion module comprises a self-attention mechanism and a depth separable convolution; the fusion processing module comprises: The splicing processing submodule is configured to splice each of the road linear features to obtain road linear splicing features; The input submodule is configured to input the road linear splicing features into the self-attention mechanism and the depth separable convolution respectively for processing, and perform channel interaction and spatial interaction in the processing process to obtain output features corresponding to the self-attention mechanism and the depth separable convolution respectively; The fusion feature submodule is configured to perform fusion processing based on each of the output features to obtain the fusion features.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the road change detection method based on mixed strip convolution according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the road change detection method based on mixed strip convolution according to any one of claims 1 to 4.

Images