Road change detection method and device and electronic equipment

Abstract

The invention provides a road change detection method and apparatus, and an electronic device. The method comprises the steps of obtaining a dual-time-phase multi-modal image pair; and inputting the multi-modal image pair into a road change detection model to obtain a road change detection result. According to the road change detection method provided by the invention, a dual-time-phase multi-modal image pair is input into a road change detection model, firstly, feature extraction is carried out on each modal image of different time phases through a plurality of encoding layers of an encoder, then feature fusion and difference feature extraction are carried out on the features extracted by each encoding layer, and a road change detection result is obtained. According to the method, dynamic aggregation of edge and semantic information of different scales is achieved, the feature expression ability of road changes in a complex environment is enhanced, then modal change features extracted by a coding layer are fused through a decoding layer of a decoder, features fused by the decoding layer are fused, self-adaptive fusion of multi-modal difference features is achieved, and the multi-modal difference feature fusion speed is improved. The complementarity of multi-modal data is fully utilized, and the accuracy of road change detection is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a road change detection method and device and electronic equipment. BACKGROUND

[0002] Road change detection based on remote sensing images is crucial for rapid post-disaster regional exploration and safety assessment, especially in emergency scenarios such as power equipment repair and road damage investigation. However, traditional manual field investigation methods are inefficient and costly, making it difficult to meet the timeliness requirements of change detection.

[0003] In recent years, the combination of deep learning and remote sensing technology has provided a new approach to road change detection. The current mainstream road change detection algorithm is mostly based on a general change detection network. Although this network can identify land cover changes, the road in remote sensing images usually presents a large span and narrow linear feature. The encoder in the general change detection network is difficult to extract complex road change features because it extracts features from single modal remote sensing data layer by layer. Moreover, the robustness of single modal remote sensing data (such as visible light images) is insufficient in complex scenarios such as light changes and occlusions, resulting in low accuracy of road change detection results. SUMMARY

[0004] The present application provides a road change detection method, device and electronic equipment to solve the problem of low accuracy of road change detection results in the prior art.

[0005] In a first aspect, the present application provides a road change detection method, comprising: obtaining a pair of multi-modal images of double time phases; the pair of multi-modal images includes a pair of visible light images and a pair of infrared images; inputting the pair of multi-modal images into a road change detection model to obtain a road change detection result output by the road change detection model; wherein the road change detection model includes an encoder and a decoder; the encoder includes a plurality of encoding layers in sequence, the encoding layers are used for feature extraction of each modal image of different time phases, the encoder is used for feature fusion and difference feature extraction of the features extracted by each encoding layer, and outputs each modal change feature corresponding to each encoding layer; the decoder includes a plurality of decoding layers in sequence, the decoding layers are used for feature fusion of each modal change feature corresponding to the encoding layer, and the decoder is used for feature fusion of the features fused by each decoding layer, and outputs a target modal fusion feature; the road change detection result is determined based on the target modal fusion feature.

[0006] In one embodiment, the plurality of encoding layers and the plurality of decoding layers have a one-to-one relationship; the road change detection model sequentially includes an encoder, a decoder and a threshold segmentation module; the inputting the multi-modal image pair into the road change detection model to obtain the road change detection result output by the road change detection model comprises: inputting each modal image pair into the encoder, extracting features of each modal image of different time phases through each encoding layer, and extracting features of each encoding layer extracted features for feature fusion and difference feature extraction, to obtain each modal change feature corresponding to each encoding layer output by the encoder; inputting each modal change feature corresponding to each encoding layer into the encoder, performing feature fusion on each modal change feature corresponding to the encoding layer of the encoding layer, and performing feature fusion on the fused features of each decoding layer, to obtain the target fusion feature output by the encoder; inputting the target fusion feature into the threshold segmentation module, normalizing the target fusion feature, and performing threshold segmentation on the normalized target fusion feature to obtain the road change detection result output by the threshold segmentation module.

[0007] In one embodiment, the encoder adopts a twin architecture with double-branch non-shared parameters, including a first branch and a second branch, the first branch and the second branch are respectively used for feature extraction of single modal images of different time phases; the first branch and the second branch both sequentially include a first encoding layer, a second encoding layer, a third encoding layer, a fourth encoding layer and a fifth encoding layer; The first encoding layer of the first branch or the second branch is used for feature extraction of the single modal image to obtain a first feature; the second encoding layer of the first branch or the second branch is used for feature extraction of the first feature to obtain a second feature; the third encoding layer of the first branch or the second branch is used for feature extraction of the second feature to obtain a third feature; the fourth encoding layer of the first branch or the second branch is used for feature extraction of the third feature to obtain a fourth feature; the fifth encoding layer of the first branch or the second branch is used for feature extraction of the fourth feature to obtain a fifth feature; The second encoding layer of the first branch or the second branch is also used for feature fusion of the second feature and the fifth feature to obtain a first level fusion feature; the third encoding layer of the first branch or the second branch is also used for feature fusion of the third feature and the fifth feature to obtain a second level fusion feature; the fourth encoding layer of the first branch or the second branch is also used for feature fusion of the fourth feature and the fifth feature to obtain a third level fusion feature.

[0008] In one embodiment, the encoder further comprises a first difference feature extraction module, a second difference feature extraction module, a third difference feature extraction module, and a fourth difference feature extraction module. The first difference feature extraction module is configured to perform pixel-by-pixel difference between the first level fusion feature corresponding to the first branch and the first level fusion feature corresponding to the second branch to obtain single-modality change features corresponding to the second encoding layer. The second difference feature extraction module is configured to perform pixel-by-pixel difference between the second level fusion feature corresponding to the first branch and the second level fusion feature corresponding to the second branch to obtain single-modality change features corresponding to the third encoding layer. The third difference feature extraction module is configured to perform pixel-by-pixel difference between the third level fusion feature corresponding to the first branch and the third level fusion feature corresponding to the second branch to obtain single-modality change features corresponding to the fourth encoding layer. The fourth difference feature extraction module is configured to perform pixel-by-pixel difference between the fifth feature corresponding to the first branch and the fifth feature corresponding to the second branch to obtain single-modality change features corresponding to the fifth encoding layer.

[0009] In one embodiment, the decoder comprises, in sequence, a fifth decoding layer, a fourth decoding layer, a third decoding layer, a second decoding layer, and a first decoding layer; the second encoding layer and the second decoding layer have a corresponding relationship; the third encoding layer and the third decoding layer have a corresponding relationship; the fourth encoding layer and the fourth decoding layer have a corresponding relationship; and the fifth encoding layer and the fifth decoding layer have a corresponding relationship. The second decoding layer is configured to perform feature fusion on the modality change features of the second encoding layer to obtain first modality fusion features; the third decoding layer is configured to perform feature fusion on the modality change features of the third encoding layer to obtain second modality fusion features; the fourth decoding layer is configured to perform feature fusion on the modality change features of the fourth encoding layer to obtain third modality fusion features; and the fifth decoding layer is configured to perform feature fusion on the modality change features of the fifth encoding layer to obtain fourth modality fusion features. The fourth decoding layer is further configured to perform feature fusion on the fourth modality fusion features and the third modality fusion features to obtain first intermediate modality fusion features; the third decoding layer is further configured to perform feature fusion on the first intermediate modality fusion features and the second modality fusion features to obtain second intermediate modality fusion features; and the second decoding layer is further configured to perform feature fusion on the second intermediate modality fusion features and the first modality fusion features to obtain target modality fusion features.

[0010] In an embodiment, the modal change features include visible light change features and infrared change features; each decoding layer performs feature fusion on the modal change features of the corresponding encoding layer to obtain modal fusion features corresponding to the decoding layer, including: performing global average pooling and global maximum pooling on the visible light change features and the infrared change features of the encoding layer respectively to obtain a first global average pooling vector and a first global maximum pooling vector corresponding to the visible light change features, and a second global average pooling vector and a second global maximum pooling vector corresponding to the infrared change features; inputting the first global average pooling vector, the first global maximum pooling vector, the second global average pooling vector and the second global maximum pooling vector into a fully connected layer to learn inter-modal channel correlation through the fully connected layer to obtain channel weights; weighting the visible light change features and the channel weights to obtain a first tensor, and weighting the infrared change features and the channel weights to obtain a second tensor; performing fusion based on the first tensor and the second tensor to obtain the modal fusion features corresponding to the decoding layer.

[0011] In an embodiment, the performing fusion based on the first tensor and the second tensor to obtain the modal fusion features corresponding to the decoding layer includes: performing global average pooling and global maximum pooling on the first tensor and the second tensor respectively to obtain a third global average pooling vector and a third global maximum pooling vector corresponding to the first tensor, and a fourth global average pooling vector and a fourth global maximum pooling vector corresponding to the second tensor; inputting the third global average pooling vector, the third global maximum pooling vector, the fourth global average pooling vector and the fourth global maximum pooling vector into a convolution layer to perform cross-modal spatial correlation modeling through the convolution layer to obtain spatial weights; weighting the first tensor and the spatial weights to obtain a first matrix, and weighting the second tensor and the spatial weights to obtain a second matrix; performing pixel-by-pixel summation on the first matrix and the second matrix to obtain the modal fusion features corresponding to the decoding layer.

[0012] In an embodiment, the loss value caused by the road change detection model during training is determined in the following manner: calculating a binary cross-entropy loss value and a hard sample mining loss value output by each decoding layer; The mean value of each binary cross-entropy loss value is calculated to obtain a first mean value, and the mean value of each hard sample mining loss value is calculated to obtain a second mean value; The first mean value and the second mean value are weighted and summed to obtain a loss value caused by the road change detection model during training.

[0013] In a second aspect, the present application further provides a road change detection device, comprising: An acquisition module is configured to acquire a dual-time-phase multi-modal image pair; the multi-modal image pair comprises a visible light image pair and an infrared image pair; A road change detection module is configured to input the multi-modal image pair into a road change detection model to obtain a road change detection result output by the road change detection model; The road change detection model comprises an encoder and a decoder; the encoder comprises a plurality of encoding layers, and the encoding layers are configured to extract features of each modality image of different time phases; the encoder is configured to perform feature fusion and difference feature extraction on the features extracted by each encoding layer, and output each modality change feature corresponding to each encoding layer; the decoder comprises a plurality of decoding layers, and the decoding layers are configured to perform feature fusion on the modality change features corresponding to the encoding layers; the decoder is configured to perform feature fusion on the features fused by each decoding layer, and output a target modality fusion feature; and the road change detection result is determined based on the target modality fusion feature.

[0014] In a third aspect, the present application provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the program, the steps of the road change detection method according to any one of the above aspects are implemented.

[0015] The road change detection method, device and electronic device provided by the present application input a dual-time-phase multi-modal image pair into a road change detection model for road change detection. Firstly, a plurality of encoding layers of an encoder are used to extract features of each modality image of different time phases, and then feature fusion and difference feature extraction are performed on the features extracted by each encoding layer. Through a cross-level feature fusion strategy, edge and semantic information of different scales are dynamically aggregated, and the sensitivity of the network to road form changes in complex environments and the multi-scale feature expression capability are enhanced. Then, decoding layers of a decoder are used to fuse the modality change features extracted by the encoding layers, and then the features fused by each decoding layer are fused to realize adaptive fusion of multi-modal difference features. The complementarity of multi-modal data is fully utilized, and then the road change detection result is determined according to the multi-modal fusion feature, which greatly improves the accuracy of road change detection. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to make the technical solutions in the present application or prior art clearer, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0017] Figure 1 FIG. 1 is a flow diagram of a road change detection method provided by the present application.

[0018] Figure 2 FIG. 2 is a schematic diagram of the overall architecture of a road change detection model provided by the present application.

[0019] Figure 3 FIG. 3 is a schematic diagram of a multi-modal feature fusion process provided by the present application.

[0020] Figure 4 FIG. 4 is a schematic diagram of a WRCD-GM dataset provided by the present application.

[0021] Figure 5 FIG. 5 is a schematic diagram of change detection result visualization based on the WRCD-GM dataset provided by the present application.

[0022] Figure 6 FIG. 6 is a structural schematic diagram of a road change detection device provided by the present application.

[0023] Figure 7 FIG. 7 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0024] In order to make the technical solutions in the present application or prior art clearer, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0025] The terms "first", "second", and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein.

[0026] The embodiments of the present application will be described below. Figures 1-7 The road change detection method, device and electronic device provided by the present application are described.

[0027] It should be noted that the road change detection method provided by the embodiment of the present application is realized based on a road change detection device. The road change detection method aims at the problems of difficulty in extracting features of narrow roads and interference of complex backgrounds in road change detection of remote sensing images, and proposes a multi-modal road change detection framework of decoupling of difference features and adaptive fusion of modalities, and focuses on solving the challenges of difficulty in extracting linear features of roads and insufficient utilization of complementarity of multi-modal data. In view of these challenge problems, three improvement directions are proposed: (1) a double-branch non-shared parameter network is adopted to decouple the difference road features of visible light and infrared modalities and suppress the interference between modalities; (2) a multi-scale pyramid-multilayer deep supervision joint optimization network is designed to enhance the capture ability of linear geometric features of roads; and (3) an adaptive fusion strategy based on cross-modal interactive attention (CMIA) is proposed to realize the complementarity enhancement and redundancy suppression of multi-modal data through dynamic feature calibration in the spatial domain channel dimension.

[0028] The embodiment of the present application takes the road change detection device as the execution main body, and describes the road change detection method.

[0029] In combination Figure 1 , Figure 1 is a flowchart of the road change detection method provided by the present application.

[0030] As Figure 1 shown, the road change detection method comprises the following steps: Step 101, acquiring a pair of multi-modal images of double time phases; the pair of multi-modal images comprises a pair of visible light images and a pair of infrared images; Step 102, inputting the pair of multi-modal images into a road change detection model to obtain a road change detection result output by the road change detection model.

[0031] Specifically, first, a target area containing a road to be detected is circled, and then a pair of multi-modal images of double time phases of the target area is acquired, which means multi-modal remote sensing images of the target area acquired at different times, and can reveal the changes of the ground surface, providing important information for the fields of environmental monitoring, urban planning, disaster assessment and the like.

[0032] The multi-modal images of the present example adopt visible light images and infrared images, of course, not limited to these two modal remote sensing images. The visible light image contains rich texture detail information, but is easily affected by environmental factors such as light and produces false changes. The infrared image has significant contrast information, such as the gray difference caused by the temperature difference between the road surface and the non-road surface, but lacks corresponding texture information. The fusion of these two modal data can integrate the complementarity of visible light modal texture detail information and infrared modal contrast information, contain more information content, reduce the influence of environmental factors on road change detection, and also reduce the misjudgment that may be brought by single modal data, thereby improving the reliability of the road change detection result.

[0033] The dual-time-phase multi-modal image pair is input to the road change detection model, that is, the dual-time-phase visible light image pair and the dual-time-phase infrared image pair are input to the road change detection model, to obtain the road change detection result output by the road change detection model.

[0034] Considering that remote sensing image road change detection usually faces narrow roads, is easily affected by environmental factors such as light, and is disturbed by complex backgrounds such as trees and buildings, a multi-modal road change detection framework of difference feature decoupling-modal adaptive fusion is proposed, as shown in Figure 2 Figure 2 is the overall architecture diagram of the road change detection model provided by the present application.

[0035] Among them, and respectively represent the visible light image and the infrared image of T1 time phase, and respectively represent the visible light image and the infrared image of T2 time phase, , , and are all input to the road change detection model for prediction. First, the encoding layer in the encoder extracts features from each modal image of different time phases, and then the encoder extracts features from each encoding layer and extracts difference features, and outputs each modal change feature corresponding to each encoding layer. Further, each modal change feature corresponding to each encoding layer output by the encoder is input to the decoder as input data. The decoding layer in the decoder has a corresponding relationship with the encoding layer, and the decoding layer performs feature fusion on each modal change feature of the corresponding encoding layer, that is, the visible light change feature and the infrared change feature are fused, and then the decoder performs feature fusion on the fused features of each decoding layer, and outputs the target modal fusion feature. Further, the target modal fusion feature is predicted to output the road change detection result.

[0036] ​The road change detection method provided by the present invention inputs a dual-phase multimodal image pair into a road change detection model to perform road change detection. First, feature extraction is performed on each modal image of different phases through multiple coding layers of an encoder. Then, feature fusion and difference feature extraction are performed on the features extracted by each coding layer. Through a cross-level feature fusion strategy, edge and semantic information of different scales are dynamically aggregated to enhance the network's sensitivity to road morphological changes in complex environments and its ability to express multi-scale features. Then, the decoding layer of a decoder fuses the modal change features extracted by the coding layer, and then the fused features of each decoding layer are fused to achieve adaptive fusion of multimodal difference features, fully utilizing the complementarity of multimodal data, and then determining the road change detection result based on the multimodal fusion features, thereby greatly improving the accuracy of road change detection.

[0037] In some embodiments, the multimodal image pair is input into a road change detection model to obtain a road change detection result output by the road change detection model. The specific process is as follows.

[0038] Specifically, the road change detection model includes an encoder, a decoder and a threshold segmentation module in sequence. The encoder is mainly used to extract difference features, the decoder is mainly used to fuse the difference features of different modalities, and the threshold segmentation module is mainly used to perform threshold segmentation based on the features after modal fusion to obtain a binary image, and then generate the road change detection results through the binarized image.

[0039] Dual-phase multimodal images 、 、 and Input to the encoder, extract features of each modal image at different phases through each coding layer, and perform feature fusion and difference feature extraction on the features extracted by each coding layer to obtain the modal change features corresponding to each coding layer output by the encoder, including visible light change features and infrared variation characteristics , Figure 2 in That is, the change characteristics, including the visible light change characteristics and infrared variation characteristics .

[0040] Furthermore, the modal change features corresponding to each coding layer are input to the encoder, that is, the visible light change features corresponding to each coding layer are and infrared variation characteristics The input is sent to the encoder, and the corresponding modal change features are fused through the encoding layer, and the features fused by each decoding layer are fused to obtain the target fusion features output by the encoder.

[0041] Further, the target fusion feature is input to a threshold segmentation module, the target fusion feature is first normalized, and then the normalized target fusion feature is threshold segmented to obtain a binary image, pixels in the target fusion feature greater than a preset threshold are determined as a change region, i.e., the corresponding position on the binary image is represented as 1, and pixels in the target fusion feature less than the preset threshold are determined as an unchanged region, i.e., the corresponding position on the binary image is represented as 0, and a road change detection result is generated according to the binary image, i.e. Figure 2 The change region is represented by white color, and the unchanged region is represented by black color.

[0042] The embodiment of the present application extracts features, fuses features, and extracts difference features through the encoder of the road change detection model, dynamically aggregates edges and semantic information of different scales, enhances the sensitivity and multi-scale feature expression capability of the network to road form changes in complex environments, fuses multi-modal features through the decoder, realizes adaptive fusion of multi-modal difference features, fully utilizes the complementarity of multi-modal data, and then determines a road change detection result according to multi-modal fusion features, thereby greatly improving the accuracy of road change detection.

[0043] In some embodiments, the specific process of feature extraction and feature fusion performed by the encoder is as follows.

[0044] Specifically, as shown in Figure 2 The encoder adopts a twin architecture with double-branch non-shared parameters, which divides the first branch and the second branch, and the first branch and the second branch can extract features based on image data at different time points. Single-modal images at different time points contain unique information of the road at different time nodes, which is crucial for detecting whether the road has changed, such as road construction, damage, expansion, etc. The first branch is used for feature extraction of single-modal images at a certain time, and the second branch is used for feature extraction of single-modal images at another time. The detailed features in the images are converted into feature vectors that can be understood and analyzed by computers. Moreover, non-shared parameters mean that the two branches can independently learn and adjust parameters according to the characteristics of images at different time points, avoiding the problems of information confusion and inaccurate feature extraction caused by parameter sharing. In this way, each branch can optimize the feature extraction of the time phase image it processes, thereby providing more rich and accurate feature information for subsequent road change detection, greatly improving the accuracy and reliability of road change detection.

[0045] Since the first branch and the second branch are twin structures, and the first branch and the second branch are both feature extraction for a single modality image of a time phase, taking the first branch as an example for specific description, the feature extraction process of the second branch is consistent with the feature extraction process of the first branch, and will not be described again.

[0046] The first branch sequentially includes a first encoding layer, a second encoding layer, a third encoding layer, a fourth encoding layer and a fifth encoding layer, that is, E1, E2, E3, E4 and E5 in the first branch. Figure 2 In order to reduce the amount of calculation, reduce the complexity of the model and extract more abstract features, the size of the encoding layer adopts a pyramid structure design. The shallow encoding layer may focus on the local details of the image, and the extracted shallow features retain the road edge details. The high-level encoding layer can learn more global and abstract features, and the extracted deep features contain high-level semantic information.

[0047] Each encoding layer includes a feature extraction module. When extracting features from a single modality image set of a time phase, the features are extracted layer by layer through the feature extraction modules of the encoding layers. Specifically, the first encoding layer extracts features from the single modality image to obtain first features ; the second encoding layer extracts features from the first features to obtain second features ; the third encoding layer extracts features from the second features to obtain third features ; the fourth encoding layer extracts features from the third features to obtain fourth features ; and the fifth encoding layer extracts features from the fourth features to obtain fifth features Since the size of the encoding layer decreases layer by layer, the process of layer-by-layer feature extraction also achieves feature compression, and channel dimension reduction is used to reduce feature redundancy.

[0048] Since the scale of the road in the remote sensing image varies, a single feature scale cannot meet the high-precision recognition performance requirement. Therefore, a multi-scale feature adaptive fusion strategy (Multi-scale Feature Extraction Module, MFEM) is adopted. In some encoding layers, a multi-scale feature fusion module is set. Based on the design concept of "deep semantic guiding shallow details", through cross-level feature interaction, the collaborative optimization of fine-grained information and global semantics is realized. Taking feature extraction of a visible light image as an example, the multi-level features output by the backbone network correspond to feature representations under different receptive fields, respectively. The shallow features retain road edge and texture details, and the deep features contain high-level semantic information. Through fusion of deep features and shallow features, "deep semantic guiding shallow details" can be realized.

[0049] Before fusion, the feature map size can be aligned by bilinear interpolation to form a unified dimensional feature space, and realize unified feature resolution. Specifically, the second feature extracted by the second encoding layer and the fifth feature extracted by the last encoding layer are fused to obtain the first level fusion feature ; the third feature extracted by the third encoding layer and the fifth feature extracted by the last encoding layer are fused to obtain the second level fusion feature ; the fourth feature extracted by the fourth encoding layer and the fifth feature extracted by the last encoding layer are fused to obtain the third level fusion feature Since the feature extracted by the first encoding layer is shallow, the feature extracted by the first encoding layer does not need to be adaptively fused, nor does it need to be fused into the decoder.

[0050] The embodiment of the application fully utilizes the advantages of the twin architecture in processing different time images independently, avoids parameter interference, and effectively integrates feature information at different abstraction levels through cross-layer feature fusion, which not only retains the fine-grained information of shallow features, but also integrates high-level semantic information of deep features. Through the cross-level feature fusion strategy, the edge and semantic information of different scales are dynamically aggregated, the sensitivity of the network to the change of road form in complex environment and the multi-scale feature expression ability are enhanced, the recognition accuracy of the road in remote sensing image is significantly improved, and the robustness of the model in small size and complex background is enhanced.

[0051] Further, the specific process of the encoder for extracting difference features is as follows.

[0052] Specifically, the encoder further includes a first difference feature extraction module, a second difference feature extraction module, a third difference feature extraction module, and a fourth difference feature extraction module, which correspond to extracting single modal change features of different time phases of the second encoding layer, the third encoding layer, the fourth encoding layer, and the fifth encoding layer, respectively.

[0053] The first difference feature extraction module performs pixel-by-pixel subtraction and absolute value operation on the first level fusion feature corresponding to the first branch of the first level and the first level fusion feature corresponding to the second branch to obtain the single modal change feature corresponding to the second encoding layer , including visible light change feature and infrared change feature .

[0054] The second difference feature extraction module subtracts the second-level fusion feature corresponding to the first branch and the second-level fusion feature corresponding to the second branch pixel by pixel and takes the absolute value to obtain the single-modal change feature corresponding to the third coding layer. , including visible light variation characteristics and infrared variation characteristics .

[0055] The third difference feature extraction module subtracts the third-level fusion feature corresponding to the first branch and the third-level fusion feature corresponding to the second branch pixel by pixel and takes the absolute value to obtain the single-modal change feature corresponding to the fourth coding layer. , including visible light variation characteristics and infrared variation characteristics .

[0056] The fourth difference feature extraction module subtracts the fifth feature corresponding to the first branch from the fifth feature corresponding to the second branch pixel by pixel and takes the absolute value to obtain the single modal change feature corresponding to the fifth coding layer. , including visible light variation characteristics and infrared variation characteristics .

[0057] The embodiment of the present invention uses the difference feature extraction module of the encoder to perform pixel-by-pixel difference operations on the features extracted by the first branch and the second branch, accurately captures the feature differences of single-modal images in different phases on each coding layer, and generates single-modal change features corresponding to each coding layer. It can effectively highlight the change information of images in different phases, enable the system to accurately identify the changes of roads at different times, and improve the efficiency and reliability of road change detection.

[0058] In some embodiments, the specific process of the decoder performing inter-modal feature fusion is described as follows.

[0059] Specifically, the decoder includes a fifth decoding layer, a fourth decoding layer, a third decoding layer, a second decoding layer and a first decoding layer in sequence, that is, Figure 2 D5, D4, D3, and D2 in the above formula do not use D1 to achieve modal feature fusion. Figure 2D1 is not shown. Each encoding layer of the encoder has a one-to-one relationship with each decoding layer of the decoder. The sizes of the corresponding encoding layers and decoding layers are consistent. Specifically, the fifth decoding layer D5 corresponds to the fifth encoding layer E5, the fourth decoding layer D4 corresponds to the fourth encoding layer E4, the third decoding layer D3 corresponds to the third encoding layer E3, and the second decoding layer D2 corresponds to the second encoding layer E2. In fact, the first decoding layer D1 also corresponds to the first encoding layer E1. However, the features extracted by the first encoding layer do not need to be fused by modal features in the decoder, so the first encoding layer E1 does not need to output modal fusion features.

[0060] In order to explore the complementary characteristics of infrared modality and visible light modality, a multimodal change feature fusion module (CMIA module) based on cross-modal interactive attention is adopted to realize the dual-modal difference feature ( and ) adaptive fusion. Specifically, the CMIA module of the second decoding layer performs feature fusion on the modal change features of the second coding layer to obtain a first modal fusion feature. The CMIA module of the third decoding layer performs feature fusion on the modal change features of the third coding layer to obtain a second modal fusion feature. The CMIA module of the fourth decoding layer performs feature fusion on the modal change features of the fourth coding layer to obtain a third modal fusion feature. The CMIA module of the fifth decoding layer performs feature fusion on the modal change features of the fifth coding layer to obtain a fourth modal fusion feature.

[0061] Furthermore, in order to fully utilize feature information of different scales and levels and improve feature expression ability and richness, a multi-scale feature fusion module is also set up in certain decoding layers. Specifically, the fourth modality fusion feature is first upsampled to unify its dimension with the third modality fusion feature of the fourth decoding layer. The fourth decoding layer then fuses the fourth modality fusion feature with the third modality fusion feature to obtain the first intermediate modality fusion feature; the first intermediate modality fusion feature is first upsampled to unify its dimension with the second modality fusion feature of the third decoding layer. The third decoding layer then fuses the first intermediate modality fusion feature with the second modality fusion feature to obtain the second intermediate modality fusion feature; the second intermediate modality fusion feature is first upsampled to unify its dimension with the first modality fusion feature of the second decoding layer.

[0062] The embodiment of the present application adopts the correspondence between the decoder and the encoder, fuses the modal change features of each encoding layer, obtains modal fusion features of different levels, and then further fuses the modal fusion features of different levels across layers. Through multi-level and multi-stage feature fusion, the modal change features extracted by different encoding layers are fully integrated, the information of different scales and abstract levels is effectively converged, the expression ability and richness of the features are enhanced, the recognition accuracy of the road in the remote sensing image is significantly improved, and the robustness of the model in small size and complex background is enhanced.

[0063] In some embodiments, the specific process of each decoding layer for feature fusion of the modal change features of its corresponding encoding layer is as follows.

[0064] Specifically, in order to fully utilize the feature information of each modality, a spatio-temporal adaptive multi-modal change feature fusion module is adopted, a channel-space collaborative optimization of cross-modal features is realized through a double-flow attention mechanism, an adaptive allocation of spatial weight and channel weight strategy is designed, the complementary enhancement of visible light modal texture detail information and infrared modal contrast information is realized in the channel dimension, and the accurate distinction of road area and background area is realized in the spatial dimension. Combined with the above Figure 3 , Figure 3 is a schematic diagram of the multi-modal feature fusion process provided by the present application.

[0065] The visible light change features and the infrared change features of the encoding layer are respectively subjected to global average pooling (GAP) and global maximum pooling (GMP) to generate channel-level statistical description vectors, thereby obtaining a first global average pooling vector and a first global maximum pooling vector corresponding to the visible light change features, and a second global average pooling vector and a second global maximum pooling vector corresponding to the infrared change features.

[0066] The formula is as follows: ; ; Among them, represents the first global average pooling vector, represents the first global maximum pooling vector, represents the second global average pooling vector, represents the second global maximum pooling vector.

[0067] Then, the channel correlation between the modes is learned through a fully connected layer (FC), and a double-modal competition mechanism is used to calculate the channel weight: wherein, denotes element-wise addition, denotes channel-wise normalization of weights, denotes channel weights. Wherein, Figure 3 The FC layer processed vector is also convoluted, and more features can be extracted therefrom.

[0068] Then, the visible light change feature and the channel weight are weighted to obtain a first tensor, and the infrared change feature and the channel weight are weighted to obtain a second tensor, as shown in the following formula: ; ; wherein, denotes the first tensor, denotes the second tensor.

[0069] This mechanism enables dynamic weighting of visible light texture details (such as road cracks) and infrared thermal radiation contrast (such as asphalt temperature difference abnormal areas).

[0070] Further, global average pooling and global maximum pooling are performed on the first tensor and the second tensor to obtain a third global average pooling vector and a third global maximum pooling vector corresponding to the first tensor, and a fourth global average pooling vector and a fourth global maximum pooling vector corresponding to the second tensor.

[0071] The formula is as follows: ; ; wherein, denotes the third global average pooling vector, denotes the third global maximum pooling vector, denotes the fourth global average pooling vector, denotes the fourth global maximum pooling vector.

[0072] Further, the channel weighted features, i.e., the first tensor and the second tensor are spliced along the channel axis, and cross-modal spatial correlation modeling is performed through convolution operation: ; wherein, denotes convolution processing, denotes nonlinear change, denotes spatial weight.

[0073] Spatial weight The shadow pseudo change and other interference are suppressed by a soft threshold mechanism, and the road structure continuity is enhanced. The spatial weight and the first tensor are weighted to obtain a first matrix, and the spatial weight and the second tensor are weighted to obtain a second matrix, and then the first matrix and the second matrix are summed pixel by pixel to obtain the modal fusion feature corresponding to the decoding layer.

[0074] The final fused modal fusion feature C can be represented as: .

[0075] The embodiment of the application fully excavates the correlation information of different modal features in the channel and space, effectively integrates the advantages of the two modalities of visible light and infrared light through channel dimension complementary enhancement and spatial dimension regional focusing, enhances the richness and accuracy of feature expression, provides more representative and discriminant feature representation for subsequent road change detection based on fused features, and thus improves the performance and reliability of road change detection.

[0076] In some embodiments, the training process of the road change monitoring model is as follows.

[0077] Specifically, a pair of multi-modal images of a target region at different time phases is collected, and a road change label is generated according to the road change between the images at different time phases.

[0078] The pair of multi-modal images and the corresponding road change label are supervised and trained. The designed multi-level pyramid feature extraction realizes the collaborative representation from local texture details to global semantic information through cross-level feature fusion. However, in the feature propagation process, the detail information of the shallow feature will gradually attenuate, resulting in the model relying too much on the deep semantic information and ignoring the pixel-level change feature. In view of the problem of detail loss caused by the attenuation of shallow feature information in pyramid feature fusion, a depth supervision strategy is introduced in the network design. This level-adaptive supervision strategy not only alleviates the dependence of single-scale features, but also significantly improves the network's ability to learn different levels of features, constructs a multi-scale feature consistency constraint mechanism, and realizes the collaborative optimization of multi-scale road change features to improve the stability of the model under the interference of light changes, noise and the like.

[0079] That is, in the training process, the modal fusion feature output by each decoding layer (the second decoding layer to the fifth decoding layer) will generate a corresponding intermediate change detection map (the first intermediate change detection map to the fourth intermediate change detection map). ) and real change map (i.e., road change label) are combined to directly constrain the intermediate feature learning process through back propagation. This design can alleviate the gradient degradation problem of deep network, and at the same time, strengthen the network's ability to represent road details (such as cracks, local collapse), thereby improving the edge continuity and semantic consistency of the change detection results.

[0080] From the mathematical expression, the supervision mechanism can be expressed as the weighted sum of the binary cross-entropy (BCE) loss of each level and the hard example mining (HEM) loss: ; ; ; wherein, represents the binary cross-entropy loss value, is the corresponding weight thereof; represents the hard example mining loss value, is the corresponding weight thereof; N represents the number of decoding layers; represents the binary cross-entropy loss value output by the i-th decoding layer, represents the hard example mining loss value output by the i-th decoding layer.

[0081] The working principle of the BCE loss is based on the concept of cross-entropy, which is an index for measuring the difference between two probability distributions. The BCE loss has the advantages of good performance in handling unbalanced data and simple calculation. The HEM loss can improve the recall rate by focusing on false negative samples, and can reduce the oscillation in the training process.

[0082] The embodiment of the present application combines the measurement of the difference between the model prediction probability distribution and the real label by the binary cross-entropy loss, and the attention to false negative samples by the hard sample mining loss, effectively balances the learning of the model to each type of sample, enables the model to learn data features more comprehensively and accurately in the training process, improves the generalization ability and processing ability of the model to complex situations, and thus improves the accuracy and reliability of road change detection.

[0083] For the above road change detection method, the existing public data set is collected for testing, and the specific description is as follows.

[0084] Due to the lack of multimodal road change detection samples in existing public datasets, based on the visible light road change detection dataset WRCD, infrared modal data is constructed through the infrared image conversion network DR-AVIT to synthesize the multimodal dataset WRCD-GM (Generated Multimodal WRCD). The WRCD-GM dataset contains dual-phase visible light-infrared image pairs and their road change labels, with a spatial resolution of 10884×13655. To adapt to the training of deep learning models, a sliding window with an overlap rate of 0.25 is used to segment the images into 512×512 pixel samples, and 1995 pairs of training sets and 980 pairs of test sets are divided according to the principle of spatial continuity. Figure 4 As shown, Figure 4 : is a schematic diagram of the WRCD-GM dataset provided by the present invention, wherein the samples include (a) a visible light image at the T1 phase, (a1) to (a3) ​​are images for different regions; (b) an infrared image at the T1 phase, (b1) to (b3) are images for different regions; (c) a visible light image at the T2 phase, (c1) to (c3) are images for different regions; (d) an infrared image at the T2 phase, (d1) to (d3) are images for different regions; and corresponding road change labels, road change label 1 is the true label of (a1), (b1) and (c1), road change label 2 is the true label of (a2), (b2) and (c2), and road change label 3 is the true label of (a3), (b3) and (c3).

[0085] To verify the effectiveness of the multi-scale feature extraction module (MFEM) and the multimodal feature fusion module (CMIA), a series of ablation experiments were conducted on the WRCD-GM dataset, including removing the MFEM and using channel splicing to achieve multimodal fusion; retaining the MFEM and introducing CMIA to replace splicing and fusion; removing the CMIA and retaining the MFEM and splicing fusion; and integrating the MFEM and CMIA at the same time.

[0086] As shown in Table 1, the introduction of CMIA (Experiment B) improves the recall, precision, and F1 of change detection by 0.38%, 1.54%, and 0.94%, respectively. This demonstrates that the spatiotemporal attention mechanism effectively suppresses visible light artifacts (such as shadow interference) and infrared thermal radiation noise by dynamically calibrating inter-modal feature contributions. In contrast, the traditional cascade fusion strategy ignores modal complementarity, resulting in a high rate of missed detections. The embedding of MFEM (Experiment C) improves precision from 71.26% to 77.16%, attributed to its cross-level feature extraction. Deep semantic features constrain the shallow feature extraction process, significantly reducing the false detection rate caused by complex backgrounds such as trees and buildings. The simultaneous integration of MFEM and CMIA (Experiment D) exhibits a significant synergistic optimization effect, achieving the best performance (IoU: 56.87%, F1: 72.50%), significantly improving over the single-module solutions (Experiments B / C), demonstrating the synergistic effect of cross-modal feature interaction and cross-level feature extraction.

[0087] Table 1

[0088] To explore the impact of multimodal data on road change detection, the performance of visible light, infrared, and multimodal fusion was tested on the WRCD-GM dataset. The results are shown in Table 2. As can be seen, visible light alone, due to its distinct geometric representation of the road, achieves the highest quantitative metrics, outperforming infrared by over 20%. Multimodal fusion offers the best overall performance, combining the high road geometric characteristics of visible light with the thermal radiation properties of infrared to achieve a better balance between recall and precision.

[0089] Table 2

[0090] To verify the advancedness of the proposed multimodal road change detection model, we compared it with five representative road change detection methods on the WRCD-GM dataset: Fully Convolutional Early Fusion (FC-EF), Fully Convolutional Siamese-difference (FC-Siam-diff), Fully Convolutional Siamese–concatenation (FC-Siam-conc), multimodal sequence model (MM-Trans, multimodel transformers), and Separable Dynamic Grouping Convolution (SepDGConv).

[0091] Figure 5 is a schematic diagram of change detection result visualization based on the WRCD-GM dataset provided by the present application. The change detection result visualization on the WRCD-GM dataset is shown in FIG. 8, and the quantitative results are shown in Table 3. Figure 5 Figure 4 The first a column to the kth column of images in FIG. 8 respectively represent the visible light images of the T1 phase (including (a1)~(a6)), the infrared images of the T1 phase (including (b1)~(b6)), the visible light images of the T2 phase (including (c1)~(c6)), the infrared images of the T2 phase (including (d1)~(d6)), the road change labels (including (e1)~(e6)), the FC-EF results (including (f1)~(f6)), the FC-Siam-conc results (including (g1)~(g6)), the FC-Siam-diff results (including (h1)~(h6)), the MM-Trans results (including (i1)~(i6)), the SepG-Net results (including (j1)~(j6)), and the results of the method of the present application (including (k1)~(k6)). In order to make the visualization effect more intuitive, four different colors are used to represent different results: true positive (white), false positive (blue), true negative (black), and false negative (red).

[0092] From the visualization Figure 4 and the quantitative results in Table 3, it can be seen that compared with other methods, the method of the present application can obtain more accurate change detection results, and has higher recognition accuracy on narrow roads, achieving the best recall rate (69.26%), precision (76.07%), IoU (56.87%), and F1 score (72.50%).

[0093] Due to the complex background of remote sensing images, remote sensing image road change detection often has misidentification, and non-road area changes are incorrectly identified as road changes, such as Figure 4 ​In the first row, FC-EF, FC-Siam-diff, and FC-Siam-conc all exhibit misidentification. FC-EF's single-branch early fusion strategy causes infrared and visible light features to mix shallowly in the network, introducing additional noise and resulting in a loss of unique information about infrared and visible light. This results in lower IoU (44.31%) and F1 scores (61.41%). FC-Siam-conc and FC-Siam-diff mitigate modal interference through dual branches, but simple feature concatenation or interpolation cannot adaptively distinguish modal contributions, resulting in an imbalance between precision (60.33%) and recall (64.85%). In contrast, the "decoupled difference features - adaptive fusion" architecture employed by this model avoids mutual interference between visible and infrared light through independent feature extraction branches and adaptively distributes modal contributions through a spatiotemporal attention mechanism. Compared to FC-EF, this model improves Rec by 8.06% and Pre by 14.44%.

[0094] Table 3

[0095] For narrow roads, such as Figure 4 In the second and fourth rows, our method can detect changes more accurately and completely, while FC-EF, FC-Siam-conc, FC-Siam-diff, MM-Trans, and SepG-Net all miss detections to varying degrees. MM-Trans uses the Transformer's global attention to model long-range dependencies, but lacks fine-grained feature optimization for small-scale roads and has weaker recognition capabilities for narrow roads ( Figure 4 The recall rate is only 59.88%. SepG-Net transmits shallow detail information through skip connections and uses dynamic group convolution to enhance multimodal feature interaction. However, its single-stream architecture lacks semantic guidance for shallow information, resulting in discontinuous detection results for narrow and long roads ( Figure 4 (j1) and (j3)). In contrast, our model enhances contextual awareness of narrow roads through a joint optimization mechanism of "multi-scale feature pyramid and multi-layer deep supervision," improving the completeness of narrow road detection. As shown in the fifth and sixth rows, our detection results are more continuous and complete compared to other methods. Compared to SepG-Net, our recall rate is improved by 3.97%, indicating that our model is more capable of recognizing road changes.

[0096] The road change detection device provided by the present invention is described below. The road change detection device described below and the road change detection method described above can be referenced to each other.

[0097] Combine Figure 6 ,Figure 6 is a structural schematic diagram of a road change detection device provided by the present application.

[0098] As shown in Figure 6 , the road change detection device comprises: The acquisition module 610 is configured to acquire a double-time-phase multi-modal image pair; the multi-modal image pair comprises a visible light image pair and an infrared image pair; The road change detection module 620 is configured to input the multi-modal image pair into a road change detection model to obtain a road change detection result output by the road change detection model; The road change detection model comprises an encoder and a decoder; the encoder comprises a plurality of encoding layers, the encoding layers are configured to extract features of each modality image of different time phases, the encoder is configured to perform feature fusion and difference feature extraction on the features extracted by each encoding layer, and output each modality change feature corresponding to each encoding layer; the decoder comprises a plurality of decoding layers, the decoding layers are configured to perform feature fusion on the modality change features corresponding to the encoding layers, and the decoder is configured to perform feature fusion on the features fused by each decoding layer, and output a target modality fusion feature; the road change detection result is determined based on the target modality fusion feature.

[0099] The road change detection device provided by the present application inputs a double-time-phase multi-modal image pair into a road change detection model for road change detection. Firstly, a plurality of encoding layers of the encoder are used to extract features of each modality image of different time phases, and then feature fusion and difference feature extraction are performed on the features extracted by each encoding layer. Through a cross-layer feature fusion strategy, edges and semantic information of different scales are dynamically aggregated, the sensitivity of the network to road form changes in complex environments and the multi-scale feature expression ability are enhanced, and then the decoding layers of the decoder are used to fuse the modality change features extracted by the encoding layers. Then, the features fused by each decoding layer are fused to realize adaptive fusion of multi-modal difference features, fully utilize the complementarity of multi-modal data, and then determine a road change detection result according to the multi-modal fusion feature, thereby greatly improving the accuracy of road change detection.

[0100] Further, the road change detection module 620 is further configured to: input each modality image pair into the encoder, extract features of each modality image of different time phases through each encoding layer, and perform feature fusion and difference feature extraction on the features extracted by each encoding layer to obtain each modality change feature corresponding to each encoding layer output by the encoder; The modal change features corresponding to each of the encoding layers are input into an encoder, the modal change features corresponding to each of the encoding layers are fused by the encoding layer, and the fused features of each of the decoding layers are fused, to obtain target fusion features output by the encoder; The target fusion features are input into a threshold segmentation module, the target fusion features are normalized, and the normalized target fusion features are threshold segmented, to obtain a road change detection result output by the threshold segmentation module.

[0101] Further, the encoder adopts a twin architecture with double-branch non-shared parameters, including a first branch and a second branch, and the first branch and the second branch are respectively used for feature extraction of single-modal images at different time phases; the first branch and the second branch both sequentially include a first encoding layer, a second encoding layer, a third encoding layer, a fourth encoding layer, and a fifth encoding layer; The first encoding layer of the first branch or the second branch is used for feature extraction of the single-modal image, to obtain first features; the second encoding layer of the first branch or the second branch is used for feature extraction of the first features, to obtain second features; the third encoding layer of the first branch or the second branch is used for feature extraction of the second features, to obtain third features; the fourth encoding layer of the first branch or the second branch is used for feature extraction of the third features, to obtain fourth features; and the fifth encoding layer of the first branch or the second branch is used for feature extraction of the fourth features, to obtain fifth features; The second encoding layer of the first branch or the second branch is further used for feature fusion of the second features and the fifth features, to obtain first-level fusion features; the third encoding layer of the first branch or the second branch is further used for feature fusion of the third features and the fifth features, to obtain second-level fusion features; and the fourth encoding layer of the first branch or the second branch is further used for feature fusion of the fourth features and the fifth features, to obtain third-level fusion features.

[0102] Further, the encoder further includes a first difference feature extraction module, a second difference feature extraction module, a third difference feature extraction module, and a fourth difference feature extraction module; The first difference feature extraction module is used for pixel-by-pixel difference between the first-level fusion features corresponding to the first branch and the first-level fusion features corresponding to the second branch, to obtain single-modal change features corresponding to the second encoding layer; The second difference feature extraction module is used for pixel-by-pixel difference between the second-level fusion features corresponding to the first branch and the second-level fusion features corresponding to the second branch, to obtain single-modal change features corresponding to the third encoding layer; The third difference feature extraction module is configured to perform pixel-by-pixel difference between the third level fusion feature corresponding to the first branch and the third level fusion feature corresponding to the second branch to obtain single-modality change features corresponding to the fourth encoding layer; The fourth difference feature extraction module is configured to perform pixel-by-pixel difference between the fifth feature corresponding to the first branch and the fifth feature corresponding to the second branch to obtain single-modality change features corresponding to the fifth encoding layer.

[0103] Further, the decoder sequentially comprises a fifth decoding layer, a fourth decoding layer, a third decoding layer, a second decoding layer and a first decoding layer; the second encoding layer and the second decoding layer have a corresponding relationship; the third encoding layer and the third decoding layer have a corresponding relationship; the fourth encoding layer and the fourth decoding layer have a corresponding relationship; the fifth encoding layer and the fifth decoding layer have a corresponding relationship; The second decoding layer is configured to perform feature fusion on the modality change features of the second encoding layer to obtain first modality fusion features; the third decoding layer is configured to perform feature fusion on the modality change features of the third encoding layer to obtain second modality fusion features; the fourth decoding layer is configured to perform feature fusion on the modality change features of the fourth encoding layer to obtain third modality fusion features; and the fifth decoding layer is configured to perform feature fusion on the modality change features of the fifth encoding layer to obtain fourth modality fusion features; The fourth decoding layer is further configured to perform feature fusion on the fourth modality fusion features and the third modality fusion features to obtain first intermediate modality fusion features; the third decoding layer is further configured to perform feature fusion on the first intermediate modality fusion features and the second modality fusion features to obtain second intermediate modality fusion features; and the second decoding layer is further configured to perform feature fusion on the second intermediate modality fusion features and the first modality fusion features to obtain target modality fusion features.

[0104] Further, the modality change features include visible light change features and infrared change features; each decoding layer performs feature fusion on the modality change features of the corresponding encoding layer to obtain modality fusion features corresponding to the decoding layer, including: respectively performing global average pooling and global maximum pooling on the visible light change features and the infrared change features of the encoding layer to obtain a first global average pooling vector and a first global maximum pooling vector corresponding to the visible light change features, and a second global average pooling vector and a second global maximum pooling vector corresponding to the infrared change features; input the first global average pooling vector, the first global maximum pooling vector, the second global average pooling vector and the second global maximum pooling vector into a full connection layer, learn inter-modal channel correlation through the full connection layer, and obtain channel weights; weight the visible light change feature and the channel weights to obtain a first tensor, and weight the infrared change feature and the channel weights to obtain a second tensor; fuse based on the first tensor and the second tensor to obtain a modal fusion feature corresponding to the decoding layer.

[0105] Further, the fusing based on the first tensor and the second tensor to obtain a modal fusion feature corresponding to the decoding layer comprises: respectively perform global average pooling and global maximum pooling on the first tensor and the second tensor to obtain a third global average pooling vector and a third global maximum pooling vector corresponding to the first tensor, and a fourth global average pooling vector and a fourth global maximum pooling vector corresponding to the second tensor; input the third global average pooling vector, the third global maximum pooling vector, the fourth global average pooling vector and the fourth global maximum pooling vector into a convolution layer, model cross-modal spatial correlation through the convolution layer, and obtain spatial weights; weight the first tensor and the spatial weights to obtain a first matrix, and weight the second tensor and the spatial weights to obtain a second matrix; perform pixel-by-pixel summation on the first matrix and the second matrix to obtain a modal fusion feature corresponding to the decoding layer.

[0106] Further, the loss value caused by the road change detection model during training is determined in the following manner: calculate binary cross-entropy loss values and hard sample mining loss values output by each decoding layer; perform mean value calculation on each binary cross-entropy loss value to obtain a first mean value, and perform mean value calculation on each hard sample mining loss value to obtain a second mean value; perform weighted summation on the first mean value and the second mean value to obtain the loss value caused by the road change detection model during training.

[0107] It should be noted that the road change detection device provided by the present application can execute the road change detection method described in any of the above embodiments during specific operation, and this embodiment will not be described here.

[0108] Figure 7 is a structural schematic diagram of an electronic device provided by the present application, as Figure 7As shown, the electronic device can include a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 complete mutual communication through the communications bus 740. The processor 710 can invoke a logical instruction in the memory 730 to execute a road change detection method, which includes: acquiring a dual-phase multi-modal image pair; the multi-modal image pair includes a visible light image pair and an infrared image pair; inputting the multi-modal image pair into a road change detection model to obtain a road change detection result output by the road change detection model; wherein the road change detection model includes an encoder and a decoder; the encoder includes a plurality of encoding layers in sequence, and the encoding layers are used for feature extraction of each modal image of different time phases; the encoder is used for feature fusion and difference feature extraction of the features extracted by each encoding layer, and outputs each modal change feature corresponding to each encoding layer; the decoder includes a plurality of decoding layers in sequence, and the decoding layers are used for feature fusion of each modal change feature corresponding to the encoding layer; the decoder is used for feature fusion of the features fused by each decoding layer, and outputs a target modal fusion feature; and the road change detection result is determined based on the target modal fusion feature.

[0109] In addition, the logical instructions in the memory 730 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0110] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, enable the computer to perform the road change detection method provided by any of the above embodiments, the method comprising: obtaining a pair of multi-modal images of double time phases; the pair of multi-modal images comprising a pair of visible light images and a pair of infrared images; inputting the pair of multi-modal images into a road change detection model to obtain a road change detection result output by the road change detection model; wherein the road change detection model comprises an encoder and a decoder; the encoder comprises a plurality of encoding layers in sequence, the encoding layers being configured to perform feature extraction on each modality image of different time phases, the encoder being configured to perform feature fusion and difference feature extraction on the features extracted by each of the encoding layers, and output each modality change feature corresponding to each of the encoding layers; the decoder comprises a plurality of decoding layers in sequence, the decoding layers being configured to perform feature fusion on the modality change features corresponding to the encoding layers, the decoder being configured to perform feature fusion on the features fused by each of the decoding layers, and output a target modality fusion feature; and the road change detection result is determined based on the target modality fusion feature.

[0111] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement a road change detection method provided by any of the above embodiments, the method comprising: obtaining a pair of multi-modal images of double time phases; the pair of multi-modal images comprising a pair of visible light images and a pair of infrared images; inputting the pair of multi-modal images into a road change detection model to obtain a road change detection result output by the road change detection model; wherein the road change detection model comprises an encoder and a decoder; the encoder comprises a plurality of encoding layers in sequence, the encoding layers being configured to perform feature extraction on each modality image of different time phases, the encoder being configured to perform feature fusion and difference feature extraction on the features extracted by each of the encoding layers, and output each modality change feature corresponding to each of the encoding layers; the decoder comprises a plurality of decoding layers in sequence, the decoding layers being configured to perform feature fusion on the modality change features corresponding to the encoding layers, the decoder being configured to perform feature fusion on the features fused by each of the decoding layers, and output a target modality fusion feature; and the road change detection result is determined based on the target modality fusion feature.

[0112] The above-described embodiments are merely illustrative for describing the present application and the units described as separate parts can or can not be physically separate, and the parts displayed as units can or can not be physical units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those of ordinary skill in the art can understand and implement it without creative labor.

[0113] Those skilled in the art can clearly understand the implementation of the embodiments by the description of the above embodiments, and the embodiments can be implemented by means of software and necessary universal hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the method described in each embodiment or some parts of the embodiment.

[0114] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some technical features thereof; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A road change detection method, characterized in that: The road change detection method comprises: Acquire a dual-phase multimodal image pair; the multimodal image pair includes a visible light image pair and an infrared image pair; Inputting the multimodal image pair into a road change detection model to obtain a road change detection result output by the road change detection model; Among them, the road change detection model includes an encoder and a decoder; the encoder includes multiple encoding layers in sequence, and the encoding layers are used to extract features of each modal image in different phases. The encoder is used to perform feature fusion and difference feature extraction on the features extracted by each encoding layer, and output each modal change feature corresponding to each encoding layer; the decoder includes multiple decoding layers in sequence, and the decoding layers are used to perform feature fusion on the modal change features corresponding to the encoding layer. The decoder is used to perform feature fusion on the features fused by each decoding layer, and output the target modal fusion feature; the road change detection result is determined based on the target modal fusion feature.

2. The road change detection method according to claim 1, characterized in that: The plurality of encoding layers and the plurality of decoding layers have a one-to-one relationship; the road change detection model includes an encoder, a decoder and a threshold segmentation module in sequence; Inputting the multimodal image pair into a road change detection model to obtain a road change detection result output by the road change detection model includes: Input each modality image pair into the encoder, perform feature extraction on each modality image of different phases through each encoding layer, and perform feature fusion and difference feature extraction on the features extracted by each encoding layer to obtain each modality change feature corresponding to each encoding layer output by the encoder; Inputting each modal change feature corresponding to each encoding layer into the encoder, performing feature fusion on each modal change feature of the corresponding encoding layer through the encoding layer, and performing feature fusion on the features fused by each decoding layer to obtain the target fusion feature output by the encoder; The target fusion features are input into a threshold segmentation module, the target fusion features are normalized, and the normalized target fusion features are threshold segmented to obtain a road change detection result output by the threshold segmentation module.

3. The road change detection method according to claim 2, characterized in that: The encoder adopts a twin architecture with two branches and non-shared parameters, including a first branch and a second branch, wherein the first branch and the second branch are respectively used to extract features from single-modal images of different phases; the first branch and the second branch each sequentially include a first coding layer, a second coding layer, a third coding layer, a fourth coding layer, and a fifth coding layer; The first encoding layer of the first branch or the second branch is used to extract features from the single-modal image to obtain a first feature; The second coding layer of the first branch or the second branch is used to perform feature extraction on the first feature to obtain a second feature extraction; the third coding layer of the first branch or the second branch is used to perform feature extraction on the second feature to obtain a third feature; the fourth coding layer of the first branch or the second branch is used to perform feature extraction on the third feature to obtain a fourth feature; The fifth coding layer of the first branch or the second branch is used to extract the fourth feature to obtain a fifth feature; The second encoding layer of the first branch or the second branch is further used to perform feature fusion on the second feature and the fifth feature to obtain a first-level fusion feature; The third encoding layer of the first branch or the second branch is further used to perform feature fusion on the third feature and the fifth feature to obtain a second-level fusion feature; The fourth coding layer of the first branch or the second branch is further used to perform feature fusion on the fourth feature and the fifth feature to obtain a third-level fusion feature.

4. The road change detection method according to claim 3, characterized in that: The encoder further includes a first difference feature extraction module, a second difference feature extraction module, a third difference feature extraction module, and a fourth difference feature extraction module; The first difference feature extraction module is used to perform pixel-by-pixel difference between the first-level fusion feature corresponding to the first branch and the first-level fusion feature corresponding to the second branch to obtain a single-modal change feature corresponding to the second coding layer; The second difference feature extraction module is used to perform pixel-by-pixel difference between the second-level fusion feature corresponding to the first branch and the second-level fusion feature corresponding to the second branch to obtain a single-modal change feature corresponding to the third coding layer; The third difference feature extraction module is used to perform pixel-by-pixel difference between the third-level fusion feature corresponding to the first branch and the third-level fusion feature corresponding to the second branch to obtain a single-modal change feature corresponding to the fourth coding layer; The fourth difference feature extraction module is used to perform pixel-by-pixel difference between the fifth feature corresponding to the first branch and the fifth feature corresponding to the second branch to obtain a single-modal change feature corresponding to the fifth coding layer.

5. The road change detection method according to claim 4, characterized in that: The decoder includes a fifth decoding layer, a fourth decoding layer, a third decoding layer, a second decoding layer, and a first decoding layer in sequence; the second coding layer and the second decoding layer have a corresponding relationship; the third coding layer and the third decoding layer have a corresponding relationship; the fourth coding layer and the fourth decoding layer have a corresponding relationship; and the fifth coding layer and the fifth decoding layer have a corresponding relationship; The second decoding layer is used to perform feature fusion on each modal change feature of the second encoding layer to obtain a first modal fusion feature; The third decoding layer is used to perform feature fusion on the modal change features of the third encoding layer to obtain a second modal fusion feature; the fourth decoding layer is used to perform feature fusion on the modal change features of the fourth encoding layer to obtain a third modal fusion feature; The fifth decoding layer is used to perform feature fusion on each modal change feature of the fifth encoding layer to obtain a fourth modal fusion feature; The fourth decoding layer is further used to perform feature fusion on the fourth modality fusion feature and the third modality fusion feature to obtain a first intermediate modality fusion feature; the third decoding layer is further used to perform feature fusion on the first intermediate modality fusion feature and the second modality fusion feature to obtain a second intermediate modality fusion feature; the second decoding layer is further used to perform feature fusion on the second intermediate modality fusion feature and the first modality fusion feature to obtain a target modality fusion feature.

6. The road change detection method according to claim 5, characterized in that: The modal change characteristics include visible light change characteristics and infrared change characteristics; Each decoding layer performs feature fusion on the modal change features of its corresponding encoding layer to obtain the modal fusion features corresponding to the decoding layer, including: Performing global average pooling and global maximum pooling on the visible light change features and the infrared change features of the coding layer, respectively, to obtain a first global average pooling vector and a first global maximum pooling vector corresponding to the visible light change features, and a second global average pooling vector and a second global maximum pooling vector corresponding to the infrared change features; Inputting the first global average pooling vector, the first global maximum pooling vector, the second global average pooling vector, and the second global maximum pooling vector into a fully connected layer, and learning inter-modal channel correlation through the fully connected layer to obtain channel weights; Weighting the visible light variation feature and the channel weight to obtain a first tensor, and weighting the infrared variation feature and the channel weight to obtain a second tensor; The first tensor and the second tensor are fused to obtain a modality fusion feature corresponding to the decoding layer.

7. The road change detection method according to claim 6, characterized in that: The fusing the first tensor and the second tensor to obtain the modality fusion feature corresponding to the decoding layer includes: Performing global average pooling and global maximum pooling on the first tensor and the second tensor, respectively, to obtain a third global average pooling vector and a third global maximum pooling vector corresponding to the first tensor, and a fourth global average pooling vector and a fourth global maximum pooling vector corresponding to the second tensor; Inputting the third global average pooling vector, the third global maximum pooling vector, the fourth global average pooling vector, and the fourth global maximum pooling vector into a convolutional layer, performing cross-modal spatial correlation modeling through the convolutional layer, and obtaining a spatial weight; Weighting the first tensor and the spatial weight to obtain a first matrix, and weighting the second tensor and the spatial weight to obtain a second matrix; The first matrix and the second matrix are summed pixel by pixel to obtain the modality fusion feature corresponding to the decoding layer.

8. The road change detection method according to any one of claims 1 to 7, characterized in that: The loss value incurred by the road change detection model during training is determined as follows: Calculate the binary cross entropy loss and hard sample mining loss for each decoding layer output; Performing mean calculation on each of the binary cross entropy loss values ​​to obtain a first mean, and performing mean calculation on each of the difficult sample mining loss values ​​to obtain a second mean; The first mean and the second mean are weightedly summed to obtain a loss value caused by the road change detection model during training.

9. A road change detection device, characterized in that: include: An acquisition module, configured to acquire a dual-phase multimodal image pair; the multimodal image pair includes a visible light image pair and an infrared image pair; a road change detection module, configured to input the multimodal image pair into a road change detection model and obtain a road change detection result output by the road change detection model; The road change detection model includes an encoder and a decoder; the encoder includes multiple encoding layers, the encoding layers are used to extract features from each modality image at different time phases, the encoder is used to perform feature fusion and difference feature extraction on the features extracted by each encoding layer, and output each modality change feature corresponding to each encoding layer; the decoder includes multiple decoding layers, the decoding layers are used to perform feature fusion on each modality change feature corresponding to the encoding layer, and the decoder is used to perform feature fusion on the features fused by each decoding layer, and output the target modality fusion feature; The road change detection result is determined based on the target modality fusion feature.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the road change detection method according to any one of claims 1 to 8 are implemented.

Images