Optical / SAR image change detection network and detection method based on multi-scale feature alignment

Abstract

The invention discloses an optical / SAR image change detection network and detection method based on multi-scale feature alignment, and the network structure comprises a multi-scale feature extraction module which carries out the multi-scale feature extraction of an optical image and an SAR image, and obtains a low-dimensional feature map and a high-dimensional feature map of the optical image and the SAR image; the low-dimensional feature fusion module is used for performing low-dimensional feature fusion based on the low-dimensional feature maps of the optical image and the SAR image to obtain a low-layer multi-source fusion feature map; the high-dimensional feature selection module performs high-dimensional feature selection and fusion based on the high-dimensional feature maps of the optical image and the SAR image to obtain a multi-source fusion feature map; and the optical / SAR image change detection module is used for carrying out image change detection based on the low-layer multi-source fusion feature map and the high-layer multi-source fusion feature map and outputting a final optical / SAR image change detection result.

Description

Technical Field

[0001] The present invention relates to the field of photogrammetry, and in particular to an optical / SAR image change detection network and detection method based on multi-scale feature alignment. Background Art

[0002] Optical imaging, based on spectral reflectance imaging, provides rich spectral information but is susceptible to weather. SAR imaging, on the other hand, uses microwaves for active imaging and can penetrate clouds and fog, but suffers from speckle noise and geometric distortion. The combined use of optical and SAR imaging can compensate for their respective shortcomings, making it possible to detect continuous regional change. However, optical and SAR imagery exhibit inherent differences in imaging principles, spatial resolution, and spectral characteristics, making it difficult to directly compare or align the features of the different modalities.

[0003] In practical applications, it is often difficult to obtain meaningful change information by directly comparing the features in optical and SAR images.

[0004] How to ensure high comparability of features while avoiding the interference of feature redundancy and effectively combining global and local features still significantly affects the accuracy of change detection models. Summary of the Invention

[0005] To overcome the problem of difficult feature alignment between optical images and SAR images in the prior art, the present invention provides an optical / SAR image change detection network and detection method based on multi-scale feature alignment. By integrating a low-dimensional feature fusion module and a high-dimensional feature selection module, the optical and SAR features are effectively aligned and fused to achieve effective utilization of features, thereby improving the accuracy of change detection.

[0006] According to one aspect of the present invention, there is provided an optical / SAR image change detection network based on multi-scale feature alignment, comprising:

[0007] The multi-scale feature extraction module performs multi-scale feature extraction on optical images and SAR images respectively to obtain low-dimensional feature maps and high-dimensional feature maps of optical images and SAR images;

[0008] The low-dimensional feature fusion module performs low-dimensional feature fusion based on the low-dimensional feature maps of optical images and SAR images to obtain low-level multi-source fusion feature maps;

[0009] The high-dimensional feature selection module selects high-dimensional features based on the high-dimensional feature maps of optical images and SAR images and then fuses them to obtain a multi-source fusion feature map;

[0010] The optical / SAR image change detection module performs image change detection based on low-level multi-source fusion feature maps and high-level multi-source fusion feature maps, and outputs the final optical / SAR image change detection results.

[0011] As a further implementation plan, the low-dimensional feature fusion module includes two branches of a twin structure, which process optical images and SAR images respectively. Among them, a single branch structure contains five coding layers, and the coding layer consists of a convolutional layer, a batch normalization layer and a ReLU activation function. The first, second, third and fourth coding layers are connected through a global maximum pooling layer, and the fourth coding layer is directly connected to the fifth coding layer; the second coding layer outputs a low-dimensional feature map, and the fifth coding layer outputs a high-dimensional feature map.

[0012] As a further embodiment, the structure of the multi-scale feature extraction module includes a first splicing layer, a first convolutional layer, a two-branch pooling layer, an addition layer, a gating module, a weighted splicing layer, a Transformer module and a second convolutional layer;

[0013] The first splicing layer, the first convolutional layer, the dual-branch pooling layer, the summing layer, and the gating module are connected in sequence. The splicing layer simultaneously inputs the low-dimensional feature maps of the optical image and the SAR image, processes them in sequence, and outputs the normalized channel attention weight vector in the gating module.

[0014] The weighted splicing layer, Transformer module and the second convolutional layer are connected in sequence. The weighted splicing layer simultaneously inputs the low-dimensional feature maps of the optical image and SAR image and the normalized channel attention weight vector, processes them in sequence, and outputs the low-level multi-source fusion feature map in the second convolutional layer.

[0015] As a further implementation plan, the gating module is composed of a third convolutional layer, a ReLU activation function, a fourth convolutional layer and a Sigmoid function in sequence. The result of the addition layer is processed in sequence by the third convolutional layer, the ReLU activation function, and the fourth convolutional layer to generate attention weights, and finally activated by the Sigmoid function to obtain a normalized channel attention weight vector.

[0016] As a further implementation scheme, the dual-branch pooling layer is a pooling layer of different types, including: a global average pooling layer and a global maximum pooling layer. The global average pooling layer processes the result of the first splicing layer to extract overall features, and the global maximum pooling layer processes the result of the first splicing layer to extract local significant features.

[0017] As a further embodiment, the structure of the high-dimensional feature selection module includes: two feature selection branches, a second splicing layer, a dilated spatial pyramid pooling module and a first upsampling layer;

[0018] The high-dimensional feature maps of optical images and SAR images are respectively input into two feature selection branches for processing to obtain selected feature maps. The splicing layer, the atrous spatial pyramid pooling module and the first upsampling layer are sequentially connected. The two selected feature maps are input into the splicing layer and processed in sequence. The first upsampling layer outputs a high-level multi-source fusion feature map.

[0019] As a further implementation scheme, the two feature selection branches are branches of the same structure, a single branch includes an SE block and a global average pooling block, one of which is used to process the high-dimensional feature map of the optical image, and the other is used to process the high-dimensional feature map of the SAR image;

[0020] The high-dimensional feature map uses SE blocks to adaptively model the dependencies between channels and enhance the response of important channels. Then, global average pooling is performed on each channel to obtain the channel importance score. A preset number of high-scoring channels are selected according to the channel importance score to obtain the selected feature map.

[0021] As a further implementation scheme, the structure of the optical / SAR image change detection module includes: a third splicing layer, a decoding layer, a second upsampling layer and a softmax layer in sequence, wherein the decoding layer is composed of two convolutional layers.

[0022] According to another aspect of the present specification, a method for detecting changes in optical / SAR images based on multi-scale feature alignment is provided, comprising:

[0023] Acquire optical and SAR images;

[0024] Perform multi-scale feature extraction on optical images and SAR images respectively to obtain low-dimensional feature maps and high-dimensional feature maps of optical images and SAR images;

[0025] Based on the low-dimensional feature maps of optical images and SAR images, low-dimensional feature fusion is performed to obtain low-level multi-source fusion feature maps;

[0026] Based on the high-dimensional feature maps of optical images and SAR images, high-dimensional feature selection is performed to obtain a selected feature map, and the selected feature map is fused to obtain a high-level multi-source fusion feature map;

[0027] Image change detection is performed based on low-level multi-source fusion feature maps and high-level multi-source fusion feature maps to obtain the final optical / SAR image change detection results.

[0028] As a further implementation scheme, the high-dimensional feature selection is specifically as follows:

[0029] The high-dimensional feature map uses SE blocks to adaptively model the dependencies between channels and enhance the response of important channels. Then, global average pooling is performed on each channel to obtain the channel importance score. A preset number of high-scoring channels are selected according to the channel importance score to obtain the selected feature map.

[0030] Compared with the existing technology, the beneficial effects of the present invention are: extracting multi-scale features within the network; for low-dimensional features, using a low-dimensional feature fusion module to fuse features, thereby combining local important features with global features to capture the relationship between cross-modal features; for high-dimensional features, using a high-dimensional feature selection module to sort channel importance, select important features, and reduce feature redundancy; decoding the results obtained by the low-dimensional feature fusion module and the high-dimensional feature selection module to form an end-to-end optical / SAR change detection network. The present invention significantly improves the accuracy of optical / SAR change detection by introducing a low-dimensional feature fusion module and a high-dimensional feature selection module to align and compare features. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 A schematic diagram of the structure of an optical / SAR image change detection network based on multi-scale feature alignment provided by an embodiment of the present invention;

[0033] Figure 2 A schematic flow chart of a method for optical / SAR image change detection based on multi-scale feature alignment in an embodiment of the present invention;

[0034] Figure 3 Schematic diagram of a multi-scale feature extraction module in an embodiment of the present invention;

[0035] Figure 4 Schematic diagram of the low-dimensional feature fusion module in an embodiment of the present invention;

[0036] Figure 5 This is a schematic diagram of a high-dimensional feature selection module in an embodiment of the present invention;

[0037] Figure 6 Schematic diagram of the optical / SAR change detection method using multi-scale feature alignment in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] It should be noted that:

[0039] The terms "including" and "having" and any variations thereof in the description and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0040] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be executed in the order described. For example, some operations / steps may be further decomposed, while others may be combined or partially combined, so the actual execution order may vary depending on the actual situation.

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0042] like Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of an optical / SAR image change detection network based on multi-scale feature alignment in an embodiment of the present invention, including: a multi-scale feature extraction module, a low-dimensional feature fusion module, a high-dimensional feature selection module and an optical / SAR image change detection module.

[0043] This paper provides an optical / SAR image change detection network based on multi-scale feature alignment, including:

[0044] The multi-scale feature extraction module performs multi-scale feature extraction on optical images and SAR images respectively to obtain low-dimensional feature maps and high-dimensional feature maps of optical images and SAR images;

[0045] The low-dimensional feature fusion module performs low-dimensional feature fusion based on the low-dimensional feature maps of optical images and SAR images to obtain low-level multi-source fusion feature maps;

[0046] The high-dimensional feature selection module selects high-dimensional features based on the high-dimensional feature maps of optical images and SAR images and then fuses them to obtain a multi-source fusion feature map;

[0047] The optical / SAR image change detection module performs image change detection based on low-level multi-source fusion feature maps and high-level multi-source fusion feature maps, and outputs the final optical / SAR image change detection results.

[0048] Furthermore, the low-dimensional feature fusion module contains two branches of a twin structure, which process optical images and SAR images respectively. Among them, a single branch structure contains five coding layers, which are composed of convolutional layers, batch normalization layers and ReLU activation functions. The first, second, third and fourth coding layers are connected through a global maximum pooling layer, and the fourth coding layer is directly connected to the fifth coding layer; the second coding layer outputs a low-dimensional feature map, and the fifth coding layer outputs a high-dimensional feature map.

[0049] Furthermore, the structure of the multi-scale feature extraction module includes a first splicing layer, a first convolutional layer, a two-branch pooling layer, an addition layer, a gating module, a weighted splicing layer, a Transformer module, and a second convolutional layer;

[0050] The first splicing layer, the first convolutional layer, the dual-branch pooling layer, the summing layer, and the gating module are connected in sequence. The splicing layer simultaneously inputs the low-dimensional feature maps of the optical image and the SAR image, processes them in sequence, and outputs the normalized channel attention weight vector in the gating module.

[0051] The weighted splicing layer, Transformer module and the second convolutional layer are connected in sequence. The weighted splicing layer simultaneously inputs the low-dimensional feature maps of the optical image and SAR image and the normalized channel attention weight vector, processes them in sequence, and outputs the low-level multi-source fusion feature map in the second convolutional layer.

[0052] Furthermore, the gating module is composed of the third convolutional layer, ReLU activation function, fourth convolutional layer and Sigmoid function in sequence. The result of the addition layer is processed in sequence by the third convolutional layer, ReLU activation function, and fourth convolutional layer to generate attention weights, and finally activated by the Sigmoid function to obtain the normalized channel attention weight vector.

[0053] Furthermore, the dual-branch pooling layer is a pooling layer of different types, including: a global average pooling layer and a global maximum pooling layer. The global average pooling layer processes the result of the first splicing layer to extract overall features, and the global maximum pooling layer processes the result of the first splicing layer to extract local significant features.

[0054] Specifically, the steps of the multi-scale feature extraction module to process the low-dimensional feature maps of optical images and SAR images are as follows:

[0055] The low-dimensional feature maps of optical images and SAR images are spliced ​​in the channel dimension in the splicing layer, and the convolution layer compresses the number of channels to obtain the initial fused body features;

[0056] The initial fused posture features are processed by the global average pooling layer and the global maximum pooling layer respectively, and the processing results are added in the addition layer to obtain the channel attention description vector;

[0057] The channel attention description vector is processed by the gating module to obtain the normalized channel attention weight vector;

[0058] The low-dimensional feature maps of optical images and SAR images are weightedly fused in the weighted fusion layer according to the normalized channel attention weight vector, and then the attention is enhanced using the Transformer module. The convolution layer is adjusted to the specified number of output channels to generate the final low-level multi-source fusion feature map.

[0059] Furthermore, the structure of the high-dimensional feature selection module includes: two feature selection branches, a second splicing layer, a dilated spatial pyramid pooling module, and a first upsampling layer;

[0060] The high-dimensional feature maps of optical images and SAR images are respectively input into two feature selection branches for processing to obtain selected feature maps. The splicing layer, the atrous spatial pyramid pooling module and the first upsampling layer are sequentially connected. The two selected feature maps are input into the splicing layer and processed in sequence. The first upsampling layer outputs a high-level multi-source fusion feature map.

[0061] Furthermore, the two feature selection branches are branches of the same structure, and a single branch includes an SE block and a global average pooling block, one of which is used to process the high-dimensional feature map of the optical image, and the other is used to process the high-dimensional feature map of the SAR image;

[0062] The high-dimensional feature map uses SE blocks to adaptively model the dependencies between channels and enhance the response of important channels. Then, global average pooling is performed on each channel to obtain the channel importance score. A preset number of high-scoring channels are selected according to the channel importance score to obtain the selected feature map.

[0063] Specifically, the steps for the high-dimensional feature selection module to process the high-dimensional feature maps of optical images and SAR images are as follows:

[0064] The high-dimensional feature maps of the optical image and the SAR image are respectively input into the feature selection branch for processing. The processing steps of the input processing branch include: adaptively modeling the inter-channel dependency through the SE block to enhance the response of important channels, then performing global average pooling on each channel to obtain a channel importance score, and selecting a preset number of high-scoring channels according to the channel importance score to obtain a selected feature map;

[0065] The two selected feature maps are spliced ​​in the channel dimension through the splicing layer, and the high-level feature map obtained by splicing is input into the void space pyramid pooling module to obtain multi-scale context information, and then adjusted to the spatial size and low-level multi-source fusion features through upsampling. Figure 1 Finally, a high-level multi-source fusion feature map is obtained.

[0066] Furthermore, the structure of the optical / SAR image change detection module includes: a third splicing layer, a decoding layer, a second upsampling layer and a softmax layer in sequence, wherein the decoding layer consists of two convolutional layers.

[0067] Specifically, the optical / SAR image change detection module works as follows:

[0068] The low-level multi-source fusion feature map and the high-level multi-source fusion feature map are input into the splicing layer for splicing in the channel dimension, and then input into the decoding layer for decoding to obtain the result feature map;

[0069] The resulting feature map is processed by the upsampling layer to align its spatial size with the input image, and then processed by the softmax layer to obtain the final change detection result.

[0070] A change detection method for optical / SAR images based on multi-scale feature alignment, such as Figure 2 Shown, including:

[0071] Acquire optical and SAR images;

[0072] Perform multi-scale feature extraction on optical images and SAR images respectively to obtain low-dimensional feature maps and high-dimensional feature maps of optical images and SAR images;

[0073] Based on the low-dimensional feature maps of optical images and SAR images, low-dimensional feature fusion is performed to obtain low-level multi-source fusion feature maps;

[0074] Based on the high-dimensional feature maps of optical images and SAR images, high-dimensional feature selection is performed to obtain a selected feature map, and the selected feature map is fused to obtain a high-level multi-source fusion feature map;

[0075] Image change detection is performed based on low-level multi-source fusion feature maps and high-level multi-source fusion feature maps to obtain the final optical / SAR image change detection results.

[0076] Furthermore, the high-dimensional feature selection is specifically as follows:

[0077] The high-dimensional feature map uses SE blocks to adaptively model the dependencies between channels and enhance the response of important channels. Then, global average pooling is performed on each channel to obtain the channel importance score. A preset number of high-scoring channels are selected according to the channel importance score to obtain the selected feature map.

[0078] Optical / SAR change detection based on multi-scale feature alignment includes the following steps:

[0079] like Figure 3 As shown, Figure 3 FIG1 is a schematic diagram of a multi-scale feature extraction module according to an embodiment of the present invention. In the network coding stage, multi-scale features of optical / SAR images are extracted respectively. The multi-scale feature extraction of optical / SAR images includes the following steps:

[0080] 1.1. For the input optical image (256×256, 3 channels), shallow features are first extracted through the encoding layer. The encoding layer consists of a convolutional layer, a batch normalization layer, and a Reluctant Unified Unit (ReLU) activation function. The convolution kernel size is 3×3 and the output channels are 64. The feature map size is then downsampled to 128×128 through a max pooling operation.

[0081] 1.2. For the extracted shallow features, they pass through two encoding layers and a global maximum pooling layer in sequence. The input feature map sizes are 64×64 and 32×32, respectively, and the corresponding number of channels is 256 and 512.

[0082] 1.3. For the features extracted in 1.2, they pass through two encoding layers in sequence, and the feature map size is saved as 32×32, and the number of channels is increased to 1024 and 2048.

[0083] 1.4. For the input SAR image (size 256×256, 1 channel), shallow features are first extracted through the encoding layer. The encoding layer consists of a convolutional layer, a batch normalization layer, and a Reluctant Unified Unit (ReLU) activation function. The convolution kernel size is 3×3 and the output channels are 64. The feature map size is then downsampled to 128×128 through a max pooling operation.

[0084] 1.5. For the extracted shallow features, they pass through two encoding layers and a global maximum pooling layer in sequence. The input feature map sizes are 64×64 and 32×32, respectively, and the corresponding number of channels is 256 and 512.

[0085] 1.6. For the features extracted in 1.5, pass them through two encoding layers in sequence, save the feature map size as 32×32, and increase the number of channels to 1024 and 2048.

[0086] Specifically, if Figure 4 As shown, Figure 4This is a schematic diagram of the low-dimensional feature fusion module in an embodiment of the present invention. The low-dimensional feature map generated in the multi-scale feature extraction stage uses the attention mechanism to generate dynamic weights, and integrates the Transformer structure to integrate global context information. The low-dimensional feature fusion includes the following steps:

[0087] 2.1. Concatenate the feature maps (64 × 64 × 256) extracted by the SAR and optical branches in the second layer along the channel dimension to obtain a feature map of size 64 × 64 × 512.

[0088] 2.2. Then, the number of channels is compressed to 256 through 1×1 convolution to obtain the initial fused features (64×64×256). Global average pooling and maximum pooling are performed on the initial fused body features and added together to obtain the channel attention description vector (1×1×256).

[0089] 2.3. Attention weights are generated through a gating module consisting of two layers of 1×1 convolution and ReLU. Finally, a sigmoid activation is performed to obtain a normalized channel attention weight vector (1×1×256). This weight is applied to the SAR and optical features, respectively, to obtain two weighted features of size (64×64×256).

[0090] 2.4. Concatenate these two sets of weighted features again to obtain a 64×64×512 feature map, flatten it into a 4096×512 sequence and input it into the Transformer module for long-range dependency modeling, outputting the attention-enhanced features.

[0091] 2.5. Finally, the output is restored to 64×64×512, and the number of channels is adjusted to the specified output channel number 48 through 1×1 convolution to generate the final low-level multi-source fusion feature map.

[0092] Specifically, if Figure 5 As shown, Figure 5 This is a schematic diagram of the high-dimensional feature selection module in an embodiment of the present invention. The high-dimensional feature map generated in the multi-scale feature extraction stage is used to assign weights to the input features using the SE (Squeeze-and-Excitation Block) mechanism, and global average pooling is used to select the top K most important channels. The high-dimensional feature selection includes the following steps:

[0093] 3.1. Input the feature map (32×32×2048) mentioned in the fifth layer of the optical branch into the high-dimensional feature selection module, and adaptively model the inter-channel dependencies through the SE block to enhance the important channel response

[0094] 3.2. Perform global average pooling on each channel to obtain the channel importance score. Finally, select the top 256 channels and retain their corresponding spatial structure, resulting in an output size of (32×32×256).

[0095] 3.3. Perform the same operation on the SAR branch feature map as on the optical branch feature map. Concatenate the two 256-channel feature maps in the channel dimension to obtain the fused high-level feature map high_level_features with a size of 32 × 32 × 512.

[0096] 3.4. Input the dilated spatial pyramid pooling module, which uses dilated convolutions with different dilation rates to obtain multi-scale context information. The output size is maintained at 32×32 and the number of channels is 256.

[0097] 3.5. Then, upsample the image using bilinear interpolation to align its spatial size to the size of the feature map output by the low-dimensional feature fusion module, i.e., adjust it to 64×64, with the number of channels remaining at 256.

[0098] Specifically, if Figure 6 As shown, Figure 6 FIG. 1 is a schematic diagram of an optical / SAR change detection method using multi-scale feature alignment according to an embodiment of the present invention. The method includes the following steps:

[0099] 4.1. Input the optical / SAR image into the multi-scale feature extraction module to obtain low-dimensional and high-dimensional features.

[0100] 4.2. Input the low-dimensional features into the "low-dimensional feature fusion module" and the high-dimensional features into the "high-dimensional feature selection module" to obtain feature maps respectively.

[0101] 4.3. The above feature maps are connected at the channel layer and input into the decoding layer. The decoding layer consists of two convolutions. The first convolution kernel size is 3×3, which converts the number of channels of the feature map from 304 to 256. The second convolution layer converts the number of channels of the feature map from 256 to 2, and the final result size is 64×64×2.

[0102] 4.4. Upsample the image using bilinear interpolation to align its spatial dimensions to the input image size, which is 256 × 256 × 2. The final change detection result is obtained through a softmax layer.

[0103] The network embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place or distributed across multiple network elements. Depending on practical needs, some or all of the modules may be selected to achieve the objectives of the embodiments. Persons of ordinary skill in the art will understand and implement the embodiments without inventive effort.

[0104] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0108] The main features of the present invention are:

[0109] Multi-scale features are extracted within the network, and low-level features and high-level features are fused. The fused features are used to replace the original fifth layer for feature extraction to extract scale context features.

[0110] In the process of spatial context feature extraction, the extraction and fusion of spatial context are integrated into the network, so that training can be performed based on suspected region features and context region features in each training to achieve the purpose of model parameter optimization.

[0111] The extraction of scale context and spatial context is integrated within the network to form an end-to-end network to complete vehicle target detection in low-light images.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. An optical / SAR image change detection network based on multi-scale feature alignment, characterized by: include: The multi-scale feature extraction module performs multi-scale feature extraction on optical images and SAR images respectively to obtain low-dimensional feature maps and high-dimensional feature maps of optical images and SAR images; The low-dimensional feature fusion module performs low-dimensional feature fusion based on the low-dimensional feature maps of optical images and SAR images to obtain low-level multi-source fusion feature maps; The high-dimensional feature selection module selects and fuses high-dimensional features based on the high-dimensional feature maps of optical images and SAR images to obtain a multi-source fusion feature map; The optical / SAR image change detection module performs image change detection based on low-level multi-source fusion feature maps and high-level multi-source fusion feature maps, and outputs the final optical / SAR image change detection results.

2. The optical / SAR image change detection network based on multi-scale feature alignment according to claim 1, characterized in that: The low-dimensional feature fusion module includes two branches of a twin structure, which process optical images and SAR images respectively. Among them, a single branch structure includes five coding layers, and the coding layer consists of a convolutional layer, a batch normalization layer and a ReLU activation function. The first, second, third and fourth coding layers are connected through a global maximum pooling layer, and the fourth coding layer is directly connected to the fifth coding layer; the second coding layer outputs a low-dimensional feature map, and the fifth coding layer outputs a high-dimensional feature map.

3. The optical / SAR image change detection network based on multi-scale feature alignment according to claim 1, characterized in that: The structure of the multi-scale feature extraction module includes a first splicing layer, a first convolutional layer, a dual-branch pooling layer, an addition layer, a gating module, a weighted splicing layer, a Transformer module and a second convolutional layer; The first splicing layer, the first convolutional layer, the dual-branch pooling layer, the summing layer, and the gating module are connected in sequence. The splicing layer simultaneously inputs the low-dimensional feature maps of the optical image and the SAR image, processes them in sequence, and outputs the normalized channel attention weight vector in the gating module. The weighted splicing layer, Transformer module and the second convolutional layer are connected in sequence. The weighted splicing layer simultaneously inputs the low-dimensional feature maps of the optical image and SAR image and the normalized channel attention weight vector, processes them in sequence, and outputs the low-level multi-source fusion feature map in the second convolutional layer.

4. The optical / SAR image change detection network based on multi-scale feature alignment according to claim 3, characterized in that: The gating module is composed of a third convolutional layer, a ReLU activation function, a fourth convolutional layer and a Sigmoid function in sequence. The result of the addition layer is processed in sequence by the third convolutional layer, the ReLU activation function and the fourth convolutional layer to generate attention weights, and finally activated by the Sigmoid function to obtain a normalized channel attention weight vector.

5. The optical / SAR image change detection network based on multi-scale feature alignment according to claim 4, characterized in that: The dual-branch pooling layer is a pooling layer of different types, including: a global average pooling layer and a global maximum pooling layer. The global average pooling layer processes the result of the first splicing layer to extract overall features, and the global maximum pooling layer processes the result of the first splicing layer to extract local significant features.

6. The optical / SAR image change detection network based on multi-scale feature alignment according to claim 1, characterized in that: The structure of the high-dimensional feature selection module includes: two feature selection branches, a second splicing layer, a dilated spatial pyramid pooling module and a first upsampling layer; The high-dimensional feature maps of optical images and SAR images are respectively input into two feature selection branches for processing to obtain selected feature maps. The splicing layer, the atrous spatial pyramid pooling module and the first upsampling layer are sequentially connected. The two selected feature maps are input into the splicing layer and processed in sequence. The first upsampling layer outputs a high-level multi-source fusion feature map.

7. The optical / SAR image change detection network based on multi-scale feature alignment according to claim 6, characterized in that: The two feature selection branches are branches of the same structure, and a single branch includes an SE block and a global average pooling block, one of which is used to process the high-dimensional feature map of the optical image, and the other is used to process the high-dimensional feature map of the SAR image; The high-dimensional feature map uses SE blocks to adaptively model the dependencies between channels and enhance the response of important channels. Then, global average pooling is performed on each channel to obtain the channel importance score. A preset number of high-scoring channels are selected according to the channel importance score to obtain the selected feature map.

8. The optical / SAR image change detection network based on multi-scale feature alignment according to claim 1, characterized in that: The structure of the optical / SAR image change detection module includes: a third splicing layer, a decoding layer, a second upsampling layer and a softmax layer in sequence, wherein the decoding layer is composed of two convolutional layers.

9. A method for optical / SAR image change detection based on multi-scale feature alignment, characterized in that: include: Acquire optical and SAR images; Perform multi-scale feature extraction on optical images and SAR images respectively to obtain low-dimensional feature maps and high-dimensional feature maps of optical images and SAR images; Based on the low-dimensional feature maps of optical images and SAR images, low-dimensional feature fusion is performed to obtain low-level multi-source fusion feature maps; Based on the high-dimensional feature maps of optical images and SAR images, high-dimensional feature selection is performed to obtain a selected feature map, and the selected feature map is fused to obtain a high-level multi-source fusion feature map; Image change detection is performed based on low-level multi-source fusion feature maps and high-level multi-source fusion feature maps to obtain the final optical / SAR image change detection results.

10. The optical / SAR image change detection method based on multi-scale feature alignment according to claim 9, characterized in that: The high-dimensional feature selection is specifically as follows: The high-dimensional feature map uses SE blocks to adaptively model the dependencies between channels and enhance the response of important channels. Then, global average pooling is performed on each channel to obtain the channel importance score. A preset number of high-scoring channels are selected according to the channel importance score to obtain the selected feature map.

Images