Ground feature change detection method based on multi-level feature adaptive alignment network

Abstract

The invention provides a surface feature change detection method based on a multi-level feature adaptive alignment network, which can be applied to the technical field of computer vision. The method comprises the following steps: inputting a reference image and a target image into a trained multi-level feature adaptive alignment network, and performing depth feature extraction on the reference image and the target image by using a trunk feature extraction network to obtain a reference feature and a target feature; eliminating the difference between the same ground feature features in the target features and the reference features by using a feature step-by-step alignment architecture to obtain target reconstruction features; and obtaining a ground object change detection result by using a ground object change prediction module based on the target reconstruction feature and the reference feature. The adaptive correction and alignment of the same ground feature features of the two stages of images are completed step by step from coarse to fine by adopting a multi-stage residual error fitting method, the inconsistency of the same ground feature features is gradually eliminated, the problem of prediction false alarm caused by texture change, offset and the like is solved, and the detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision technologies such as deformable attention mechanism, target change detection / segmentation, remote sensing image processing, and more particularly to a method for detecting ground object changes based on a multi-level feature adaptive alignment network. Background Art

[0002] In recent years, the widespread use of high-resolution satellite data and aerial data has opened up new ways for fine-grained monitoring of land cover and land use. As one of the main research contents in the field of remote sensing, remote sensing object change detection aims to identify the difference information of object changes in remote sensing scenes by comparing two registered remote sensing images of the same geographical location taken at different times and assigning a binary label (i.e., changed or not changed) to each pixel. Due to the deterioration of the natural environment, remote sensing object change detection has become increasingly important in many fields, such as disaster monitoring, urban planning, environmental surveys, etc. In response to the need to process massive amounts of remote sensing data, high-accuracy and automated remote sensing object change detection algorithms can significantly reduce the cost and time consumption of manual interpretation, and therefore have received increasing attention.

[0003] At present, the ground object change detection method based on high-resolution optical remote sensing images is still a challenging task, which is mainly reflected in the following two aspects: 1) The complexity of ground objects in remote sensing scenes; 2) The inconsistency of remote sensing imaging conditions. It is precisely because of the complexity and diversity of remote sensing ground object scenes that when the platform imaging conditions (imaging angle, lighting conditions, position deviation and other factors) change, the same ground object in two different images will undergo significant changes in geometric texture, brightness and color, and even problems such as ground object position offset may occur. Limited by the above-mentioned remote sensing observation conditions, in actual applications, there are many unchanged ground object targets that change due to factors such as "texture changes", "color changes" and "position offsets", which lead to false alarms in detection, causing a sharp deterioration in the algorithm's false alarm rate and reducing the algorithm's detection accuracy, limiting the performance improvement of the current ground object change detection algorithm and its further widespread application in actual tasks. For the comparison of "color changes" of the same ground object in two remote sensing images, see Figure 1 , "Texture Change" comparison see Figure 2 and "position offset" comparison see Figure 3 .

[0004] At present, the mainstream remote sensing ground object change detection algorithms in academia are mainly divided into methods based on deep convolutional neural networks and methods based on Transformer networks. The methods based on deep convolutional neural networks focus on the extraction and fusion of contextual semantic features of ground object targets to cope with the complexity of ground object targets and changes in imaging conditions in remote sensing scenes. This type of method mainly extracts semantic features by inputting two phases of high-resolution images into the feature extraction network, then fuses features through a multi-level feature fusion structure, and finally makes recognition decisions. In order to obtain a larger network receptive field, existing methods include using deeper convolutional network models or stacking more convolutional layers, such as ResNet, etc.; dilated convolution and attention mechanism. The methods based on Transformer networks mainly introduce the Transformer structure to model the long-distance contextual semantic feature relationship between the two phases of images, and use high-level semantic features to guide the enhancement of the original pixel space feature expression ability, realize feature selective fusion, and improve the network recognition ability. The Transformer network was first proposed in 2017 and has been widely used in the field of natural language processing (NLP) to solve sequence-to-sequence prediction tasks. At present, the Transformer network structure has been widely used in the field of computer vision (CV) and has shown better performance than convolutional networks.

[0005] The above two methods are essentially to fully exploit the contextual semantic features of the image, so as to better model and fuse the contextual features of the two images in the space-time domain, improve the feature representation ability of the network model, and better solve the problem of complex ground object target feature extraction in remote sensing scenes, showing certain performance and application prospects. The above Transformer-based ground object change detection method adopts a hybrid architecture of convolutional network + Transformer network, with convolutional network as feature extractor and Transformer network as feature fusion processing module. At present, there is no ground object change detection method based on a complete Transformer structure. In the face of the above-mentioned problem of high false alarm of ground object change detection due to differences in imaging conditions, the above methods have not been designed and processed in a targeted manner. Therefore, the performance of the above methods is limited and the performance is not satisfactory, and there is still a lot of room for improvement. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] In order to solve the problems existing in the prior art of the ground object change detection method based on deep learning, the embodiment of the present invention provides a ground object change detection method based on a multi-level feature adaptive alignment network, which uses a ground object change detection network based on multi-level feature adaptive alignment, and adopts a multi-level residual fitting method to complete the adaptive correction and alignment of the same ground object features in the two images from coarse to fine, and gradually eliminate the inconsistency of the same ground object features. The test results show that this scheme can better solve the problem of prediction false alarms caused by texture changes, offsets, etc., and greatly improve the detection accuracy.

[0008] (II) Technical solution

[0009] In view of the above problems, an embodiment of the present invention provides a method for detecting ground object changes based on a multi-level feature adaptive alignment network.

[0010] According to a first aspect of the present invention, a method for detecting object changes based on a multi-level feature adaptive alignment network is provided, the method comprising: acquiring a reference image and a target image; inputting the reference image and the target image into a trained multi-level feature adaptive alignment network, wherein the multi-level feature adaptive alignment network comprises a backbone feature extraction network, a feature step-by-step alignment architecture and an object change prediction module; using the backbone feature extraction network, performing deep feature extraction on the reference image and the target image to obtain reference features and target features; using the feature step-by-step alignment architecture, eliminating the differences between the same object features in the target features and the reference features to obtain target reconstruction features; and based on the target reconstruction features and the reference features, using the object change prediction module to predict object changes to obtain object change detection results.

[0011] In some exemplary embodiments, the backbone feature extraction network includes a lightweight Swin Transformer network, wherein the Swin Transformer network includes four levels of stacked feature extraction modules, namely, Stage_0 module, Stage_1 module, Stage_2 module and Stage_3 module.

[0012] In some exemplary embodiments, a backbone feature extraction network is used to perform deep feature extraction on a reference image and a target image to obtain reference features and target features, respectively, including: setting stacking parameters of four modules according to preset parameters; outputting corresponding levels of ground object target features and reference features in the Stage_1 module, the Stage_2 module, and the Stage_3 module, respectively.

[0013] In some exemplary embodiments, the feature level-by-level alignment architecture includes a multi-level adaptive feature alignment module from coarse to fine, wherein the adaptive feature alignment module includes Φ net The network and multi-head attention mechanism structure use the feature level-by-level alignment architecture to eliminate the differences between the target features and the reference features. Specifically, it includes: obtaining the sampling information offset of the feature alignment module of the previous level; based on the sampling information offset of the feature alignment module of the previous level, using the bilinear interpolation algorithm, sampling the target features in three-dimensional space to generate the first deformable sampled ground object features; based on the first deformable sampled ground object features and the reference features, using Ф net network, generate a three-dimensional sampling offset residual; add the three-dimensional sampling offset residual to the sampling information offset of the previous level feature alignment module to obtain the sampling information offset of the current feature alignment module; based on the position information in the sampling information offset of the current feature alignment module, use a bilinear interpolation algorithm to perform feature sampling on the target feature in three-dimensional space to generate a second deformable sampling of the ground object target feature; and use a multi-head attention mechanism structure to selectively aggregate and reconstruct the second deformable sampling of the ground object target feature and the reference feature to generate and output the target reconstruction feature.

[0014] In some exemplary embodiments, Ф net The network includes a 7*7 convolutional layer, a normalization layer, an activation layer, and a 1*1 convolutional layer. Based on the ground object features and reference features of the first deformable sampling, the Ф net The network generates a three-dimensional sampling offset residual, including: using a 7*7 convolution layer and a normalization layer to capture the local features of the image; based on the local features of the image, using an activation layer and a 1*1 convolution layer to generate a three-dimensional sampling offset residual.

[0015] In some exemplary embodiments, the land object change prediction module includes a classification network, which includes a first convolutional layer, a normalization layer, an activation layer, and a second convolutional layer stacked in sequence. Based on the target reconstruction feature and the reference feature, the land object change prediction module is used to predict the land object change to obtain a land object change detection result, which specifically includes: obtaining the target reconstruction feature of the current feature alignment module and its corresponding reference feature; performing pixel-by-pixel absolute value subtraction on the target reconstruction feature and its corresponding reference feature to obtain a change feature map of the current feature alignment module; adding the change feature maps of all feature alignment modules pixel by pixel to obtain a total change feature map; upsampling the total change feature map by 4 times to obtain a doubled change feature map; and inputting the doubled change feature map into the classification network to obtain a land object change detection result.

[0016] In some exemplary embodiments, training a multi-level feature adaptive alignment network includes: acquiring a reference image, a target image, and a ground object change truth map; based on the reference image, the target image, and the ground object change truth map, training the multi-level feature adaptive alignment network using a cross entropy loss function, wherein the multi-level feature adaptive alignment network includes a parameter-initialized data set classification network.

[0017] The second aspect of the present invention provides a ground object change detection device based on a multi-level feature adaptive alignment network, comprising: an acquisition module, used to acquire a reference image and a target image; an input module, used to input the reference image and the target image into a trained multi-level feature adaptive alignment network, wherein the multi-level feature adaptive alignment network includes a backbone feature extraction network, a feature step-by-step alignment architecture, and a ground object change prediction module; a feature extraction module, used to use the backbone feature extraction network to perform deep feature extraction on the reference image and the target image to obtain reference features and target features; a feature alignment module, used to use the feature step-by-step alignment architecture to eliminate the differences between the same ground object features in the target features and the reference features to obtain target reconstruction features; and a result detection module, used to predict ground object changes based on the target reconstruction features and the reference features using the ground object change prediction module to obtain ground object change detection results.

[0018] A third aspect of the present invention provides an electronic device, comprising: one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0019] The fourth aspect of the present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above method.

[0020] (III) Beneficial effects

[0021] It can be seen from the above technical solutions that the method for detecting changes in objects based on a multi-level feature adaptive alignment network provided by the embodiments of the present invention has at least one of the following beneficial effects:

[0022] (1) A ground object change detection method using a full Transformer network is proposed. The lightweight Swin Transformer network is used to completely replace the convolutional network as a feature extractor to achieve feature extraction of complex remote sensing ground object images. A ground object change detection network based on multi-level feature adaptive alignment is constructed, abandoning the conventional method of using convolutional networks as feature extractors. This method effectively solves the problem of high false alarm of ground object change detection caused by the difference / inconsistency of the features of the same ground object in two phases of images, and comprehensively improves the overall performance of the algorithm.

[0023] (2) An adaptive feature alignment module (AFAM) based on a deformable attention mechanism is proposed. The adaptive feature aggregation characteristics of the deformable attention mechanism are utilized to realize adaptive sampling, selective aggregation and alignment of the ground object features of the two phases of images under data-driven conditions. This can eliminate the inconsistency of the same ground object features and improve the adaptability of the algorithm to changes in the same ground object features.

[0024] (3) A feature alignment architecture from coarse to fine and from low to high was constructed: Based on the idea of ​​multi-level residual fitting, three groups of AFAM modules were hierarchically embedded in the backbone network, and the sampling offsets were coupled step by step in a residual manner. The AFAM modules were used to fit and align the inconsistencies of the same ground object features caused by deformation or displacement. Overall, the three groups of AFAM modules were stacked from top to bottom to form a hierarchical feature alignment architecture from coarse to fine and from low to high levels, so as to continuously eliminate the differences and inconsistencies between the features of the same ground object, greatly reducing the false alarm rate of ground object changes and improving the overall performance of the algorithm.

[0025] (4) Compared with the current mainstream ground object change detection methods based on deep learning, the AlignFormer algorithm proposed in the embodiments of the present invention shows obvious advantages in comprehensive indicators such as accuracy (Pre%), recall rate (Rec%) and F1-Score (%). BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0027] Figure 1 A schematic diagram showing the comparison of the "color changes" of the same ground objects in two phases of remote sensing images;

[0028] Figure 2 A schematic diagram showing the comparison of the "texture changes" of the same ground objects in two phases of remote sensing images;

[0029] Figure 3 A schematic diagram showing the comparison of the “position offset” of the same ground object in two phases of remote sensing images;

[0030] Figure 4 A schematic diagram of a process flow of a method for detecting changes in ground objects based on a multi-level feature adaptive alignment network according to an embodiment of the present invention is shown;

[0031] Figure 5 A schematic diagram of a multi-level feature adaptive alignment network according to an embodiment of the present invention is shown;

[0032] Figure 6A schematic diagram of a structure of an adaptive feature alignment module according to an embodiment of the present invention is shown;

[0033] Figure 7 A schematic diagram of a process of eliminating the difference between the same ground feature in the target feature and the reference feature by using the feature level-by-level alignment architecture according to an embodiment of the present invention is shown;

[0034] Figure 8 A schematic diagram of a process of predicting ground object changes based on target reconstruction features and reference features using a ground object change prediction module to obtain ground object change detection results according to an embodiment of the present invention is shown;

[0035] Fig. 9 A schematic structural block diagram of a ground feature change detection device 800 based on a multi-level feature adaptive alignment network according to an embodiment of the present invention is shown; and

[0036] Fig.10 A block diagram of an electronic device for a method for detecting changes in land objects based on a multi-level feature adaptive alignment network according to an embodiment of the present invention is schematically shown.

[0037] Illustration Description:

[0038] 600-electronic device; 601-processor; 602-read only memory (ROM); 603-random access memory (RAM); 604-bus; 605-input / output (I / O) interface; 606-input part; 607-output part; 608-storage part; 609-communication part; 610-drive; 611-removable medium. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Figure 4 The flowchart of a method for detecting changes in ground objects based on a multi-level feature adaptive alignment network according to an embodiment of the present invention is schematically shown.

[0041] like Figure 4 As shown, a method for detecting changes in objects based on a multi-level feature adaptive alignment network according to an embodiment of the present invention includes steps S110-S150.

[0042] In step S110 , a reference image and a target image are acquired.

[0043] In step S120, the reference image and the target image are input into the trained multi-level feature adaptive alignment network, wherein the multi-level feature adaptive alignment network includes a backbone feature extraction network, a feature level-by-level alignment architecture, and a ground object change prediction module. The structure of the multi-level feature adaptive alignment network is shown in Figure 5 . A ground object change detection network based on multi-scale feature adaptive alignment (Multi-scale Residual Alignment Transformer, referred to as Alignformer).

[0044] In step S130, a backbone feature extraction network is used to perform deep feature extraction on the reference image and the target image to obtain reference features and target features.

[0045] In some exemplary embodiments, the backbone feature extraction network includes a lightweight Swin Transformer network, wherein the Swin Transformer network includes four levels of feature extraction modules stacked, namely, Stage_0 module, Stage_1 module, Stage_2 module, and Stage_3 module. Step S130 includes setting stacking parameters of the four modules according to preset parameters; and outputting corresponding levels of ground object target features and reference features in the Stage_1 module, the Stage_2 module, and the Stage_3 module, respectively.

[0046] For example, the backbone feature extraction network uses a lightweight Swin Transformer network instead of a convolutional network as the backbone feature extraction network, and completes the two-phase complex image (reference image / Ref Image, I r Target Image, I t ; The image size is H×W×3), the lightweight Swin Transformer network is mainly composed of four hierarchical feature extraction modules (Stage_0 module, Stage_1 module, Stage-2 module and Stage_3 module), where low-level features contain rich local detail features, and high-level features contain more global high-level semantic features. The number of module stacking parameters for each stage from the shallow to the deep layer of the network is set to (2,2,2,2), respectively, and the corresponding level of ground object target features are output after the Stage_1 module, the Stage_2 module and the Stage_3 module network stages. With reference feature :

[0047] (1)

[0048] (2)

[0049] (3)

[0050] Among them, h=H / 8, w=W / 8, C1=192, C2=384 and C3=769.

[0051] In step S140 , the feature level-by-level alignment architecture is used to eliminate the differences between the target feature and the same ground feature in the reference feature to obtain the target reconstructed feature.

[0052] In some exemplary embodiments, the feature level-by-level alignment architecture includes a multi-level adaptive feature alignment module (AFAM) from coarse to fine, wherein the adaptive feature alignment module includes Φ net Network and multi-head attention mechanism structure, see Figure 6 Step S140 includes steps S141-S146, see Figure 7 .

[0053] In step S141, the sampling information offset of the previous level feature alignment module is obtained.

[0054] In step S142, based on the sampling information offset of the previous level feature alignment module, a bilinear interpolation algorithm is used to perform feature sampling on the target feature in three-dimensional space to generate the first deformable sampled ground object target feature.

[0055] In step S143, based on the ground object features and reference features of the first deformable sampling, Φ net Network,generates 3D sampling offset residuals.

[0056] In step S144, the three-dimensional sampling offset residual is added to the sampling information offset of the previous level feature alignment module to obtain the sampling information offset of the current feature alignment module.

[0057] In step S145, based on the position information in the sampling information offset of the current feature alignment module, a bilinear interpolation algorithm is used to perform feature sampling on the target feature in three-dimensional space to generate a second deformable sampled ground object target feature.

[0058] In step S146, a multi-head attention mechanism structure is used to selectively aggregate and reconstruct the target features of the second deformable sampling and the reference features to generate and output the target reconstructed features.

[0059] For example, Figure 5As shown on the right side of the network architecture diagram, based on the multi-level residual fitting idea, a feature alignment framework from coarse to fine is constructed, making full use of the advantages of multi-level features. The three groups of AFAM-1, AFAM-2 and AFAM-3 feature alignment modules are embedded in the backbone network respectively; the AFAM_l module generates a coarse feature sampling offset based on the upper level AFAM_l-1 module. Prior information, combined with its own prediction of fine offset residuals The deformable attention mechanism is used to fit and align the inconsistencies in the features of the same object in two images caused by deformation or displacement. Overall, the three groups of AFAM modules are stacked from top to bottom to form a hierarchical feature alignment architecture from coarse to fine, so as to continuously eliminate the differences and inconsistencies between the features of the same object.

[0060] The input of AFAM_l module is and :

[0061] (4)

[0062] (5)

[0063] Among them, Conv() represents a 1*1 convolutional layer, which is used to compress the number of feature channels; U2() represents a two-fold upsampling operation on the feature map; Concat() represents a concatenation operation along the channel dimension of the feature map.

[0064] Further, the structure of the AFAM module according to the embodiment of the present invention is shown in Figure 6 The Adaptive Feature Alignment (AFAM) module based on the deformable attention mechanism can solve the problem that the features of the same object in the two images are inconsistent due to factors such as texture changes and position offsets, resulting in a high false alarm rate. By using its feature adaptive alignment capability, it can use the reference features in the reference image (Ref Image) ) as the benchmark, the target image (Target Image) is flexibly focused and selectively resampled under data-driven conditions. ), the AFAM module completes feature aggregation by adaptively learning feature weights, thereby fitting and eliminating the texture

[0065] Differences and inconsistencies in the characteristics of ground objects caused by changes or displacements.

[0066] 1) AFAM model input

[0067] like Figure 6 As shown, the input of the first layer AFAM_1 module specifically includes the ground object target features Reference feature map And the sampling offset output by the upper level AFAM_l-1 module , where l∈{1,2,3}:

[0068] (6)

[0069] (7)

[0070] (8)

[0071] in, represents the ground object target features output by the first layer of the backbone feature network, represents the ground feature reference features output by the first layer of the backbone feature network, Represents the sampling offset of the AFAM_l-1 module, where Indicates the position offset of the target feature relative to the reference feature of the same feature.

[0072] 2) Feature deformation sampling based on offset residual

[0073] According to the rough sampling 3D offset of the previous level AFAM_l-1 module The provided position prior information is used, bilinear interpolation is used, and the Wrap function is used to wrap the target features. Perform feature sampling in the height-width-channel three-dimensional space to generate the first deformable sampling of ground object target features , where the Wrap function is used to place each selected element in the specified HTML content or element:

[0074] (9)

[0075] Next, the deformed sampled ground feature and reference features After the Concat operation, input Ф net The network is calculated and generates a fine sampling offset residual ; In this scheme, Ф net The network structure first uses a 7*7 convolutional layer and an LN layer to capture local features of the image; then a GELU activation layer and a 1*1 convolutional layer are used to calculate and generate the three-dimensional sampling offset residual. (The position offset covers the height-width-channel three-dimensional space range):

[0076] (10)

[0077] Based on 3D migration residual Calculate the sampling offset generated by the l-th layer AFAM module ,in, Indicates the position offset of the target feature relative to the reference feature of the same object:

[0078] (11)

[0079] Based on sampling offset The position information of the target features is obtained by using bilinear interpolation method. Perform feature sampling in the height-width-channel three-dimensional space to generate the second deformable sampling of ground object target features :

[0080] (12)

[0081] 3) Feature Adaptive Aggregation and Reconstruction

[0082] Finally, the target features after the second deformable sampling are And reference features Input the multi-head self-attention mechanism (MSA) to selectively aggregate and reconstruct the target features, generate and output the target reconstructed features :

[0083] (13)

[0084] in, , , .

[0085] In step S150, based on the target reconstruction features and the reference features, the ground object change prediction module is used to predict the ground object change to obtain a ground object change detection result.

[0086] In some exemplary embodiments, the ground feature change prediction module includes a classification network, the classification network includes a first convolution layer, a normalization layer, an activation layer, and a second convolution layer stacked in sequence, and step S150 includes steps S151-S155, see Figure 8 .

[0087] In step S151, the target reconstruction feature of the current feature alignment module and its corresponding reference feature are obtained.

[0088] In step S152, pixel-by-pixel absolute value subtraction is performed on the target reconstructed feature and its corresponding reference feature to obtain a change feature map of the current feature alignment module.

[0089] In step S153, the change feature maps of all feature alignment modules are added pixel by pixel to obtain a total change feature map.

[0090] In step S154, the total variation characteristic graph is upsampled by a factor of 4 to obtain a doubled variation characteristic graph.

[0091] In step S155, the doubled change feature map is input into the classification network to obtain the ground object change detection result.

[0092] For example, the ground feature change prediction module according to an embodiment of the present invention can be seen in Figure 5 , the output of the l-th level AFAM_l module is , by comparing with the reference feature Perform pixel-by-pixel absolute value subtraction to obtain the change feature map :

[0093] (14)

[0094] Then the three sets of feature maps Add pixel by pixel, And perform 4 times upsampling, and finally input the features into the classification network for prediction output:

[0095] (15)

[0096] in, Indicates that the feature map is quadrupled. Represents a classification network, which is mainly composed of a 1*1 convolutional layer, a BN normalization layer, a RELU activation layer, and a 1*1 convolutional layer stacked in sequence.

[0097] According to the training method of the ground object change detection network based on multi-level feature adaptive alignment according to the embodiment of the present invention, the network training is carried out with the input reference image, target image and ground object change true value map, the cross entropy function is used as the loss function, the network is initialized with the Image 100K dataset classification network pre-training model, the AdamW algorithm is used as the network parameter optimizer, and the initial value of the learning rate is set to 5.0×10 −3 And following the linear attenuation rule, the number of network training Epoch iterations is set to 200.

[0098] In order to test the ground object change detection network based on multi-level feature adaptive alignment according to the embodiment of the present invention, the public dataset LEVIR-CD was selected as the test data, and the current mainstream and advanced ground object change detection algorithms in the academic community were compared to verify the effectiveness of this method. The selected detection algorithms include: 1) methods based on convolutional networks, a total of 10; 2) methods based on convolutional networks + Transformer hybrid models, a total of 1; AlignFormer algorithm proposed in the embodiment of the present invention. Comprehensive evaluation indicators are used to measure the comprehensive performance of the algorithm, including accuracy (Pre%), recall (Rec%) and F1 score (F1-Score%), among which,

[0099] Pre = TP / (TP + FP) (16)

[0100] Rec=TP / (TP+FN) (17)

[0101] Among them, TP represents the number of true positive pixel samples in the prediction image, FP represents the number of false positive pixel samples in the prediction image, and FN represents the number of false negative pixel samples in the prediction image.

[0102] In order to reduce the false alarm rate of ground object change detection, an embodiment of the present invention proposes an adaptive feature alignment module based on deformable attention mechanism (Adaptive Feature Alignment module based DeformableAttention, abbreviated as AFAM) to solve the problem of inconsistency of ground object features in two phases of images caused by factors such as "texture change", "color change" and "position offset". The adaptive feature sampling and aggregation characteristics of the deformable attention mechanism are used to realize adaptive sampling, selective aggregation and alignment of ground object features in two phases of images under data-driven, which can eliminate the inconsistency of the same ground object features and improve the adaptability of the algorithm to the changes of the same ground object features in the two phases of images. At the same time, the embodiment of the present invention uses a lightweight Swin Transformer network as the backbone feature extraction network, and hierarchically embeds three groups of AFAM modules into the backbone network to form a ground object change detection network based on multi-level feature adaptive alignment (Multi-scale ResidualAlignmentTransformer, referred to as AlignFormer, realizes the step-by-step feature fitting alignment from coarse to fine and from low-level to high-level features by adding and connecting the sampling offsets step by step in a residual manner. Finally, the aligned multi-level feature maps are subtracted position by position to generate change feature maps, which are then sent to the classifier to complete the prediction of ground object change detection results. This method abandons the conventional method of using convolutional networks as feature extractors and adopts a complete Transformer structure for ground object change detection tasks. It can solve the problem of high false alarm rate of change detection caused by the above problems and comprehensively improve the overall performance of the algorithm.

[0103] The (partial) performance test results of different algorithms on the LEVIR-CD dataset are shown in Table 1.

[0104] Table 1 Comparison of performance test results of different algorithms on the LEVIR-CD dataset

[0105]

[0106] The test results of public data sets show that compared with the most advanced ground object change detection method, the method of the present invention can effectively reduce the false alarm rate of ground object changes, greatly improve the accuracy of ground object change detection, and the algorithm performance has obvious advantages.

[0107] Fig. 9 The structure block diagram of a ground object change detection device 800 based on a multi-level feature adaptive alignment network according to an embodiment of the present invention is schematically shown.

[0108] like Fig. 9As shown, a ground feature change detection device 800 based on a multi-level feature adaptive alignment network according to an embodiment of the present invention includes an acquisition module 810, an input module 820, a feature extraction module 830, a feature alignment module 840 and a result detection module 850.

[0109] The acquisition module 810 is used to acquire a reference image and a target image.

[0110] The input module 820 is used to input the reference image and the target image into the trained multi-level feature adaptive alignment network, wherein the multi-level feature adaptive alignment network includes a backbone feature extraction network, a feature level-by-level alignment architecture, and a ground object change prediction module.

[0111] The feature extraction module 830 is used to use the backbone feature extraction network to perform deep feature extraction on the reference image and the target image to obtain reference features and target features.

[0112] The feature alignment module 840 is used to eliminate the differences between the target features and the same ground features in the reference features by using the feature level-by-level alignment architecture to obtain the target reconstructed features.

[0113] The result detection module 850 is used to predict the ground object change based on the target reconstruction feature and the reference feature using the ground object change prediction module to obtain the ground object change detection result.

[0114] In some specific embodiments, at least one of the acquisition module 810, the input module 820, the feature extraction module 830, the feature alignment module 840, and the result detection module 850 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging the circuit, or in any one of the three implementation modes of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 810, the input module 820, the feature extraction module 830, the feature alignment module 840, and the result detection module 850 may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function may be executed.

[0115] Fig.10 A block diagram of an electronic device for a method for detecting changes in land objects based on a multi-level feature adaptive alignment network according to an embodiment of the present invention is schematically shown.

[0116] like Fig.10As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an on-board memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0117] In RAM603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM602 and RAM603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to an embodiment of the present invention by executing the program in ROM602 and / or RAM603. It should be noted that the program can also be stored in one or more memories other than ROM602 and RAM603. Processor 601 can also perform various operations of the method flow according to an embodiment of the present invention by executing the program stored in one or more memories.

[0118] In some specific embodiments, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0119] An embodiment of the present invention shows a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0120] In some specific embodiments, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM602 and / or RAM603 described above and / or one or more memories other than ROM602 and RAM603.

[0121] An embodiment of the present invention shows a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0122] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0123] In some specific embodiments, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0124] In some specific embodiments, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0125] In some specific embodiments, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0126] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0128] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for detecting ground object changes based on a multi-level feature adaptive alignment network, characterized in that: The method comprises: Acquire a reference image and a target image; Inputting the reference image and the target image into a trained multi-level feature adaptive alignment network, wherein the multi-level feature adaptive alignment network includes a backbone feature extraction network, a feature level-by-level alignment architecture, and a ground object change prediction module; Using the backbone feature extraction network, deep feature extraction is performed on the reference image and the target image to obtain reference features and target features; Using the feature level-by-level alignment architecture, the difference between the target feature and the same ground feature in the reference feature is eliminated to obtain a target reconstructed feature; and Based on the target reconstruction feature and the reference feature, a ground object change prediction module is used to predict the ground object change to obtain a ground object change detection result.

2. The method according to claim 1, characterized in that The backbone feature extraction network includes a lightweight Swin Transformer network, wherein the Swin Transformer network includes four levels of feature extraction modules stacked together, namely, a Stage_0 module, a Stage_1 module, a Stage_2 module, and a Stage_3 module.

3. The method according to claim 2, characterized in that The use of the backbone feature extraction network to perform deep feature extraction on the reference image and the target image to obtain reference features and target features respectively specifically includes: Set the stacking parameters of the four modules according to the preset parameters; The Stage_1 module, the Stage_2 module and the Stage_3 module output the corresponding level of ground object target features and reference features respectively.

4. The method according to claim 3, characterized in that The feature level-by-level alignment architecture includes a multi-level adaptive feature alignment module from coarse to fine, wherein the adaptive feature alignment module includes Ф net Network and multi-head attention mechanism structure, The step of using the feature-by-feature alignment architecture to eliminate the difference between the target feature and the same ground feature in the reference feature specifically includes: Get the sampling information offset of the previous level feature alignment module; Based on the sampling information offset of the previous level feature alignment module, the target feature is sampled in three-dimensional space using a bilinear interpolation algorithm to generate a first deformable sampled ground object target feature; Based on the ground object feature of the first deformable sampling and the reference feature, using the Φ net Network,generates 3D sampling offset residuals; Adding the three-dimensional sampling offset residual to the sampling information offset of the previous level feature alignment module to obtain the sampling information offset of the current feature alignment module; Based on the position information in the sampling information offset of the current feature alignment module, the target feature is sampled in three-dimensional space using a bilinear interpolation algorithm to generate a second deformable sampled ground object target feature; and The multi-head attention mechanism structure is used to selectively aggregate and reconstruct the ground object target features of the second deformable sampling and the reference features to generate and output target reconstruction features.

5. The method according to claim 4, characterized in that The Φ net The network includes a 7*7 convolutional layer, a normalization layer, an activation layer, and a 1*1 convolutional layer. The ground object target feature based on the first deformable sampling and the reference feature, using the Φ net Network, generates three-dimensional sampling offset residuals, including: Using the 7*7 convolutional layer and the normalization layer to capture local features of the image; Based on the local features of the image, the activation layer and the 1*1 convolution layer are used to generate a three-dimensional sampling offset residual.

6. The method according to claim 1, characterized in that The ground feature change prediction module includes a classification network, and the classification network includes a first convolution layer, a normalization layer, an activation layer, and a second convolution layer stacked in sequence. The method of predicting the ground object change based on the target reconstruction feature and the reference feature using a ground object change prediction module to obtain a ground object change detection result specifically includes: Obtain the target reconstruction feature of the current feature alignment module and its corresponding reference feature; Performing pixel-by-pixel absolute value subtraction on the target reconstructed feature and its corresponding reference feature to obtain a change feature map of the current feature alignment module; After adding the change feature maps of all feature alignment modules pixel by pixel, the total change feature map is obtained; Upsampling the total variation characteristic graph by a factor of 4 to obtain a doubled variation characteristic graph; and The doubled change feature map is input into the classification network to obtain a ground object change detection result.

7. The method according to claim 1, characterized in that Training the multi-level feature adaptive alignment network includes: Obtain reference images, target images, and ground object change truth maps; Based on the reference image, the target image and the ground object change true value map, the multi-level feature adaptive alignment network is trained using a cross entropy loss function, Wherein, the multi-level feature adaptive alignment network includes a parameter-initialized data set classification network.

8. A ground object change detection device based on a multi-level feature adaptive alignment network, characterized in that: The device comprises: An acquisition module, used for acquiring a reference image and a target image; An input module, used for inputting the reference image and the target image into a trained multi-level feature adaptive alignment network, wherein the multi-level feature adaptive alignment network includes a backbone feature extraction network, a feature level-by-level alignment architecture, and a ground object change prediction module; A feature extraction module, used to use the backbone feature extraction network to perform deep feature extraction on the reference image and the target image to obtain reference features and target features; A feature alignment module, used to eliminate the difference between the target feature and the same ground feature in the reference feature by using the feature level-by-level alignment architecture, so as to obtain a target reconstructed feature; and The result detection module is used to predict the ground object change based on the target reconstruction feature and the reference feature using the ground object change prediction module to obtain the ground object change detection result.

9. An electronic device, wherein: include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 7.

Images