A multi-scale changing target retrieval method based on spatiotemporal modeling

Abstract

A multi-scale changing object retrieval method based on spatiotemporal modeling includes: acquiring data with multiple spatial and temporal dimensions and segmenting it into several subgraphs; extracting features from each subgraph using a pretrained CNN or Transformer model to obtain an abstract feature map; applying a cross-temporal joint global attention mechanism to the abstract feature maps obtained from corresponding subgraphs in an image pair to obtain a corresponding interaction feature map; performing a dual-branch joint pixel-by-pixel subtraction and parameterized learning on the interaction feature map to obtain a corresponding difference feature map; performing multi-scale fusion on the difference feature maps obtained from corresponding subgraphs in the image pair to obtain a fused feature map; and sequentially upsampling and convolutional transformation on the fused feature maps obtained from corresponding subgraphs in the image pair to obtain a change detection map. This invention helps improve the accuracy and generalization performance of multi-scale changing object retrieval.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a multi-scale changing target retrieval method based on space-time modeling. Background Art

[0002] Multi-scale changing object retrieval refers to the process of identifying changes at the same spatial location from different time series. As a key technology for monitoring regional conditions, changing object retrieval plays a significant role in numerous tasks, including urban construction mapping, natural disaster assessment, and environmental change monitoring. Since time series data are captured at different times, there may be a large amount of task-irrelevant interference between them, such as seasonal changes, varying exposure levels, and building renovations. In such cases, fast, stable, and robust methods are needed to process multi-dimensional spatial and temporal data acquired at different times.

[0003] Traditional methods for retrieval of changing objects, based on manually extracted features, do not provide ideal representational results. Deep learning networks, however, can capture abstract features through nonlinear operations, making them more representative. For example, convolutional neural networks (CNNs) can extract local features from data using a sliding window mechanism, while Transformers rely on a self-attention mechanism to dynamically compute global context. However, CNNs focus only on local features and ignore global dependencies, while Transformers capture similar global features at each layer, resulting in redundancy. Therefore, in this paper, we propose implementing an improved self-attention mechanism on convolutional feature maps to capture features with different representations. Summary of the Invention

[0004] The purpose of the present invention is to address the above problems and propose a multi-scale changing target retrieval method based on spatio-temporal modeling. It aims to retrieve multi-scale changing targets in spatial-temporal multi-dimensional data, suppress irrelevant differences through a cross-temporal joint global attention mechanism, and construct a dual-branch learning difference of joint pixel-by-pixel subtraction and parameterized learning. Finally, multi-level feature fusion is completed through a lightweight global spatial attention mechanism, which helps to improve the accuracy and generalization performance of multi-scale changing target retrieval.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention proposes a multi-scale variable target retrieval method based on space-time modeling, which includes the following steps:

[0007] S1, split the remote sensing image pair X1 and X2 into several image blocks of the same spatial size;

[0008] S2. Use the pre-trained network ResNet18 model to extract abstract feature maps F1 and F2 for each sub-graph. The output sizes of each stage of the ResNet18 model include 4W×4H×d, 2W×2H×d, and W×H×d, where W, H, and d are width, height, and channel depth, respectively.

[0009] S3. Model F1 and F2 together through a cross-temporal joint global attention mechanism. The cross-temporal joint global attention mechanism operates as follows:

[0010] S31. Linearly expand the feature maps F1 and F2 of the same size to obtain vector groups (Q1, K1, V1) and (Q2, K2, V2) in the form of (query Q vector, key K vector, value V vector). Taking W×H×d as an example, the sizes of Q, K and V are all N×d, where N=W×H.

[0011] S32, compress Q1 and Q2 into (N / 2)×d through a linear function, and then concatenate them into a new query vector Q joint , size is N×d;

[0012] S33, based on the exchanged query vector Q joint Calculate the global attention output F respectively with the key-value pair vector groups (K1, V1) and (K2, V2) in step S31 joint-1 and F joint-2 , the dimensions are W×H×d;

[0013] S4. For feature maps F of the same size joint-1 and F joint-2 Perform a two-branch difference capture operation with joint pixel-wise subtraction and parameterized learning to obtain the difference feature map F sub and F cat ;

[0014] The dual-branch difference capture operation is specifically as follows:

[0015] S41. In the pixel-by-pixel subtraction branch, calculate the feature map F of the same size joint-1 and F joint-2 The pixel-by-pixel difference between them is used to obtain the difference feature map F sub ;

[0016] S42. In the parameterized learning branch, we first stack the feature maps F of the same size along the channel direction. joint-1 and F joint-2 Then, multiple convolutional layers with the same size are used to capture nonlinear features and obtain the difference feature map F cat ;

[0017] S5. Difference feature map F for multiple sizessub and F cat , using lightweight global spatial attention for multi-scale feature map fusion, with multi-scale difference feature maps F including sizes W×H×d, 2W×2H×d and 4W×4H×d sub For example, the feature map fusion operation is as follows:

[0018] S51, the feature map F of size W×H×d sub-1 Interpolation upsampling and feature map F with size 2W×2H×d sub-2 Stacking is done along the channel direction, and then preliminary fusion is performed through lightweight global spatial attention;

[0019] S52, interpolate and upsample the feature map initially fused in step S51, and combine it with the feature map F of size 4W×4H×d sub-3 Stacking along the channel direction and then further fusion is completed through lightweight global spatial attention;

[0020] S6. Finally, the feature map after feature fusion is interpolated and upsampled to the size of the image block in step S1, and the changed target pixels are changed to 1 and the non-changed area pixels are 0 through convolution transformation and Sigmoid function to complete the changed target detection.

[0021] Preferably, in step S1, the size of the image block is 256×256.

[0022] Preferably, in step S2, the output sizes of each stage of the ResNet18 model are 64×64×64, 32×32×128 and 16×16×256.

[0023] Preferably, in step 32, Q1 and Q2 are compressed into (N / 2)×d by a linear function, and then concatenated into a new query vector Q joint , the calculation formula is as follows:

[0024] Q joint =Concat(Linear(Q1),Linear(Q2)) (1)

[0025] Among them, Linear(·) is a linear function, and Concat(·) is a concatenation operation along the channel direction.

[0026] Preferably, in step 33, the query vector Q after the exchange joint Calculate the global attention output F respectively with the key-value pair vector groups (K1, V1) and (K2, V2) in step S31 joint-1 and F joint-2 , the calculation formula is as follows:

[0027]

[0028]

[0029] Among them, Softmax(·) is a nonlinear mapping function, and T is a transposition operation.

[0030] Preferably, in step 42, in the branch of parameterized learning, first stack the feature maps F of the same size along the channel direction. joint-1 and F joint-2 Then, multiple convolutional layers with the same size are used to capture nonlinear features and obtain the difference feature map F cat , the calculation formula is as follows:

[0031] F cat =Conv(Concat(F joint-1 ,F joint-2 )) (4)

[0032] Among them, Conv(·) is the convolution operation.

[0033] Preferably, in step 51, the feature map F with a size of W×H×d is sub-1 Interpolation upsampling and feature map F with size 2W×2H×d sub-2 The stacking is performed along the channel direction, and then the initial fusion is completed through lightweight global spatial attention. The calculation formula is as follows:

[0034] F fuse =LSA(Concat(F sub-1 ,F sub-2 )) (5)

[0035] LSA(X)=Softmax(Linear(Q)·Pooling(K T ))·Pooling(V) (6)

[0036] Where Pooling(·) is the spatial pooling operation.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] This method aims to retrieve multi-scale changing targets in data with spatial-temporal multi-dimensionality, and uses a cross-temporal joint global attention mechanism to suppress task-irrelevant differences between data. It also performs differential learning by combining pixel-by-pixel subtraction and parameterized learning to improve the model's potential while maintaining stability and reliability. Finally, a lightweight global spatial attention mechanism is used to complete multi-level feature fusion, improving positioning while refining boundaries, thereby achieving accurate retrieval of changing targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of multi-scale variable target retrieval based on spatiotemporal modeling of the present invention;

[0040] Figure 2 Flowchart of the cross-temporal joint global attention mechanism of the present invention;

[0041] Figure 3 Flowchart of the lightweight global spatial attention mechanism of the present invention.

[0042] Figure 4 2 is a diagram showing the detection results of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0044] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application.

[0045] This application targets the spatial-temporal multi-dimensional characteristics of data in multi-scale changing target retrieval, and achieves accurate multi-scale changing target retrieval by constructing a cross-temporal joint global attention mechanism, a dual-branch joint pixel-by-pixel subtraction and parameterized learning, and a lightweight global spatial attention mechanism.

[0046] like Figure 1-4 As shown in FIG, the multi-scale changing target retrieval method based on space-time modeling includes the following steps:

[0047] S1, split the remote sensing image pair X1 and X2 into several image blocks of the same spatial size;

[0048] In one embodiment, in step S1, the size of the sub-image is 256×256, or can be adjusted according to actual needs.

[0049] S2. Use the pre-trained network ResNet18 model to extract abstract feature maps F1 and F2 for each sub-graph. The output sizes of each stage of the ResNet18 model include 4W×4H×d, 2W×2H×d, and W×H×d, where W, H, and d are width, height, and channel depth, respectively.

[0050] In one embodiment, in step S2, the output sizes of each stage of the ResNet18 model are 64×64×64, 32×32×128, and 16×16×256.

[0051] S3. Model F1 and F2 together through a cross-temporal joint global attention mechanism. The cross-temporal joint global attention mechanism operates as follows:

[0052] S31. Linearly expand the feature maps F1 and F2 of the same size to obtain vector groups (Q1, K1, V1) and (Q2, K2, V2) in the form of (query Q vector, key K vector, value V vector). Taking W×H×d as an example, the sizes of Q, K and V are all N×d, where N=W×H.

[0053] S32, compress Q1 and Q2 into (N / 2)×d through a linear function, and then concatenate them into a new query vector Q joint , size is N×d;

[0054] In one embodiment, in step 32, Q1 and Q2 are compressed into (N / 2)×d by a linear function, and then concatenated into a new query vector Q j o int , the calculation formula is as follows:

[0055] Q joint =Concat(Linear(Q1),Linear(Q2)) (1)

[0056] Among them, Linear(·) is a linear function, and Concat(·) is a concatenation operation along the channel direction.

[0057] S33, based on the exchanged query vector Q joint Calculate the global attention output F respectively with the key-value pair vector groups (K1, V1) and (K2, V2) in step S31 joint-1 and F joint-2 , the dimensions are W×H×d;

[0058] In one embodiment, in step 33, the query vector Q after the exchange joint Calculate the global attention output F respectively with the key-value pair vector groups (K1, V1) and (K2, V2) in step S31 joint-1 and F joint-2 , the calculation formula is as follows:

[0059]

[0060]

[0061] Among them, Softmax(·) is a nonlinear mapping function, and T is a transposition operation.

[0062] S4. For feature maps F of the same size joint-1 and F joint-2 Perform a two-branch difference capture operation with joint pixel-wise subtraction and parameterized learning to obtain the difference feature map F sub and F cat ;

[0063] The dual-branch difference capture operation is specifically as follows:

[0064] S41. In the pixel-by-pixel subtraction branch, calculate the feature map F of the same size joint-1 and F joint-2 The pixel-by-pixel difference between them is used to obtain the difference feature map F sub ;

[0065] S42. In the parameterized learning branch, we first stack the feature maps F of the same size along the channel direction. joint-1 and F joint-2 Then, multiple convolutional layers with the same size are used to capture nonlinear features and obtain the difference feature map F cat ;

[0066] In one embodiment, in step 42, in the parameterized learning branch, first, feature maps F of the same size are stacked along the channel direction. joint-1 and F joint-2 Then, multiple convolutional layers with the same size are used to capture nonlinear features and obtain the difference feature map F cat , the calculation formula is as follows:

[0067] F cat =Conv(Concat(F joint-1 ,F joint-2 )) (4)

[0068] Among them, Conv(·) is the convolution operation.

[0069] S5. Difference feature map F for multiple sizes sub and F cat , using lightweight global spatial attention for multi-scale feature map fusion, with multi-scale difference feature maps F including sizes W×H×d, 2W×2H×d and 4W×4H×d sub For example, the feature map fusion operation is as follows:

[0070] S51, the feature map F of size W×H×d sub-1 Interpolation upsampling and feature map F with size 2W×2H×d sub-2Stacking is done along the channel direction, and then preliminary fusion is performed through lightweight global spatial attention;

[0071] In one embodiment, in step 51, the feature map F with a size of W×H×d is sub-1 Interpolation upsampling and feature map F with size 2W×2H×d sub-2 The stacking is performed along the channel direction, and then the initial fusion is completed through lightweight global spatial attention. The calculation formula is as follows:

[0072] F fuse =LSA(Concat(F sub-1 ,F sub-2 )) (5)

[0073] LSA(X)=Softmax(Linear(Q)·Pooling(K T ))·Pooling(V) (6)

[0074] Where Pooling(·) is the spatial pooling operation.

[0075] S52, interpolate and upsample the feature map initially fused in step S51, and combine it with the feature map F of size 4W×4H×d sub-3 Stacking is done along the channel direction, and then further fusion is done through lightweight global spatial attention;

[0076] S6. Finally, the feature map after feature fusion is interpolated and upsampled to the size of the image block in step S1, and the changed target pixels are changed to 1 and the non-changed area pixels are 0 through convolution transformation and Sigmoid function to complete the changed target detection.

[0077] As a preferred embodiment, an experiment was conducted using images collected from a suburban area of ​​Guangzhou from 2006 to 2019. The multi-scale variable target retrieval method based on spatiotemporal modeling provided in this application was used. The images in the detection dataset contain three bands: R, G, and B, with a resolution of 0.55m. The specific steps are as follows:

[0078] Step 1: Load images collected at different times in a suburban area of ​​Guangzhou from 2006 to 2019, such as Figure 4 As shown, X1 is the image before the change, X2 is the image after the change, (a)-(f) are six groups of images, the three bands shown are selected (default R, G, B), the image pairs and their corresponding detection label maps are cut into 256×256, and the data is divided into three parts: training, validation, and testing.

[0079] Step 2: Pre-train a CNN model (specifically, a Resnet-18 network) on the ImageNet classification dataset. The CNN model can also be replaced with a Transformer model.

[0080] Step 3: Load the training set and validation set data after cutting in step 1 and feed them into the pre-trained CNN model to capture feature maps F1 and F2 with sizes of 64×64×64, 32×32×128, and 16×16×256. F1 and F2 are multi-scale abstract feature maps of corresponding sub-images in the image pair.

[0081] Step 4: Perform a cross-temporal joint global attention mechanism on the feature maps F1 and F2 to obtain the interactive feature maps F with sizes of 64×64×64, 32×32×128, and 16×16×256. joint-1 and F joint-2 .

[0082] Step 5: For feature maps F of the same size joint-1 and F joint-2 Perform a two-branch difference capture operation with joint pixel-wise subtraction and parameterized learning to obtain the difference feature map F sub and F cat .

[0083] Step 6: Targeting multi-scale difference feature maps F sub and F cat , use lightweight global spatial attention to perform multi-scale feature map fusion and obtain a feature map with a scale of 64×64×64.

[0084] Step 7: After upsampling the fused feature map by a factor of 4, performing a convolution operation, and applying a sigmoid function, a change detection map of size 256 × 256 × 1 is obtained. Model training is completed by minimizing the loss between the detection prediction map and the label map using the cross-entropy loss function and the stochastic gradient descent algorithm. This technique is well known to those skilled in the art and will not be detailed here.

[0085] Step 8: Load the trained parameters of this method, obtain the predicted output on the test set, and compare the change detection results with the test set labels. Calculate the precision, recall, F1 score, intersection-over-union ratio, and overall accuracy, as shown in the following example: Figure 4 As shown in Figure 1, there are six image pairs (a) to (f), which are the comparison results of the change detection images of the proposed method (the method of this application) and the reference method (dual time converter). The detection evaluation results are shown in Table 1.

[0086] Table 1

[0087]

[0088] Among them, the dual time converter is a method proposed in the prior art. Reference: H.Chen, Z.Qi and Z.Shi, "Remote Sensing Image Change Detection With Transformers," in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-14, 2022, Art no. 5607514, doi: 10.1109 / TGRS.2021.3095166. The detection results of the proposed method in Table 1 surpass the results of the dual time converter method in the prior art in all aspects. Figure 4 The results show that the proposed method has fewer missed detections (light gray pixels) and false detections (dark gray pixels), as well as higher accuracy (white pixels), which further verifies the effectiveness of our method in changing target retrieval.

[0089] This method aims to retrieve multi-scale changing targets in data with spatial-temporal multi-dimensionality, and uses a cross-temporal joint global attention mechanism to suppress task-irrelevant differences between data. It also performs differential learning by combining pixel-by-pixel subtraction and parameterized learning to improve the model's potential while maintaining stability and reliability. Finally, a lightweight global spatial attention mechanism is used to complete multi-level feature fusion, improving positioning while refining boundaries, thereby achieving accurate retrieval of changing targets.

[0090] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The above-described embodiments merely represent specific and detailed examples of the present application and should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A multi-scale target retrieval method based on spatiotemporal modeling, characterized by: The multi-scale changing target retrieval method based on space-time modeling includes the following steps: S1, split the remote sensing image pair X1 and X2 into several image blocks of the same spatial size; S2. Use the pre-trained network ResNet18 model to extract abstract feature maps F1 and F2 for each sub-graph. The output sizes of each stage of the ResNet18 model include 4W×4H×d, 2W×2H×d, and W×H×d, where W, H, and d are width, height, and channel depth, respectively. S3. Model F1 and F2 together through a cross-temporal joint global attention mechanism. The cross-temporal joint global attention mechanism operates as follows: S31. Linearly expand the feature maps F1 and F2 of the same size to obtain vector groups (Q1, K1, V1) and (Q2, K2, V2) in the form of (query Q vector, key K vector, value V vector). Taking W×H×d as an example, the sizes of Q, K and V are all N×d, where N=W×H. S32, compress Q1 and Q2 into (N / 2)×d through a linear function, and then concatenate them into a new query vector Q joint , size is N×d; S33, based on the exchanged query vector Q joint Calculate the global attention output F respectively with the key-value pair vector groups (K1, V1) and (K2, V2) in step S31 joint-1 and F joint-2 , the dimensions are W×H×d; S4. For feature maps F of the same size joint-1 and F joint-2 Perform a two-branch difference capture operation with joint pixel-wise subtraction and parameterized learning to obtain the difference feature map F sub and F cat ; The dual-branch difference capture operation is specifically as follows: S41. In the pixel-by-pixel subtraction branch, calculate the feature map F of the same size joint-1 and F joint-2 The pixel-by-pixel difference between them is used to obtain the difference feature map F sub ; S42. In the parameterized learning branch, we first stack the feature maps F of the same size along the channel direction. joint-1 and F joint-2 Then, multiple convolutional layers with the same size are used to capture nonlinear features and obtain the difference feature map F cat ; S5. Difference feature map F for multiple sizes sub and F cat , using lightweight global spatial attention for multi-scale feature map fusion, with multi-scale difference feature maps F including sizes W×H×d, 2W×2H×d and 4W×4H×d sub For example, the feature map fusion operation is as follows: S51, the feature map F of size W×H×d sub-1 Interpolation upsampling and feature map F with size 2W×2H×d sub-2 Stacking is done along the channel direction, and then preliminary fusion is performed through lightweight global spatial attention; S52, interpolate and upsample the feature map initially fused in step S51, and combine it with the feature map F of size 4W×4H×d sub-3 Stacking along the channel direction and then further fusion is completed through lightweight global spatial attention; S6. Finally, the feature map after feature fusion is interpolated and upsampled to the size of the image block in step S1, and the changed target pixels are changed to 1 and the non-changed area pixels are 0 through convolution transformation and Sigmoid function to complete the changed target detection.

2. The multi-scale variable object retrieval method based on spatiotemporal modeling according to claim 1, characterized in that: In step S1, the size of the image block is 256×256.

3. The multi-scale variable object retrieval method based on spatiotemporal modeling according to claim 1, characterized in that: In step S2, the output sizes of each stage of the ResNet18 model are 64×64×64, 32×32×128, and 16×16×256.

4. The multi-scale variable object retrieval method based on spatiotemporal modeling according to claim 1, characterized in that: In step 32, Q1 and Q2 are compressed into (N / 2)×d by a linear function, and then concatenated into a new query vector Q joint , the calculation formula is as follows: Q joint =Concat(Linear(Q1),Linear(Q2)) (1) Among them, Linear(·) is a linear function, and Concat(·) is a concatenation operation along the channel direction.

5. The multi-scale variable target retrieval method based on space-time modeling according to claim 4, characterized in that: In step 33, the query vector Q after the exchange joint Calculate the global attention output F respectively with the key-value pair vector groups (K1, V1) and (K2, V2) in step S31 joint-1 and F joint-2 , the calculation formula is as follows: Among them, Softmax(·) is a nonlinear mapping function, and T is a transposition operation.

6. The multi-scale variable object retrieval method based on spatiotemporal modeling according to claim 5, characterized in that: In step 42, in the parameterized learning branch, the feature maps F of the same size are first stacked along the channel direction. joint-1 and F joint-2 Then, multiple convolutional layers with the same size are used to capture nonlinear features and obtain the difference feature map F cat , the calculation formula is as follows: F cat =Conv(Concat(F joint-1 ,F joint-2 )) (4) Among them, Conv(·) is the convolution operation.

7. The multi-scale variable object retrieval method based on space-time modeling according to claim 6, characterized in that: In step 51, the feature map F with a size of W×H×d is sub-1 Interpolation upsampling and feature map F with size 2W×2H×d sub-2 The stacking is performed along the channel direction, and then the initial fusion is completed through lightweight global spatial attention. The calculation formula is as follows: F fuse =LSA(Concat(F sub-1 ,F sub-2 )) (5) LSA(X)=Softmax(Linear(Q)·Pooling(K T ))·Pooling(V) (6) Where Pooling(·) is the spatial pooling operation.

Images