Remote sensing video hash retrieval method based on key frame fusion and attention mechanism

Abstract

The application discloses a remote sensing video hash retrieval method based on key frame fusion and an attention mechanism, and mainly solves the problems that existing methods cannot fully capture the category-level semantics of remote sensing videos and the semantic information is insufficiently utilized. A new video space-time information extraction structure is designed to extract the data representation of three-dimensional video data in two-dimensional space, and an attention mechanism is introduced under the structure of a convolutional neural network to extract the corresponding hash code of the video in the Hamming space. The application not only captures the relative semantic correlation of hash codes in different modes, learns the relative semantic correlation of deep features, but also enhances the category-level semantics of hash codes and reduces the quantization error between class hash codes and hash codes. The application fully utilizes the space-time semantic information of remote sensing videos, and further improves the retrieval performance.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing video retrieval, specifically relating to a remote sensing video hash retrieval method based on keyframe fusion and attention mechanism. Background Technology

[0002] With the rapid development of Earth observation technology, remote sensing video retrieval has received widespread attention in the field of remote sensing data processing. Therefore, extracting useful information from large-scale remote sensing imagery is crucial. To uncover this useful information, many researchers have focused on remote sensing data retrieval. This is because remote sensing data retrieval can quickly retrieve useful information and has been applied in many areas. Remote sensing video retrieval is a branch of general video retrieval, focusing more specifically on the content retrieved from remote sensing video data.

[0003] With the explosive growth of remote sensing data, efficient remote sensing data analysis techniques have received urgent attention in processing remote sensing data. The task of remote sensing video retrieval is to retrieve relevant remote sensing videos using remote sensing video data. Due to the large volume of remote sensing video data and the significant differences in information across different scales, users find it difficult to quickly obtain useful information. How to address the multi-scale problem of remote sensing video data is a major challenge for remote sensing video retrieval tasks.

[0004] In recent years, many scholars have used deep learning methods to solve the problem of remote sensing video data retrieval. A common approach is to encode all remote sensing video data into their corresponding features and then calculate the similarity between different videos in a common representation space. Although existing remote sensing video retrieval methods have made some progress, they still have two shortcomings: 1) Remote sensing videos typically contain complex temporal and spatial information, and most existing methods use video frames for image feature extraction, failing to utilize the temporal dimension of the video. 2) Remote sensing videos contain one more dimension of data than remote sensing images, and existing methods still suffer from relatively high time complexity when retrieving them. Summary of the Invention

[0005] The purpose of this invention is to address the aforementioned shortcomings by proposing a remote sensing video hash retrieval method based on keyframe fusion and attention mechanisms. This method learns the semantic information of remote sensing video data, generates hash codes using an improved convolutional neural network model, and finally retrieves a given number of remote sensing video items using similarity calculation. This invention fully utilizes the spatiotemporal semantic information of remote sensing videos to further improve retrieval performance.

[0006] The technical solution adopted by this invention to solve this problem is: a remote sensing video hash retrieval method based on keyframe fusion and attention mechanism. This invention first designs a novel video spatiotemporal information extraction structure to extract the data representation of 3D video data in 2D space. Secondly, it introduces an attention mechanism under the inherent convolutional neural network structure to extract the corresponding hash code of the video in Hamming space. This invention mainly includes three implementation steps: extraction, learning, and selection. Given a remote sensing video to be queried, the representation features of the remote sensing video are first extracted. Then, hash code learning is performed using fixed similarity relationships between similar remote sensing videos. Finally, similarity calculation is used to obtain K similar videos. In addition, this invention introduces temporal attention and spatial attention mechanisms, which can select the importance of key information in the remote sensing video. The specific steps are as follows:

[0007] Step 1: Split the training dataset and the test dataset;

[0008] Step 2: Construct a video keyframe extraction and fusion module, including a keyframe extraction module and a video frame fusion module based on a spatial attention mechanism;

[0009] The keyframe extraction and fusion module comprises a keyframe extraction module and a video frame fusion module based on a spatial attention mechanism. The keyframe extraction module processes the following steps: It uses image histograms to perform K-means clustering on frames from a video, sorts them according to the inter-frame differences in the LUV color space, filters them based on their brightness scores, and finally selects keyframes based on the Laplacian variance (i.e., blur detection ranking). Next, the video frame fusion module fuses the keyframes using 3D convolution, and finally inputs the results into the spatial attention module to extract more important video semantic information.

[0010] Step 3: Construct a convolutional neural network to compute the feature representation and hash code of the video:

[0011] The convolutional neural network comprises a semantic feature extraction part and a feature hash code mapping part. The semantic feature extraction part consists of five sequentially connected convolutional blocks. Each convolutional block contains a different number of channel attention convolutional blocks connected via residual modules. The first convolutional block contains a 7×7 convolutional layer, the second contains 3 channel attention convolutional blocks, the third contains 8 channel attention convolutional blocks, the fourth contains 36 channel attention convolutional blocks, and the fifth contains 3 channel attention convolutional blocks. Each channel attention convolutional block includes a first 1×1 residual convolutional layer, a first channel attention layer, a second 3×3 convolutional layer, a second channel attention layer, and a third 1×1 convolutional layer. The hash mapping part consists of two fully connected layers: a 1×1000 fully connected layer activated by the sigmoid function and a 1×k fully connected layer activated by the tanh function, where k is the length of the mapped hash code.

[0012] Step 4: Train the overall network model, calculate the objective function of the overall network model, and update the initial parameters of the overall network model;

[0013] Step 5: Use the trained overall network model to calculate the hash codes of the samples in the test dataset, sort the Hamming distances between the hash codes of the query samples and each sample in the training dataset from largest to smallest, calculate the precision of the top n samples in the ranking list, and obtain the average precision index MAP and the top n retrieval results.

[0014] Furthermore, the specific implementation method of step 2 is as follows;

[0015] The input video V consists of a series of consecutive frames. Where F i For each video frame, N is the number of video frames, and J cluster centers are randomly selected. Calculate the Euclidean distance d = Argmin║x i -y j ║2, where x i y j For different frame samples, Argmin║║2 is the objective function for minimizing the Euclidean distance. The centers of the new clusters are recalculated using the average value. If the cluster centers change, the above steps are repeated iteratively until the cluster centers remain constant. Keyframes are then extracted based on the obtained cluster centers.

[0016] After obtaining the corresponding frames, they are input into a 3D convolutional layer for fusion, and then input into a spatial attention module for processing. The specific formula is as follows:

[0017]

[0018] Among them, F sa f is the result of spatial attention processing on the fused frames. i These are keyframes extracted by a clustering algorithm. Conv3D() performs 3D convolution, k is the number of keyframes, ⊙ represents element-wise dot product, Cat() performs vector concatenation, and M... sa This refers to spatial attention information, and the specific formula is as follows:

[0019] M sa =σ(Conv2([P avg (f); P max (f)]))

[0020] Where f represents the feature map input to the spatial attention module, and P avg For average pooling layers, P max For max pooling, Cat() is the frame fusion function, Conv2() is a 2D convolution, and σ is the Sigmoid activation function.

[0021] Furthermore, in step 3, the channel attention layer M ca The specific formula for (f) is:

[0022] M ca (f)=σ(MLP(P avg (f))+MLP(P max (f)))

[0023] Where f represents the feature map input to the channel attention layer, P avg For average pooling layers, P max For max pooling, σ() represents the sigmoid function, and MLP is a shared multilayer perceptron.

[0024] Furthermore, the objective function in step 4 consists of a video feature representation and a hash code representation. The specific formula for the video feature representation is as follows:

[0025] L(B)=l(x,y)={l1,…,l N} T ,

[0026]

[0027] Where w represents the weight settings for different classes, which are adjusted according to the sample size of each class, and ignore_index represents the class for which no loss is calculated; this class does not exist in this method. x is the score of the predicted value on the target class. n,c The target value is the score of the predicted value in other classes, x is the input, y is the target value, C is the number of classes, and N is the batch size. The objective function aims to preserve the one-to-one correspondence between the video and the corresponding class in the original space.

[0028] The specific formula for representing a hash code item is as follows:

[0029]

[0030] in, Let H represent the L3 paradigm, H represent the hash code information, and sgn represent the step function. Therefore, the overall objective function is expressed as follows:

[0031]

[0032]

[0033] Here, α is a hyperparameter, which is obtained by training the model to obtain the weight parameters W and bias parameters B of the backbone network.

[0034] Furthermore, the SGD algorithm is used for optimization when training the overall network model, with a learning rate set to 10. -3 The momentum is set to 0.9, and the weight decay is set to 5 × 10. -4 The batch size is set to 32, the hash code length k is set to 64, the initial weights of the convolutional neural network are initialized using pre-trained weights, α is set to 0.1, and the training is performed for 50 iterations, with the learning rate dynamically decreasing every 10 iterations. The weight parameters W and bias parameters B of the network are obtained by training the model.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] The method proposed in this invention not only captures the semantic relevance of videos in the time dimension and learns the relative semantic relevance of deep features, but also enhances the category-level semantics of hash codes, reduces the quantization error between hash codes and hash codes, and further improves retrieval accuracy. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the process of the present invention.

[0038] Figure 2 This is a conceptual illustration of the present invention.

[0039] Figure 3 This is a network structure diagram of the present invention.

[0040] Figure 4 This is an example of how the present invention retrieves similar remote sensing videos on the Drone-Action remote sensing dataset.

[0041] Figure 5 This is an example of how the present invention retrieves similar remote sensing videos on the ERA remote sensing dataset. Detailed Implementation

[0042] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0043] The environment used in this embodiment is a GeForce GTX Titan X GPU, an Intel Core i7-5930K, a 3.50GHz CPU, 64GB of RAM, and a Linux operating system. It is developed using Python and the open-source library PyTorch.

[0044] The first step is to divide the training dataset and the test dataset:

[0045] Using the Era and Drone-Action datasets, 80% of these datasets were selected as the training dataset I. train The remaining 20% ​​is used as the test dataset I.test ;

[0046] The second step is to use the keyframe extraction and fusion module to extract key information from the video:

[0047] Frame sampling is performed within the video shot, acquiring one frame per second. The sequence number of the sampled video frame is stored in a container, and the HSV vector of each sampled frame is calculated. The similarity of each sampled frame is calculated. Assuming the first sampled frame in the shot is the center of the first cluster, it is then set as the center of the next cluster at regular intervals. By comparing the similarity of each sampled frame with each initial sampled cluster center, the sampled frame is assigned to the cluster with the most similarity. Furthermore, by comparing the similarity of consecutive video frames, a threshold is set. When the cumulative difference exceeds the threshold, a new cluster is added, achieving adaptive cluster addition. After the above operations, the sum of the similarities between the cluster center and all sampled frames in the cluster is compared to see if it is the highest within the cluster. If not, the cluster center is reassigned until the frame has the highest similarity with all frames in the cluster. At this point, we can basically determine that it meets the keyframe characteristics. This algorithm can obtain the optimal cluster center in 3-4 iterations.

[0048] After obtaining the corresponding frames, they are input into the spatial attention module for processing. The specific formula is as follows:

[0049]

[0050] Among them, F sa f is the result of spatial attention processing on the fused frames. i These are the keyframes extracted by the clustering algorithm. Conv3D() performs 3D convolution, k is the number of keyframes, ⊙ represents element-wise dot product, and Cat() performs vector concatenation. sa This refers to spatial attention information, and the specific formula is as follows:

[0051] M sa =σ(Conv2([P avg (f); P max (f)]))

[0052] Where f represents the feature map input to the channel attention convolution block, P avg For average pooling layers, P max For max pooling, Cat() is the frame fusion function, Conv2() is a 2D convolution, and σ is the Sigmoid function.

[0053] The third step is to construct a remote sensing video convolutional neural network to calculate the video's feature representation and hash code:

[0054] The convolutional neural network comprises a semantic feature extraction part and a feature hash code mapping part. The semantic feature extraction part consists of five sequentially connected convolutional blocks. Each convolutional block contains a different number of channel attention convolutional blocks connected via residual modules. The first convolutional block contains a 7×7 convolutional layer, the second contains 3 channel attention convolutional blocks, the third contains 8 channel attention convolutional blocks, the fourth contains 36 channel attention convolutional blocks, and the fifth contains 3 channel attention convolutional blocks. Each channel attention convolutional block includes a first 1×1 residual convolutional layer, a first channel attention layer, a second 3×3 convolutional layer, a second channel attention layer, and a third 1×1 convolutional layer. The hash mapping part consists of two fully connected layers: a 1×1000 fully connected layer activated by the sigmoid function and a 1×k fully connected layer activated by the tanh function, where k is the length of the mapped hash code.

[0055] Channel attention layer M ca (f) The specific formula is:

[0056] M ca (f)=σ(MLP(P avg (f))+MLP(P max (f)))

[0057] Where f represents the feature map input to the channel attention layer, P avg For average pooling layers, P max For max pooling, σ() represents the sigmoid function, and MLP is a shared multilayer perceptron.

[0058] The fourth step is to calculate the objective function and update the initial parameters of the network: the overall objective function consists of video category representations and hash layer representations.

[0059] To generate more efficient hash codes, this invention not only captures the semantic relevance of remote sensing video hash codes but also learns the relative semantic relevance of deep features, which can effectively improve the retrieval accuracy of remote sensing videos. Category-level semantics can enhance the potential relevance of similar hash codes. To capture the category-level semantics of remote sensing videos, the specific expression formula for the video feature representation is as follows:

[0060] L(B)=l(x,y)={l1,…,l N} T ,

[0061]

[0062] Where w represents the weight settings for different classes, which are adjusted according to the sample size of each class, and ignore_index represents the classes for which no loss is calculated. x is the score of the predicted value on the target class. n,c For the predicted score on other classes, x is the input, y is the target value, C is the number of classes, and N is the batch size;

[0063] The objective function aims to preserve the one-to-one correspondence between videos and their corresponding categories in the original space.

[0064] The above loss function calculation may introduce quantization errors. This invention proposes a penalty term for the hash layer to reduce the quantization error between the hash-like code and the hash code. The specific formula for the hash layer representation is as follows:

[0065]

[0066] in, denoted by L3 paradigm, H represents hash code information, and sgn represents step function.

[0067] Therefore, the formula for the overall objective function is expressed as follows:

[0068]

[0069] Here, α is a hyperparameter.

[0070] Specifically, in this embodiment, the SGD algorithm is used for optimization, and the learning rate is set to 10. -3 The momentum is set to 0.9, and the weight decay is set to 5 × 10. -4 The batch size is set to 32, the hash code length k is set to 64, the initial weights of the convolutional neural network are initialized using pre-trained weights, α is set to 0.1, and training is performed for 50 iterations, with the learning rate dynamically decreasing every 10 iterations. By training the model, the network's weight parameters W and bias parameters B are obtained, thus completing the process of this embodiment.

[0071] Step 5: Calculate and test retrieval accuracy.

[0072] The network trained above is used to calculate the hash codes of samples in the test video dataset. The Hamming distance between the hash codes of the query samples and each sample in the dataset is sorted from largest to smallest. The accuracy of the top n samples in the ranking list is calculated, and the average precision index (MAP) and the top n search results are obtained.

[0073] To evaluate the effectiveness of the method of this invention, its retrieval performance was compared with several state-of-the-art methods, including CNNH, DHN, DPSH, DTSH, DAPH, DSDH, LCDSH, DCH, ISDH, QSMIH, Uns_GH, DFH, DSHSD, IDHN, DBDH, PCDH, and Uns_Bihalf. This experiment used 64-bit hash codes and the Drone-Action and ERA datasets. The CNNH method utilizes a convolutional neural network to simultaneously learn the feature representation of the image and the hash function, then projects their corresponding features into a common representation space. The DHN, DPSH, DTSH, DAPH, DSDH, LCDSH, DCH, ISDH, QSMIH, Uns_GH, DFH, DSHSD, IDHN, DBDH, PCDH, and Uns_Bihalf methods were executed as described in the original paper.

[0074] Table 1

[0075]

[0076] Table 1 shows the comparative experimental results of this invention and other methods on the remote sensing video retrieval task on the Drone-Action dataset, where mAP is the average accuracy index and P@K is the accuracy of the top K items in the retrieval list.

[0077] Table 2

[0078]

[0079]

[0080] Table 2 shows the comparative experimental results of this invention and other methods on the remote sensing video retrieval task on the ERA dataset, where mAP is the average accuracy index and P@K is the accuracy of the top K items in the retrieval list.

[0081] The embodiments of the present invention are merely illustrative examples and are not intended to limit the scope of the invention. Those skilled in the art may make different modifications or equivalent substitutions based on this method, and all variations similar to those of the present invention fall within the protection scope of the claims of the present invention.

Claims

1. A remote sensing video hash retrieval method based on key frame fusion and attention mechanism, characterized in that, The method comprises the following steps: Step 1, dividing the training data set and the test data set; Step 2, constructing a video key frame extraction and fusion module, including a key frame screening and extraction module and a frame information fusion module; The key frame extraction and fusion module comprises a key frame extraction module and a video frame fusion module based on a spatial attention mechanism; wherein the key frame extraction module uses an image histogram to perform K-means clustering on frames from a video, sorts according to the inter-frame difference of the LUV color space of different frames, filters according to the brightness score of the frames, and finally selects key frames according to the Laplacian variance, i.e. fuzzy detection sorting; in the video frame fusion module, key frames are fused by three-dimensional convolution, and finally input into a spatial attention module to obtain more important video semantic information; Step 3, constructing a convolutional neural network for calculating the feature representation and hash code of the video: The convolutional neural network comprises a semantic feature extraction part and a feature hash code mapping part, wherein the semantic feature extraction part comprises five convolutional blocks connected in sequence, each of which comprises different numbers of channel attention convolutional blocks connected through a residual module, and the hash mapping part comprises two fully connected layers; Step 4, training the overall network model, calculating the objective function of the overall network model and updating the initial parameters of the overall network model; The objective function in step 4 is composed of a video feature representation term and a hash code representation term, and the specific expression formula of the video feature representation term is as follows: where w is different class weight settings adjusted according to the sample size of each class, ignore_index is a class not to calculate loss, is the score of the prediction value on the target class, is the score of the prediction value on other classes, x is the input, y is the target value, C is the number of classes, and N is the batch size; The specific expression formula of the hash code representation term is as follows wherein, represents the L3 norm, H represents the hash code information, and sgn represents the step function; thus, the formula expression of the total objective function is as follows: wherein, is a hyperparameter, the weight parameters W and the bias parameters B of the network are obtained by training the model; Step 5, using the trained overall network model to calculate the hash code of the samples in the test data set, sorting the Hamming distance between the query sample and the hash code of each sample in the training data set from large to small, calculating the top n precision of the ranking list, and obtaining the average precision indicator MAP and the top n retrieval results.

2. The remote sensing video hash retrieval method based on key frame fusion and attention mechanism according to claim 1, characterized in that: The specific implementation method in step 2 is as follows; The input video V is composed of a series of continuous frames , randomly selecting J cluster centers , calculating the Euclidean distance , wherein , for different frame samples, the objective function with the minimum Euclidean distance, recalculating the center of the new class by using the average value, repeating the above steps if the cluster center changes until the cluster center is unchanged, and extracting the key frame according to the obtained cluster center; After obtaining the corresponding key frames, input them into three-dimensional convolution for fusion, and then input them into a spatial attention module for processing, and the specific formula is as follows: wherein F sa is the result of the fusion frame after spatial attention processing, is the key frame extracted by the clustering algorithm, is a three-dimensional convolution, and k is the number of key frames, is an element-wise dot product, is vector splicing, is spatial attention information, and the specific formula is as follows: wherein, represents a feature map input to the spatial attention module, is an average pooling layer, is a max pooling layer, is a frame fusion function, is a two-dimensional convolution, is a Sigmoid function.

3. The keyframe fusion and attention mechanism based remote sensing video hash retrieval method of claim 1, wherein: Channel attention convolutional layer The specific formula is: wherein, denotes the feature map input to the channel attention layer, is an average pooling layer, is a max pooling layer, denotes a sigmoid function, and MLP is a shared multi-layer perceptron.

4. The keyframe fusion and attention mechanism based remote sensing video hash retrieval method of claim 1, wherein: When training the overall network model, the SGD algorithm is used for optimization, the learning rate is set to 10 -3 , the momentum is set to 0.9, the weight decay is set to 5x10 -4 , the batch size is set to 32, the length k of the hash code is set to 64, the initial weights of the convolutional neural network are initialized using pre-trained weights, , the setting is 0.1, and 50 rounds of iterations are trained, wherein the learning rate is dynamically reduced every 10 rounds, and the weight parameters W and the bias parameters B of the network are obtained by training the model.

5. The keyframe fusion and attention mechanism based remote sensing video hash retrieval method of claim 1, wherein: The Era and Drone-Action datasets are used in Step 1, with 80% of the dataset selected as the training dataset , and the remaining 20% as the test dataset .

6. The keyframe fusion and attention mechanism based remote sensing video hash retrieval method of claim 1, wherein: In step 3, the first convolutional block contains one 7x7 convolutional layer, the second convolutional block contains three channel attention convolutional blocks, the third convolutional block contains eight channel attention convolutional blocks, the fourth convolutional block contains thirty-six channel attention convolutional blocks, and the fifth convolutional block contains three channel attention convolutional blocks; the channel attention convolutional block comprises a first 1x1 residual convolutional layer, a first channel attention layer, a second 3x3 convolutional layer, a second channel attention layer, and a third 1x1 convolutional layer.

7. The keyframe fusion and attention mechanism based remote sensing video hash retrieval method of claim 1, wherein: In step 3, the hash mapping part specifically comprises a 1x1000 fully connected layer activated by a sigmoid function and a 1xk fully connected layer activated by a tanh function, and k is the length of the mapped hash code.

Images