Remote sensing ship image retrieval method based on spatial invariance

Abstract

The present application belongs to the technical field of remote sensing image processing, and discloses a remote sensing ship image retrieval method based on spatial invariance. The remote sensing ship image set is divided into a training set, a verification set and a test set. Remote sensing ship images are extracted from the training set for pretreatment. After the overall network model is constructed, training is carried out. Finally, the trained network model is evaluated to obtain the retrieval result. The present application first designs a module based on spatial invariance. A deep convolutional neural network is used to extract continuous real-valued descriptors. A spatial transformation attention mechanism is introduced to enhance the model's adaptation ability to spatial invariance features and learning efficiency. Secondly, a module based on self-attention hash is proposed. Multi-scale image embedding is used to improve image representation efficiency, optimize attention regularization in the visual encoder, and effectively solve the quantization loss problem in hash mapping. Experimental results show that the retrieval performance of the present application on the GGWS, DSCR, FGSC-23 and FGSCR-42 data sets is better than that of existing deep feature-based methods.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image processing, and particularly relates to a remote sensing ship image retrieval method based on spatial invariance. BACKGROUND

[0002] Remote sensing ship image retrieval technology is crucial for improving the efficiency of maritime traffic management and safety monitoring, quickly identifying suspicious ships, and correcting route deviations. It also plays an important role in optimizing logistics management, performing maritime rescue missions, and implementing environmental monitoring. With the continuous advancement of remote sensing technology, the increasing number of ship images poses new challenges to traditional remote sensing ship image retrieval technology.

[0003] Existing ship image retrieval methods mainly include traditional manual feature extraction and deep learning-based methods. Traditional methods rely on manually designed features, often accompanied by a large number of uncertain factors, such as the generalization problem of designed features. While directly extracting features using convolutional neural networks can improve some robustness, the extracted features may lack certain emphasis, causing simple convolution and pooling operations to focus only on general features. In real remote sensing ship retrieval scenarios, there are further requirements, such as rotation invariance of ships and multi-scale of remote sensing. Due to the influence of distance and angle during remote sensing image capture, the scale and appearance features of ships change, which poses a challenge to existing feature extraction methods. At the same time, the background information in the image often does not help the identification and retrieval of ships, and may even cause interference. In the face of these challenges, researchers are exploring more efficient and accurate remote sensing ship image retrieval technology. SUMMARY

[0004] In view of the deficiencies in the background art, and considering the characteristics of remote sensing ship images, the present application proposes a remote sensing ship image retrieval method based on spatial invariance.

[0005] The technical solution adopted by the present application is a remote sensing ship image retrieval method based on spatial invariance, the steps of which are as follows:

[0006] Step S1, integrate existing remote sensing ship images and divide them into training set, validation set and test set according to different categories of pictures;

[0007] Step S2, pre-process the remote sensing ship images in the training set;

[0008] Step S3, based on the spatial transformation attention module, feature extraction network and feature descriptor extraction module, randomly pick the pre-processed remote sensing ship images as input to construct an overall network model;

[0009] Step S4, train the overall network model;

[0010] Step S5, using the trained overall network model to evaluate the remote sensing ship image data to obtain the retrieval result.

[0011] Preferably, the existing data set in step S1 adopts four data sets, a remote sensing ship image data set DSCR of a warship, a remote sensing ship image data set FGSCR-42 composed of a plurality of target detection data sets and a remote sensing ship image data set CGWX taken by a "Jilin No. 1" satellite, and a remote sensing ship image data set FGSC-23 provided by Google Earth image and high-resolution satellite, and the proportion of each data set is different. After integrating different categories of pictures in the four data sets, the proportion is divided into training set, validation set and test set.

[0012] Preferably, before the remote sensing ship image is randomly extracted from the training set as the input of the network in step S2, the remote sensing ship image is preprocessed for better extraction of features in the remote sensing ship image. The preprocessing method: adopting random flip, rotation, adjusting the brightness, saturation and contrast of the remote sensing ship image, for data enhancement processing, so that the network pays more attention to extracting semantic features in the remote sensing ship image, and improves the generalization and robustness of the network.

[0013] Preferably, in step S3, the process of constructing the overall network model is as follows:

[0014] S3a, constructing a spatial transformation attention module by using a spatial transformation module and an attention layer;

[0015] S3b, constructing a remote sensing ship feature extraction network through generalized average pooling and attention mechanism;

[0016] S3c, constructing a remote sensing ship feature descriptor extraction module.

[0017] Preferably, the process of constructing the spatial transformation attention module in step S3a is as follows:

[0018] The spatial transformation attention module can learn the spatial invariance of data before inputting the backbone feature network, and the attention mechanism can reduce the difficulty of model training caused by interference information. The spatial transformation attention module can be placed at the front position of the convolutional neural network, so that the network can perform spatial transformation and redundant information elimination before feature extraction.

[0019] The spatial transformation attention module is composed of a spatial transformation module and an attention layer.

[0020] (1) The construction process of the spatial transformation module is as follows:

[0021] First, the remote sensing ship image is preprocessed to adjust the brightness, saturation and contrast of the remote sensing ship image, and the remote sensing ship image is rotated and flipped to increase the diversity of the data set. Obtain the position transformation parameter through two-dimensional affine transformation The specific calculation formula is as follows:

[0022] ;

[0023] Wherein, is the position transformation parameter, is the input remote ship image, , corresponding to the high, wide, RGB three channels of the remote ship image respectively, indicates six parameters of the affine transformation matrix, indicates a linear layer used to regress the image transformation parameter, and the linear layer uses two fully connected layers for regression;

[0024] Then, the feature map of the input remote ship image is transformed using the sampling grid method, and each pixel uses the calculation centering on the input feature map, and is converted into the changed remote ship image through clipping, translation, rotation, scaling and inclination The specific calculation formula is as follows:

[0025] ;

[0026] Wherein, is the changed remote ship image;

[0027] If the calculated coordinate value is not on the original image grid, the bilinear interpolation is used to calculate the value of the corresponding point;

[0028] (2) Construction process of attention layer:

[0029] Then, the changed remote ship image is used as the input of the attention layer, and the convolution layer is used to perform feature dimensioning operation on the changed remote ship image to obtain the dimensioned feature map , and then the dimensioned feature map is subjected to maximum pooling and average pooling to obtain two feature maps with different attention points, then the convolution layer is used to reduce the dimension and splice the two feature maps in the channel dimension, and then the convolution is used to generate the final spatial attention feature map, and finally the function is scaled to (0-1) to be used as the channel attention mechanism weight, and finally multiplied with the changed remote ship image to obtain the final remote ship image feature map, and the specific calculation formula is as follows:

[0030] ;

[0031] wherein is the last remote sensing ship image spatial transformation attention feature map input into the remote sensing ship feature extraction network, is a function, is a feature splicing operation, is an average pooling operation, is a maximum pooling operation, and are convolutional layer dimension reduction operations, is a feature map after dimension increase.

[0032] Preferably, the process of constructing the remote sensing ship feature extraction network in step S3b is as follows:

[0033] (1) The calculation process of generalized average pooling:

[0034] Since the maximum pooling will lose part of the activated information, in order to improve the feature representation ability, the generalized average pooling (Generalized Mean Pooling, GeM) is used instead of the maximum pooling layer inside, and the generalized average pooling The calculation formula is as follows:

[0035] ;

[0036] wherein, is the data of different images and different channels, is the number of data channels, is the parameter of generalized average pooling;

[0037] (2) The construction process of attention mechanism:

[0038] In order to further improve the model representation ability, the relationship between the attention among the channels is considered, and the attention mechanism is added on the channel: first, the spatial dimension is compressed, and the maximum average pooling is used to average each channel into a value, the value has a global receptive field, secondly, the correlation between the above channels is obtained by using the weight learning of the full connection layer to get the attention map, finally the original input is multiplied by the attention map to get the output of the feature extraction network, that is, the high-level semantic feature of the remote sensing ship image, the specific calculation formula is as follows:

[0039] ;

[0040] wherein, is the original input, that is, the output feature of the fourth layer in the feature extraction network as the original input of the attention mechanism, is a full connection layer, is a maximum average pooling operation, is a 1x1 convolution operation, is a "softmax" function. The operation process of the attention mechanism is as follows.

[0041] Preferably, the process of constructing the remote sensing ship feature descriptor extraction module in step S3c is as follows:

[0042] The remote sensing ship image feature map obtained by the remote sensing ship feature extraction network is converted into an image descriptor : first, a convolution is used to obtain features, wherein is the number of cluster centers, is the feature dimension, then a Softmax function is used to generate the weight coefficient of the difference matrix, and finally a VLAD Core is used to realize, that is, the coefficient is multiplied by the residual (the difference between the remote sensing ship image feature vector x and the cluster center c), and finally two normalization implementations are performed, including intra-class normalization and inter-class normalization, and the specific calculation formula is as follows:

[0043] ;

[0044] wherein, is the first dimension of the th local feature descriptor in the remote sensing ship image high-level semantic feature , is the th cluster center in the VLAD algorithm, , , and are all trainable parameters, is a twice normalization function, is a descriptor limiting function, that is, the descriptor is flattened and then mapped into a vector by a fully connected layer, is generally taken as 512, so that the local feature descriptor can be effectively encoded into a fixed-length global vector, and finally a remote sensing ship image descriptor is obtained.

[0045] Preferably, the process of training the overall network model in step S4 is as follows:

[0046] The remote sensing ship image is input into the model to obtain the corresponding ship feature descriptor, and then the loss function calculation in the remote sensing ship image retrieval model is performed, the loss result is back propagated, and the Pytorch framework is used to automatically update and optimize the model parameters until the training is completed.

[0047] ​To enhance both intra-class compactness and inter-class variability, an additive angular margin loss (ArcFace) is used, which establishes an angular cosine margin between the feature vector and the class label, so that the angle between the feature descriptors of different ship classes increases, while the angle between the feature descriptors of the same ship class decreases; both the semantic features of the ship class and the accuracy of the mapped feature descriptors are utilized;

[0048] An additive angular margin loss (ArcFace) is used: ArcFace first normalizes the remote sensing ship image descriptor to a unit vector, then maps the feature vector to a specific angle interval through a fully connected layer for each data sample, so that the feature vectors of the same class are closer to the predefined angle, finally, the class probability of each class is calculated through the Softmax function, and it is used to calculate the loss, the specific calculation is as follows:

[0049] ;

[0050] wherein, represents the number of training samples, i.e. the number of samples in the remote sensing ship image dataset, represents the index of the current remote sensing ship image, ranging from 1 to , represents the class label of the th sample, and represents the class of the remote sensing ship image, represents the angle between the feature descriptor of the th remote sensing ship image and its ship class label , represents the angular cosine margin, which is used to enhance the difference between the feature vectors, represents the scaled Softmax function, which is used to control the scale of the feature vector, The loss function can effectively utilize the semantic information of the remote sensing ship image class to learn the network parameters;

[0051] In order to improve the generalization of the model, all the input remote sensing ship images are first scaled to 224x224 size, and then the values of the RGB three channels are normalized and standardized. Standardization can reduce the instability problem caused by difference values, that is, subtract the average value of each input channel from the channel and divide it by the channel standard deviation. Because the mean and standard deviation of the remote sensing ship image are quite different from the natural image, referring to the experimental data set and the public data set of 12w remote sensing ship images, the mean is changed to [0.294, 0.323, 0.318], and the standard deviation is changed to [0.207, 0.194, 0.191]. Finally, the common image enhancement method is introduced, that is, random rotation, random flip, and random padding block with a probability of 0.1. When training with the Adam optimizer, the parameters are optimized.

[0052] Among them, the training batch size of the training is set to 100, the initial learning rate is set to 2e-5, and when the loss function no longer has a significant downward trend, the training process is terminated in advance.

[0053] Preferably, the step S5 of using the trained overall network model to evaluate the remote sensing ship image data to obtain the retrieval result is as follows:

[0054] In the remote sensing ship image retrieval task, the average precision mAP is calculated according to the precision and recall, the average precision mAP (mean Average Precision) is used to evaluate the performance of ship retrieval, so as to evaluate the performance of the network model on the unseen data, and verify the generalization ability of the network model, which is used to evaluate the overall performance of the model according to the mAP in the subsequent test set experiment part;

[0055] The precision (Precision) refers to the proportion of the correct ship category images in all the retrieved remote sensing ship images, and the specific calculation formula of the precision is as follows:

[0056] ;

[0057] Among them, represents the number of positive ship prediction results as positive examples of ships, represents the number of negative ship prediction results as positive examples of ships.

[0058] The recall (Recall) is used to evaluate the proportion of similar ships retrieved in all similar ships in the retrieval process, and the specific calculation formula of the recall is as follows:

[0059] ;

[0060] Among them, denotes the number of sample prediction results of the positive ship indicating that the ship is a negative example;

[0061] The retrieval results are sorted in descending order of confidence, and then the confidence of each result is taken as a threshold in turn, and each time a precision and recall can be obtained, and all points are plotted on the precision-recall graph, and the PR curve is obtained by connecting each point to obtain a line graph, and the area of the curve is (Average Precision, AP), and the specific calculation process of the average precision mAP is as follows:

[0062] ;

[0063] wherein, is the number of retrieval lists, is the of each retrieval sample.

[0064] Compared with the prior art, the present application proposes a remote sensing ship image retrieval method based on spatial invariance, and has the following advantages:

[0065] (1) The present application uses a spatial transformation attention module, which reduces the sensitivity of the model to spatial transformations such as translation, scaling and rotation, and can dynamically perform spatial transformation on the image, improving the adaptability of the model to changes in remote sensing ship images and enhancing the generalization and robustness of the model under different conditions;

[0066] (2) The present application focuses on key features using the attention mechanism, reduces background interference by combining channel attention and spatial attention, effectively deals with the problem of complex background and scale change, and improves the accuracy of feature extraction and retrieval performance;

[0067] (3) The present application method can adapt to the spatial invariance of the image, ensuring that ship images under different angles and scales can be accurately identified, and significantly improving the precision of remote sensing ship image retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 is the network structure diagram of the remote sensing ship image retrieval method based on spatial invariance of the present application;

[0069] Figure 2 is a comparison chart of retrieval performance results of the method proposed by the present application and other methods on different data sets, i.e., the influence of the dimension of the real-valued continuous feature descriptor of the remote sensing ship image on the retrieval precision is evaluated on four remote sensing ship image data sets;

[0070] Figure 3For comparison of the visualization results of w / o ST model, w / o GA model, w / o DE model and full model, the t-SNE algorithm is used to reduce the dimension of the remote sensing ship image descriptor data and the category data and to visualize and map them to two-dimensional space; thus, the following figures are obtained Figure 3 (a) t-SNE visualization retrieval result image of w / o ST model on FGSC-23 dataset, Figure 3 (b) t-SNE visualization retrieval result image of w / o GA model on FGSC-23 dataset, Figure 3 (c) t-SNE visualization retrieval result image of w / o DE model on FGSC-23 dataset, Figure 3 (d) t-SNE visualization retrieval result image of full model on FGSC-23 dataset;

[0071] Figure 4 Remote sensing ship image retrieval result of w / o ST model on DSCR dataset;

[0072] Figure 5 Remote sensing ship image retrieval result of w / o GA model on DSCR dataset;

[0073] Figure 6 Remote sensing ship image retrieval result of w / o DE model on DSCR dataset;

[0074] Figure 7 Remote sensing ship image retrieval result of full model on DSCR dataset. DETAILED DESCRIPTION

[0075] The technical solutions in the embodiments of the present application will be further described in a clear and complete manner with reference to the drawings in the embodiments of the present application. It should be noted that the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0076] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings of the specification: In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the advantages of the present application will be further illustrated by comparing embodiments with reference to the drawings and specific embodiments.

[0077] The present application proposes a remote sensing ship image retrieval method based on spatial invariance, and the steps of the method are described in detail as follows:

[0078] Step S1, integrate existing remote sensing ship images and divide them into training set, validation set and test set according to different categories of pictures;

[0079] Further, the existing data set in step S1 adopts four data sets, i.e., a remote sensing ship image data set DSCR of warships, a remote sensing ship image data set FGSCR-42 composed of a plurality of target detection data sets and cropped images of Google Earth images and high-resolution satellite images provided by FGSC-23, and a remote sensing ship image data set CGWX taken by a "Jilin No. 1" satellite. Since the proportions of the respective data sets are different, after integrating different categories of pictures in the four data sets, the pictures are divided into training set, validation set and test set according to a ratio of 6:2:2.

[0080] Step S2, pre-process the remote sensing ship images in the training set;

[0081] Further, before randomly extracting a remote sensing ship image from the training set as the input of the network in step S2, the remote sensing ship image is pre-processed for better extraction of features in the remote sensing ship image. The pre-processing method is to use random flipping, rotation, adjustment of brightness, saturation and contrast of the remote sensing ship image to enhance the data, so that the network focuses more on extracting semantic features in the remote sensing ship image and improves the generalization and robustness of the network.

[0082] Step S3, based on the spatial transformation attention module, the feature extraction network and the feature descriptor extraction module, randomly pick a pre-processed remote sensing ship image as the input to construct an overall network model;

[0083] Further, in step S3, the process of constructing the overall network model is as follows:

[0084] S3a, construct a spatial transformation attention module using a spatial transformation module and an attention layer;

[0085] Further, the process of constructing the spatial transformation attention module in step S3a is as follows:

[0086] The spatial transformation attention module can learn spatial invariance of data before inputting the backbone feature network, and the attention mechanism can reduce the difficulty of model training caused by interference information. The spatial transformation attention module can be placed at the front position of the convolutional neural network, so that the network can perform spatial transformation and redundant information elimination before feature extraction.

[0087] The spatial transformation attention module is composed of a spatial transformation module and an attention layer.

[0088] (1) Construction process of the spatial transformation module:

[0089] Firstly, the remote sensing ship image is obtained The position transformation parameters are obtained by two-dimensional affine transformation The specific calculation formula is as follows:

[0090] ;

[0091] Wherein, is the position transformation parameter, is the input remote sensing ship image, , corresponding to the height, width, RGB three channels of the remote sensing ship image respectively, indicates six parameters of the affine transformation matrix, indicates the linear layer used to regress the image transformation parameters, and two fully connected layers are used for regression here;

[0092] Then, the feature map of the input remote sensing ship image is transformed by using the sampling grid method, and each pixel uses the calculation centering on the input feature map, and after cutting, translation, rotation, scaling and inclination, the changed remote sensing ship image The specific calculation formula is as follows:

[0093] ;

[0094] Wherein, is the changed remote sensing ship image;

[0095] If the calculated coordinate value is not on the original image grid, the bilinear interpolation is used to calculate the value of the corresponding point;

[0096] (2) Construction process of attention layer:

[0097] Then, the changed remote sensing ship image is used as the input of the attention layer, and the convolution layer is used to perform feature dimensioning operation on the changed remote sensing ship image to obtain the dimensioned feature map , and then the dimensioned feature map is maximum-pooled and average-pooled to obtain two feature maps with different attention points, then the convolution layer is used to reduce the dimension and splice the two feature maps in the channel dimension, and then the convolution is used to generate the final spatial attention feature map, and finally the function is scaled to (0-1) as the channel attention mechanism weight, and finally multiplied with the changed remote sensing ship image to obtain the final remote sensing ship image feature map, and the specific calculation formula is as follows:

[0098] ;

[0099] wherein is the last remote sensing ship image spatial transformation attention feature map input into the remote sensing ship feature extraction network, is a function, is a feature concatenation operation, is an average pooling operation, is a max pooling operation, and are convolutional layer dimension reduction operations, is a feature map after dimension increase.

[0100] S3b, constructing a remote sensing ship feature extraction network through generalized average pooling and attention mechanism;

[0101] Further, the process of constructing the remote sensing ship feature extraction network in the step S3b is as follows:

[0102] (1) Calculation process of generalized average pooling:

[0103] Since the max pooling will lose part of the activated information, in order to improve the feature representation ability, the generalized average pooling (Generalized Mean Pooling, GeM) is used instead of the max pooling layer inside, and the generalized average pooling The calculation formula is as follows:

[0104] ;

[0105] wherein, is the data of different images and different channels, is the number of data channels, is the parameter of generalized average pooling;

[0106] (2) Construction process of attention mechanism:

[0107] In order to further improve the model representation ability, the relationship between the attention among the channels is considered, and the attention mechanism is added on the channel: first, the spatial dimension is compressed, and the maximum average pooling is used to average each channel into a value. The value has a global receptive field, secondly, the correlation between the above channels is obtained by using the weight learning of the full connection layer to obtain the attention map, and finally the original input is multiplied by the attention map to obtain the output of the feature extraction network, that is, the high-level semantic feature of the remote sensing ship image, and the specific calculation formula is as follows:

[0108] ;

[0109] wherein, is the output feature of the fourth layer in the feature extraction network as the original input of the attention mechanism, is a full connection layer, is a max-pooling operation, is a 1x1 convolution operation, is the operation process of the attention mechanism.

[0110] S3c, constructing a remote sensing ship feature descriptor extraction module;

[0111] Further, the process of constructing the remote sensing ship feature descriptor extraction module in step S3c is as follows:

[0112] The remote sensing ship image feature map obtained by the remote sensing ship feature extraction network is converted into an image descriptor : first, a convolution is used to obtain features, where is the number of cluster centers, is the feature dimension, then a Softmax function is used to generate the weight coefficient of the difference matrix, and finally a VLAD Core is used to realize, i.e. multiplying the coefficient with the residual (the difference between the remote sensing ship image feature vector x and the cluster center c), and finally implementing twice normalization, including intra-class normalization and extra-class normalization, the specific calculation formula is as follows:

[0113]

[0114] wherein, is the first dimension of the th local feature descriptor in the remote sensing ship image high-level semantic feature , is the th cluster center in the VLAD algorithm, , and are trainable parameters, is a twice normalization function, is a descriptor limiting function, i.e. the descriptor is flattened and then mapped into a vector through a full connection layer, is generally taken as 512, so that the local feature descriptor can be effectively encoded into a fixed-length global vector, and finally the remote sensing ship image descriptor is obtained. Step S4, training the overall network model;

[0115] Step S4, training the overall network model; ​​​​

[0116] Further, the step S4 of training the overall network model is as follows:

[0117] The remote sensing ship image is input into the model to obtain the corresponding ship feature descriptor, and then the loss function in the remote sensing ship image retrieval model is calculated. According to the loss result, the model parameters are automatically updated and optimized by the Pytorch framework until the training is completed. The training process of the ship image retrieval network is shown in Table 1.

[0118] Table 1 Process table of ship image retrieval network training process

[0119]

[0120] In order to enhance the intra-class compactness and inter-class difference at the same time, the additive angular margin loss (ArcFace) is used. The core idea is to establish an angle cosine interval between the feature vector and the class label, so that the angle between the feature descriptors of different ship classes is increased, and the angle between the feature descriptors of the same ship class is reduced. Both the semantic features of the ship class and the accuracy of the mapped feature descriptor are improved.

[0121] The additive angular margin loss (ArcFace) is used: ArcFace first normalizes the remote sensing ship image descriptor to a unit vector, then maps the feature vector to a specific angle interval through a fully connected layer for each data sample, so that the feature vectors of the same class are closer to the predefined angle, and finally, the class probability of each class is calculated through the Softmax function, which is used to calculate the loss. The specific calculation is as follows:

[0122] ;

[0123] wherein, N represents the number of training samples, i.e. the number of samples in the remote sensing ship image dataset, i represents the index of the current remote sensing ship image, ranging from 1 to N, , y i represents the class label of the i-th sample, and represents the class of the remote sensing ship image, cos θ i represents the angle between the feature descriptor of the i-th remote sensing ship image and its ship class label y i, m represents the angle cosine interval for enhancing the difference between feature vectors, σ represents the scaled Softmax function for controlling the scale of the feature vector, ​​​The loss function can effectively utilize the semantic information of the remote sensing ship image category to learn network parameters.

[0124] In order to improve the generalization of the model, all input remote sensing ship images are first scaled to 224x224 size, and then the values of the RGB three channels are normalized and standardized to 0-1. Standardization can reduce the instability problem caused by difference values, that is, subtract the average value of each input channel from the channel and divide it by the channel standard deviation. The commonly used method is to use the mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] of the natural image ImageNet

[58] dataset. However, the mean and standard deviation of the remote sensing ship image are quite different from the natural image. Referring to the experimental dataset and the public dataset of 12w remote sensing ship images, the mean is changed to [0.294, 0.323, 0.318], and the standard deviation is changed to [0.207, 0.194, 0.191]. Finally, the commonly used image enhancement method is introduced, that is, random rotation, random flip, and random padding block with a probability of 0.1. When training with the Adam optimizer, the parameters are optimized.

[0125] The training batch size is set to 100, the initial learning rate is set to 2e-5, and the training process is terminated in advance when the loss function no longer has a significant downward trend.

[0126] Step S5, using the trained overall network model to evaluate the remote sensing ship image data to obtain a retrieval result;

[0127] Further, the process of using the trained overall network model to evaluate the remote sensing ship image data in step S5 to obtain a retrieval result is as follows:

[0128] In the remote sensing ship image retrieval task, the average precision mAP (mean Average Precision) is mainly used to evaluate the performance of ship retrieval, and the average precision mAP is calculated according to the precision and recall, so as to evaluate the performance of the network model on unseen data, and to verify the generalization ability of the network model. In the subsequent test set experiment part, the overall performance of the model can be evaluated according to the mAP;

[0129] The precision (Precision) refers to the proportion of the correct ship category images in all retrieved remote sensing ship images. The specific calculation formula of the precision is as follows:

[0130] ;

[0131] Among them, denotes the number of positive ship prediction results that are positive ship examples, denotes the number of negative ship prediction results that are positive ship examples;

[0132] Recall is used to evaluate the proportion of similar ships retrieved in all similar ships in the retrieval process, and the specific calculation formula of Recall is as follows:

[0133] ;

[0134] wherein, denotes the number of positive ship prediction results that are positive ship examples;

[0135] The retrieval results are sorted in descending order of confidence, and then the confidence of each result is taken as a threshold, and a precision and recall can be obtained each time, and all points are plotted on the precision-recall graph, and the PR curve is obtained by connecting each point to obtain a line graph, and the area of the curve is (Average Precision, AP), and the specific calculation process is as follows:

[0136] ;

[0137] wherein, is the number of retrieval lists, is the of each retrieval sample.

[0138] The experiment of the application uses the trained overall network model to obtain retrieval results:

[0139] (1) Retrieval mAP comparison experiment of four data sets on the remote sensing ship image data set under different methods

[0140] In order to verify the effectiveness of the method proposed in the application, CGWX, DSCR

[55] , FGSC-23

[56] and FGSCR-42

[57] in the test set are selected, and comparison experiments are performed on the four data sets; and in order to be rigorous in the experiment, the remote sensing ship image descriptor is set to a 512-dimensional vector, and the retrieval mAP of the four data sets on the remote sensing ship image data set under different methods is compared, and the detailed comparison experiment results are shown in Table 2:

[0141] Table 2 Comparison of retrieval mAP of different methods on remote sensing ship image data set

[0142]

[0143] As can be seen from Table 2, the mAP values of the method proposed in the application on the four data sets are all higher than those of other methods, indicating that the method has good performance in remote sensing ship image retrieval;

[0144] (2) Comparison experiment of mAP results of four data sets under different models

[0145] In order to evaluate the importance and effectiveness of different modules in the remote sensing ship image retrieval method based on spatial invariance proposed in the application, an ablation experiment is performed on different data sets; the full model proposed in the application is compared with the following three reduced versions: 1) the model using the method of the application to remove the spatial transformation attention module, denoted as w / o ST; 2) the model using the method of the application without considering the generalized attention module in the feature extraction network, denoted as w / o GA; 3) the model using the method of the application to remove the feature descriptor extraction module, denoted as w / o DE; the mAP results of the data sets under different models are shown in Table 3:

[0146] Table 3 mAP results of data sets under different models

[0147]

[0148] Table 3 shows the mAP results of the 512-bit continuous real-valued image feature descriptor experiment of the w / o ST model, the w / o GA model, the w / o DE model and the full model on the CGWX, DSCR, FGSC-23 and FGSCR-42 data sets; on the DSCR data set, removing the spatial transformation attention module, the attention module in the feature extraction network and the feature descriptor extraction module reduces the mAP to 70.6%, 85.2% and 76.6% respectively; it can be seen that removing the spatial transformation attention module has the greatest impact on accuracy, followed by the attention module in the feature extraction network and the feature descriptor extraction module, which also play a certain role; for example, the introduction of the spatial transformation attention module can increase the mAP of the FGSCR-42 data set from 75.2% to 96.1%. The feature descriptor extraction module increases the mAP of the FGSC-23 data set from 76.1% to 81.5%;

[0149] Based on the above experimental results, it can be seen that the retrieval performance of the application on the GGWS, DSCR, FGSC-23 and FGSCR-42 data sets is better than that of the existing deep feature-based method;

[0150] In specific implementation, the above process can be realized by computer software technology to automatically run the process.

[0151] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that such additions and modifications be included within the scope of the application. It is the following claims, including any amendments thereto, which define the scope of the application.

[0152] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for retrieving remote sensing ship images based on spatial invariance, characterized in that, The application relates to a remote sensing ship image retrieval method based on a spatial transformation attention module, a feature extraction network and a feature descriptor extraction module. Step S1: integrating existing remote sensing ship images and dividing the images into a training set, a verification set and a test set according to different categories; Step S2: preprocessing remote sensing ship images in the training set; Step S3: based on the spatial transformation attention module, the feature extraction network and the feature descriptor extraction module, randomly picking a remote sensing ship image after preprocessing as input to construct an overall network model; The process of constructing the overall network model is as follows: S3a: constructing a spatial transformation attention module by using a spatial transformation module and an attention layer; The process of constructing the spatial transformation attention module in the step S3a is as follows: (1) the construction process of the spatial transformation module: First, the remote sensing ship image is preprocessed The position transformation parameters are obtained by two-dimensional affine transformation The specific calculation formula is as follows: ; wherein, is a position transformation parameter, is a remote sensing ship image, , respectively correspond to the height, width, RGB three channels of the remote sensing ship image, denotes six parameters of an affine transformation matrix, denotes a linear layer used to regress out the image transformation parameter, and the linear layer uses two fully connected layers for regression; Then, the feature map of the input remote sensing ship image is transformed using the method of the sampling grid, each pixel uses the calculation centered on the input feature map, and is transformed into a changed remote sensing ship image through clipping, translation, rotation, scaling and tilt conversion The specific calculation formula is as follows: ; wherein, is the changed remote sensing ship image; If the calculated coordinate value is not on the original image grid, the value of the corresponding point is calculated by using bilinear interpolation; (2) the construction process of the attention layer: Then, the changed remote sensing ship image is input into the convolutional layer As the input of the attention layer, the changed remote sensing ship image is input into the convolutional layer The convolutional layer is used to change the remote sensing ship image The feature map is obtained by performing the feature dimension operation The maximum pooling and average pooling are performed on the feature map to obtain two feature maps with different attention points The convolutional layer is used to reduce the dimension and splice the two feature maps in the channel dimension The convolutional layer is used to generate the final spatial attention feature map The function is used to scale to 0-1 as the channel attention mechanism weight The changed remote sensing ship image is multiplied by the final remote sensing ship image feature map to obtain the final remote sensing ship image feature map, and the specific calculation formula is as follows: ； In the formula is the last remote sensing ship image spatial transformation attention feature map input into the remote sensing ship feature extraction network, is is a function, is a feature splicing operation, is an average pooling operation, is a maximum pooling operation, and are all convolutional layer dimension reduction operations, is a feature map after dimension increasing; S3b: constructing a remote sensing ship feature extraction network by using generalized average pooling and an attention mechanism; S3c: constructing a remote sensing ship feature descriptor extraction module; Step S4: training the overall network model; Step S5: using the trained overall network model to evaluate remote sensing ship image data to obtain a retrieval result.

2. The spatial-invariance-based remote-sensing ship image retrieval method according to claim 1, characterized in that, The remote sensing ship image data set in the step S1 adopts four data sets, and since the proportions of the data sets are different, after integrating different category images in the four data sets, the images are divided into the training set, the verification set and the test set according to the proportions.

3. The spatial-invariance-based remote-sensing ship image retrieval method according to claim 1, characterized in that, The preprocessing mode in the step S2 is: enhancing the data.

4. The spatial-invariance-based remote-sensing ship image retrieval method according to claim 3, characterized in that, The enhancing mode is: randomly flipping, rotating and adjusting the brightness, saturation and contrast of the remote sensing ship image, so that the network can better extract semantic features in the remote sensing ship image.

5. The spatial-invariance-based remote-sensing ship image retrieval method according to claim 1, characterized in that, The process of constructing the remote sensing ship feature extraction network in the step S3b is as follows: (1) the calculation process of the generalized average pooling: The generalized average pooling is used instead of the maximum pooling layer in the inside to reserve more activation information, so as to improve the feature representation capability The calculation formula is as follows: ； wherein, is data for different channels of different images, is the number of data channels, is a parameter for generalized average pooling; (2) the construction process of the attention mechanism: First, the spatial dimension is compressed, and each channel is averaged into a value using average pooling. Second, the correlation between the channels is learned using the weights of the fully connected layer to obtain an attention map. Finally, the original input is multiplied by the attention map to obtain the output of the feature extraction network, i.e., the high-level semantic features of the remote sensing ship image The specific calculation formula is as follows: ; wherein, is the original input, i.e. the output features of the fourth layer in the feature extraction network as the original input of the attention mechanism, is a fully connected layer, is an average pooling operation, is a 1x1 convolution operation, is the operation process of the attention mechanism.

Images