Remote sensing ship image retrieval method based on spatial invariance

Abstract

The invention belongs to the technical field of remote sensing image processing, and discloses a remote sensing ship image retrieval method based on spatial invariance, which comprises the following steps: integrating and dividing a remote sensing ship image into a training set, a verification set and a test set, extracting the remote sensing ship image from the training set for preprocessing, constructing an overall network model and then training, and finally evaluating the trained network model to obtain a retrieval result. According to the method, firstly, a module based on spatial invariance is designed, a continuous real value descriptor is extracted by using a deep convolutional neural network, a spatial transformation attention mechanism is introduced, and the adaptive ability and learning efficiency of the model to spatial invariance features are enhanced; secondly, a module based on self-attention hash is provided, image representation efficiency is improved through multi-scale image embedding, attention regularization in a visual encoder is optimized, and the problem of quantization loss in hash mapping is effectively solved; experimental results show that the retrieval performance of the method on GGWS, DSCR, FGSC-23 and FGSCR-42 data sets is superior to that of an existing method based on depth features.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing, and in particular relates to a remote sensing ship image retrieval method based on space invariance. Background Art

[0002] Remote sensing ship image retrieval technology is crucial to improving the efficiency of maritime traffic management and safety monitoring, quickly identifying suspicious ships, and correcting route deviations. It also plays an important role in optimizing logistics management, performing maritime rescue missions, and implementing environmental monitoring. With the continuous advancement of remote sensing technology, the increase in the number of ship images has brought new challenges to traditional remote sensing ship image retrieval technology.

[0003] Existing ship image retrieval methods are mainly divided into traditional manual feature extraction and deep learning-based methods. Among them, traditional methods rely on manually designed features, which are often accompanied by a large number of uncertain factors, such as the generalization problem of the designed features. Although directly extracting features with convolutional neural networks can improve some robustness, the extracted features may lack a certain focus, making simple convolution and pooling operations only focus on general features. In real remote sensing ship retrieval scenarios, there will be further requirements, such as the rotation invariance of ships and the multi-scale of remote sensing. Since remote sensing images are affected by distance and angle when shooting, the scale and appearance characteristics of ships will change, which will pose a challenge to existing feature extraction methods. At the same time, the background information in the image is often of no practical help to the identification and retrieval of ships, and may even cause interference. Facing these challenges, researchers are prompted to explore more efficient and accurate remote sensing ship image retrieval technology. Summary of the invention

[0004] Aiming at the deficiencies existing in the background technology and taking into account the characteristics of remote sensing ship images, the present invention proposes a remote sensing ship image retrieval method based on spatial invariance.

[0005] The technical solution adopted by the present invention is a remote sensing ship image retrieval method based on spatial invariance, and the steps are as follows: Step S1, integrating existing remote sensing ship images and dividing them into training sets, verification sets and test sets according to different categories of images; Step S2, preprocessing the remote sensing ship images in the training set; Step S3, based on the spatial transformation attention module, the feature extraction network and the feature descriptor extraction module, randomly select the pre-processed remote sensing ship image as input to build the overall network model; Step S4, training the overall network model; Step S5, using the trained overall network model to evaluate the remote sensing ship image data to obtain retrieval results.

[0006] Preferably, the existing data set in step S1 uses four data sets, namely, the remote sensing ship image data set DSCR of warship remote sensing ship, the remote sensing ship image data set FGSC-23 provided by Google Earth images and high-resolution satellites, the remote sensing ship image data set FGSCR-42 composed of cropped images of multiple target detection data sets, and the remote sensing ship image data set CGWX taken by the "Jilin-1" satellite. Since the division ratios of each data set are different, after integrating the different categories of pictures in the four data sets used, they are divided into training set, verification set and test set according to the ratio.

[0007] Preferably, before randomly extracting remote sensing ship images from the training set as network input in step S2, the remote sensing ship images are preprocessed to better extract features in the remote sensing ship images. The preprocessing method: randomly flipping and rotating, adjusting the brightness, saturation and contrast of the remote sensing ship images, is used to enhance the data, so that the network is more focused on extracting semantic features in the remote sensing ship images, thereby improving network generalization and robustness.

[0008] Preferably, in step S3, the process of constructing the overall network model is as follows; S3a, constructs a spatial transformation attention module using the spatial transformation module and the attention layer; S3b, constructing a remote sensing ship feature extraction network through generalized average pooling and attention mechanism; S3c, construct remote sensing ship feature descriptor extraction module.

[0009] Preferably, the process of constructing the spatial transformation attention module in step S3a is as follows: The spatial transformation attention module can learn the spatial invariance of data before inputting into the backbone feature network, and the attention mechanism can reduce the difficulty of model training caused by interference information; the spatial transformation attention module can be placed at the front position of the convolutional neural network, so that the network can perform spatial transformation and eliminate redundant information before extracting features; The spatial transformer attention module consists of a spatial transformer module and an attention layer; (1) Construction process of spatial transformation module: First, remote sensing ship images Obtain position transformation parameters through two-dimensional affine transformation , the specific calculation formula is as follows: ; in, is the position transformation parameter, To input remote sensing ship images, , They correspond to the height, width, and RGB channels of remote sensing ship images respectively. Represents the six parameters of the affine transformation matrix, Represents a linear layer used to regress image transformation parameters. This linear layer uses two fully connected layers for regression; Next, the sampling grid method is used to transform the feature map of the input remote sensing ship image. Each pixel is calculated using the input feature map as the center, and is converted into a changed remote sensing ship image after cropping, translation, rotation, scaling and tilting. , the specific calculation formula is as follows: ; in, is the remote sensing ship image after the change; If the calculated coordinate value is not on the grid of the original image, bilinear interpolation is used to calculate the value of the corresponding point; (2) Construction process of attention layer: Next, the changed remote sensing ship image is used As the input of the attention layer, we use Convolutional layer on the changed remote sensing ship image Perform feature dimension upgrade operation to obtain the feature map after dimension upgrade , and then perform maximum pooling and average pooling on the feature map after dimensionality increase to obtain two feature maps with different focus points, and then use The convolutional layer of is reduced in dimension and concatenated with the first two feature maps in the channel dimension, and then used The convolution is used to produce the final spatial attention feature map, and finally the The function is scaled to (0-1) as the channel attention mechanism weight, and finally combined with the changed remote sensing ship image Multiply them to obtain the final remote sensing ship image feature map. The specific calculation formula is as follows: ; In the formula That is, the final remote sensing ship image spatial transformation attention feature map input into the remote sensing ship feature extraction network. for function, is the feature concatenation operation, is the average pooling operation, is the maximum pooling operation, and These are all convolutional layer dimensionality reduction operations. It is the feature map after dimension increase.

[0010] Preferably, the process of constructing the remote sensing ship feature extraction network in step S3b is as follows: (1) The calculation process of generalized average pooling: Since the maximum pooling will lose some activation information, in order to improve the feature representation ability, the generalized mean pooling (GeM) is used instead of the maximum pooling layer. The calculation formula is as follows: ; in, For data of different channels of different images, is the number of data channels, is the parameter of generalized average pooling; (2) Construction process of attention mechanism: In order to further improve the model representation ability, the relationship between the attention of channels is considered and the attention mechanism is added to the channel: first, the spatial dimension is compressed, and each channel is averaged into a value using the maximum average pooling, which has a global receptive field. Secondly, the weight of the fully connected layer is used to learn the correlation between the above channels to obtain the attention map. Finally, the original input is multiplied by the attention map to obtain the output of the feature extraction network, that is, the high-level semantic features of the remote sensing ship image. , the specific calculation formula is as follows: ; in, is the original input, that is, the output feature of the fourth layer in the feature extraction network is used as the original input of the attention mechanism. is the fully connected layer, is the maximum average pooling operation, is a 1x1 convolution operation, for" ”The operation process of the attention mechanism.

[0011] Preferably, the process of constructing the remote sensing ship feature descriptor extraction module in step S3c is as follows: Convert the remote sensing ship image feature map obtained by the remote sensing ship feature extraction network into an image descriptor :First use one Convolution gets The characteristics of is the number of cluster centers, is the feature dimension, and then the weight coefficient of the difference matrix is ​​generated through the Softmax function. Finally, a VLAD Core is used to implement it, that is, the coefficient is multiplied by the residual (the difference between the remote sensing ship image feature vector x and the cluster center c), and finally it is implemented through two normalizations, including in-class normalization and out-of-class normalization. The specific calculation formula is as follows: ; in, It is the high-level semantic feature of remote sensing ship image The The local feature descriptor dimensions, It is the first Cluster centers, , and are all trainable parameters. is a twice normalized function, is the descriptor-qualified function, that is The descriptor is flattened and then mapped through the fully connected layer into The vector of Generally, 512 is used so that the local feature descriptor can be effectively encoded into a global vector of fixed length, and finally the remote sensing ship image descriptor is obtained. .

[0012] Preferably, the process of training the overall network model in step S4 is as follows: The remote sensing ship image is input into the model to obtain the corresponding ship feature descriptor, and then the loss function in the remote sensing ship image retrieval model is calculated. The loss result is back-propagated, and the Pytorch framework automatically updates the optimization model parameters until the training is completed. In order to enhance both intra-class compactness and inter-class differences, the additive angular margin loss (ArcFace) is used. The core idea is to establish an angle cosine interval between the feature vector and the class label, so that the angle between the feature descriptors of different ship categories is increased, while the angle between the feature descriptors of the same ship category is reduced; it not only utilizes the semantic features of the ship category, but also improves the accuracy of the mapped feature descriptors; Additive angular margin loss (ArcFace) is used: ArcFace first descripts the remote sensing ship image Normalize it to make it a unit vector. Then, for each data sample, map the feature vector to a specific angle interval through a fully connected layer, so that the feature vectors of the same category are closer to the predefined angle. Finally, calculate the category probability of each category through the Softmax function and use it to calculate the loss. The specific calculation is as follows: ; in, represents the number of training samples, that is, the number of samples in the remote sensing ship image dataset, Indicates the index of the current remote sensing ship image, ranging from 1 to , Indicates The category label of the samples indicates the category of the remote sensing ship image. Indicates Feature descriptors of remote sensing ship images and their ship category labels The angle of Represents the angle cosine interval, which is used to enhance the difference between feature vectors. represents the scaled Softmax function, which is used to control the scale of the feature vector. The loss function can effectively use the semantic information of remote sensing ship image categories to learn network parameters; In order to improve the generalization of the model, all input remote sensing ship images first use common data enhancement methods: first scale them to 224×224 size, then perform 0-1 normalization and standardization calculation on the values ​​of the three RGB channels. Standardization can reduce the instability caused by the difference value, that is, subtract the channel mean from each input channel and divide it by the channel standard deviation; because the mean and standard deviation of remote sensing ship images are quite different from those of natural images, the reference experimental data set and the public data set contain a total of 12w remote sensing ship images, and the mean is changed to [0.294, 0.323, 0.318], and the standard deviation is changed to [0.207, 0.194, 0.191]; finally, the common image enhancement method is introduced, that is, random rotation, random flipping, and random filling blocks are performed with a probability of 0.1; and the Adam optimizer is used for parameter optimization during training; The training batch size is set to 100 and the initial learning rate is Set to 2e-5, when the loss function no longer has a clear downward trend, the training process is terminated early.

[0013] Preferably, the process of evaluating the remote sensing ship image data using the trained overall network model in step S5 to obtain the retrieval result is as follows: In the remote sensing ship image retrieval task, the mean average precision (mAP) is calculated based on the precision and recall rate, and the mean average precision (mAP) is used to evaluate the ship retrieval performance, so as to evaluate the performance of the network model on unseen data, thereby verifying the generalization ability of the network model. In the subsequent test set experiment, the overall performance of the model can be evaluated based on mAP. Precision refers to the proportion of images with correct ship categories in all search results among the retrieved remote sensing ship images. The specific calculation formula for precision is as follows: ; in, Indicates the number of positive ship prediction results as positive examples of ships, Indicates the number of negative ship prediction results that are positive ship examples; Recall is used to evaluate the proportion of similar ships that can be retrieved among all similar ships during the retrieval process. The specific calculation formula for recall is as follows: ; in, Indicates the number of samples whose prediction results for positive ship are negative examples of ship; Sort the search results by confidence from high to low, and then use the confidence of each result as the threshold in turn. Each time, a precision and recall rate can be obtained. All points are plotted on the precision-recall rate graph. The PR curve is a line graph obtained by connecting each point. The area of ​​the curve is (Average Precision, AP). The specific calculation process of the average precision mAP is as follows: ; in, To retrieve the number of lists, For each retrieved sample .

[0014] Compared with the prior art, the present invention proposes a remote sensing ship image retrieval method based on spatial invariance, and the present invention has the following advantages: (1) The present invention uses a spatial transformation attention module to reduce the model's sensitivity to spatial transformations such as translation, scaling, and rotation, and can dynamically perform spatial transformations on images, thereby improving the model's ability to adapt to changes in remote sensing ship images and enhancing the model's generalization and robustness under different conditions; (2) The present invention uses the attention mechanism to focus on key features. By combining channel attention and spatial attention, it reduces background interference, effectively copes with problems of complex background and scale changes, and improves the accuracy of feature extraction and retrieval performance. (3) The proposed method can adapt to the spatial invariance of images, ensuring that ship images at different viewing angles and scales can be accurately identified, significantly improving the accuracy of remote sensing ship image retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1A network structure diagram of a remote sensing ship image retrieval method based on spatial invariance of the present invention; Figure 2 This is a comparison chart of the retrieval performance results of the method proposed in the present invention and other methods on different data sets, that is, the impact of the dimension of the real-valued continuous feature descriptor of remote sensing ship images on the retrieval accuracy is evaluated on four remote sensing ship image data sets; Figure 3 In order to compare the visualization results of w / o ST model, w / o GA model, w / o DE model and the full model, the t-SNE algorithm was used to reduce the dimension of the remote sensing ship image descriptor data and category data and map them to two-dimensional space; Figure 3 (a) t-SNE visualization of retrieval results of w / o ST model on FGSC-23 dataset, Figure 3 (b) t-SNE visualization of retrieval results of the w / o GA model on the FGSC-23 dataset. Figure 3 (c) t-SNE visualization of retrieval results of the w / oDE model on the FGSC-23 dataset. Figure 3 (d) t-SNE visualization of the retrieval result image of the full model on the FGSC-23 dataset; Figure 4 The results of remote sensing ship image retrieval on DSCR dataset w / o ST model; Figure 5 The results of remote sensing ship image retrieval on DSCR dataset with / o GA model; Figure 6 The remote sensing ship image retrieval results on the DSCR dataset w / o DE model; Figure 7 This is the remote sensing ship image retrieval result of the full model on the DSCR dataset. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to further clearly and completely describe the technical solutions in the embodiments of the present application. It should be noted that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0017] In order to make the invention objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application are further described in detail in conjunction with the drawings in the specification: In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the advantages of the present invention will be further illustrated by comparing the embodiments in conjunction with the drawings and specific implementation methods.

[0018] The present invention proposes a remote sensing ship image retrieval method based on spatial invariance. The steps of the method are described in detail below: Step S1, integrating existing remote sensing ship images and dividing them into training sets, verification sets and test sets according to different categories of images; Furthermore, the existing data set in step S1 uses four data sets, namely, the remote sensing ship image data set DSCR of warships, the remote sensing ship image data set FGSC-23 provided by Google Earth images and high-resolution satellites, the remote sensing ship image data set FGSCR-42 composed of cropped images of multiple target detection data sets, and the remote sensing ship image data set CGWX taken by the "Jilin-1" satellite. Since the division ratios of each data set are different, after integrating the different categories of pictures in the four data sets used, they are divided into training set, verification set and test set in a ratio of 6:2:2.

[0019] Step S2, preprocessing the remote sensing ship images in the training set; Furthermore, before randomly extracting remote sensing ship images from the training set as network input in step S2, the remote sensing ship images are preprocessed to better extract features in the remote sensing ship images. The preprocessing method: randomly flipping and rotating, adjusting the brightness, saturation and contrast of the remote sensing ship images, is used to enhance the data, so that the network is more focused on extracting semantic features in the remote sensing ship images, thereby improving network generalization and robustness.

[0020] Step S3, based on the spatial transformation attention module, the feature extraction network and the feature descriptor extraction module, randomly select the pre-processed remote sensing ship image as input to build the overall network model; Furthermore, in step S3, the process of constructing the overall network model is as follows; S3a, constructs a spatial transformation attention module using the spatial transformation module and the attention layer; Furthermore, the process of constructing the spatial transformation attention module in step S3a is as follows: The spatial transformation attention module can learn the spatial invariance of data before inputting into the backbone feature network, and the attention mechanism can reduce the difficulty of model training caused by interference information; the spatial transformation attention module can be placed at the front position of the convolutional neural network, so that the network can perform spatial transformation and eliminate redundant information before extracting features; The spatial transformer attention module consists of a spatial transformer module and an attention layer; (1) Construction process of spatial transformation module: First, remote sensing ship images Obtain position transformation parameters through two-dimensional affine transformation , the specific calculation formula is as follows: ; in, is the position transformation parameter, To input remote sensing ship images, , They correspond to the height, width, and RGB channels of remote sensing ship images respectively. Represents the six parameters of the affine transformation matrix, Represents the linear layer used to regress the image transformation parameters. Here, two fully connected layers are used for regression; Next, the sampling grid method is used to transform the feature map of the input remote sensing ship image. Each pixel is calculated using the input feature map as the center, and is converted into a changed remote sensing ship image after cropping, translation, rotation, scaling and tilting. , the specific calculation formula is as follows: ; in, is the remote sensing ship image after the change; If the calculated coordinate value is not on the grid of the original image, bilinear interpolation is used to calculate the value of the corresponding point; (2) Construction process of attention layer: Next, the changed remote sensing ship image is used As the input of the attention layer, we use Convolutional layer on the changed remote sensing ship image Perform feature dimension upgrade operation to obtain the feature map after dimension upgrade , and then perform maximum pooling and average pooling on the feature map after dimensionality increase to obtain two feature maps with different focus points, and then use The convolutional layer of is reduced in dimension and concatenated with the first two feature maps in the channel dimension, and then used The convolution is used to produce the final spatial attention feature map, and finally the The function is scaled to (0-1) as the channel attention mechanism weight, and finally combined with the changed remote sensing ship image Multiply them to obtain the final remote sensing ship image feature map. The specific calculation formula is as follows: ; In the formula That is, the final remote sensing ship image spatial transformation attention feature map input into the remote sensing ship feature extraction network. for function, is the feature concatenation operation, is the average pooling operation, is the maximum pooling operation, and These are all convolutional layer dimensionality reduction operations. It is the feature map after dimension increase.

[0021] S3b, constructing a remote sensing ship feature extraction network through generalized average pooling and attention mechanism; Furthermore, the process of constructing the remote sensing ship feature extraction network in step S3b is as follows: (1) The calculation process of generalized average pooling: Since the maximum pooling will lose some activation information, in order to improve the feature representation ability, the generalized mean pooling (GeM) is used instead of the maximum pooling layer. The calculation formula is as follows: ; in, For data of different channels of different images, is the number of data channels, is the parameter of generalized average pooling; (2) Construction process of attention mechanism: In order to further improve the model representation ability, the relationship between the attention of channels is considered and the attention mechanism is added to the channel: first, the spatial dimension is compressed, and each channel is averaged into a value using the maximum average pooling, which has a global receptive field. Secondly, the weight of the fully connected layer is used to learn the correlation between the above channels to obtain the attention map. Finally, the original input is multiplied by the attention map to obtain the output of the feature extraction network, that is, the high-level semantic features of the remote sensing ship image. , the specific calculation formula is as follows: ; in, is the original input, that is, the output feature of the fourth layer in the feature extraction network is used as the original input of the attention mechanism. is the fully connected layer, is the maximum average pooling operation, is a 1x1 convolution operation, for" ”The operation process of the attention mechanism.

[0022] S3c, constructing remote sensing ship feature descriptor extraction module; Furthermore, the process of constructing the remote sensing ship feature descriptor extraction module in step S3c is as follows: Convert the remote sensing ship image feature map obtained by the remote sensing ship feature extraction network into an image descriptor :First use one Convolution gets The characteristics of is the number of cluster centers, is the feature dimension, and then the weight coefficient of the difference matrix is ​​generated through the Softmax function. Finally, a VLAD Core is used to implement it, that is, the coefficient is multiplied by the residual (the difference between the remote sensing ship image feature vector x and the cluster center c), and finally it is implemented through two normalizations, including in-class normalization and out-of-class normalization. The specific calculation formula is as follows: ; in, It is the high-level semantic feature of remote sensing ship image The The local feature descriptor dimensions, It is the first Cluster centers, , and are all trainable parameters. is a twice normalized function, is the descriptor-qualified function, that is The descriptor is flattened and then mapped through the fully connected layer into The vector of Generally, 512 is used so that the local feature descriptor can be effectively encoded into a global vector of fixed length, and finally the remote sensing ship image descriptor is obtained. .

[0023] Step S4, training the overall network model; Furthermore, the process of training the overall network model in step S4 is as follows: The remote sensing ship image is input into the model to obtain the corresponding ship feature descriptor, and then the loss function in the remote sensing ship image retrieval model is calculated. The loss result is back-propagated, and then the Pytorch framework automatically updates the optimization model parameters until the training is completed. The training process of the ship image retrieval network is shown in Table 1. Table 1 Flowchart of the ship image retrieval network training process

[0024] In order to enhance both intra-class compactness and inter-class differences, the additive angular margin loss (ArcFace) is used. The core idea is to establish an angle cosine interval between the feature vector and the class label, so that the angle between the feature descriptors of different ship categories is increased, while the angle between the feature descriptors of the same ship category is reduced; it not only utilizes the semantic features of the ship category, but also improves the accuracy of the mapped feature descriptors; Additive angular margin loss (ArcFace) is used: ArcFace first descripts the remote sensing ship image Normalize it to make it a unit vector. Then, for each data sample, map the feature vector to a specific angle interval through a fully connected layer, so that the feature vectors of the same category are closer to the predefined angle. Finally, calculate the category probability of each category through the Softmax function and use it to calculate the loss. The specific calculation is as follows: ; in, represents the number of training samples, that is, the number of samples in the remote sensing ship image dataset, Indicates the index of the current remote sensing ship image, ranging from 1 to , Indicates The category label of the samples indicates the category of the remote sensing ship image. Indicates Feature descriptors of remote sensing ship images and their ship category labels The angle of Represents the angle cosine interval, which is used to enhance the difference between feature vectors. represents the scaled Softmax function, which is used to control the scale of the feature vector. The loss function can effectively use the semantic information of remote sensing ship image categories to learn network parameters; In order to improve the generalization of the model, all input remote sensing ship images are first enhanced using common data augmentation methods: first they are scaled to 224×224, and then the values ​​of the three RGB channels are normalized and standardized to 0-1. Standardization can reduce the instability caused by the difference value, that is, the channel mean is subtracted from each input channel and divided by the channel standard deviation. The common method is to use the mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] of the natural image ImageNet

[58] dataset. However, the mean and standard deviation of remote sensing ship images are quite different from those of natural images. Referring to the experimental dataset and the public dataset, a total of 120,000 remote sensing ship images, the mean is changed to [0.294, 0.323, 0.318] and the standard deviation is changed to [0.207, 0.194, 0.191]; finally, a common image enhancement method is introduced, that is, random rotation, random flipping, and random filling blocks are performed with a probability of 0.1; and the Adam optimizer is used to optimize the parameters during training; The training batch size is set to 100 and the initial learning rate is Set to 2e-5, when the loss function no longer has a clear downward trend, the training process is terminated early.

[0025] Step S5, using the trained overall network model to evaluate the remote sensing ship image data to obtain a retrieval result; Furthermore, the process of evaluating the remote sensing ship image data using the trained overall network model in step S5 to obtain the retrieval result is as follows: In the remote sensing ship image retrieval task, the mean average precision (mAP) is mainly used to evaluate the ship retrieval performance. The mean average precision (mAP) is calculated based on the precision and recall rate to evaluate the performance of the network model on unseen data, thereby verifying the generalization ability of the network model. In the subsequent test set experiment, the overall performance of the model can be evaluated based on mAP. Precision refers to the proportion of images with correct ship categories in all search results among the retrieved remote sensing ship images. The specific calculation formula for precision is as follows: ; in, Indicates the number of positive ship prediction results as positive examples of ships, Indicates the number of negative ship prediction results that are positive ship examples; Recall is used to evaluate the proportion of similar ships that can be retrieved among all similar ships during the retrieval process. The specific calculation formula for recall is as follows: ; in, Indicates the number of samples whose prediction results for positive ship are negative examples of ship; Sort the search results from high to low according to the confidence, and then use the confidence of each result as the threshold in turn. Each time, a precision and recall rate can be obtained. All points are plotted on the precision-recall rate graph. The PR curve is a line graph obtained by connecting each point. The area of ​​the curve is (Average Precision, AP). According to the average precision mAP evaluation, the specific calculation process is as follows: ; in, To retrieve the number of lists, For each retrieved sample .

[0026] The present invention uses the trained overall network model to obtain the experiment of retrieval results: (1) Comparative experiment on retrieval mAP of four datasets on remote sensing ship image dataset using different methods In order to verify the effectiveness of the proposed method, CGWX, DSCR

[55] , FGSC-23

[56] , and FGSCR-42

[57] were selected from the test set, and comparative experiments were carried out on these four datasets. In addition, for the rigor of the experiment, the dimension of the remote sensing ship image descriptor was uniformly set to a 512-dimensional vector, and the retrieval mAP of the four datasets on the remote sensing ship image dataset was compared under different methods. The detailed comparative experimental results are shown in Table 2: Table 2 Comparison of retrieval mAP of different methods on remote sensing ship image dataset

[0027] As can be seen from Table 2, the mAP values ​​of the method proposed in this paper on the four data sets are higher than those of other methods, indicating that this method has better performance in remote sensing ship image retrieval; (2) Comparative experiment of mAP results of four datasets under different models In order to evaluate the importance and effectiveness of different modules in the remote sensing ship image retrieval method based on spatial invariance proposed in the present invention, ablation experiments were carried out on different data sets; the proposed full model was compared with the following three reduced versions: 1) the model using the method of the present invention to remove the spatial transformation attention module, denoted as w / o ST; 2) the model using the method of the present invention without considering the generalized attention module in the feature extraction network, denoted as w / o GA; 3) the model using the method of the present invention to remove the feature descriptor extraction module, denoted as w / o DE; the mAP results of the data sets under different models are shown in Table 3: Table 3 mAP results of datasets under different models

[0028] Table 3 shows the mAP results of the 512-bit continuous real-valued image feature descriptor experiments on the CGWX, DSCR, FGSC-23, and FGSCR-42 datasets for the w / o ST model, w / o GA model, w / o DE model, and the full model; on the DSCR dataset, removing the spatial transformation attention module, the feature extraction attention module, and the feature descriptor extraction module reduces the mAP to 70.6%, 85.2%, and 76.6%, respectively; it can be seen that removing the spatial transformation attention module has the greatest impact on the accuracy, followed by the feature extraction attention module and the feature descriptor extraction module can also play a certain role; for example, the introduction of the spatial transformation attention module can increase the mAP of the FGSCR-42 dataset from 75.2% to 96.1%. The feature descriptor extraction module increases the mAP of the FGSC-23 dataset from 76.1% to 81.5%; In summary, the experimental results show that the retrieval performance of the proposed method on the GGWS, DSCR, FGSC-23 and FGSCR-42 datasets is better than the existing deep feature-based methods; During specific implementation, the above process can be automatically operated using computer software technology.

[0029] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0030] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A remote sensing ship image retrieval method based on spatial invariance, characterized in that: include: Step S1, integrating existing remote sensing ship images and dividing them into training sets, verification sets and test sets according to different categories of images; Step S2, preprocessing the remote sensing ship images in the training set; Step S3, based on the spatial transformation attention module, the feature extraction network and the feature descriptor extraction module, randomly select the pre-processed remote sensing ship image as input to build the overall network model; Step S4, training the overall network model; Step S5, using the trained overall network model to evaluate the remote sensing ship image data to obtain retrieval results.

2. The remote sensing ship image retrieval method based on spatial invariance according to claim 1 is characterized in that: In step S1, the existing remote sensing ship image dataset uses four datasets. Since the division ratios of each dataset are different, after integrating the different categories of pictures in the four datasets, they are divided into a training set, a validation set and a test set according to the ratio.

3. The remote sensing ship image retrieval method based on spatial invariance according to claim 1 is characterized in that: The preprocessing method in step S2 is: enhancing the data.

4. The remote sensing ship image retrieval method based on spatial invariance according to claim 3 is characterized in that: The enhancement processing method is: randomly flipping, rotating and adjusting the brightness, saturation and contrast of the remote sensing ship image, so that the network can better extract the semantic features in the remote sensing ship image.

5. The remote sensing ship image retrieval method based on spatial invariance according to claim 1 is characterized in that: In step S3, the process of constructing the overall network model is as follows: S3a, constructs a spatial transformation attention module using the spatial transformation module and the attention layer; S3b, constructing a remote sensing ship feature extraction network through generalized average pooling and attention mechanism; S3c, construct remote sensing ship feature descriptor extraction module.

6. The remote sensing ship image retrieval method based on spatial invariance according to claim 5 is characterized in that: The process of constructing the spatial transformation attention module in step S3a is as follows: (1) Construction process of spatial transformation module: First, remote sensing ship images Obtain position transformation parameters through two-dimensional affine transformation , the specific calculation formula is as follows: ； in, is the position transformation parameter, For remote sensing ship images, , They correspond to the height, width, and RGB channels of remote sensing ship images respectively. Represents the six parameters of the affine transformation matrix, Represents a linear layer used to regress image transformation parameters. This linear layer uses two fully connected layers for regression; Next, the sampling grid method is used to transform the feature map of the input remote sensing ship image. Each pixel is calculated using the input feature map as the center, and is converted into a changed remote sensing ship image after cropping, translation, rotation, scaling and tilting. , the specific calculation formula is as follows: ； in, is the remote sensing ship image after the change; If the calculated coordinate value is not on the grid of the original image, bilinear interpolation is used to calculate the value of the corresponding point; (2) Construction process of attention layer: Next, the changed remote sensing ship image is used As the input of the attention layer, we use The convolutional layer is used to transform the remote sensing ship image after the change Perform feature dimension upgrade operation to obtain the feature map after dimension upgrade , and then perform maximum pooling and average pooling on the feature map after dimensionality increase to obtain two feature maps with different focus points, and then use The convolutional layer of is reduced in dimension and concatenated with the first two feature maps in the channel dimension, and then used The convolution is used to produce the final spatial attention feature map, and finally the The function is scaled to (0-1) as the channel attention mechanism weight, and finally combined with the changed remote sensing ship image Multiply them to obtain the final remote sensing ship image feature map. The specific calculation formula is as follows: ； In the formula That is, the final remote sensing ship image spatial transformation attention feature map input into the remote sensing ship feature extraction network. for function, is the feature concatenation operation, is the average pooling operation, is the maximum pooling operation, and These are all convolutional layer dimensionality reduction operations. It is the feature map after dimension increase.

7. The remote sensing ship image retrieval method based on spatial invariance according to claim 5 is characterized in that: The process of constructing the remote sensing ship feature extraction network in step S3b is as follows: (1) The calculation process of generalized average pooling: Generalized average pooling is used instead of the maximum pooling layer to retain more activation information to improve feature representation capabilities. The calculation formula is as follows: ； in, For data of different channels of different images, is the number of data channels, is the parameter of generalized average pooling; (2) Construction process of attention mechanism: First, the spatial dimension is compressed, and each channel is averaged into one value using the maximum average pooling. Then, the weights of the fully connected layer are used to learn the correlation between the above channels to obtain the attention map. Finally, the original input is multiplied by the attention map to obtain the output of the feature extraction network, that is, the high-level semantic features of the remote sensing ship image. , the specific calculation formula is as follows: ； in, is the original input, that is, the output feature of the fourth layer in the feature extraction network is used as the original input of the attention mechanism. is the fully connected layer, is the maximum average pooling operation, is a 1x1 convolution operation, for" ”The operation process of the attention mechanism.

8. The remote sensing ship image retrieval method based on spatial invariance according to claim 5 is characterized in that: The process of constructing the remote sensing ship feature descriptor extraction module in step S3c is as follows: Convert the remote sensing ship image feature map obtained by the remote sensing ship feature extraction network into an image descriptor :First use one Convolution gets The characteristics of is the number of cluster centers, is the feature dimension, and then the weight coefficient of the difference matrix is ​​generated through the Softmax function. Finally, a VLAD Core is used to implement it, that is, the coefficient is multiplied by the residual of the remote sensing ship image feature vector x and the cluster center c. Finally, it is implemented through two normalizations, including in-class normalization and out-of-class normalization. The specific calculation formula is as follows: ； in, It is the high-level semantic feature of remote sensing ship image The The local feature descriptor dimensions, It is the first Cluster centers, , and are all trainable parameters. is a twice normalized function, is the descriptor-qualified function, that is The descriptor is flattened and then mapped through the fully connected layer into The vector of Generally, 512 is used so that the local feature descriptor can be effectively encoded into a global vector of fixed length, and finally the remote sensing ship image descriptor is obtained. .

9. The remote sensing ship image retrieval method based on spatial invariance according to claim 1 is characterized in that: The process of training the overall network model in step S4 is as follows: inputting the remote sensing ship image into the overall network model to obtain the corresponding ship feature descriptor, then calculating the loss function in the remote sensing ship image retrieval model, back-propagating according to the loss result, and then automatically updating the optimization model parameters by the Pytorch framework until the training is completed; First, the additive angular margin loss, ArcFace, is used: ArcFace first descriptors the remote sensing ship image Normalize it to make it a unit vector. Then, for each data sample, map the feature vector to a specific angle interval through a fully connected layer, so that the feature vectors of the same category are closer to the predefined angle. Finally, calculate the category probability of each category through the Softmax function and use it to calculate the loss. The specific calculation formula is as follows: ； in, represents the number of training samples, that is, the number of samples in the remote sensing ship image dataset, Indicates the index of the current remote sensing ship image, ranging from 1 to , Indicates The category label of the samples indicates the category of the remote sensing ship image. Indicates Feature descriptors of remote sensing ship images and their ship category labels The angle of Represents the angle cosine interval, which is used to enhance the difference between feature vectors. Represents the scaled Softmax function, which is used to control the scale of the feature vector; Next, all input remote sensing ship images are first subjected to common data enhancement methods to improve the generalization of the model: first, the input remote sensing ship images are scaled to 224×224, and then the values ​​of the three RGB channels are normalized and standardized from 0 to 1. Standardization can reduce the instability caused by the difference value, that is, the channel mean is subtracted from each input channel and divided by the channel standard deviation. Because the mean and standard deviation of remote sensing ship images are quite different from those of natural images, a total of 12w remote sensing ship images are referenced from the experimental data set and the public data set, with the mean being [0.294, 0.323, 0.318] and the standard deviation being [0.207, 0.194, 0.191]. Finally, common image enhancement methods are introduced, that is, random rotation, random flipping, and random filling blocks are performed with a probability of 0.1, and then the parameters are optimized during training through the Adam optimizer. The training batch size is set to 100 and the initial learning rate is Set to 2e-5, when the loss function no longer has a clear downward trend, the training process is terminated early.

Images