Ship image retrieval method and device, electronic equipment, storage medium and product

Abstract

The invention provides a ship image retrieval method and device, electronic equipment, a storage medium and a product, and relates to the technical field of image processing, and the method comprises the steps: obtaining a to-be-retrieved ship image and a ship sample database; respectively inputting the to-be-retrieved ship image and all ship sample images in the ship sample database into a ship visual language model to obtain a hash code corresponding to the to-be-retrieved ship image output by the ship visual language model and all sample hash codes corresponding to all ship sample images in the ship sample database; determining all Hamming similarities according to the hash code and all the sample hash codes, and determining a similarity ranking list according to all the Hamming similarities; and determining a target retrieval result according to the ranking list of all the Hamming similarities. According to the technical scheme, the hash code and the sample hash code are obtained based on the ship visual language model, the target retrieval result is determined, and the accuracy of image cross-modal retrieval is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a ship image retrieval method, device, electronic equipment, storage medium and product. Background Art

[0002] With the rapid development of earth observation technology, a large amount of remote sensing image data is now generated, and efficient and convenient data management strategies are urgently needed.

[0003] In the prior art, for the retrieval of cross-modal remote sensing text images, there is a text-to-image retrieval method, which mainly uses the high-level semantic information inherent in the text to overcome the limitations of image-to-image retrieval, and realizes the retrieval of cross-modal remote sensing ship images through text features. There is also an image-to-image retrieval method, which mainly uses detailed visual information from the image itself for retrieval. The above-mentioned text-to-image retrieval method cannot capture the rich and detailed image information existing in the remote sensing image, and the image-to-image retrieval method ignores the abstract semantic information provided by the text tags, thus resulting in a decrease in the accuracy of cross-modal retrieval of remote sensing images. Summary of the invention

[0004] The present invention provides a ship image retrieval method, device, electronic device, storage medium and product, which are used to solve the defect in the prior art that the accuracy of cross-modal retrieval of remote sensing images is reduced due to simple text-to-image retrieval or image-to-image retrieval, and implement a ship visual language model combining images and texts obtained by training based on a ship full-color image dataset, a ship multispectral image dataset and label texts to perform multimodal comparative learning, cross-modal semantic fusion and hash learning processing on the ship image to be retrieved and the ship sample database, and finally determine the target retrieval result based on the obtained hash code and the sample hash code, so as to improve the accuracy of cross-modal retrieval of the ship image to be retrieved in the ship sample database.

[0005] The present invention provides a method for retrieving ship images, comprising the following steps.

[0006] Obtain a ship image to be retrieved and a ship sample database; wherein the ship sample database contains at least two ship sample images, and the ship sample images are ship images of different modalities from the ship image to be retrieved.

[0007] The ship image to be retrieved and all the ship sample images in the ship sample database are respectively input into the ship visual language model to obtain the hash code corresponding to the ship image to be retrieved and all the sample hash codes corresponding to all the ship sample images in the ship sample database output by the ship visual language model; wherein, the ship visual language model is trained based on the acquired multimodal ship sample pattern dataset including a ship full-color image dataset, a ship multispectral image dataset and a label text, and the ship visual language model is a model that processes the ship image to be retrieved and the ship sample database based on multimodal contrast learning, cross-modal semantic fusion and hash learning to obtain the corresponding hash code and sample hash code.

[0008] All Hamming similarities are determined based on the hash code and all sample hash codes, respectively, and a similarity ranking list is determined based on all Hamming similarities.

[0009] The target retrieval result is determined according to the ranking list of all Hamming similarities; wherein the target retrieval result is a set of sample ship images similar to the ship image to be retrieved, selected based on all similarities in the ranking list.

[0010] According to a ship image retrieval method provided by the present invention, a ship visual language model is trained based on the following steps: obtaining a multimodal ship sample pattern dataset; wherein the multimodal ship sample pattern dataset includes a ship panchromatic image dataset, a ship multispectral image dataset and a label text, and the label text is a paired label text corresponding to a paired sample determined by pairing the ship panchromatic image dataset and the ship multispectral image dataset; inputting the ship panchromatic image dataset, the ship multispectral image dataset and the label text into a basic ship visual language model to obtain a dataset hash code and a target loss function output by the basic ship visual language model; wherein the basic ship visual language model is a predetermined basic model for training; and optimizing the basic ship visual language model according to the dataset hash code and the target loss function to obtain a ship visual language model.

[0011] According to a ship image retrieval method provided by the present invention, a ship panchromatic image dataset, a ship multispectral image dataset and a label text are input into a basic ship visual language model to obtain a dataset hash code and a target loss function output by the basic ship visual language model, including: inputting the ship panchromatic image dataset and the ship multispectral image dataset into a multimodal contrast learning network in the basic ship visual language model to obtain image features and contrast loss output by the multimodal contrast learning network; the image features are image enhancement processing and similarity loss performed by the multimodal contrast learning network on the ship panchromatic image dataset and the ship multispectral image dataset. The image features and label texts are input into the cross-modal semantic fusion network in the basic ship visual language model to obtain the fusion features and fusion loss output by the cross-modal semantic fusion network; wherein, the fusion features are obtained by aligning and fusing the image features and label texts through the cross-modal semantic fusion network; the fusion features are input into the hash learning network in the basic ship visual language model to obtain the dataset hash code and hash loss output by the hash learning network; wherein, the dataset hash code is obtained by hash learning the fusion features through the hash learning network; the target loss function is determined according to the contrast loss, fusion loss and hash loss.

[0012] According to a ship image retrieval method provided by the present invention, fusion features are input into a hash learning network in a basic ship visual language model to obtain a data set hash code output by the hash learning network, including: inputting the fusion features into a hash learning layer in the hash learning network to obtain a hash feature output by the hash learning layer; wherein the hash learning layer is a layer that processes the fusion features based on a nonlinear activation function; inputting the hash features into a binarization layer in the hash learning network to obtain a data set hash code output by the binarization layer; wherein the binarization layer is a layer that performs binary processing on the hash features based on a sign function.

[0013] According to a ship image retrieval method provided by the present invention, all Hamming similarities are determined according to a hash code and all sample hash codes, including: multiplying the hash code and all sample hash codes respectively to obtain all Hamming similarities.

[0014] According to a retrieval method for ship images provided by the present invention, a target retrieval result is determined according to a ranking list of all Hamming similarities, including: determining all candidate similarities according to all Hamming similarities in the ranking list and a similarity threshold; wherein all candidate similarities are Hamming similarities in the ranking list whose Hamming similarities are greater than a similarity threshold, and the similarity threshold is a preset threshold; and determining all candidate similarities corresponding to all ship sample images as target retrieval results.

[0015] The present invention also provides a ship image retrieval device, comprising the following modules.

[0016] The image acquisition module is used to acquire the ship image to be retrieved and the ship sample database; wherein the ship sample database contains at least two ship sample images, and the ship sample images are ship images of different modalities from the ship image to be retrieved.

[0017] The hash code output module is used to input the ship image to be retrieved and all the ship sample images in the ship sample database into the ship visual language model respectively, and obtain the hash code corresponding to the ship image to be retrieved output by the ship visual language model and all the sample hash codes corresponding to all the ship sample images in the ship sample database; wherein the ship visual language model is trained based on the acquired multimodal ship sample pattern data set including the ship full-color image data set, the ship multispectral image data set and the label text, and the ship visual language model is a model that processes the ship image to be retrieved and the ship sample database based on multimodal contrast learning, cross-modal semantic fusion and hash learning to obtain the corresponding hash code and sample hash code.

[0018] The list determination module is used to determine all Hamming similarities according to the hash code and all sample hash codes respectively, and determine a similarity ranking list according to all Hamming similarities.

[0019] The result determination module is used to determine the target retrieval result according to the ranking list of all Hamming similarities; wherein the target retrieval result is a set of sample ship images similar to the ship image to be retrieved, selected based on all similarities in the ranking list.

[0020] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned ship image retrieval methods is implemented.

[0021] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned ship image retrieval methods.

[0022] The present invention also provides a computer program product, comprising a computer program, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned ship image retrieval methods.

[0023] The present invention provides a method, device, electronic device, storage medium and product for searching ship images. The method comprises the following steps: obtaining a ship image to be searched and a ship sample database; wherein the ship sample database contains at least two ship sample images, and the ship sample images are ship images of different modalities from the ship image to be searched; the ship image to be searched and all the ship sample images in the ship sample database are respectively input into a ship visual language model, and a hash code corresponding to the ship image to be searched output by the ship visual language model and all sample hash codes corresponding to all the ship sample images in the ship sample database are obtained; wherein the ship visual language model is based on the obtained full-color image of the ship. The ship visual language model is trained on a multimodal ship sample pattern data set, a ship multispectral image data set and a multimodal ship sample pattern data set with labeled texts. The ship visual language model is a model that processes the ship images to be retrieved and the ship sample database based on multimodal contrastive learning, cross-modal semantic fusion and hash learning to obtain corresponding hash codes and sample hash codes; all Hamming similarities are determined according to the hash codes and all sample hash codes, respectively, and a similarity ranking list is determined according to all Hamming similarities; a target retrieval result is determined according to the ranking list of all Hamming similarities; wherein the target retrieval result is a set of ship sample images similar to the ship image to be retrieved, selected based on all similarities in the ranking list. The technical solution of the present invention is used to solve the defect in the prior art that the accuracy of cross-modal retrieval of remote sensing images is reduced by simply performing text-to-image retrieval or image-to-image retrieval, and realizes a ship visual language model that combines images and texts obtained by training based on a ship panchromatic image dataset, a ship multispectral image dataset and label text, and performs multimodal comparative learning, cross-modal semantic fusion and hash learning processing on the ship images to be retrieved and the ship sample database. Finally, the target retrieval result is determined based on the obtained hash code and the sample hash code, thereby improving the accuracy of cross-modal retrieval of the ship images to be retrieved in the ship sample database. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a flow chart of the ship image retrieval method provided by the present invention.

[0026] Figure 2 It is a structural schematic diagram of the ship visual language model provided by the present invention.

[0027] Figure 3It is a schematic diagram of the alignment and fusion method provided by the present invention.

[0028] Figure 4 It is a schematic diagram of the attention mechanism fusion method provided by the present invention.

[0029] Figure 5 It is a schematic diagram of image samples provided by the present invention.

[0030] Figure 6 It is a schematic diagram of the structure of the ship image retrieval device provided by the present invention.

[0031] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Combine the following Figure 1 The ship image retrieval method provided by the present invention is described. The ship image retrieval method provided by the present invention can be applicable to the retrieval of cross-modal remote sensing ship images. The execution subject of the method can be an electronic device or a ship image retrieval device set in the electronic device. The ship image retrieval device can be implemented by software, hardware or a combination of both. Figure 1 FIG. 1 is a flow chart of the ship image retrieval method provided by the present invention. Figure 1 As shown, the method includes the following steps 101, 102, 103 and 104.

[0034] Step 101: Obtain ship images to be retrieved and a ship sample database.

[0035] In this step, the ship sample database contains at least two ship sample images, and the ship sample images are ship images of different modalities from the ship image to be retrieved.

[0036] Specifically, a ship image to be retrieved is obtained, and at the same time, a ship sample database containing at least two ship sample images of different modalities from the ship image to be retrieved is obtained.

[0037] Step 102: input the ship image to be retrieved and all the ship sample images in the ship sample database into the ship visual language model respectively, and obtain the hash code corresponding to the ship image to be retrieved output by the ship visual language model and all the sample hash codes corresponding to all the ship sample images in the ship sample database.

[0038] In this step, the ship visual language model is trained based on a multimodal ship sample pattern dataset including a ship panchromatic image dataset, a ship multispectral image dataset and label text. The ship visual language model is a model that processes the ship images to be retrieved and the ship sample database based on multimodal contrast learning, cross-modal semantic fusion and hash learning to obtain corresponding hash codes and sample hash codes.

[0039] Specifically, based on the trained ship visual language model, the ship image to be retrieved and all the ship sample images in the ship sample database are respectively input into the ship visual language model, and the ship image to be retrieved and all the ship sample images in the ship sample database are processed by multimodal contrast learning, cross-modal semantic fusion and hash learning based on the trained ship visual language model, so as to obtain the hash code corresponding to the ship image to be retrieved and all sample hash codes corresponding to all the ship sample images in the ship sample database.

[0040] In a specific embodiment, the ship visual language model is trained based on the following steps: obtaining a multimodal ship sample pattern dataset; wherein the multimodal ship sample pattern dataset includes a ship panchromatic image dataset, a ship multispectral image dataset and a label text, and the label text is a paired label text corresponding to a paired sample determined by pairing the ship panchromatic image dataset and the ship multispectral image dataset; inputting the ship panchromatic image dataset, the ship multispectral image dataset and the label text into a basic ship visual language model to obtain a dataset hash code and a target loss function output by the basic ship visual language model; wherein the basic ship visual language model is a predetermined basic model for training; and optimizing the basic ship visual language model according to the dataset hash code and the target loss function to obtain a ship visual language model.

[0041] In this step, the basic ship visual language model is a language model obtained in advance by the user for image recognition and processing, which is not limited in this embodiment.

[0042] Specifically, Figure 2 is a schematic diagram of the structure of the ship visual language model provided by the present invention, such as Figure 2 As shown in Figure 2, the obtained ship visual language model includes a multimodal contrastive learning network, a cross-modal semantic fusion network, and a hash learning network.

[0043] In a specific implementation, a ship panchromatic image dataset, a ship multispectral image dataset and a label text are input into a basic ship visual language model to obtain a dataset hash code and a target loss function output by the basic ship visual language model, including: inputting the ship panchromatic image dataset and the ship multispectral image dataset into a multimodal contrast learning network in the basic ship visual language model to obtain image features and contrast losses output by the multimodal contrast learning network; the image features are obtained by the multimodal contrast learning network performing image enhancement processing and similarity measurement on the ship panchromatic image dataset and the ship multispectral image dataset; the image features and the label text are input into a cross-modal semantic fusion network in the basic ship visual language model to obtain fusion features and fusion losses output by the cross-modal semantic fusion network; wherein the fusion features are obtained by the cross-modal semantic fusion network aligning and fusing the image features and the label text; the fusion features are input into a hash learning network in the basic ship visual language model to obtain a dataset hash code and a hash loss output by the hash learning network; wherein the dataset hash code is obtained by the hash learning network performing hash learning on the fusion features; and the target loss function is determined according to the contrast loss, the fusion loss and the hash loss.

[0044] In a specific implementation, a ship panchromatic image dataset and a ship multispectral image dataset are input into a multimodal contrastive learning network in a basic ship visual language model to obtain image features and contrast losses output by the multimodal contrastive learning network.

[0045] In this step, the image features are obtained by performing image enhancement processing and similarity measurement on the ship panchromatic image dataset and the ship multispectral image dataset using a multimodal contrast learning network.

[0046] The multimodal contrastive learning network includes a twin image encoder for extracting high-dimensional image embeddings such as the ship panchromatic image dataset and the ship multispectral image dataset, a similarity measurement module, and a contrast loss applied to image embeddings in the ship panchromatic image dataset and the ship multispectral image dataset. The twin image encoder includes a panchromatic vision branch and a multispectral vision branch. The multimodal contrastive learning network aims to reduce the distance between similar image embeddings within the same modality and between different modalities, effectively align feature representations, and bridge the heterogeneity gap between modalities.

[0047] Specifically, first, the multimodal image dataset is formally defined as , where N represents the label text in the multimodal dataset The total number of paired samples, each pair consists of a panchromatic image in the ship panchromatic image dataset and a corresponding multispectral image in the ship multispectral image dataset. Specifically, Represents the first Full-color images, Represents the height of the full-color image, represents the width of the full-color image, Indicates the number of channels of a full-color image. Represents a single-band grayscale image. Similarly, Represents the first Multispectral images, Indicates the number of spectral bands. Indicates the corresponding The label text of paired samples, where The set of predefined semantic categories is used for panchromatic images and multispectral images, wherein the set of predefined language categories may be, for example, a destroyer, a tanker, etc., which is not limited in this embodiment. In order to improve the robustness of feature extraction, the ship panchromatic image dataset and the ship multispectral image dataset are input into the multimodal contrast learning network in the basic ship visual language model, and data enhancement is performed on all panchromatic images in the ship panchromatic image dataset and all multispectral images in the ship multispectral image dataset through the multimodal contrast learning network, and the enhanced ship panchromatic image enhanced dataset is obtained. and ship multispectral image enhancement dataset .

[0048] Exemplary, ship panchromatic image enhancement dataset , Ship multispectral image enhancement dataset ,in, It means that common enhancement techniques are included, such as random cropping and flipping, which are not limited in this embodiment. The enhanced data set of the full-color image of the ship after enhancement processing It is then processed by the full-color branch of the twin image encoder, which uses a shared backbone network , thus obtaining the full-color image features in the image features ,in, , Represents the dimension of the feature space. Similarly, the enhanced ship multispectral image dataset after enhancement processing It is then processed by the multispectral branch of the twin image encoder, which uses a shared backbone network , thus obtaining the multispectral image features in the image features ,in, , and the weight of the full-color branch during the processing and the weights of the multispectral branches is shared and uses the parameters of the pre-trained base ship visual language model for the weights of the full color branch and the weights of the multispectral branches Initialize.

[0049] However, in the process of feature embedding, it is necessary to determine the similarity of features for feature embedding. Specifically, the similarity measurement module is responsible for quantifying the similarity of feature embeddings of the same modality (intra-modality) or different modalities (inter-modality). The similarity measurement module supports the use of cosine similarity or Pearson correlation coefficient to measure the relationship between feature vectors, which is not limited in this embodiment.

[0050] Specifically, the similarity metric is calculated as follows. For example, given the feature embedding from the full color branch and , and feature embedding from the multispectral branch and , following the first method, use cosine similarity to calculate the two feature embeddings and The similarity between them is shown in formula (1).

[0051] (1) In formula (1), express and The dot product of express The Euclidean norm of , express The Euclidean norm of .

[0052] The second method uses the standardized Pearson correlation coefficient to calculate the embedding of two features and The similarity between them is shown in formula (2).

[0053] (2) In formula (2), express and The covariance between express The standard deviation of express The Pearson correlation coefficient is used to capture the linear relationship between features, making it particularly suitable for aligning multimodal representations, especially when there are modality-specific scale and variance differences, which are not limited in this embodiment.

[0054] Then, the contrast loss It is applied to maximize the similarity of samples of the same type within and between modalities, thereby alleviating the variance within the same type and the difference between cross-modal categories.

[0055] For example, the calculation of contrast loss is as follows. In order to learn high-dimensional embeddings with discriminative and aligned features within the same modality, a contrast loss function is used. Batches of paired samples, where represents the number of different categories in the ship panchromatic image dataset and the ship multispectral image dataset, let represents full color embedding, , represents multi-spectral embedding, a batch of one modality contains two groups of samples: the first group Include images, each image represents a unique category, and the second group Include images, each image represents the same category as the first set of images, and is paired in the same order. In other words, within the same modality, the first Sample and The samples belong to the same category.

[0056] The intra-modality loss maximizes the similarity of similar samples within a single modality. The calculation of is shown in formula (3).

[0057] (3) In formula (3), Indicates that it can be instantiated using image features of cosine similarity or image features of standardized Pearson correlation coefficient. Represents the temperature parameter, which is used to control the steepness of the similarity distribution.

[0058] Therefore, for multispectral modalities, the intra-multispectral modal loss The calculation of is shown in formula (4).

[0059] (4) In summary, the total intra-modal loss is the sum of the panchromatic intra-modal loss and the multispectral intra-modal loss, that is, the total intra-modal loss .

[0060] After computing the overall intra-modality loss, in order to learn aligned high-dimensional embeddings between panchromatic and multispectral modalities, a cross-modal contrastive loss is further defined by utilizing four sets of multimodal features.

[0061] For the cross-modal contrast loss, first, for a batch of Paired samples, let , , and Represent two sets of full-color features respectively, let and , and Represents two sets of multispectral features, with the same index Features and Belong to the same category, for example, can be and Belong to the same category. The cross-modal contrast loss is calculated based on the comparison of four pairs of cross-modal features. For example, the first contrast loss can be The calculation of is shown in formula (5).

[0062] (5) Similarly, the second contrast loss , the third contrast loss And the fourth contrast loss The calculation is also performed in the same way as formula (5), so that according to the first contrast loss , the second contrast loss , the third contrast loss And the fourth contrast loss The sum determines the total cross-modal contrast loss The total cross-modal contrast loss It encourages the embedding alignment of samples of the same category between the two modalities, while reducing the similarity between samples of different categories.

[0063] Finally, if Figure 2 As shown, the contrast loss between high-dimensional image embeddings is determined is the total intra-modal loss Compared with the total cross-modal contrast loss The sum of .

[0064] In a specific implementation, image features and label text are input into a cross-modal semantic fusion network in a basic ship visual language model to obtain fusion features and fusion losses output by the cross-modal semantic fusion network.

[0065] In this step, the fusion feature is obtained by aligning and fusing the image features and the label text using a cross-modal semantic fusion network. The cross-modal semantic fusion network is a network designed to enhance the fusion of high-level semantic information of the label text.

[0066] Specifically, the image features and label text are input into the cross-modal semantic fusion network in the basic ship visual language model to obtain the fusion features and fusion loss output by the cross-modal semantic fusion network. First, a pre-trained text editor is used to extract text features offline. For example, for each ship label text , change the ship label text Expanded to descriptive text prompts, such as {a remote sensing image of , a type of ship}, and encode descriptive textual cues into high-level semantic embeddings . High-level semantic embedding It is then fused with image features through a semantic embedding fusion module.

[0067] Fusion is usually based on alignment fusion and attention fusion, which is not limited in this embodiment.

[0068] Image and text feature fusion is performed based on alignment fusion. Figure 3 is a schematic diagram of the alignment and fusion method provided by the present invention, such as Figure 3 As shown, the frozen text features and image (panchromatic or multispectral) features Alignment is performed via a contrastive loss mechanism with ground-truncated accuracy to enforce semantic correspondence between modalities. Alignment ensures that semantically similar text and image pairs are closer in the embedding space, while semantically dissimilar pairs are pushed further away. Figure 3 As shown, the similarity matrix Each value in .in, A similarity measure representing image features (e.g., determined by cosine similarity or normalized Pearson correlation coefficient). Alignment is done by fusion loss Fusion loss The calculation of is shown in formula (6).

[0069] (6) The benefit of this setting is that it ensures the alignment of text and image features in the semantic space. Through alignment-based fusion, image features of different modalities are not only enhanced with high-level semantic information, but also encouraged to maintain consistency with their corresponding text descriptions. This effectively bridges the gap between modalities and enhances the model's ability to associate semantically related visual and textual features, which is very important for explainable cross-modal retrieval.

[0070] Specifically, for the fusion method of the attention mechanism, Figure 4 is a schematic diagram of the attention mechanism fusion method provided by the present invention, such as Figure 4 In the attention-based fusion mechanism, the text features are integrated into the image features of the image, and the image features can be the full-color image features from the full-color modality. Or multispectral image features of multispectral modalities In order to introduce the attention mechanism fusion, frozen text features are used As a semantic anchor point. The fusion process is described as follows: represents the full-color image feature, Represents the multispectral image features, where , represents the batch size, is the dimension of the feature. represents precomputed text features, Indicates the number of categories in the dataset. Full-color image features in the input image features or multispectral image features First, normalization is performed to obtain the normalized full-color normalized image features and multispectral normalized image features . Full color normalized image features The calculation process of is shown in formula (7). Multispectral normalized image features The calculation process of is shown in formula (8).

[0071] (7) (8) Get the normalized full-color normalized image features and multispectral normalized image features After that, the text features are also normalized to obtain the normalized text features , normalized text features The calculation process of is shown in formula (9).

[0072] (9) Then, a text-guided attention mechanism is applied to compute the correlation between image features and text features. Compute full-color normalized image features Or multispectral normalized image features With normalized text features The attention value between , the calculation formula of the attention value A is shown in formula (10).

[0073] (10) In formula (10), the attention value is the attention value matrix, , express or .

[0074] Furthermore, an activation function is introduced, such as softmax. The attention weight is calculated by the softmax operation , , In determining the attention weights Afterwards, according to the attention weight and text features Perform matrix multiplication to obtain semantic features by calculating the weighted sum of text features and attention weights . Semantic features , Finally, the semantic features and the original input image features are input into the semantic embedding fusion module for feature concatenation to obtain the fused features. , fusion features , , Represents full-color image features or multispectral image features .

[0075] The benefit of this setting is that the attention-based fusion mechanism not only promotes the integration of higher-level semantic information from the label text, but also enables the cross-modal semantic fusion network to adaptively adjust and focus on the ship area that is most relevant to the text. Combined with alignment-based fusion, it helps to reduce background noise in multi-modal ship images. Therefore, attention-based fusion not only enhances the semantic understanding of the image, but also reduces the impact of irrelevant background elements in some cases.

[0076] In a specific implementation, the fusion features are input into the hash learning network in the basic ship visual language model to obtain the data set hash code and hash loss output by the hash learning network. The target loss function is determined based on the contrast loss, fusion loss and hash loss.

[0077] Among them, the data set hash code is obtained by hash learning the fusion features by the hash learning network.

[0078] In a specific implementation, the fused features are input into a hash learning network in a basic ship visual language model to obtain a hash code of a data set output by the hash learning network, including: inputting the fused features into a hash learning layer in the hash learning network to obtain a hash feature output by the hash learning layer; wherein the hash learning layer is a layer that processes the fused features based on a nonlinear activation function; inputting the hash features into a binarization layer in the hash learning network to obtain a hash code of a data set output by the binarization layer; wherein the binarization layer is a layer that performs binary processing on the hash features based on a sign function.

[0079] Specifically, after obtaining the fusion feature Afterwards, the fusion features Input into the hash learning network in the basic ship visual language model, perform hash learning through the hash learning layer of the hash learning network, and obtain the compact hash features output by the hash learning layer . Hash Features in, Indicates normalization processing. represents a fully connected network with a nonlinear activation function, Make sure the dataset hash code value falls within the range [-1,1], Indicates the length of the dataset hash code.

[0080] Furthermore, hash features The same contrastive loss as in the multimodal contrastive learning network is used for training. This ensures that semantically similar samples are closer in the hash space, while semantically dissimilar samples are farther away. This design compresses features into a low-dimensional space while maintaining the semantic relationship between different modalities, making the retrieval task efficient.

[0081] Among them, the hash loss is composed of two parts: ,in, represents the intra-modal alignment loss, represents the inter-modality alignment loss.

[0082] In determining the contrast loss , fusion loss and hash loss Afterwards, according to the contrast loss , fusion loss and hash loss Determine the target loss function . Objective loss function ,in, is a hyperparameter, a hyperparameter Used to control the relative importance of hash loss in the overall objective, the default value is set to 1.

[0083] The advantage of this setting is that by jointly optimizing the contrast loss , fusion loss and hash loss , thereby obtaining a ship visual language model so that the ship visual language model can learn semantically rich and aligned representations between panchromatic and multispectral modalities, and encode these representations into a hash space that is both discriminative and computationally efficient, suitable for large-scale retrieval tasks.

[0084] Furthermore, in the hash retrieval process, the learned hash features are Input the binarization layer in the hash learning network, and use the symbol function to hash the features based on the binarization layer Binarization is performed to obtain the dataset hash code output by the binarization layer. The dataset hash code is a binary hash code. ,in, The function is to Each element in The mapping is shown in formula (11).

[0085] (11) Finally, the basic ship visual language model is further optimized through the dataset hash code and target loss function to obtain the ship visual language model.

[0086] After obtaining the ship visual language model, the ship image to be retrieved and all the ship sample images in the ship sample database are respectively input into the ship visual language model, and the image features are extracted through the multimodal contrast learning network of the ship visual language model. Then, the image features and text features are fused through the cross-modal semantic fusion network to obtain the fused features. Finally, the fused features are input into the hash learning network, and the fused features are hashed through the hash learning network to obtain the hash code corresponding to the ship image to be retrieved output by the ship visual language model and all sample hash codes corresponding to all the ship sample images in the ship sample database.

[0087] The advantage of this setting is that by performing feature fusion and introducing hash feature calculation, a compact and robust hash code is generated for subsequent efficient retrieval, thereby achieving efficient and effective cross-modal ship image retrieval.

[0088] Step 103: determine all Hamming similarities according to the hash code and all sample hash codes respectively, and determine a similarity ranking list according to all Hamming similarities.

[0089] In a specific implementation, all Hamming similarities are determined according to the hash code and all sample hash codes respectively, including: multiplying the hash code and all sample hash codes respectively to obtain all Hamming similarities.

[0090] Specifically, after obtaining the hash code corresponding to the to-be-retrieved ship image output by the ship visual language model and all sample hash codes corresponding to all ship sample images in the ship sample database, the hash codes are respectively And each sample hash code The product of , we get all the Hamming similarities. , Hamming similarity The value range is arrive .in, Indicates the length, and a larger value indicates that the similarity between the corresponding hash codes is higher. After all the Hamming similarities are obtained, all the Hamming similarities are sorted in order of size to determine the similarity ranking list.

[0091] Step 104: Determine the target search result according to the ranking list of all Hamming similarities.

[0092] In this step, the target retrieval result is a set of sample ship images similar to the ship image to be retrieved, which are selected based on all similarities in the ranking list.

[0093] In a specific embodiment, the target retrieval result is determined according to a ranked list of all Hamming similarities, including: determining all candidate similarities according to all Hamming similarities in the ranked list and a similarity threshold; wherein all candidate similarities are all Hamming similarities in the ranked list whose Hamming similarities are greater than a similarity threshold, and the similarity threshold is a pre-set threshold; and determining all ship sample images corresponding to all candidate similarities as the target retrieval result.

[0094] Specifically, after obtaining the similarity ranking list, all Hamming similarities in the similarity ranking list are compared with the similarity threshold, and the Hamming similarities greater than the similarity threshold among all Hamming similarities are determined as candidate similarities, and the ship sample images in the ship sample database corresponding to the candidate similarities are determined as target retrieval results.

[0095] Exemplarily, a ship image to be retrieved and a ship sample database are obtained; the ship image to be retrieved and all ship sample images in the ship sample database are respectively input into a ship visual language model to obtain a hash code corresponding to the ship image to be retrieved output by the ship visual language model and all sample hash codes corresponding to all ship sample images in the ship sample database; all Hamming similarities are determined according to the hash code and all sample hash codes, respectively, and a similarity ranking list is determined according to all Hamming similarities; a target retrieval result is determined according to the ranking list of all Hamming similarities.

[0096] Table 1 Experimental environment configuration

[0097] The retrieval of target retrieval results by using the ship visual language model can be verified by the following embodiments. All experimental verification environments are carried out in the same environment. The detailed information of the experimental environment configuration is shown in Table 1, and this embodiment does not limit this.

[0098] Figure 5 is a schematic diagram of an image sample provided by the present invention, such as Figure 5As shown, the ship image samples used in the experimental data set are some public ship images, such as Destroyers image samples, Littoral combatships image samples, Combat boats image samples, Bulk carriers image samples, Container ships image samples and Oil tankers image samples, etc. This embodiment does not limit this.

[0099] The experiment was conducted on the only publicly available cross-modal remote sensing ship image retrieval dataset MRSSID. The cross-modal remote sensing ship image retrieval dataset MRSSID includes 2632 pairs of panchromatic and multispectral ship images acquired by the GF-2 multispectral sensor and the GF-2 panchromatic sensor. Each pair of ship image slices represents a combination of panchromatic and multispectral images captured simultaneously in the same area. In the experiment, the spatial resolution of the multispectral image is 4 meters and the number of spectral channels is 4. The spatial resolution of the panchromatic image is 1 meter, and the number of spectral channels is only 1, which is not limited in this embodiment.

[0100] In the experimental setting, for the data augmentation part, the data augmentation strategy includes random cropping and horizontal flipping. All images are first scaled to 256×256 pixels and randomly cropped to 224×224 pixels. In order to exploit the knowledge of multimodal data, the ship visual language model is used as the backbone of the twin image encoder.

[0101] Specifically, we use the Contrastive Language- Image Pre-Training (CLIP)-ResNet-50 as the twin image encoder, and the pre-trained language model CLIP-Transformer in the ship visual language model as the text encoder. The model is pre-trained on 400M general image-text pairs. In order to solve the problem of unbalanced classes, the class-balanced resampling method is used. At the same time, the default training round is set to 50, the default setting γ=1, and the learning parameter Initialized to 0.07. In the proposed ship visual language model, the Adam optimizer with decay rates β1=0.9 and β2=0.999 is used for model optimization. The initial learning rate of the twin image encoder is set to , and the initial learning rate of the hash learning network is set to .

[0102] A learning rate warm-up strategy was adopted in the first training round, where the learning rate was linearly increased from 0 to a predefined initial value.

[0103] After the warm-up phase, a cosine learning rate schedule without restarts is applied for the remaining training epochs, gradually decreasing the learning rate to improve convergence and model performance.

[0104] The final experimental results are compared in Table 2.

[0105] Table 2

[0106] In Table 2, FSISR stands for the latest state-of-the-art (SOTA) cross-modal hashing method. To ensure a fair comparison with the latest SOTA methods, the backbone of FSISR is also replaced and the modified version is denoted as FSISR-r. The comparative experimental results of the proposed ship visual language model and the SOTA methods are shown in Table 2. Table 2 provides the mean average precision (mAP) values ​​for two cross-modal retrieval tasks: PAN→MS (panchromatic to multispectral) and MS→PAN (multispectral to panchromatic), which are averaged over various hash code lengths (16, 24, 32, 48, and 64 bits). Table 2 also shows the experimental results of three variants of the proposed ship visual language model: SFCH-ali, which uses alignment-based fusion in the semantic embedding fusion module; SFCH-att, which exploits attention-based fusion; and SFCH-fus, which combines these two strategies.

[0107] As can be seen from Table 2, the proposed ship visual language model achieves better performance on both tasks. In particular, the ship visual language model surpasses FSISR, the latest SOTA methods, and a fair comparison using the same backbone FSISR-r. Specifically, SFCH-att improves FSISR-r by 3.45% on average, and SFCH-fus improves FSISR-r by 3.13% on average, further verifying the superiority of the proposed ship visual language model. The superior performance of the ship visual language model can be attributed to the integration of its multimodal contrastive learning network, hash learning network, and cross-modal semantic fusion network. The multimodal contrastive learning network and hash learning network effectively align high-dimensional image embeddings and hash features of different modalities by optimizing their representations in a shared latent space, ensuring that the features are compact and semantically consistent. In addition, the cross-modal semantic fusion network exploits high-level semantic information to achieve a more comprehensive and adaptive integration of complementary features. These results show that the proposed ship visual language model effectively bridges the gap between modalities, leveraging semantic fusion and contrastive learning strategies to achieve superior retrieval performance over a range of hash code lengths.

[0108] Table 2 also shows the experimental results of three variants of the proposed ship visual language model: SFCH-ali, SFCH-att, and SFCH-fus. The results show that SFCH-ali, which only uses alignment-based fusion, achieves relatively low performance compared to the other two variants. In contrast, SFCH-att and SFCH-fus, which include attention-based fusion mechanisms, show better performance. SFCH-att and SFCH-fus are almost comparable in terms of accuracy. However, SFCH-att lacks supervised alignment with semantic information, which limits its interpretability at the semantic level. In contrast, SFCH-fus, which combines alignment-based and attention-based fusion strategies, achieves semantic-level interpretability without sacrificing accuracy.

[0109] The present invention provides a method for retrieving ship images, which comprises obtaining a ship image to be retrieved and a ship sample database; wherein the ship sample database contains at least two ship sample images, and the ship sample images are ship images of different modalities from the ship image to be retrieved; the ship image to be retrieved and all the ship sample images in the ship sample database are respectively input into a ship visual language model, and a hash code corresponding to the ship image to be retrieved output by the ship visual language model and all sample hash codes corresponding to all the ship sample images in the ship sample database are obtained; wherein the ship visual language model is based on the obtained ship panchromatic image dataset, the ship multi-photon image dataset, and the ship image data set. The ship visual language model is trained with a multimodal ship sample pattern dataset of spectral image dataset and label text. The ship visual language model is a model that processes the ship image to be retrieved and the ship sample database based on multimodal contrastive learning, cross-modal semantic fusion and hash learning to obtain corresponding hash codes and sample hash codes; all Hamming similarities are determined according to the hash code and all sample hash codes respectively, and a similarity ranking list is determined according to all Hamming similarities; a target retrieval result is determined according to the ranking list of all Hamming similarities; wherein the target retrieval result is a set of ship sample images similar to the ship image to be retrieved, selected based on all similarities in the ranking list. In the above-mentioned embodiment, the technical scheme of the present invention is used to solve the defect in the prior art that the accuracy of cross-modal retrieval of remote sensing images is reduced by simply performing text-to-image retrieval or image-to-image retrieval, and to implement a ship visual language model combining images and texts obtained by training based on a ship panchromatic image dataset, a ship multispectral image dataset and label text, to perform multimodal comparative learning, cross-modal semantic fusion and hash learning processing on the ship images to be retrieved and the ship sample database, and finally determine the target retrieval result based on the obtained hash code and the sample hash code, thereby improving the accuracy of cross-modal retrieval of the ship images to be retrieved in the ship sample database.

[0110] The following is a description of a ship image retrieval device provided by the present invention. The ship image retrieval device described below and the ship image retrieval method described above can be referenced to each other.

[0111] Figure 6 is a schematic diagram of the structure of the ship image retrieval device provided by the present invention, referring to Figure 6 As shown, the ship image retrieval device 600 includes: an image acquisition module 601, a hash code output module 602, a list determination module 603 and a result determination module 604.

[0112] The image acquisition module 601 is used to acquire the ship image to be retrieved and the ship sample database; wherein the ship sample database contains at least two ship sample images, and the ship sample images are ship images of different modalities from the ship image to be retrieved.

[0113] The hash code output module 602 is used to input the ship image to be retrieved and all the ship sample images in the ship sample database into the ship visual language model respectively, and obtain the hash code corresponding to the ship image to be retrieved output by the ship visual language model and all the sample hash codes corresponding to all the ship sample images in the ship sample database; wherein the ship visual language model is trained based on the acquired multimodal ship sample pattern data set including the ship full-color image data set, the ship multispectral image data set and the label text, and the ship visual language model is a model that processes the ship image to be retrieved and the ship sample database based on multimodal contrast learning, cross-modal semantic fusion and hash learning to obtain the corresponding hash code and sample hash code.

[0114] The list determination module 603 is used to determine all Hamming similarities according to the hash code and all sample hash codes respectively, and determine a similarity ranking list according to all Hamming similarities.

[0115] The result determination module 604 is used to determine the target retrieval result according to the ranking list of all Hamming similarities; wherein the target retrieval result is a set of sample ship images similar to the ship image to be retrieved, selected based on all similarities in the ranking list.

[0116] In an exemplary embodiment, the device further includes: a model training module. The model training module is used to: obtain a multimodal ship sample pattern data set; wherein the multimodal ship sample pattern data set includes a ship panchromatic image data set, a ship multispectral image data set and a label text, and the label text is a paired label text corresponding to a paired sample determined by pairing the ship panchromatic image data set and the ship multispectral image data set; input the ship panchromatic image data set, the ship multispectral image data set and the label text into a basic ship visual language model to obtain a data set hash code and a target loss function output by the basic ship visual language model; wherein the basic ship visual language model is a predetermined basic model for training; and optimize the basic ship visual language model according to the data set hash code and the target loss function to obtain a ship visual language model.

[0117] In an exemplary embodiment, the model training module inputs the ship panchromatic image dataset, the ship multispectral image dataset and the label text into the basic ship visual language model to obtain the dataset hash code and the target loss function output by the basic ship visual language model, and is specifically used to: input the ship panchromatic image dataset and the ship multispectral image dataset into the multimodal contrast learning network in the basic ship visual language model to obtain the image features and contrast loss output by the multimodal contrast learning network; the image features are the image enhancement processing and similarity loss of the ship panchromatic image dataset and the ship multispectral image dataset by the multimodal contrast learning network. The measurement is obtained; the image features and label texts are input into the cross-modal semantic fusion network in the basic ship visual language model to obtain the fusion features and fusion losses output by the cross-modal semantic fusion network; wherein, the fusion features are obtained by aligning and fusing the image features and label texts through the cross-modal semantic fusion network; the fusion features are input into the hash learning network in the basic ship visual language model to obtain the data set hash code and hash loss output by the hash learning network; wherein, the data set hash code is obtained by hash learning the fusion features through the hash learning network; the target loss function is determined according to the contrast loss, fusion loss and hash loss.

[0118] In an exemplary embodiment, the model training module inputs the fused features into the hash learning network in the basic ship visual language model to obtain the hash code of the data set output by the hash learning network, and is specifically used to: input the fused features into the hash learning layer in the hash learning network to obtain the hash features output by the hash learning layer; wherein the hash learning layer is a layer that processes the fused features based on a nonlinear activation function; input the hash features into the binarization layer in the hash learning network to obtain the hash code of the data set output by the binarization layer; wherein the binarization layer is a layer that performs binary processing on the hash features based on a sign function.

[0119] In an exemplary embodiment, the list determination module 603 determines all Hamming similarities according to the hash code and all sample hash codes respectively, and is specifically configured to: multiply the hash code and all sample hash codes respectively to obtain all Hamming similarities.

[0120] In an exemplary embodiment, the result determination module 604 is specifically used to: determine all candidate similarities based on all Hamming similarities in the ranking list and the similarity threshold; wherein all candidate similarities are all Hamming similarities in the ranking list whose Hamming similarities are greater than the similarity threshold, and the similarity threshold is a pre-set threshold; and determine all candidate similarities corresponding to all ship sample images as target retrieval results.

[0121] The device of this embodiment can be used to execute the method of any embodiment in the embodiment of the ship image retrieval method side. Its specific implementation process and technical effects are similar to those in the embodiment of the ship image retrieval method side. For details, please refer to the detailed description in the embodiment of the ship image retrieval method side, which will not be repeated here.

[0122] Figure 7 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the ship image retrieval method, the method comprising: obtaining a ship image to be retrieved and a ship sample database; wherein the ship sample database contains at least two ship sample images, and the ship sample images are ship images of different modalities from the ship image to be retrieved; the ship image to be retrieved and all the ship sample images in the ship sample database are respectively input into the ship visual language model, and the hash code corresponding to the ship image to be retrieved output by the ship visual language model and all the sample hash codes corresponding to all the ship sample images in the ship sample database are obtained; wherein the ship visual language model is based on the acquired ship image containing the entire ship. The ship visual language model is trained with a color image dataset, a ship multispectral image dataset and a multimodal ship sample pattern dataset with labeled texts. The ship visual language model is a model that processes the ship image to be retrieved and the ship sample database based on multimodal contrast learning, cross-modal semantic fusion and hash learning to obtain corresponding hash codes and sample hash codes; all Hamming similarities are determined according to the hash code and all sample hash codes respectively, and a similarity ranking list is determined according to all Hamming similarities; a target retrieval result is determined according to the ranking list of all Hamming similarities; wherein the target retrieval result is a set of ship sample images similar to the ship image to be retrieved, which are selected based on all similarities in the ranking list.

[0123] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0124] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the ship image retrieval method provided by the above methods, the method including: obtaining a ship image to be retrieved and a ship sample database; wherein the ship sample database contains at least two ship sample images, and the ship sample images are ship images of different modalities from the ship image to be retrieved; the ship image to be retrieved and all the ship sample images in the ship sample database are respectively input into a ship visual language model, and a hash code corresponding to the ship image to be retrieved output by the ship visual language model and a hash code corresponding to all the ship sample images in the ship sample database are obtained. all sample hash codes of ; wherein, the ship visual language model is trained based on a multimodal ship sample pattern dataset including a ship panchromatic image dataset, a ship multispectral image dataset and a label text, and the ship visual language model is a model that processes the ship image to be retrieved and the ship sample database based on multimodal contrast learning, cross-modal semantic fusion and hash learning to obtain corresponding hash codes and sample hash codes; all Hamming similarities are determined according to the hash code and all sample hash codes respectively, and a similarity ranking list is determined according to all Hamming similarities; a target retrieval result is determined according to the ranking list of all Hamming similarities; wherein the target retrieval result is a set of ship sample images similar to the ship image to be retrieved, which are selected based on all similarities in the ranking list.

[0125] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the ship image retrieval method provided by the above methods, the method comprising: obtaining a ship image to be retrieved and a ship sample database; wherein the ship sample database contains at least two ship sample images, and the ship sample images are ship images of different modalities from the ship image to be retrieved; inputting the ship image to be retrieved and all the ship sample images in the ship sample database into a ship visual language model respectively, and obtaining a hash code corresponding to the ship image to be retrieved output by the ship visual language model and all sample hash codes corresponding to all the ship sample images in the ship sample database; wherein the ship The ship visual language model is trained based on a multimodal ship sample pattern dataset including a ship panchromatic image dataset, a ship multispectral image dataset and label text. The ship visual language model is a model that processes the ship image to be retrieved and the ship sample database based on multimodal contrast learning, cross-modal semantic fusion and hash learning to obtain corresponding hash codes and sample hash codes; all Hamming similarities are determined according to the hash code and all sample hash codes respectively, and a similarity ranking list is determined according to all Hamming similarities; a target retrieval result is determined according to the ranking list of all Hamming similarities; wherein the target retrieval result is a set of ship sample images similar to the ship image to be retrieved, selected based on all similarities in the ranking list.

[0126] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ship image retrieval method, characterized in that: include: Acquire a ship image to be retrieved and a ship sample database; wherein the ship sample database contains at least two ship sample images, and the ship sample images are ship images of different modalities from the ship image to be retrieved; The ship image to be retrieved and all the ship sample images in the ship sample database are respectively input into a ship visual language model to obtain a hash code corresponding to the ship image to be retrieved output by the ship visual language model and all sample hash codes corresponding to all the ship sample images in the ship sample database; wherein the ship visual language model is obtained by training based on an acquired multimodal ship sample pattern data set including a ship panchromatic image data set, a ship multispectral image data set and label texts, and the ship visual language model is a model that processes the ship image to be retrieved and the ship sample database based on multimodal contrast learning, cross-modal semantic fusion and hash learning to obtain the corresponding hash code and the sample hash code; Determine all Hamming similarities according to the hash code and all the sample hash codes respectively, and determine a similarity ranking list according to all the Hamming similarities; The target retrieval result is determined according to the ranking list of all the Hamming similarities; wherein the target retrieval result is a set of the sample ship images similar to the ship image to be retrieved, selected based on all the similarities in the ranking list.

2. The ship image retrieval method according to claim 1, characterized in that: The ship visual language model is trained based on the following steps: Acquire the multimodal ship sample pattern dataset; wherein the multimodal ship sample pattern dataset includes a ship panchromatic image dataset, a ship multispectral image dataset and a label text, and the label text is a paired label text corresponding to a paired sample determined by pairing the ship panchromatic image dataset and the ship multispectral image dataset; Inputting the ship panchromatic image dataset, the ship multispectral image dataset and the label text into a basic ship visual language model to obtain a dataset hash code and a target loss function output by the basic ship visual language model; wherein the basic ship visual language model is a predetermined basic model for training; The basic ship visual language model is optimized according to the data set hash code and the target loss function to obtain the ship visual language model.

3. The ship image retrieval method according to claim 2, characterized in that: The step of inputting the ship panchromatic image dataset, the ship multispectral image dataset and the label text into a basic ship visual language model to obtain a dataset hash code and a target loss function output by the basic ship visual language model includes: Inputting the ship panchromatic image dataset and the ship multispectral image dataset into the multimodal contrastive learning network in the basic ship visual language model, and obtaining image features and contrast losses output by the multimodal contrastive learning network; the image features are obtained by the multimodal contrastive learning network performing image enhancement processing and similarity measurement on the ship panchromatic image dataset and the ship multispectral image dataset; Inputting the image features and the label text into the cross-modal semantic fusion network in the basic ship visual language model to obtain fusion features and fusion losses output by the cross-modal semantic fusion network; wherein the fusion features are obtained by aligning and fusing the image features and the label text by the cross-modal semantic fusion network; Input the fused features into the hash learning network in the basic ship visual language model to obtain the data set hash code and hash loss output by the hash learning network; wherein the data set hash code is obtained by the hash learning network performing hash learning on the fused features; The target loss function is determined according to the contrast loss, the fusion loss and the hash loss.

4. The ship image retrieval method according to claim 3, characterized in that: Inputting the fusion feature into the hash learning network in the basic ship visual language model to obtain the data set hash code output by the hash learning network includes: Inputting the fused features into the hash learning layer in the hash learning network to obtain the hash features output by the hash learning layer; wherein the hash learning layer is a layer that processes the fused features based on a nonlinear activation function; The hash feature is input into a binarization layer in the hash learning network to obtain a hash code of the data set output by the binarization layer; wherein the binarization layer is a layer that performs binary processing on the hash feature based on a sign function.

5. The ship image retrieval method according to any one of claims 1 to 3, characterized in that: The determining all Hamming similarities according to the hash code and all the sample hash codes respectively includes: The hash code and all the sample hash codes are multiplied respectively to obtain all the Hamming similarities.

6. The ship image retrieval method according to any one of claims 1 to 4, characterized in that: The step of determining the target search result according to the ranking list of all the Hamming similarities comprises: Determine all candidate similarities according to all the Hamming similarities in the ranking list and the similarity threshold; wherein all the candidate similarities are the Hamming similarities in the ranking list whose Hamming similarities are greater than the similarity threshold, and the similarity threshold is a preset threshold; All the candidate similarities corresponding to all the ship sample images are determined as the target retrieval results.

7. A ship image retrieval device, characterized in that: include: An image acquisition module, used to acquire a ship image to be retrieved and a ship sample database; wherein the ship sample database contains at least two ship sample images, and the ship sample images are ship images of different modalities from the ship image to be retrieved; A hash code output module is used to input the ship image to be retrieved and all the ship sample images in the ship sample database into a ship visual language model respectively, to obtain the hash code corresponding to the ship image to be retrieved output by the ship visual language model and all sample hash codes corresponding to all the ship sample images in the ship sample database; wherein the ship visual language model is obtained by training based on an acquired multimodal ship sample pattern data set including a ship panchromatic image data set, a ship multispectral image data set and label text, and the ship visual language model is a model that processes the ship image to be retrieved and the ship sample database based on multimodal contrast learning, cross-modal semantic fusion and hash learning to obtain the corresponding hash code and sample hash code; A list determination module, used to determine all Hamming similarities according to the hash code and all the sample hash codes respectively, and determine a similarity ranking list according to all the Hamming similarities; A result determination module is used to determine a target retrieval result according to a ranking list of all the Hamming similarities; wherein the target retrieval result is a set of sample ship images similar to the ship image to be retrieved, selected based on all the similarities in the ranking list.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for retrieving a ship image according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for retrieving a ship image according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for retrieving a ship image according to any one of claims 1 to 6 is implemented.

Images