Cross-modal remote sensing ship search method and device

Abstract

The application relates to the technical field of computer vision and remote sensing image processing, and provides a cross-modal remote sensing ship searching method and device.The method comprises the following steps: acquiring a first modal remote sensing image of a target ship to be searched; inputting the first modal remote sensing image and a plurality of second modal remote sensing images in a search library into a pre-trained target cross-modal ship searching model; outputting a predicted bounding box and an identity feature corresponding to a candidate target region in each remote sensing image through the target cross-modal ship searching model; the target cross-modal ship searching model is obtained through joint foreground segmentation, target detection and identity feature comparison training; based on the similarity between the identity feature corresponding to the target ship in the first modal remote sensing image and the identity features corresponding to the candidate target regions in each second modal remote sensing image, a candidate target matched with the target ship and a predicted bounding box thereof are determined, and a search result is obtained, so that the accuracy of cross-modal remote sensing ship searching can be improved to a certain extent.

Description

Technical Field

[0001] This application relates to the fields of computer vision and remote sensing image processing technology, and in particular to a cross-modal remote sensing ship search method and apparatus. Background Technology

[0002] With the development of sky-based multi-source Earth observation networks, cross-modal ship search using visible light images and synthetic aperture radar (SAR) images has become a key technology for maritime surveillance. Its main task is to accurately recall the same ship from an image library of another modality when given a query image of one modality.

[0003] Currently, relevant cross-modal ship search methods include: one is a search method based on common feature space mapping, which designs a cross-modal loss function to project the features of optical and SAR images onto the same metric space and directly calculates feature similarity for remote sensing ship search; the other is a search method based on generative adversarial networks, which attempts to convert an image of one modality into the style of another modality and then uses a single-modal retrieval model for remote sensing ship search.

[0004] However, the aforementioned remote sensing ship search methods suffer from significant domain gaps in feature distribution across different modes due to the fundamental differences in imaging mechanisms between optical reflection and microwave scattering. This can easily lead to cross-modal semantic misalignment, resulting in low accuracy in cross-modal remote sensing ship search. Summary of the Invention

[0005] This application provides a cross-modal remote sensing ship search method and apparatus, which can improve the accuracy of cross-modal remote sensing ship search to a certain extent.

[0006] This application provides a cross-modal remote sensing ship search method, including: Acquire the first modal remote sensing image of the target ship to be searched; The first modal remote sensing image and multiple second modal remote sensing images from the search library are input into a pre-trained target cross-modal ship search model. The target cross-modal ship search model outputs the predicted bounding boxes and identity features corresponding to candidate target regions in each remote sensing image. The remote sensing images are either the first modal remote sensing image or images from the second modal remote sensing images. The target cross-modal ship search model is trained by jointly performing foreground segmentation, target detection, and identity feature comparison. Based on the similarity between the identity features corresponding to the target ship in the first modality remote sensing image and the identity features corresponding to each candidate target region in each of the second modality remote sensing images, candidate targets matching the target ship and their predicted bounding boxes are determined, and search results are obtained.

[0007] According to the cross-modal remote sensing ship search method provided in this application, the method involves inputting a first-modal remote sensing image and multiple second-modal remote sensing images from a search library into a pre-trained target cross-modal ship search model, and outputting the predicted bounding boxes and identity features corresponding to candidate target regions in each remote sensing image through the target cross-modal ship search model, including: For each of the aforementioned remote sensing images, the remote sensing images are input into the target backbone network of the target cross-modal ship search model, and the basic feature map of the remote sensing images is extracted through the target backbone network; and candidate target regions of the remote sensing images are generated based on the basic feature map, and the region of interest features corresponding to the candidate target regions are extracted through the region of interest alignment operation; The region of interest features corresponding to the candidate target region are input into the target search head network in the target cross-modal ship search model, and the predicted bounding box and the identity features corresponding to the candidate target region are output through the target search head network.

[0008] According to the cross-modal remote sensing ship search method provided in this application, the target cross-modal ship search model is obtained through joint foreground segmentation, target detection, and identity feature comparison training, including: Multiple remote sensing image sample pairs are acquired, wherein each remote sensing image sample pair includes a first modal remote sensing image sample and a second modal remote sensing image sample. Each remote sensing image sample is labeled with the ship's true horizontal bounding box and the corresponding true category label, and has a corresponding labeled mask image. For each pair of remote sensing image samples, the pair is input into the initial backbone network of the cross-modal ship search model to be trained. The initial backbone network determines candidate target regions in each remote sensing image sample. Each candidate target region is mapped onto a corresponding basic feature map. Through region of interest alignment, region of interest features of each candidate target region are extracted. The region of interest features of each candidate target region are input into the mask auxiliary head network and the search head network of the cross-modal ship search model, respectively. The mask auxiliary head network outputs the foreground mask image corresponding to each candidate target region. The search head network outputs the predicted bounding box, predicted class probability, and initial identity features corresponding to each candidate target region. Based on the true horizontal bounding boxes, true class labels, labeled mask images, predicted bounding boxes, predicted class probabilities, and initial identity features corresponding to each remote sensing image sample pair, the model parameters of the cross-modal ship search model are updated until the target cross-modal ship search model is trained.

[0009] According to the cross-modal remote sensing ship search method provided in this application, the acquisition of multiple remote sensing image sample pairs includes: Acquire a multimodal remote sensing ship dataset; wherein the multimodal remote sensing ship dataset includes multiple first modality remote sensing image samples and multiple second modality remote sensing image samples; Based on the category histograms corresponding to each remote sensing image sample, the category similarity between each first modality remote sensing image sample and each second modality remote sensing image sample is determined, and a cost matrix is ​​constructed based on the determined multiple category similarities. The Hungarian algorithm is used to solve the cost matrix globally for optimal matching, and the pairing scheme with the minimum total cost is found to obtain multiple pairs of remote sensing image samples.

[0010] According to the cross-modal remote sensing ship search method provided in this application, the method involves updating the model parameters of the cross-modal ship search model based on the ground truth horizontal bounding box, ground truth class label, labeled mask image, predicted bounding box, predicted class probability, and initial identity features corresponding to each remote sensing image sample pair, until the target cross-modal ship search model is trained, including: For each of the candidate target regions, based on the intersection-union ratio (IU) between the candidate target region and the true horizontal bounding box corresponding to the remote sensing image sample to which it belongs, the top K candidate target regions with the highest IU are determined from all candidate target regions, as well as positive sample candidate target regions with IU greater than a preset threshold and negative sample candidate target regions with IU less than the preset threshold. Based on the foreground mask images of the first K candidate target regions, construct the corresponding mask loss for the remote sensing image sample pairs; Based on the predicted bounding box and predicted class probability corresponding to the positive sample candidate target region, the true horizontal bounding box and true class label corresponding to the remote sensing image sample, and the predicted class probability and true class probability corresponding to the negative sample candidate target region, construct the target detection loss for the remote sensing image sample. The initial identity features corresponding to the positive sample candidate target regions are normalized to obtain normalized identity features. Based on the normalized identity features corresponding to the positive sample candidate target region, construct the identity master supervision loss and prototype quintuple contrast loss for the remote sensing image sample pair. The target loss corresponding to the remote sensing image sample pair is obtained by weighted summing of the mask loss, target detection loss, identity master supervision loss and prototype quintuple contrast loss. Based on the target loss corresponding to each remote sensing image sample pair, the model parameters of the cross-modal ship search model are updated until the target cross-modal ship search model is trained.

[0011] According to the cross-modal remote sensing ship search method provided in this application, the step of constructing the mask loss corresponding to the remote sensing image sample pairs based on the foreground mask images of the top K candidate target regions includes: For each of the candidate target regions, the labeled mask image corresponding to the remote sensing image sample to which the candidate target region belongs is projected onto the local coordinate system of the feature space where the candidate target region is located to obtain the projected real mask corresponding to the candidate target region. Based on the foreground mask image of the candidate target region and the corresponding projected real mask, the mask loss of the candidate target region is determined; Based on the mask loss of each candidate target region, the mask loss corresponding to the remote sensing image sample pair is determined.

[0012] According to the cross-modal remote sensing ship search method provided in this application, the step of constructing the target detection loss for the remote sensing image sample based on the predicted bounding box and predicted class probability corresponding to the positive sample candidate target region, the true horizontal bounding box and true class label corresponding to the remote sensing image sample, and the predicted class probability and true class probability corresponding to the negative sample candidate target region includes: Based on the predicted bounding boxes corresponding to the positive sample candidate target regions and the true horizontal bounding boxes corresponding to the remote sensing image samples, the bounding box regression loss is determined. The first classification loss is determined based on the predicted class probability corresponding to the positive sample candidate target region and the true class label corresponding to the remote sensing image sample. The bounding box regression loss and the first classification loss are weighted and summed to obtain the target detection loss corresponding to the positive sample candidate target region; Based on the predicted class probability corresponding to the negative sample candidate target region and the real class label corresponding to the remote sensing image sample, a second classification loss is determined, and the second classification loss is used as the target detection loss corresponding to the negative sample candidate target region. Based on the target detection loss corresponding to the positive sample candidate target region and the target detection loss corresponding to the negative sample candidate target region, the target detection loss corresponding to the remote sensing image sample pair is determined.

[0013] According to the cross-modal remote sensing ship search method provided in this application, the step of constructing the identity master supervision loss corresponding to the remote sensing image sample pair based on the normalized identity features corresponding to the positive sample candidate target region includes: Determine the first similarity between the normalized identity features and all class prototypes in the preset class feature lookup table; Determine the second similarity between the normalized identity features and all negative features in a preset negative sample circular queue; Based on the first similarity and the second similarity, the identity master supervision loss corresponding to the remote sensing image sample pair is determined.

[0014] According to the cross-modal remote sensing ship search method provided in this application, determining the first similarity between the normalized identity features and all class prototypes in a preset class feature lookup table includes: Obtain a preset class feature lookup table, which stores multiple class prototypes, each class prototype corresponding to a ship identity category; Determine the cosine similarity or dot product similarity between the normalized identity features and each class prototype to obtain multiple similarity values; The vector or set formed by the multiple similarity values ​​is determined as the first similarity.

[0015] According to the cross-modal remote sensing ship search method provided in this application, determining the second similarity between the normalized identity features and all negative features in a preset negative sample circular queue includes: Obtain a preset negative sample circular queue, which stores multiple negative features, each of which is an identity feature of a different identity category than the normalized identity feature; Determine the cosine similarity or dot product similarity between the normalized identity feature and each negative feature in the negative sample circular queue to obtain multiple negative similarity values; The vector or set formed by the plurality of negative similarity values ​​is determined as the second similarity.

[0016] According to the cross-modal remote sensing ship search method provided in this application, the step of determining the identity master supervision loss corresponding to the remote sensing image sample pair based on the first similarity and the second similarity includes: The identity master supervision loss for the remote sensing image sample pair is determined based on the following formula; in, This represents the normalized identity feature. The true identity category of the positive sample candidate target region in the feature lookup table. The corresponding class prototype, This represents the first similarity between the normalized identity features and the corresponding class prototype. Indicates the first class in the feature lookup table Prototypical characteristics of class identity This indicates that the normalized identity features are related to the first... The first similarity between the prototype features of class identity, This indicates the number of negative samples in the circular queue. k One negative feature, The second similarity represents the normalized identity feature and the k-th negative feature. This represents the temperature scaling factor, used to control the smoothness of the similarity probability distribution.

[0017] According to the cross-modal remote sensing ship search method provided in this application, based on the normalized identity features corresponding to the positive sample candidate target region, a prototype quintuple contrast loss is constructed for the remote sensing image sample pair, including: Determine the first distance between the normalized identity feature and the same-domain positive prototype in the preset cross-domain prototype memory, the second distance between the normalized identity feature and the same-domain negative prototype in the cross-domain prototype memory, the third distance between the normalized identity feature and the cross-domain positive prototype in the cross-domain prototype memory, and the fourth distance between the normalized identity feature and the cross-domain negative prototype in the cross-domain prototype memory. Based on the first distance, the second distance, the third distance, and the fourth distance, the prototype quintuple contrast loss corresponding to the remote sensing image sample pair is determined.

[0018] According to the cross-modal remote sensing ship search method provided in this application, the step of determining the prototype quintuple contrast loss corresponding to the remote sensing image sample pair based on the first distance, the second distance, the third distance, and the fourth distance includes: in, This represents the prototype quintuple contrast loss. Indicates the third distance, This represents the normalized identity feature. This represents the cross-domain prototype in the cross-domain prototype memory. This represents the fourth distance. This represents the cross-domain negative prototype in the cross-domain prototype memory. Indicates the first distance. This represents the co-domain prototype in the cross-domain prototype memory. Indicates the second distance. This represents the co-domain negative prototype in the cross-domain prototype memory. These are boundary parameters used to force the setting of the margin between the positive and negative prototypes. Distance between features is typically measured using Euclidean distance or cosine distance.

[0019] This application also provides a cross-modal remote sensing ship search device, including: The acquisition unit is used to acquire the first modal remote sensing image of the target ship to be searched; The processing unit is configured to input the first modal remote sensing image and multiple second modal remote sensing images from the search library into a pre-trained target cross-modal ship search model, and output the predicted bounding boxes and identity features corresponding to candidate target regions in each remote sensing image through the target cross-modal ship search model; wherein, the remote sensing image is an image from the first modal remote sensing image or the second modal remote sensing image; the target cross-modal ship search model is obtained through joint foreground segmentation, target detection, and identity feature comparison training; The processing unit is further configured to determine candidate targets and their predicted bounding boxes that match the target ship based on the similarity between the identity features corresponding to the target ship in the first modality remote sensing image and the identity features corresponding to each candidate target region in each of the second modality remote sensing images, thereby obtaining search results.

[0020] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the cross-modal remote sensing ship search method as described above.

[0021] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-modal remote sensing ship search method as described above.

[0022] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the cross-modal remote sensing ship search method as described above.

[0023] The cross-modal remote sensing ship search method and apparatus provided in this application acquire a first modal remote sensing image of the target ship to be searched; input the first modal remote sensing image and multiple second modal remote sensing images from the search library into a pre-trained target cross-modal ship search model; output the predicted bounding boxes and identity features corresponding to candidate target regions in each remote sensing image through the target cross-modal ship search model; wherein, the remote sensing image is an image from either the first modal remote sensing image or the second modal remote sensing image; the target cross-modal ship search model is trained by joint foreground segmentation, target detection, and identity feature comparison; based on the similarity between the identity features corresponding to the target ship in the first modal remote sensing image and the identity features corresponding to each candidate target region in each second modal remote sensing image, candidate targets matching the target ship and their predicted bounding boxes are determined, and search results are obtained. This cross-modal ship search model, trained by combining foreground segmentation, target detection, and identity feature comparison, can effectively eliminate cross-modal semantic misalignment caused by differences in optical and SAR imaging mechanisms, enhance the consistency and discriminativeness of features of the same ship in different modalities, and thus improve the accuracy of cross-modal remote sensing ship search to a certain extent. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating a cross-modal remote sensing ship search method provided in an embodiment of this application.

[0026] Figure 2 This is a flowchart illustrating the training process of a target cross-modal ship search model provided in an embodiment of this application.

[0027] Figure 3 This is a schematic diagram of the structure of a cross-modal remote sensing ship search device provided in an embodiment of this application.

[0028] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0031] The primary task of cross-modal ship search methods is to accurately retrieve the same ship from an image database of another modality, given a query image in one modality. Currently, related cross-modal ship search methods, such as those based on common feature space mapping or generative adversarial networks, suffer from low accuracy due to the fundamental differences in imaging mechanisms between optical reflection and microwave scattering. These differences result in significant domain gaps in the feature distribution of the same ship across different modalities, easily leading to cross-modal semantic misalignment.

[0032] To improve the accuracy of cross-modal remote sensing ship search to a certain extent, this application provides a cross-modal remote sensing ship search method. The following specific embodiments will describe the cross-modal remote sensing ship search method provided in this application in detail. It is understood that these specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0033] Figure 1 A flowchart illustrating a cross-modal remote sensing ship search method provided in this application embodiment is shown below. For example, please refer to... Figure 1 As shown, this cross-modal remote sensing ship search method may include: S101. Obtain the first modal remote sensing image of the target ship to be searched.

[0034] For example, the first modal remote sensing image can be either an optical remote sensing image or a synthetic aperture radar (SAR) remote sensing image. This first modal remote sensing image includes the target ship to be searched, i.e., the ship whose identity the user wishes to identify from another modal image library through a cross-modal search task.

[0035] For example, the methods for acquiring the first modal remote sensing image may include: reading the pre-acquired first modal remote sensing image from a local storage device, receiving real-time first modal remote sensing images from platforms such as satellites or drones via a network, or retrieving historical images of a specified target ship from a remote sensing image database as the first modal remote sensing image, etc. The specific settings can be configured according to the actual application scenario.

[0036] For example, the first modal remote sensing image is usually a single image, which may include one or more ship targets. Users can specify specific query targets by selecting targets in the first modal remote sensing image or by automatic detection by the system.

[0037] For example, in this embodiment of the application, an optical remote sensing image can be used as the first modality remote sensing image. The user provides an optical image containing a specific ship and uses this first modality remote sensing image as the query input to search for the same ship in the SAR image. Conversely, a SAR image can also be used as the first modality remote sensing image. The user provides an optical image containing a specific ship and uses this first modality remote sensing image as the query input to search for the same ship in the optical image. The specific method can be chosen according to actual needs.

[0038] S102. Input the first modal remote sensing image and multiple second modal remote sensing images from the search library into the pre-trained target cross-modal ship search model, and output the predicted bounding boxes and identity features corresponding to the candidate target regions in each remote sensing image through the target cross-modal ship search model.

[0039] The remote sensing images are either images from the first modality remote sensing image or the second modality remote sensing image; the target cross-modality ship search model is obtained through joint foreground segmentation, target detection, and identity feature comparison training.

[0040] For example, the second modality remote sensing image belongs to a different imaging modality than the first modality remote sensing image. For instance, if the first modality is an optical remote sensing image, then the second modality is a synthetic aperture radar (SAR) image; and vice versa. The search library contains multiple second modality remote sensing images, and the goal is to find targets from these second modality remote sensing images that match the identity of the queried ship.

[0041] For example, in this embodiment of the application, a first modality remote sensing image and multiple second modality remote sensing images from the search library are input into a pre-trained target cross-modal ship search model. The target cross-modal ship search model outputs the predicted bounding boxes and identity features corresponding to candidate target regions in each remote sensing image, including: For each remote sensing image, the following operations are performed independently: The remote sensing image is input into the target backbone network of the target cross-modal ship search model. The target backbone network extracts the basic feature map of the remote sensing image. This basic feature map preserves the spatial structure and semantic information of the modal remote sensing image, providing a feature basis for subsequent target detection. Candidate target regions (Regions of Interest, RoIs) of the remote sensing image are generated based on the basic feature map. The RoIAlign operation is used to extract the features of the corresponding regions of interest of the candidate target regions, such as 14×14×C, to unify the spatial size of the subsequent network input. The features of the corresponding regions of interest of the candidate target regions are input into the target search head network of the target cross-modal ship search model. The target search head network outputs the predicted bounding box (i.e., the precise position coordinates of the ship in the image) and identity features (a high-dimensional vector used to represent the ship's identity, usually after L2 normalization) corresponding to the candidate target regions.

[0042] For example, the target backbone network can be ResNet, VGG network, etc., and the specific configuration can be made according to actual needs.

[0043] For example, if the first modal remote sensing image contains multiple candidate target regions, one of them can be selected as the target ship. For instance, the ship in the candidate target region corresponding to the target box specified by the user can be selected as the target ship, or the ship in the first candidate target region output by the target search head network can be selected as the target ship, and its identity features can be extracted as query features to find targets in the second modal remote sensing image that match the identity of the query ship.

[0044] It is understood that, in the embodiments of this application, for each second modality remote sensing image in the search library, the target backbone network can output multiple candidate target regions, each candidate target region corresponding to an identity feature and a predicted bounding box, and the identity features of these candidate target regions can be collectively referred to as candidate features.

[0045] S103. Based on the similarity between the identity features corresponding to the target ship in the first modality remote sensing image and the identity features corresponding to each candidate target region in each second modality remote sensing image, determine the candidate targets that match the target ship and their predicted bounding boxes, and obtain the search results.

[0046] For example, when calculating the similarity between the identity features corresponding to the target ship in the first modality remote sensing image and the identity features corresponding to each candidate target region in each second modality remote sensing image, L2 normalization can be performed on the identity features corresponding to the target ship and the identity features corresponding to each candidate target region in each second modality remote sensing image. Then, the cosine similarity between the query feature (i.e., the identity feature corresponding to the target ship in the first modality remote sensing image) and each candidate identity feature (i.e., the identity feature corresponding to each candidate target region in each second modality remote sensing image) can be calculated. Afterwards, all candidate targets can be sorted in descending order based on the similarity score. The candidate target with the highest score is determined as the target matching the target ship, and its predicted bounding box and the identifier of the corresponding second modality remote sensing image are output as the search result. By performing L2 normalization on identity features and using cosine similarity measurement, the semantic consistency between cross-modal features can be accurately measured. On this basis, by combining descending order ranking and the highest score matching strategy, mismatches caused by feature scale differences or noise interference can be effectively avoided, thereby further improving the accuracy and reliability of cross-modal remote sensing ship search.

[0047] The similarity score ranges from [-1, 1]. Generally, a higher similarity score indicates that the two features are more similar, and the greater the likelihood that the corresponding candidate target and the queried ship are the same.

[0048] As can be seen, in this embodiment, a first modal remote sensing image of the target ship to be searched is acquired; the first modal remote sensing image and multiple second modal remote sensing images from the search library are input into a pre-trained target cross-modal ship search model, and the target cross-modal ship search model outputs the predicted bounding boxes and identity features corresponding to the candidate target regions in each remote sensing image; wherein, the remote sensing image is an image from the first modal remote sensing image or the second modal remote sensing image; the target cross-modal ship search model is trained by joint foreground segmentation, target detection and identity feature comparison; based on the similarity between the identity features corresponding to the target ship in the first modal remote sensing image and the identity features corresponding to each candidate target region in each second modal remote sensing image, the candidate targets matching the target ship and their predicted bounding boxes are determined, and the search results are obtained. This cross-modal ship search model, trained by combining foreground segmentation, target detection, and identity feature comparison, can effectively eliminate cross-modal semantic misalignment caused by differences in optical and SAR imaging mechanisms, enhance the consistency and discriminativeness of features of the same ship in different modalities, and thus improve the accuracy of cross-modal remote sensing ship search to a certain extent.

[0049] Based on the above Figure 1 The illustrated embodiment, for example, in this application embodiment, the target cross-modal ship search model is obtained through joint foreground segmentation, target detection, and identity feature comparison training. For a specific implementation, please refer to [link to relevant documentation]. Figure 2 The example shown.

[0050] Figure 2 A flowchart illustrating the training process of a target cross-modal ship search model provided in this application embodiment is shown below. For example, please refer to... Figure 2 As shown, the training process may include: S201. Obtain multiple remote sensing image sample pairs, wherein each remote sensing image sample pair includes a first modal remote sensing image sample and a second modal remote sensing image sample. Each remote sensing image sample is labeled with the ship's true horizontal bounding box and the corresponding true category label, and has a corresponding labeled mask image.

[0051] For example, the ship's true horizontal bounding box and corresponding true category label are detected and labeled based on the Horizontal Bounding Box (HBB) format. The labeled mask image is obtained by rasterizing a polygonal fine-grained annotation file based on JSON format, providing the coordinate points of the ship target's outline polygons. Additionally, modal domain labels can be set for each remote sensing image sample: optical remote sensing image sample domain label Dopt=0, SAR remote sensing image sample domain label Dsar=1, to support subsequent cross-domain alignment and prototype maintenance.

[0052] For example, in this embodiment of the application, when acquiring multiple remote sensing image sample pairs, a multimodal remote sensing ship dataset can be acquired. This dataset includes multiple first-modal remote sensing image samples and multiple second-modal remote sensing image samples. Based on the category histograms corresponding to each remote sensing image sample, the category similarity between each first-modal and each second-modal remote sensing image sample is determined, and a cost matrix is ​​constructed based on the determined category similarities. The Hungarian algorithm is used to solve the cost matrix for global optimal matching, finding the pairing scheme with the minimum total cost, thus obtaining multiple remote sensing image sample pairs. By statistically analyzing the image category histograms and using the Hungarian algorithm for global optimal matching, the problem of missing cross-modal positive samples within a batch caused by traditional random sampling is effectively solved, gradient oscillations during training are eliminated, and stable, high-quality remote sensing image sample pairs are provided for subsequent comparative learning. This can, to a certain extent, improve the training efficiency and feature alignment capability of the cross-modal ship search model.

[0053] For example, the construction method for the category histogram corresponding to the remote sensing image sample is as follows: For each remote sensing image sample, the ship identity categories and their frequencies are counted, forming a multi-dimensional vector. Each dimension of the multi-dimensional vector corresponds to an identity category, and the dimension value is the number of times that category appears in the remote sensing image sample. The category similarity between the category histograms of two remote sensing image samples is then considered. It is the cosine of the angle between two vectors, and its value ranges from [0,1]. The greater the class similarity, the more similar the class distributions of the two remote sensing image samples are.

[0054] For example, when constructing a cost matrix based on the determined similarities of multiple categories, the similarity matrix composed of multiple category similarities can be denoted as S, and the cost matrix can be the cost matrix of bipartite graph matching, denoted as C, whose elements are defined as follows: .in, This represents the class similarity between the i-th first-modality remote sensing image sample and the j-th second-modality remote sensing image sample. This represents the cost required to pair the i-th first-modality remote sensing image sample with the j-th second-modality remote sensing image sample. The higher the class similarity, the lower the cost (closer to 0); the lower the class similarity, the higher the cost (closer to 1).

[0055] The Hungarian algorithm aims to minimize the total cost of all pairings, seeking the globally optimal pairing scheme, i.e., the scheme where all pairings are minimized. The minimum sum is equivalent to all paired values. The sum is maximized. In this way, the two remote sensing image samples in each remote sensing image sample pair have the closest class distribution, thereby ensuring stable input of paired cross-modal positive samples in the training batch, thus providing a high-quality data foundation for subsequent identity feature comparison learning.

[0056] S202. For each pair of remote sensing image samples, the remote sensing image sample pairs are input into the initial backbone network of the cross-modal ship search model to be trained. The initial backbone network determines the candidate target regions in each remote sensing image sample. Each candidate target region is mapped onto the corresponding basic feature map. Through the region of interest alignment operation, the region of interest features of each candidate target region are extracted. The region of interest features of each candidate target region are input into the mask auxiliary head network and the search head network of the cross-modal ship search model, respectively. The mask auxiliary head network outputs the foreground mask image corresponding to each candidate target region. The search head network outputs the predicted bounding box, predicted class probability, and initial identity features corresponding to each candidate target region.

[0057] To improve the semantic purity of the foreground, for example, in this embodiment of the application, a polygon-guided region of interest masking auxiliary head network is innovatively designed. This masking auxiliary head network is introduced in parallel with the search head network. By using polygon annotations to perform fine-grained masking supervision on candidate target regions during the training phase, the network is forced to focus its attention on the foreground pixels of the ship target, effectively removing ocean clutter and port background, avoiding noise interference in the cross-domain alignment process, thereby significantly improving the purity and discriminativeness of the retrieved features.

[0058] For example, the masking auxiliary head network can be composed of 3 convolutional layers cascaded together, specifically: 3×3 convolution + 3×3 convolution + 1×1 convolution, with the number of intermediate channels set to 256 and the pooling size set to 14×14.

[0059] For example, the search head network typically consists of fully connected or convolutional layers and contains three parallel branches: bounding box regression branch, category classification branch, and identity feature extraction branch.

[0060] For example, the above-mentioned region of interest alignment operation avoids quantization errors by preserving floating-point coordinates and using interpolation methods, such as bilinear interpolation. By performing the region of interest alignment operation, the region of interest features of each candidate target region in the remote sensing image sample pair can be obtained. In this way, the region of interest alignment operation can avoid the two quantization rounding errors in traditional RoI Pooling, retain more accurate spatial location information, and thus improve the accuracy of mask prediction and target localization.

[0061] The region of interest (ROI) features of each candidate target region are input into the masking head network. When the masking head network outputs the foreground mask image corresponding to each candidate target region, the ROI features of each candidate target region are sequentially passed through 3×3 convolution + 3×3 convolution + 1×1 convolution, finally outputting a feature map of size 14×14×1, which is the foreground mask image corresponding to that candidate target region. Each pixel value in this foreground mask image is mapped to the (0,1) interval by the Sigmoid activation function, representing the probability that the pixel belongs to the ship's foreground.

[0062] The region of interest (ROI) features of each candidate target region are input into the search head network of the cross-modal ship search model. When the search head network outputs the predicted bounding box, predicted class probability, and initial identity features corresponding to each candidate target region, the ROI features of each candidate target region are first converted into a fixed-length one-dimensional vector (e.g., 256-dimensional) through global average pooling or flattening operations, and then input into the three branches respectively: Bounding box regression branch: This branch outputs a 4-dimensional vector representing the coordinates of the predicted bounding box, typically (x, y, w, h) or (xmin, ymin, xmax, ymax), used to pinpoint the precise location of the ship in the remote sensing image sample.

[0063] Category classification branch: Used to output a 2D vector (activated by Softmax) representing the probability that the candidate target region belongs to "ship" or "background", used to distinguish the foreground and background of remote sensing image samples.

[0064] Identity feature extraction branch: This branch outputs a D-dimensional vector (e.g., 256-dimensional or 512-dimensional), which is the initial identity feature used to represent the identity information of the candidate target. This initial identity feature is typically L2 normalized in subsequent steps and then used to calculate the identity main supervision loss and the prototype quintuple contrastive loss.

[0065] S203. Based on the true horizontal bounding boxes, true class labels, labeled mask images, predicted bounding boxes, predicted class probabilities, and initial identity features corresponding to each remote sensing image sample pair, update the model parameters of the cross-modal ship search model until the target cross-modal ship search model is trained.

[0066] For example, in this embodiment of the application, for each candidate target region, based on the intersection-union ratio (IU) between the candidate target region and the ground truth horizontal bounding box corresponding to its remote sensing image sample, the top K candidate target regions with the highest IU, positive sample candidate target regions with IU greater than a preset threshold, and negative sample candidate target regions with IU less than a preset threshold can be determined from all candidate target regions. Based on the foreground mask images of the top K candidate target regions, a mask loss corresponding to the remote sensing image sample pair is constructed. Based on the predicted bounding box and predicted class probability corresponding to the positive sample candidate target region, the ground truth horizontal bounding box and ground truth class label corresponding to its remote sensing image sample, and the predicted class probability corresponding to the negative sample candidate target region, the mask loss is constructed. The system predicts the class probability and the true class probability, and constructs the target detection loss corresponding to the remote sensing image sample pairs. It normalizes the initial identity features corresponding to the positive candidate target regions to obtain normalized identity features. Based on the normalized identity features corresponding to the positive candidate target regions, it constructs the identity master supervision loss and prototype quintuple contrast loss corresponding to the remote sensing image sample pairs. It then performs a weighted summation of the mask loss, target detection loss, identity master supervision loss, and prototype quintuple contrast loss corresponding to the remote sensing image sample pairs to obtain the target loss corresponding to each remote sensing image sample pair. Based on the target loss corresponding to each remote sensing image sample pair, it updates the model parameters of the cross-modal ship search model until a target cross-modal ship search model is trained.

[0067] For example, the true class label of a positive sample candidate target region is 1, and the true class label of a negative sample candidate target region is 0.

[0068] For each candidate target region, calculate the intersection-over-union (IoU) ratio between the candidate target region and all the true horizontal bounding boxes in the corresponding remote sensing image sample, and take the maximum value as the IoU score of the candidate region.

[0069] The top K candidate target regions: The top K candidate target regions with the highest IoU (e.g., K=16) are selected specifically for mask loss calculation. These selected candidate target regions are most likely to contain ships; fine-grained foreground segmentation supervision of them can improve feature purity.

[0070] Positive candidate target regions: If the IoU is greater than a preset threshold, such as 0.5, it is determined as a positive candidate target region and participates in the calculation of subsequent bounding box regression loss, classification loss and identity comparison loss.

[0071] Negative sample candidate target region: If the IoU is less than the preset threshold, it is determined as a negative sample candidate target region. It only participates in the subsequent classification loss and does not participate in the calculation of bounding box regression loss and identity comparison loss.

[0072] For example, in this embodiment of the application, the above-mentioned construction of a remote sensing image sample pair corresponding to the mask loss based on the foreground mask images of the first K candidate target regions is used to supervise the mask-assisted head network to perform pixel-level foreground segmentation. This can learn to accurately activate the ship body region and suppress background clutter, thereby providing high-quality candidate region features for subsequent identity feature extraction and cross-modal alignment, and may include: For each candidate target region, the labeled mask image corresponding to the remote sensing image sample to which the candidate target region belongs is projected onto the local coordinate system of the feature space where the candidate target region is located to obtain the projected true mask corresponding to the candidate target region; based on the foreground mask image of the candidate target region and the corresponding projected true mask, the mask loss of the candidate target region is determined; based on the mask loss of each candidate target region, the mask loss corresponding to the remote sensing image sample pair is determined.

[0073] For example, see Equation 1 below, mask loss The binary cross-entropy loss with logistic regression can be used for calculation: (1) in, The mask loss represents the candidate target region, used to measure the difference between the foreground mask image and the projected true mask for that region. The mask loss for each remote sensing image sample pair can be obtained by averaging (or summing) the mask losses of all candidate target regions. N represents the total number of pixels in the foreground mask image. If the size of the foreground mask image output by the mask auxiliary head network is 14×14, then N=196. i represents the pixel index in the mask image, ranging from 1 to N, and is used to traverse each pixel position. This represents the value of the i-th pixel in the projected true mask. =1 indicates that the pixel belongs to the foreground of the ship. =0 indicates that the pixel belongs to the background. This represents the predicted value of the i-th pixel in the foreground mask image. This represents the activation function, which will predict the value. Mapping to the (0,1) interval transforms the probability that the pixel belongs to the ship's foreground.

[0074] For example, in this embodiment of the application, the target detection loss for a remote sensing image sample is constructed based on the predicted bounding box and predicted class probability corresponding to the positive sample candidate target region, the true horizontal bounding box and true class label corresponding to the remote sensing image sample, and the predicted class probability and true class probability corresponding to the negative sample candidate target region. This can include: Based on the predicted bounding boxes corresponding to positive candidate target regions and the ground truth horizontal bounding boxes corresponding to remote sensing image samples, a bounding box regression loss is determined. Based on the predicted class probabilities corresponding to positive candidate target regions and the ground truth class labels corresponding to remote sensing image samples, a first classification loss is determined. The bounding box regression loss and the first classification loss are weighted and summed to obtain the target detection loss corresponding to the positive candidate target regions. Based on the predicted class probabilities corresponding to negative candidate target regions and the ground truth class labels corresponding to remote sensing image samples, a second classification loss is determined, which serves as the target detection loss for negative candidate target regions. Based on the target detection losses corresponding to both positive and negative candidate target regions, a target detection loss for each pair of remote sensing image samples is determined. This loss is used to supervise the search head network in simultaneously learning accurate bounding box regression and foreground / background classification, ensuring accurate ship location and differentiation from the background. This provides reliable target regions for subsequent identity feature extraction and cross-modal alignment, laying the foundation for improving the accuracy of cross-modal remote sensing ship search.

[0075] For example, when determining the bounding box regression loss based on the predicted bounding boxes corresponding to the positive sample candidate target regions and the true horizontal bounding boxes corresponding to the remote sensing image samples, the following formula 2 can be used: (2) in, This represents the bounding box regression loss. This represents the i-th coordinate component of the predicted bounding box. This represents the i-th coordinate component of the true horizontal bounding box. This represents the Smooth L1 loss function.

[0076] For example, the first classification loss is determined based on the predicted class probability corresponding to the positive sample candidate target region and the true class label corresponding to the remote sensing image sample, as shown in Formula 3 below: (3) in, This represents the first category loss. Indicates the true category label, =1, This indicates the predicted category probability.

[0077] For example, the second classification loss is determined based on the predicted class probability corresponding to the negative sample candidate target region and the true class label corresponding to the remote sensing image sample, as shown in Formula 4 below: (4) in, This represents the second category loss. Indicates the true category label, =0, This indicates the predicted category probability.

[0078] For example, when determining the target detection loss corresponding to a remote sensing image sample pair based on the target detection loss corresponding to the positive sample candidate target region and the target detection loss corresponding to the negative sample candidate target region, the target detection loss corresponding to the positive sample candidate target region and the target detection loss corresponding to the negative sample candidate target region can be averaged (or summed) to obtain the target detection loss corresponding to the remote sensing image sample pair.

[0079] For example, in the embodiments of this application, the above-mentioned construction of the identity master supervision loss for remote sensing image sample pairs based on the normalized identity features corresponding to the positive sample candidate target regions may include: The first similarity between the normalized identity features and all class prototypes in the preset class feature lookup table is determined; the second similarity between the normalized identity features and all negative features in the preset negative sample circular queue is determined. Based on the first and second similarities, the identity master supervision loss corresponding to the remote sensing image sample pair is determined. This loss is used to supervise the learning of highly discriminative identity features by the master supervision branch of the retrieval system. This allows the features of the same ship in different candidate regions to be tightly aggregated in the feature space, while the features of different ships are separated from each other. This provides a feature representation with strong identity discrimination capability for cross-modal similarity matching, which can effectively improve the accuracy of cross-modal ship search.

[0080] The supervised retrieval main branch maintains two structures in memory: a class feature lookup table and a negative sample circular queue. The class feature lookup table matrix... This is used to store the class prototypes of all known identities, i.e., positive features. D represents the feature dimension of the class prototypes of known identities, and C represents the number of identity categories; negative sample circular queue matrix. It is used to store non-target features, i.e., negative features, that have been calculated in recent training batches. The feature dimension representing the non-target feature. The number of prototype features representing a known identity; This represents the number of negative features in the negative sample circular queue.

[0081] For example, in this embodiment of the application, when determining the first similarity between the normalized identity feature and all class prototypes in the preset class feature lookup table, the preset class feature lookup table can be obtained first. The class feature lookup table stores multiple class prototypes, and each class prototype corresponds to a ship identity category. The cosine similarity or dot product similarity between the normalized identity feature and each class prototype is determined to obtain multiple similarity values. The vector or set composed of multiple similarity values ​​is determined as the first similarity.

[0082] For example, in this embodiment of the application, when determining the second similarity between the normalized identity feature and all negative features in the preset negative sample circular queue, the preset negative sample circular queue can be obtained. The negative sample circular queue stores multiple negative features, each of which is an identity feature of a different identity category than the normalized identity feature. The cosine similarity or dot product similarity between the normalized identity feature and each negative feature in the negative sample circular queue is determined to obtain multiple negative similarity values. The vector or set composed of the multiple negative similarity values ​​is determined as the second similarity.

[0083] For example, based on the first similarity and the second similarity, the identity-based master supervision loss for remote sensing image sample pairs is determined. For the following example, see Formula 5: (5) in, Indicates normalized identity characteristics. For the true identity category of the positive sample candidate target region in the feature lookup table. The corresponding class prototype, This represents the first similarity between normalized identity features and the corresponding class prototype. The first in the class feature lookup table Prototypical characteristics of class identity Indicates normalized identity features and the first The first similarity between the prototype features of class identity. This indicates the number of negative samples in the circular queue. k One negative feature, This represents the second similarity between the normalized identity feature and the k-th negative feature. This represents the temperature scaling factor, used to control the smoothness of the similarity probability distribution.

[0084] During the backpropagation and weight update phases, the corresponding prototype features in the lookup table V for the search class feature are evolved online using a moving average mechanism: ,in, This indicates the current training round, such as the real identity category in the feature lookup table V after the t-th round update. Prototype features, This indicates the corresponding real identity category in the feature lookup table V after the previous update. Prototype features, Indicates normalized identity characteristics. Indicates momentum update parameters; circular queue The first-in-first-out principle is used to continuously dequeue and enqueue operations. This moving average mechanism trains the current feature to move closer to its global identity prototype, while providing rich dynamic negative samples in the feature space for contrastive learning.

[0085] For example, in an embodiment of this application, in order to explicitly eliminate cross-modal semantic misalignment, a cross-domain prototype quintuple contrast constraint branch is injected during the training phase of the cross-modal remote sensing ship search model. This mechanism maintains a memory-sized... Cross-domain prototype memory This means that a prototype feature is maintained independently for each identity category in the optical domain and the SAR domain, where C represents the number of identity categories and D represents the feature dimension.

[0086] It should be noted that during the training phase, an initial prototype memory of identity features corresponding to positive sample candidate target regions can be established. The prototype features are all normalized, which can unify the feature measurement scale, stabilize the contrastive learning process, and enhance the discriminativeness of the features.

[0087] For example, in the embodiments of this application, the above-mentioned construction of the prototype quintuple contrast loss corresponding to the remote sensing image sample pair based on the normalized identity features corresponding to the positive sample candidate target region may include: Determine the first distance between the normalized identity features and the co-domain positive prototypes in the preset cross-domain prototype memory, the second distance between the normalized identity features and the co-domain negative prototypes in the cross-domain prototype memory, the third distance between the normalized identity features and the cross-domain positive prototypes in the cross-domain prototype memory, and the fourth distance between the normalized identity features and the cross-domain negative prototypes in the cross-domain prototype memory; based on the first, second, third, and fourth distances, determine the contrast loss of the corresponding prototype quintuple for the remote sensing image sample pairs.

[0088] For example, let's assume normalized identity features. The domain tag is , =0 indicates the optical domain. =1 indicates the SAR domain, and the true identity category is... With normalized identity characteristics Using these as anchors, four types of relational constraints are constructed in the cross-domain prototype memory: same-domain prototypes, etc. This refers to the normalized identity characteristics. Prototype features of the same modality and identity; negative prototypes in the same domain This refers to the normalized identity characteristics. Prototype features of the same modality but different identities; cross-domain positive prototypes This refers to the normalized identity characteristics. Prototype features of different modalities but the same identity; cross-domain negative prototypes This refers to the normalized identity characteristics. Prototype features of different modalities and different identities.

[0089] For example, based on the first distance, second distance, third distance, and fourth distance, the prototype quintuple contrast loss corresponding to the remote sensing image sample pairs with boundary constraints is determined. Please refer to the following formula: in, This represents the prototype quintuple contrast loss. Indicates the third distance. Indicates normalized identity characteristics. This represents a cross-domain prototype in the cross-domain prototype memory. Indicates the fourth distance. This represents the cross-domain negative prototype in the cross-domain prototype memory. Indicates the first distance. This represents a co-domain prototype in a cross-domain prototype memory. Indicates the second distance. This represents the co-domain negative prototype in the cross-domain prototype memory. These are boundary parameters used to force the setting of the margin between the positive and negative prototypes. Distance between features is typically measured using Euclidean distance or cosine distance.

[0090] It should be noted that, in the embodiments of this application, the boundary parameters This is used to enforce a margin between positive and negative prototypes, requiring that the distance between positive and negative prototypes be at least *m* smaller than the distance between negative prototypes. The distance from the current feature to a cross-domain (or same-domain) prototype of the same identity must be less than the distance to a cross-domain (or same-domain) prototype of different identities minus the boundary value *m*. In other words, the above prototype quintuple contrastive loss... The four loss parameters constrained the normalized identity features. The distance to the same identity prototype across domains (or within the same domain) must be less than the distance to different identity prototypes across domains (or within the same domain) minus the boundary value. Cross-domain prototype memory The prototype in the model also uses a momentum parameter of... The mechanism is used to update the moving average.

[0091] By constructing a cross-domain prototype memory library with "domain × category" states, positive and negative sample relationships within the same domain and across domains are simultaneously constructed in the normalized feature space. This allows optical features and SAR features of the same identity to be tightly aggregated in the feature space, while features of different identities are effectively separated. This overcomes the coarse-grainedness and one-sidedness of traditional single loss optimization and achieves fine-grained cross-modal semantic alignment, thereby improving the accuracy of cross-modal remote sensing ship search.

[0092] Based on the above description, the mask loss corresponding to each pair of remote sensing image samples is determined. Target detection loss Losses due to identity supervision Loss compared to the prototype quintuple By performing a weighted summation, the target loss corresponding to the remote sensing image sample is obtained. Please refer to the following formula: in, , , These represent the mask loss respectively. Target detection loss Losses due to identity supervision Loss compared to the prototype quintuple Each has its corresponding weighting coefficient.

[0093] By combining the above formulas to obtain the target loss corresponding to each remote sensing image sample pair, the model parameters of the cross-modal ship search model can be updated based on the target loss corresponding to each remote sensing image sample pair until the target cross-modal ship search model is trained. In this way, through multi-task joint optimization, the mask loss, target detection loss, identity master supervision loss and prototype quintuple contrast loss are weighted and summed and the model parameters are updated end-to-end. This enables the cross-modal ship search model to simultaneously learn accurate target localization, clean foreground semantic segmentation, highly discriminative identity features and cross-modal semantic alignment, thereby effectively eliminating semantic misalignment caused by the difference between optical and SAR imaging mechanisms and improving the accuracy and robustness of cross-modal remote sensing ship search.

[0094] The cross-modal remote sensing ship search device provided in this application will be described below. The cross-modal remote sensing ship search device described below can be referred to in correspondence with the cross-modal remote sensing ship search method described above.

[0095] Figure 3 A schematic diagram of a cross-modal remote sensing ship search device provided in this application embodiment is shown below. For example, please refer to... Figure 3 As shown, the cross-modal remote sensing ship search device 30 may include: Acquisition unit 301 is used to acquire the first modal remote sensing image of the target ship to be searched; Processing unit 302 is configured to input the first modal remote sensing image and multiple second modal remote sensing images from the search library into a pre-trained target cross-modal ship search model, and output the predicted bounding boxes and identity features corresponding to candidate target regions in each remote sensing image through the target cross-modal ship search model; wherein, the remote sensing image is an image from the first modal remote sensing image or the second modal remote sensing image; the target cross-modal ship search model is obtained by training through joint foreground segmentation, target detection, and identity feature comparison; The processing unit 302 is further configured to determine candidate targets and their predicted bounding boxes that match the target ship based on the similarity between the identity features corresponding to the target ship in the first modal remote sensing image and the identity features corresponding to each candidate target region in each of the second modal remote sensing images, thereby obtaining search results.

[0096] For example, in an embodiment of this application, the processing unit 302 is configured to input both the first modal remote sensing image and multiple second modal remote sensing images from the search library into a pre-trained target cross-modal ship search model, and output the predicted bounding boxes and identity features corresponding to candidate target regions in each remote sensing image through the target cross-modal ship search model, including: For each of the aforementioned remote sensing images, the remote sensing images are input into the target backbone network of the target cross-modal ship search model, and the basic feature map of the remote sensing images is extracted through the target backbone network; and candidate target regions of the remote sensing images are generated based on the basic feature map, and the region of interest features corresponding to the candidate target regions are extracted through the region of interest alignment operation; The region of interest features corresponding to the candidate target region are input into the target search head network in the target cross-modal ship search model, and the predicted bounding box and the identity features corresponding to the candidate target region are output through the target search head network.

[0097] For example, in an embodiment of this application, the target cross-modal ship search model is obtained through joint foreground segmentation, target detection, and identity feature comparison training, including: Multiple remote sensing image sample pairs are acquired, wherein each remote sensing image sample pair includes a first modal remote sensing image sample and a second modal remote sensing image sample. Each remote sensing image sample is labeled with the ship's true horizontal bounding box and the corresponding true category label, and has a corresponding labeled mask image. For each pair of remote sensing image samples, the pair is input into the initial backbone network of the cross-modal ship search model to be trained. The initial backbone network determines candidate target regions in each remote sensing image sample. Each candidate target region is mapped onto a corresponding basic feature map. Through region of interest alignment, region of interest features of each candidate target region are extracted. The region of interest features of each candidate target region are input into the mask auxiliary head network and the search head network of the cross-modal ship search model, respectively. The mask auxiliary head network outputs the foreground mask image corresponding to each candidate target region. The search head network outputs the predicted bounding box, predicted class probability, and initial identity features corresponding to each candidate target region. Based on the true horizontal bounding boxes, true class labels, labeled mask images, predicted bounding boxes, predicted class probabilities, and initial identity features corresponding to each remote sensing image sample pair, the model parameters of the cross-modal ship search model are updated until the target cross-modal ship search model is trained.

[0098] For example, in an embodiment of this application, acquiring multiple remote sensing image sample pairs includes: Acquire a multimodal remote sensing ship dataset; wherein the multimodal remote sensing ship dataset includes multiple first modality remote sensing image samples and multiple second modality remote sensing image samples; Based on the category histograms corresponding to each remote sensing image sample, the category similarity between each first modality remote sensing image sample and each second modality remote sensing image sample is determined, and a cost matrix is ​​constructed based on the determined multiple category similarities. The Hungarian algorithm is used to solve the cost matrix globally for optimal matching, and the pairing scheme with the minimum total cost is found to obtain multiple pairs of remote sensing image samples.

[0099] For example, in this embodiment of the application, updating the model parameters of the cross-modal ship search model based on the ground truth horizontal bounding box, ground truth class label, labeled mask image, predicted bounding box, predicted class probability, and initial identity features corresponding to each of the remote sensing image sample pairs, until the target cross-modal ship search model is trained, includes: For each of the candidate target regions, based on the intersection-union ratio (IU) between the candidate target region and the true horizontal bounding box corresponding to the remote sensing image sample to which it belongs, the top K candidate target regions with the highest IU are determined from all candidate target regions, as well as positive sample candidate target regions with IU greater than a preset threshold and negative sample candidate target regions with IU less than the preset threshold. Based on the foreground mask images of the first K candidate target regions, construct the corresponding mask loss for the remote sensing image sample pairs; Based on the predicted bounding box and predicted class probability corresponding to the positive sample candidate target region, the true horizontal bounding box and true class label corresponding to the remote sensing image sample, and the predicted class probability and true class probability corresponding to the negative sample candidate target region, construct the target detection loss for the remote sensing image sample. The initial identity features corresponding to the positive sample candidate target regions are normalized to obtain normalized identity features. Based on the normalized identity features corresponding to the positive sample candidate target region, construct the identity master supervision loss and prototype quintuple contrast loss for the remote sensing image sample pair. The target loss corresponding to the remote sensing image sample pair is obtained by weighted summing of the mask loss, target detection loss, identity master supervision loss and prototype quintuple contrast loss. Based on the target loss corresponding to each remote sensing image sample pair, the model parameters of the cross-modal ship search model are updated until the target cross-modal ship search model is trained.

[0100] For example, in an embodiment of this application, constructing the mask loss corresponding to the remote sensing image sample pair based on the foreground mask images of the top K candidate target regions includes: For each of the candidate target regions, the labeled mask image corresponding to the remote sensing image sample to which the candidate target region belongs is projected onto the local coordinate system of the feature space where the candidate target region is located to obtain the projected real mask corresponding to the candidate target region. Based on the foreground mask image of the candidate target region and the corresponding projected real mask, the mask loss of the candidate target region is determined; Based on the mask loss of each candidate target region, the mask loss corresponding to the remote sensing image sample pair is determined.

[0101] For example, in this embodiment of the application, the step of constructing the target detection loss for the remote sensing image sample based on the predicted bounding box and predicted class probability corresponding to the positive sample candidate target region, the true horizontal bounding box and true class label corresponding to the remote sensing image sample, and the predicted class probability and true class probability corresponding to the negative sample candidate target region includes: Based on the predicted bounding boxes corresponding to the positive sample candidate target regions and the true horizontal bounding boxes corresponding to the remote sensing image samples, the bounding box regression loss is determined. The first classification loss is determined based on the predicted class probability corresponding to the positive sample candidate target region and the true class label corresponding to the remote sensing image sample. The bounding box regression loss and the first classification loss are weighted and summed to obtain the target detection loss corresponding to the positive sample candidate target region; Based on the predicted class probability corresponding to the negative sample candidate target region and the real class label corresponding to the remote sensing image sample, a second classification loss is determined, and the second classification loss is used as the target detection loss corresponding to the negative sample candidate target region. Based on the target detection loss corresponding to the positive sample candidate target region and the target detection loss corresponding to the negative sample candidate target region, the target detection loss corresponding to the remote sensing image sample pair is determined.

[0102] For example, in an embodiment of this application, constructing the identity master supervision loss corresponding to the remote sensing image sample pair based on the normalized identity features corresponding to the positive sample candidate target region includes: Determine the first similarity between the normalized identity features and all class prototypes in the preset class feature lookup table; Determine the second similarity between the normalized identity features and all negative features in a preset negative sample circular queue; Based on the first similarity and the second similarity, the identity master supervision loss corresponding to the remote sensing image sample pair is determined.

[0103] For example, in an embodiment of this application, determining the first similarity between the normalized identity feature and all class prototypes in a preset class feature lookup table includes: Obtain a preset class feature lookup table, which stores multiple class prototypes, each class prototype corresponding to a ship identity category; Determine the cosine similarity or dot product similarity between the normalized identity features and each class prototype to obtain multiple similarity values; The vector or set formed by the multiple similarity values ​​is determined as the first similarity.

[0104] For example, in an embodiment of this application, determining the second similarity between the normalized identity feature and all negative features in a preset negative sample circular queue includes: Obtain a preset negative sample circular queue, which stores multiple negative features, each of which is an identity feature of a different identity category than the normalized identity feature; Determine the cosine similarity or dot product similarity between the normalized identity feature and each negative feature in the negative sample circular queue to obtain multiple negative similarity values; The vector or set formed by the plurality of negative similarity values ​​is determined as the second similarity.

[0105] For example, in an embodiment of this application, determining the identity master supervision loss corresponding to the remote sensing image sample pair based on the first similarity and the second similarity includes: The identity master supervision loss for the remote sensing image sample pair is determined based on the following formula; in, This represents the normalized identity feature. The true identity category of the positive sample candidate target region in the feature lookup table. The corresponding class prototype, This represents the first similarity between the normalized identity features and the corresponding class prototype. Indicates the first class in the feature lookup table Prototypical characteristics of class identity This indicates that the normalized identity features are related to the first... The first similarity between the prototype features of class identity, This indicates the number of negative samples in the circular queue. k One negative feature, The second similarity represents the normalized identity feature and the k-th negative feature. This represents the temperature scaling factor, used to control the smoothness of the similarity probability distribution.

[0106] For example, in an embodiment of this application, based on the normalized identity features corresponding to the positive sample candidate target region, a prototype quintuple contrast loss corresponding to the remote sensing image sample pair is constructed, including: Determine the first distance between the normalized identity feature and the same-domain positive prototype in the preset cross-domain prototype memory, the second distance between the normalized identity feature and the same-domain negative prototype in the cross-domain prototype memory, the third distance between the normalized identity feature and the cross-domain positive prototype in the cross-domain prototype memory, and the fourth distance between the normalized identity feature and the cross-domain negative prototype in the cross-domain prototype memory. Based on the first distance, the second distance, the third distance, and the fourth distance, the prototype quintuple contrast loss corresponding to the remote sensing image sample pair is determined.

[0107] For example, in an embodiment of this application, determining the prototype quintuple contrast loss corresponding to the remote sensing image sample pair based on the first distance, the second distance, the third distance, and the fourth distance includes: in, This represents the prototype quintuple contrast loss. Indicates the third distance, This represents the normalized identity feature. This represents the cross-domain prototype in the cross-domain prototype memory. This represents the fourth distance. This represents the cross-domain negative prototype in the cross-domain prototype memory. Indicates the first distance. This represents the co-domain prototype in the cross-domain prototype memory. Indicates the second distance. This represents the co-domain negative prototype in the cross-domain prototype memory. These are boundary parameters used to force the setting of the margin between the positive and negative prototypes. Distance between features is typically measured using Euclidean distance or cosine distance.

[0108] The cross-modal remote sensing ship search device 30 provided in this application embodiment can execute the technical solution of the cross-modal remote sensing ship search method in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the above-mentioned cross-modal remote sensing ship search method. Please refer to the implementation principle and beneficial effects of the above-mentioned cross-modal remote sensing ship search method. It will not be repeated here.

[0109] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application, such as... Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communications bus 440. The processor 410 can call logical instructions in the memory 430 to execute a cross-modal remote sensing ship search method. This method includes: acquiring a first modal remote sensing image of the target ship to be searched; inputting the first modal remote sensing image and multiple second modal remote sensing images from a search library into a pre-trained target cross-modal ship search model; outputting predicted bounding boxes and identity features corresponding to candidate target regions in each remote sensing image through the target cross-modal ship search model; wherein the remote sensing image is an image from either the first modal remote sensing image or the second modal remote sensing image; the target cross-modal ship search model is trained through joint foreground segmentation, target detection, and identity feature comparison; based on the similarity between the identity features corresponding to the target ship in the first modal remote sensing image and the identity features corresponding to each candidate target region in each of the second modal remote sensing images, determining candidate targets matching the target ship and their predicted bounding boxes, and obtaining search results.

[0110] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cross-modal remote sensing ship search method provided by the above methods. The method includes: acquiring a first modal remote sensing image of the target ship to be searched; inputting the first modal remote sensing image and multiple second modal remote sensing images from a search library into a pre-trained target cross-modal ship search model; outputting the predicted bounding boxes and identity features corresponding to candidate target regions in each remote sensing image through the target cross-modal ship search model; wherein, the remote sensing image is an image in the first modal remote sensing image or a second modal remote sensing image; the target cross-modal ship search model is obtained by training through joint foreground segmentation, target detection, and identity feature comparison; and determining candidate targets and their predicted bounding boxes that match the target ship based on the similarity between the identity features corresponding to the target ship in the first modal remote sensing image and the identity features corresponding to each candidate target region in each of the second modal remote sensing images, thereby obtaining search results.

[0112] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the cross-modal remote sensing ship search method provided by the above methods. The method includes: acquiring a first modal remote sensing image of a target ship to be searched; inputting the first modal remote sensing image and multiple second modal remote sensing images from a search library into a pre-trained target cross-modal ship search model; outputting predicted bounding boxes and identity features corresponding to candidate target regions in each remote sensing image through the target cross-modal ship search model; wherein, the remote sensing image is an image from the first modal remote sensing image or the second modal remote sensing image; the target cross-modal ship search model is obtained through joint foreground segmentation, target detection, and identity feature comparison training; determining candidate targets and their predicted bounding boxes matching the target ship based on the similarity between the identity features corresponding to the target ship in the first modal remote sensing image and the identity features corresponding to each candidate target region in each of the second modal remote sensing images, thereby obtaining search results.

[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A cross-modal remote sensing ship search method, characterized in that, include: Acquire the first modal remote sensing image of the target ship to be searched; The first modal remote sensing image and multiple second modal remote sensing images from the search library are input into a pre-trained target cross-modal ship search model. The target cross-modal ship search model outputs the predicted bounding boxes and identity features corresponding to candidate target regions in each remote sensing image. The remote sensing images are either the first modal remote sensing image or images from the second modal remote sensing images. The target cross-modal ship search model is trained by jointly performing foreground segmentation, target detection, and identity feature comparison. Based on the similarity between the identity features corresponding to the target ship in the first modality remote sensing image and the identity features corresponding to each candidate target region in each of the second modality remote sensing images, candidate targets matching the target ship and their predicted bounding boxes are determined, and search results are obtained.

2. The method according to claim 1, characterized in that, The step of inputting the first modality remote sensing image and multiple second modality remote sensing images from the search library into a pre-trained target cross-modality ship search model, and outputting the predicted bounding boxes and identity features corresponding to candidate target regions in each remote sensing image through the target cross-modality ship search model, includes: For each of the aforementioned remote sensing images, the remote sensing images are input into the target backbone network of the target cross-modal ship search model, and the basic feature map of the remote sensing images is extracted through the target backbone network; and candidate target regions of the remote sensing images are generated based on the basic feature map, and the region of interest features corresponding to the candidate target regions are extracted through the region of interest alignment operation; The region of interest features corresponding to the candidate target region are input into the target search head network in the target cross-modal ship search model, and the predicted bounding box and the identity features corresponding to the candidate target region are output through the target search head network.

3. The method according to claim 1 or 2, characterized in that, The target cross-modal ship search model is obtained through joint foreground segmentation, target detection, and identity feature comparison training, including: Multiple remote sensing image sample pairs are acquired, wherein each remote sensing image sample pair includes a first modal remote sensing image sample and a second modal remote sensing image sample. Each remote sensing image sample is labeled with the ship's true horizontal bounding box and the corresponding true category label, and has a corresponding labeled mask image. For each pair of remote sensing image samples, the pair is input into the initial backbone network of the cross-modal ship search model to be trained. The initial backbone network determines candidate target regions in each remote sensing image sample. Each candidate target region is mapped onto a corresponding base feature map. Through region of interest alignment, region of interest features of each candidate target region are extracted. The region of interest features of each candidate target region are input into the mask auxiliary head network and the search head network of the cross-modal ship search model, respectively. The mask auxiliary head network outputs the foreground mask image corresponding to each candidate target region. The search head network outputs the predicted bounding box, predicted class probability, and initial identity features corresponding to each candidate target region. Based on the true horizontal bounding boxes, true class labels, labeled mask images, predicted bounding boxes, predicted class probabilities, and initial identity features corresponding to each remote sensing image sample pair, the model parameters of the cross-modal ship search model are updated until the target cross-modal ship search model is trained.

4. The method according to claim 3, characterized in that, The acquisition of multiple remote sensing image sample pairs includes: Acquire a multimodal remote sensing ship dataset; wherein the multimodal remote sensing ship dataset includes multiple first modality remote sensing image samples and multiple second modality remote sensing image samples; Based on the category histograms corresponding to each remote sensing image sample, the category similarity between each first modality remote sensing image sample and each second modality remote sensing image sample is determined, and a cost matrix is ​​constructed based on the determined multiple category similarities. The Hungarian algorithm is used to solve the cost matrix globally for optimal matching, and the pairing scheme with the minimum total cost is found to obtain multiple pairs of remote sensing image samples.

5. The method according to claim 3, characterized in that, The step of updating the model parameters of the cross-modal ship search model based on the ground truth horizontal bounding box, ground truth class label, labeled mask image, predicted bounding box, predicted class probability, and initial identity features corresponding to each remote sensing image sample pair, until the target cross-modal ship search model is trained, includes: For each of the candidate target regions, based on the intersection-union ratio (IU) between the candidate target region and the true horizontal bounding box corresponding to the remote sensing image sample to which it belongs, the top K candidate target regions with the highest IU are determined from all candidate target regions, as well as positive sample candidate target regions with IU greater than a preset threshold and negative sample candidate target regions with IU less than the preset threshold. Based on the foreground mask images of the first K candidate target regions, construct the corresponding mask loss for the remote sensing image sample pairs; Based on the predicted bounding box and predicted class probability corresponding to the positive sample candidate target region, the true horizontal bounding box and true class label corresponding to the remote sensing image sample, and the predicted class probability and true class probability corresponding to the negative sample candidate target region, construct the target detection loss for the remote sensing image sample. The initial identity features corresponding to the positive sample candidate target regions are normalized to obtain normalized identity features. Based on the normalized identity features corresponding to the positive sample candidate target region, construct the identity master supervision loss and prototype quintuple contrast loss for the remote sensing image sample pair. The target loss corresponding to the remote sensing image sample pair is obtained by weighted summing of the mask loss, target detection loss, identity master supervision loss and prototype quintuple contrast loss. Based on the target loss corresponding to each remote sensing image sample pair, the model parameters of the cross-modal ship search model are updated until the target cross-modal ship search model is trained.

6. The method according to claim 5, characterized in that, The step of constructing the mask loss corresponding to the remote sensing image sample pair based on the foreground mask images of the first K candidate target regions includes: For each of the candidate target regions, the labeled mask image corresponding to the remote sensing image sample to which the candidate target region belongs is projected onto the local coordinate system of the feature space where the candidate target region is located to obtain the projected real mask corresponding to the candidate target region. Based on the foreground mask image of the candidate target region and the corresponding projected real mask, the mask loss of the candidate target region is determined; Based on the mask loss of each candidate target region, the mask loss corresponding to the remote sensing image sample pair is determined.

7. The method according to claim 5, characterized in that, The step of constructing the target detection loss for the remote sensing image sample based on the predicted bounding box and predicted class probability corresponding to the positive sample candidate target region, the true horizontal bounding box and true class label corresponding to the remote sensing image sample, and the predicted class probability and true class probability corresponding to the negative sample candidate target region includes: Based on the predicted bounding boxes corresponding to the positive sample candidate target regions and the true horizontal bounding boxes corresponding to the remote sensing image samples, the bounding box regression loss is determined. The first classification loss is determined based on the predicted class probability corresponding to the positive sample candidate target region and the true class label corresponding to the remote sensing image sample. The bounding box regression loss and the first classification loss are weighted and summed to obtain the target detection loss corresponding to the positive sample candidate target region; Based on the predicted class probability corresponding to the negative sample candidate target region and the real class label corresponding to the remote sensing image sample, a second classification loss is determined, and the second classification loss is used as the target detection loss corresponding to the negative sample candidate target region. Based on the target detection loss corresponding to the positive sample candidate target region and the target detection loss corresponding to the negative sample candidate target region, the target detection loss corresponding to the remote sensing image sample pair is determined.

8. The method according to claim 5, characterized in that, The step of constructing the identity master supervision loss for the remote sensing image sample pairs based on the normalized identity features corresponding to the positive sample candidate target regions includes: Determine the first similarity between the normalized identity features and all class prototypes in the preset class feature lookup table; Determine the second similarity between the normalized identity features and all negative features in a preset negative sample circular queue; Based on the first similarity and the second similarity, the identity master supervision loss corresponding to the remote sensing image sample pair is determined.

9. The method according to claim 8, characterized in that, Determining the first similarity between the normalized identity features and all class prototypes in a preset class feature lookup table includes: Obtain a preset class feature lookup table, which stores multiple class prototypes, each class prototype corresponding to a ship identity category; Determine the cosine similarity or dot product similarity between the normalized identity features and each class prototype to obtain multiple similarity values; The vector or set formed by the multiple similarity values ​​is determined as the first similarity.

10. The method according to claim 8, characterized in that, Determining the second similarity between the normalized identity features and all negative features in a preset negative sample circular queue includes: Obtain a preset negative sample circular queue, which stores multiple negative features, each of which is an identity feature of a different identity category than the normalized identity feature; Determine the cosine similarity or dot product similarity between the normalized identity feature and each negative feature in the negative sample circular queue to obtain multiple negative similarity values; The vector or set formed by the plurality of negative similarity values ​​is determined as the second similarity.

11. The method according to claim 10, characterized in that, The step of determining the identity-based master supervision loss for the remote sensing image sample pair based on the first similarity and the second similarity includes: The identity master supervision loss for the remote sensing image sample pair is determined based on the following formula; in, This represents the normalized identity feature. The true identity category of the positive sample candidate target region in the feature lookup table. The corresponding class prototype, This represents the first similarity between the normalized identity features and the corresponding class prototype. Indicates the first class in the feature lookup table Prototypical characteristics of class identity This indicates that the normalized identity features are related to the first... The first similarity between the prototype features of class identity, This indicates the number of negative samples in the circular queue. k One negative feature, The second similarity represents the normalized identity feature and the k-th negative feature. This represents the temperature scaling factor, used to control the smoothness of the similarity probability distribution.

12. The method according to claim 5, characterized in that, Based on the normalized identity features corresponding to the positive sample candidate target regions, a prototype quintuple contrast loss is constructed for the remote sensing image sample pairs, including: Determine the first distance between the normalized identity feature and the same-domain positive prototype in the preset cross-domain prototype memory, the second distance between the normalized identity feature and the same-domain negative prototype in the cross-domain prototype memory, the third distance between the normalized identity feature and the cross-domain positive prototype in the cross-domain prototype memory, and the fourth distance between the normalized identity feature and the cross-domain negative prototype in the cross-domain prototype memory. Based on the first distance, the second distance, the third distance, and the fourth distance, the prototype quintuple contrast loss corresponding to the remote sensing image sample pair is determined.

13. The method according to claim 12, characterized in that, The step of determining the prototype quintuple contrast loss corresponding to the remote sensing image sample pair based on the first distance, the second distance, the third distance, and the fourth distance includes: in, This represents the prototype quintuple contrast loss. Indicates the third distance, This represents the normalized identity feature. This represents the cross-domain prototype in the cross-domain prototype memory. This represents the fourth distance. This represents the cross-domain negative prototype in the cross-domain prototype memory. Indicates the first distance. This represents the co-domain prototype in the cross-domain prototype memory. Indicates the second distance. This represents the co-domain negative prototype in the cross-domain prototype memory. These are boundary parameters used to force the setting of the margin between the positive and negative prototypes. Distance between features is typically measured using Euclidean distance or cosine distance.

14. A cross-modal remote sensing ship search device, characterized in that, include: The acquisition unit is used to acquire the first modal remote sensing image of the target ship to be searched; The processing unit is configured to input the first modal remote sensing image and multiple second modal remote sensing images from the search library into a pre-trained target cross-modal ship search model, and output the predicted bounding boxes and identity features corresponding to candidate target regions in each remote sensing image through the target cross-modal ship search model; wherein, the remote sensing image is an image from the first modal remote sensing image or the second modal remote sensing image; the target cross-modal ship search model is obtained through joint foreground segmentation, target detection, and identity feature comparison training; The processing unit is further configured to determine candidate targets and their predicted bounding boxes that match the target ship based on the similarity between the identity features corresponding to the target ship in the first modality remote sensing image and the identity features corresponding to each candidate target region in each of the second modality remote sensing images, thereby obtaining search results.

Images