Ship identification method and device based on latent semantics, storage medium and product

Abstract

The application discloses a ship identification method and device based on implicit semantics, a storage medium and a product, and relates to the field of small sample target detection. Image feature vectors and text feature vectors are obtained by acquiring an image data set; corresponding implicit semantic features are obtained; image feature vectors after removing differences are obtained; implicit semantic features and image feature vectors with consistent dimensions are obtained; keys and values of the implicit semantic features and keys and values of the image feature vectors are obtained; fusion features are obtained; ship class prototype features are constructed; N RoI bounding box features are obtained; an adjacency matrix is obtained; updated ship class prototype features are obtained; a pre-training maritime small sample target identification model is optimized by using a contrast learning method according to the updated ship class prototype features, a ship identification model based on implicit semantics is obtained, and the application improves the identification accuracy of the identification model for ship targets.

Description

Technical Field

[0001] This invention relates to the field of small sample target detection, and in particular to a ship identification method, device, storage medium and product based on implicit semantics. Background Technology

[0002] In marine scenarios, intelligent ships need to perform tasks in dangerous sea conditions, requiring them to effectively identify targets even in harsh environments. However, due to complex environmental factors such as lighting, weather, and sea conditions, collecting rich training datasets of marine objects is a challenging task, making it difficult for recognition models to acquire sufficient training data. Furthermore, relying solely on visual information to extract ship target features may prevent recognition models from fully understanding new categories of ship targets. Especially in Few-Shot Object Detection (FSOD) settings, the visual information of new ship categories exhibits significant within-class variance, failing to accurately represent the ships themselves. Combining textual and image information can enhance ship target recognition performance. Textual information can provide prior knowledge related to the target, such as its size, shape, color, or speed, helping the recognition model to more accurately identify and locate targets.

[0003] In existing research on small-sample ship target recognition at sea, some works face similar problems. With the scarcity of maritime ship image data, the visual information that recognition models can acquire becomes limited. In this situation, methods relying solely on visual information processing may not provide sufficient accuracy. Currently, one approach to address this issue is to introduce other forms of information, such as semantic information extracted from text descriptions. By fusing semantic information, the deficiencies of visual information can be supplemented, resulting in a more comprehensive and accurate target understanding.

[0004] To supplement semantic information, textual descriptions are associated with images. Semantic information described using natural language can provide explanations of the category, attributes, and context of objects in the image. These textual descriptions can be converted into semantic embeddings by a text encoder and then interact with image features, enabling the recognition model to better understand image content and extract meaningful visual features. Specifically, semantic representations of different categories are obtained through a word embedding model. Simultaneously, to enhance the visual features extracted from new objects, a semantic enhancement module is designed to perform a nonlinear transformation that maps semantic representations in the semantic space to visual attention in the visual space.

[0005] Semantic information typically relies on specific datasets or knowledge bases. The applicability of this semantic information may be limited when using different datasets, and generalization to new scenarios may encounter difficulties. Furthermore, effectively fusing and aligning the visual and semantic information of ships is also a challenge, requiring the design of appropriate models and algorithms. This is because the representational differences and semantic correspondences between different information sources need to be processed and modeled to ensure effective fusion. Summary of the Invention

[0006] The purpose of this invention is to provide a ship identification method, device, storage medium, and product based on implicit semantics, which improves the accuracy of the identification model in identifying ship targets.

[0007] To achieve the above objectives, the present invention provides the following solution:

[0008] A ship identification method based on implicit semantics, the ship identification method based on implicit semantics includes:

[0009] Obtain an image dataset; the image dataset includes: image data and text data; the image data and text data are in one-to-one correspondence, and the text data is used to describe the image data; use an encoder to encode the image dataset to obtain image feature vectors and text feature vectors; the encoder includes: CLIP text encoder and convolutional neural network;

[0010] The CLIP text encoder is used to extract text feature vectors corresponding to ship target label information in the image; then, an arbitrary ship category is constructed based on the ship target label information; and then the ship category is fed into the CLIP text encoder to obtain the corresponding implicit semantic features.

[0011] The difference between the image feature vector and the first image feature distribution extracted by the pre-trained marine small sample target recognition model is obtained by KL divergence comparison, and the difference is minimized to obtain the image feature vector after removing the difference;

[0012] Based on the implicit semantic features and the image feature vector after removing differences, the implicit semantic features are transformed using a multilayer perceptron to obtain implicit semantic features and image feature vectors with consistent dimensions.

[0013] Based on the dimensionally consistent latent semantic features and image feature vectors, the keys and values ​​of the latent semantic features, as well as the keys and values ​​of the image feature vectors, are obtained.

[0014] By using cross-attention, the keys and values ​​of the implicit semantic features and the keys and values ​​of the image feature vector are interactively fused to obtain fused features;

[0015] Based on the fusion features, construct prototype features for ships;

[0016] The ship-type prototype features are filtered to obtain N RoI bounding box features;

[0017] The first similarity between the ship-type prototype features and the N RoI bounding box features is obtained using cosine similarity, and the second similarity between the N RoI bounding box features is obtained; based on the first similarity and the second similarity, the adjacency matrix is ​​obtained;

[0018] Based on the adjacency matrix, the prototype features of the ship class are dynamically updated to obtain the updated prototype features of the ship class.

[0019] Based on the updated ship class prototype features, a target recognition model is created by optimizing the pre-trained small marine samples using a contrastive learning method, resulting in a ship recognition model based on implicit semantics; the ship recognition model based on implicit semantics is used to identify ships.

[0020] Optionally, based on the dimensionally consistent latent semantic features and image feature vectors, the keys and values ​​of the latent semantic features, as well as the keys and values ​​of the image feature vectors, are obtained, specifically including:

[0021] The key and value of the latent semantic features are encoded into a first mapping pair using a second encoder; the key and value of the image feature vector are encoded into a second mapping pair using a second encoder.

[0022] The second encoder takes one or more image feature vectors, or one or more image feature vectors as input, and then outputs two feature maps for each input feature through two parallel 3×3 convolutional layers: key and value;

[0023] Key mapping is used to measure the similarity between the latent semantic features of image feature vectors and text feature vectors; key mapping is used to learn the degree of matching information between the latent semantic features of image feature vectors and text feature vectors.

[0024] Value mappings are used to store detailed information about identifying implicit semantic features and image feature vectors.

[0025] Optionally, the soft weights of the text feature vectors are obtained by calculating the similarity between the keys of the latent semantic features and the keys of the image feature vectors;

[0026] The soft weights are normalized using the Softmax function to obtain the final weight values;

[0027] The final weights are weighted and summed to obtain image feature values ​​containing textual semantic information.

[0028] The final weight is added to the image feature value with a preset weight to obtain the final visual feature containing rich semantic information.

[0029] Optionally, ship-type prototype features are constructed based on the fused features, specifically including:

[0030] We obtain initial prototypes for K ship categories, extract raw image features from a given support set S using a backbone network, and then obtain the target instance embedding representation f using region-of-interest pooling and ground truth bounding boxes. i The ship class prototype features for any ship category are obtained by using the mean method; the specific calculation process is as follows:

[0031]

[0032] Among them, y i S is the label for the target true bounding box. K This indicates that the Kth class in the set is supported.

[0033] Optionally, the ship-type prototype features are filtered to obtain N RoI bounding box features, specifically including:

[0034] Based on the predicted score of the prototype features of the ship category for the current category, the IoU score u between the predicted bounding box and the corresponding ground truth bounding box for the current category is used. i Feature filtering is performed on N RoI bounding box features, using features greater than a threshold. The candidate bounding boxes are used to update the ship type prototype features, resulting in B RoI bounding box features; the selection formula is:

[0035]

[0036] B is less than or equal to N, and both B and N are positive integers.

[0037] Optionally, cosine similarity is used to obtain a first similarity between the ship-type prototype features and the N RoI bounding box features, and a second similarity between the N RoI bounding box features; based on the first similarity and the second similarity, an adjacency matrix is ​​obtained, specifically including:

[0038] C subgraphs are constructed using ship class prototype features and B filtered RoI bounding box features; the node set V of each subgraph consists of a ship class prototype feature and its corresponding RoI bounding box feature.

[0039] Normalize the adjacency matrix A to obtain the normalized adjacency matrix A'; the calculation process is as follows:

[0040]

[0041]

[0042] Where E represents the edge set, C represents the number of categories, D is the degree matrix, and x i Representing RoI bounding box features, It is represented as A.

[0043] Optionally, the ship class prototype features are dynamically updated based on the adjacency matrix to obtain updated ship class prototype features, specifically including:

[0044] Based on the adjacency matrix A, the feature similarity is obtained. Then, GCN is used to aggregate the feature information of the candidate boxes into the corresponding ship class prototype features, corresponding to each ship class prototype feature c. K The effect of each layer of GCN is equal to the weighted sum of the RoI bounding box features of its corresponding category. The calculation process for dynamically updating the prototype features of each ship class is as follows:

[0045]

[0046] Among them, B k This indicates the number of category K features among the B RoI bounding box features;

[0047] By improving the contrast loss between ship instance features and ship prototype features through feature selection and positive / negative sample decoupling, the ship recognition model based on latent semantics can match similar image feature vectors and text feature vectors together, while separating dissimilar image feature vectors and text feature vectors. The calculation process of the contrast loss for ship instance features is as follows:

[0048]

[0049] The contrastive loss of the prototype features of ships is expressed as:

[0050]

[0051] Where N is the total number of RoI bounding box features in the mini-batch. This indicates a feature normalization operation. This represents the inner product between the features of the i-th and j-th RoI bounding boxes in the projected hypersphere. Let τ represent the inner product between the i-th RoI bounding box feature and the ship class prototype feature of its corresponding category, and let τ represent the temperature hyperparameter.

[0052] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement any of the ship identification methods with implicit semantics.

[0053] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements a ship identification method with implicit semantics as described in any one of the claims.

[0054] A computer program product includes a computer program that, when executed by a processor, implements a ship identification method with any of the implicit semantics described above.

[0055] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0056] To effectively integrate text data into image data, the Contrastive Language-Image Pre-training (CLIP) model is introduced into the identification of small-sample ship targets at sea. CLIP is used to obtain the implicit semantic features of ship targets to assist the image feature vector in identifying small-sample ship targets, supplementing some implicit semantic features that are difficult to capture in the image feature vector, and improving the ship target recognition ability of the implicit semantic-based ship recognition model.

[0057] The CLIP text encoder is used to obtain latent semantic features for each ship category, which can represent the semantic knowledge of each ship category. Simultaneously, to better integrate these latent semantic features into the image feature vector, the KL (Kullback-Leibler) divergence method is used to optimize the distribution difference between the image features extracted by CLIP and those extracted by the few-sample model. Then, a ship latent semantic feature fusion module based on cross-attention is used to integrate the latent semantic features into the image feature embedding, enriching the latent semantic features of the ship image feature embedding. This enables the model to learn the semantic knowledge of ships, enhances the model's recognition performance for few-sample ship targets at sea, and further optimizes the perception capabilities of intelligent ships. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart of the ship identification method based on implicit semantics according to the present invention;

[0060] Figure 2 This is a recognition result on a small sample target dataset at sea.

[0061] Figure 3This is another recognition result on a small sample target dataset at sea;

[0062] Figure 4 This is a diagram of the internal structure of a computer device. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] The purpose of this invention is to provide a ship recognition method, device, storage medium, and product based on implicit semantics, which overcomes the problem that the visual features extracted from images during the training process of small sample ship targets at sea are incomplete. By introducing CLIP into the recognition of small sample ship targets at sea, the expressive power of visual features is enhanced, and the implicit semantic features of ship visual images are enriched, so as to further optimize the ship recognition model based on implicit semantics for recognizing small sample ship targets at sea.

[0065] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] like Figure 1 As shown, this invention provides a ship identification method based on implicit semantics, the specific process of which is as follows:

[0067] Step S1: Obtain an image dataset; the image dataset includes: image data and text data; the image data and text data correspond one-to-one, and the text data is used to describe the image data; the image dataset is encoded using an encoder to obtain image feature vectors and text feature vectors; the encoder includes: CLIP text encoder and convolutional neural network;

[0068] In one example, the input image dataset is encoded using a CLIP text encoder (Transformer network) to encode the text data, and a general convolutional neural network (CNN) to encode the image data.

[0069] Step S2: Use the CLIP text encoder to extract text feature vectors corresponding to the ship target label information in the image; then construct an arbitrary ship category based on the ship target label information; then feed the ship category into the CLIP text encoder to obtain the corresponding implicit semantic features;

[0070] In one example, as a multimodal trained model, the CLIP text encoder can be used to find the text data that best represents the image data, or to find the most suitable image data for a given text data.

[0071] Step S3: Use KL divergence comparison to obtain the difference between the image feature vector and the first image feature distribution extracted by the pre-trained marine small sample target recognition model, and minimize the difference to obtain the image feature vector after removing the difference;

[0072] In one example, because the CLIP text encoder aligns image and text data during training, and the region features of the image and the text features that need to interact are extracted in their respective feature spaces, feature interaction alone in multimodal encoding would be difficult to model. KL divergence is a method for measuring the difference between two probability distributions. It can be used to compare the similarity or difference between two probability distributions, and also to evaluate how closely one probability distribution approximates another.

[0073] Step S4: Based on the implicit semantic features and the image feature vector after removing differences, perform dimensional transformation on the implicit semantic features using a multilayer perceptron to obtain implicit semantic features and image feature vectors with consistent dimensions;

[0074] In one example, for the input implicit semantic features and the image feature vector after removing the differences, since the dimensions of the features extracted by different models may be inconsistent, a cross-attention method is subsequently used to prepare for the fusion of the implicit semantic features and the image feature vector after removing the differences, so that the implicit semantic knowledge contained in the text and the image information can fully interact.

[0075] Step S5: Based on the consistent latent semantic features and image feature vectors, obtain the keys and values ​​of the latent semantic features, and the keys and values ​​of the image feature vectors; specifically including:

[0076] Step S51: Use the second encoder to encode the keys and values ​​of the latent semantic features into a first mapping pair; use the second encoder to encode the keys and values ​​of the image feature vector into a second mapping pair;

[0077] Step S52: The second encoder takes one or more image feature vectors, or one or more image feature vectors as input, and then outputs two feature maps for each input feature through two parallel 3×3 convolutional layers: key and value;

[0078] Step S53: Key mapping is used to measure the similarity between the latent semantic features of image feature vectors and text feature vectors; key mapping is used to learn the degree of matching information between the latent semantic features of image feature vectors and text feature vectors.

[0079] Step S54: The value map is used to store detailed information about identifying implicit semantic features and image feature vectors.

[0080] In one example, a second encoder is used to encode key-value pairs. The second encoder takes one or more features as input and then outputs two feature maps—key and value—for each input feature through two parallel 3×3 convolutional layers to reduce the dimensionality of the input features and save computational cost. The key map pairs are used to measure the similarity between image and text features, helping to determine where information is more important. Therefore, the key map can learn the degree of matching information related to the image and text, while the value map stores detailed information about identifying implicit semantic features and image feature vectors.

[0081] Step S6: The keys and values ​​of the implicit semantic features and the keys and values ​​of the image feature vector are interactively fused through cross-attention to obtain fused features;

[0082] Step S61: Obtain the soft weights of the text feature vectors by calculating the similarity between the keys of the latent semantic features and the keys of the image feature vectors;

[0083] Step S62: Normalize the soft weights using the Softmax function to obtain the final weight values;

[0084] Step S63: The final weights are weighted and summed to obtain image feature values ​​containing textual semantic information;

[0085] Step S64: Add the image feature value and the image feature value map with a preset weight to obtain the final visual feature containing rich semantic information. The preset weight value can be 1, which will not be elaborated here.

[0086] Step S7: Construct ship-type prototype features based on the fused features; specifically including:

[0087] We obtain initial prototypes for K ship categories, extract raw image features from a given support set S using a backbone network, and then obtain the target instance embedding representation f using region-of-interest pooling and ground truth bounding boxes. i The ship class prototype features of the Kth ship category are obtained by using the mean method; the specific calculation process is as follows:

[0088]

[0089] Among them, y i S is the label for the target true bounding box. K This indicates the Kth class in the support set, where the support set S is a pre-defined training set.

[0090] Step S8: Filter the ship-type prototype features to obtain N RoI bounding box features; specifically including:

[0091] Based on the predicted score of the prototype features of the ship category for the current category, the IoU score u between the predicted bounding box and the corresponding ground truth bounding box for the current category is used. i Feature filtering is performed on N RoI bounding box features, using features greater than a threshold. The candidate bounding boxes are used to update the prototype features of the ship class, resulting in B RoI bounding box features; the prediction score is obtained by training the backbone network, and the model will give the prediction score after training.

[0092] The screening formula is:

[0093]

[0094] B is less than or equal to N, and both B and N are positive integers.

[0095] Step S9: Obtain the first similarity between the ship-type prototype features and the N RoI bounding box features using cosine similarity, and the second similarity between the N RoI bounding box features; obtain the adjacency matrix based on the first similarity and the second similarity; specifically including:

[0096] Step S91: Construct C subgraphs using the ship class prototype feature and the filtered B RoI bounding box features; the node set V of each subgraph consists of a ship class prototype feature and its corresponding category RoI bounding box feature;

[0097] Step S92: Normalize the adjacency matrix A to obtain the normalized adjacency matrix A'; the calculation process is as follows:

[0098]

[0099]

[0100] Where C represents the number of categories, D is the degree matrix, and x i Representing RoI bounding box features, It is represented as A.

[0101] Step S10: Dynamically update the ship class prototype features based on the adjacency matrix to obtain the updated ship class prototype features; specifically including:

[0102] Based on the adjacency matrix A, the feature similarity is obtained. Then, GCN is used to aggregate the feature information of the candidate boxes into the corresponding ship class prototype features, corresponding to each ship class prototype feature c. K The effect of each layer of GCN is equal to the weighted sum of the RoI bounding box features of its corresponding category. The calculation process for dynamically updating the prototype features of each ship class is as follows:

[0103]

[0104] Among them, B k This indicates the number of category K features among the B RoI bounding box features;

[0105] Step S11: Based on the updated ship class prototype features, use contrastive learning to optimize the pre-trained small marine sample target recognition model to obtain a ship recognition model based on implicit semantics; the ship recognition model based on implicit semantics is used to identify ships.

[0106] By improving the contrast loss of ship instance features and ship prototype features through feature filtering and positive-negative sample decoupling, the ship recognition model based on latent semantics can match similar image feature vectors and text feature vectors together, and separate dissimilar image feature vectors and text feature vectors.

[0107] The process for calculating the comparative loss of ship-type instance features is as follows:

[0108]

[0109] The contrastive loss of the prototype features of ships is expressed as:

[0110]

[0111] Where N is the total number of RoI bounding box features in the mini-batch. This indicates a feature normalization operation. This represents the inner product between the features of the i-th and j-th RoI bounding boxes in the projected hypersphere. This represents the inner product between the i-th RoI bounding box feature and its corresponding ship class prototype feature, and τ represents the temperature hyperparameter, which can be 256, etc. for mini-batch, but will not be elaborated here.

[0112] Example 1

[0113] Experiments were conducted on a small-sample marine target dataset. The experimental dataset contained 10278 images, with 7853 images in the training set and 2425 images in the test set. During the base class training phase, four vessel types with abundant samples—container ships, bulk carriers, cruise ships, and sailboats—were selected as the base classes for model training. Furthermore, no base class and new class vessels appeared in the same sample in the training data. In the fine-tuning phase, three new classes with fewer samples—fishing vessels, small boats, and aircraft carriers—were selected. A balanced base class and new class dataset was used, comprising N classes and K labeled samples, where N = 7 and K = 1 / 2 / 3 / 5 / 10.

[0114] The experimental parameters for base class training and new class fine-tuning are listed in Tables 1 and 2. `learningrate` represents the learning rate, initially set to 0.0025 for base class training, then changing to 0.00025 and 0.000025 at epochs 96000 and 106400, respectively. For new class training, it initially sets to 0.01, changing to 0.001 at epoch 1300 for 1-shot training, epoch 1700 for 2-shot training, epoch 2100 for 3-shot training, epoch 2500 for 5-shot training, and epoch 3700 for 10-shot training. `total epoch` represents the total number of training epochs; `weightdecay` represents the weight decay value; `momentum` represents the momentum value; `SGD` represents the optimizer; and `batch size` represents the number of samples selected in one training iteration. The model was trained on a single RTX 3090 GPU.

[0115] Table 1

[0116] Learning rate 0.005 Total number of training rounds 120000 Number of samples selected in one training session 4 Weight decay value <![CDATA[5.0e -5 ]]> Momentum value 0.09

[0117] Table 2

[0118]

[0119] Figure 2 and Figure 3 The results of the ship identification method based on implicit semantics are shown. From Figure 2 It can be seen that the ship identification model based on implicit semantics can accurately identify new types of maritime targets, such as aircraft carriers, even when they appear in complex backgrounds. Figure 3 It can be seen that even with incomplete new and base class ship targets, the ship recognition model based on implicit semantics can still effectively identify them. Therefore, the proposed optimization method for small-sample ship target recognition at sea based on implicit semantic knowledge fusion can further effectively optimize the perception capabilities of intelligent ships in small-sample scenarios.

[0120] This invention addresses the problem that visual features extracted solely from images during training for small-sample maritime vessel target recognition are incomplete. It proposes a ship recognition method based on latent semantics to enrich the semantic information of ship visual images, thereby further optimizing the model's ability to recognize small-sample maritime vessel targets. By introducing CLIP (Text-Induced Profiling) into the recognition of small-sample maritime vessel targets, the expressive power of visual features is enhanced. On one hand, the CLIP text encoder acquires latent semantic knowledge about the vessel, and KL divergence is used to optimize the distribution of image features. On the other hand, a cross-attention-based latent semantic knowledge fusion module embeds latent semantic features into the image features, enriching the semantic expressive power of visual features. The method proposed in this patent can effectively optimize the recognition performance for small-sample maritime vessel targets.

[0121] In one embodiment, a computer device is provided, which may be a database, and its internal structure diagram may be as follows: Figure 4 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores pending transactions. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a ship identification method based on implicit semantics.

[0122] In one embodiment, a computer device is also provided, including a memory and a processor for a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the steps in the above method embodiments.

[0123] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0124] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0125] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0126] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A ship identification method based on implicit semantics, characterized in that, The ship identification method based on implicit semantics includes: Obtain an image dataset; the image dataset includes: image data and text data; the image data and text data are in one-to-one correspondence, and the text data is used to describe the image data; use an encoder to encode the image dataset to obtain image feature vectors and text feature vectors; the encoder includes: CLIP text encoder and convolutional neural network; The CLIP text encoder is used to extract text feature vectors corresponding to ship target label information in the image; then, an arbitrary ship category is constructed based on the ship target label information; and then the ship category is fed into the CLIP text encoder to obtain the corresponding implicit semantic features. The difference between the image feature vector and the first image feature distribution extracted by the pre-trained marine small sample target recognition model is obtained by KL divergence comparison, and the difference is minimized to obtain the image feature vector after removing the difference; Based on the implicit semantic features and the image feature vector after removing differences, the implicit semantic features are transformed using a multilayer perceptron to obtain implicit semantic features and image feature vectors with consistent dimensions. Based on the dimensionally consistent latent semantic features and image feature vectors, the keys and values ​​of the latent semantic features, as well as the keys and values ​​of the image feature vectors, are obtained. By using cross-attention, the keys and values ​​of the implicit semantic features and the keys and values ​​of the image feature vector are interactively fused to obtain fused features; Based on the fusion features, construct prototype features for ships; The ship-type prototype features are filtered to obtain N RoI bounding box features; The first similarity between the ship-type prototype features and the N RoI bounding box features is obtained using cosine similarity, and the second similarity between the N RoI bounding box features is obtained; based on the first similarity and the second similarity, the adjacency matrix is ​​obtained; Based on the adjacency matrix, the prototype features of the ship class are dynamically updated to obtain the updated prototype features of the ship class. Based on the updated ship class prototype features, a target recognition model is created by optimizing the pre-trained small marine samples using a contrastive learning method, resulting in a ship recognition model based on implicit semantics; the ship recognition model based on implicit semantics is used to identify ships.

2. The ship identification method based on implicit semantics according to claim 1, characterized in that, Based on the consistent latent semantic features and image feature vectors, the keys and values ​​of the latent semantic features, as well as the keys and values ​​of the image feature vectors, are obtained, specifically including: The key and value of the latent semantic features are encoded into a first mapping pair using a second encoder; the key and value of the image feature vector are encoded into a second mapping pair using a second encoder. The second encoder takes one or more image feature vectors, or one or more image feature vectors as input, and then outputs two feature maps for each input feature through two parallel 3×3 convolutional layers: key and value; Key mapping is used to measure the similarity between the latent semantic features of image feature vectors and text feature vectors; key mapping is used to learn the degree of matching information between the latent semantic features of image feature vectors and text feature vectors. Value mappings are used to store detailed information about identifying implicit semantic features and image feature vectors.

3. The ship identification method based on implicit semantics according to claim 2, characterized in that, The soft weights of the text feature vectors are obtained by calculating the similarity between the keys of the latent semantic features and the keys of the image feature vectors. The soft weights are normalized using the Softmax function to obtain the final weight values; The final weights are weighted and summed to obtain image feature values ​​containing textual semantic information. The image feature values ​​are added to the image feature map with preset weights to obtain the final visual features containing rich semantic information.

4. The ship identification method based on implicit semantics according to claim 1, characterized in that, Based on the fusion features, prototype features for ships are constructed, specifically including: We obtain initial prototypes for K ship categories, extract raw image features from a given support set S using a backbone network, and then obtain the target instance embedding representation f using region-of-interest pooling and ground truth bounding boxes. i The ship class prototype features of the Kth ship category are obtained by using the mean method; the specific calculation process is as follows: Where yi is the label of the target ground truth bounding box, and S K This indicates that the Kth class in the set is supported.

5. The ship identification method based on implicit semantics according to claim 4, characterized in that, The ship-type prototype features are filtered to obtain N RoI bounding box features, specifically including: Based on the predicted score of the prototype features of the ship category for the current category, the IoU score u between the predicted bounding box and the corresponding ground truth bounding box for the current category is used. i Feature filtering is performed on N RoI bounding box features, using features greater than a threshold. The candidate bounding boxes are used to update the ship type prototype features, resulting in B RoI bounding box features; the selection formula is: B is less than or equal to N, and both B and N are positive integers.

6. The ship identification method based on implicit semantics according to claim 5, characterized in that, Cosine similarity is used to obtain the first similarity between the prototype features of the ship class and the features of N RoI bounding boxes, and the second similarity between the features of N RoI bounding boxes; Based on the first similarity and the second similarity, the adjacency matrix is ​​obtained, specifically including: C subgraphs are constructed using ship class prototype features and B filtered RoI bounding box features; the node set V of each subgraph consists of a ship class prototype feature and its corresponding RoI bounding box feature. Normalize the adjacency matrix A to obtain the normalized adjacency matrix A'; the calculation process is as follows: Where E represents the edge set, C represents the number of categories, D is the degree matrix, and x i Representing RoI bounding box features, It is represented as A.

7. The ship identification method based on implicit semantics according to claim 6, characterized in that, Based on the adjacency matrix, the ship class prototype features are dynamically updated to obtain the updated ship class prototype features, specifically including: Based on the adjacency matrix A, the feature similarity is obtained. Then, GCN is used to aggregate the feature information of the candidate boxes into the corresponding ship class prototype features, corresponding to each ship class prototype feature c. K The effect of each layer of GCN is equal to the weighted sum of the RoI bounding box features of its corresponding category. The calculation process for dynamically updating the prototype features of each ship class is as follows: Among them, B k This indicates the number of category K features among the B RoI bounding box features; By improving the contrast loss between ship instance features and ship prototype features through feature selection and positive / negative sample decoupling, the ship recognition model based on latent semantics can match similar image feature vectors and text feature vectors together, while separating dissimilar image feature vectors and text feature vectors. The calculation process of the contrast loss for ship instance features is as follows: The contrastive loss of the prototype features of ships is expressed as: Where N is the total number of RoI bounding box features in the mini-batch. This indicates a feature normalization operation. This represents the inner product between the features of the i-th and j-th RoI bounding boxes in the projected hypersphere. Let τ represent the inner product between the i-th RoI bounding box feature and the ship class prototype feature of its corresponding category, and let τ represent the temperature hyperparameter.

8. A computer device comprising: The memory and processor contain a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the ship identification method with implicit semantics as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the ship identification method with implicit semantics as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the ship identification method with implicit semantics as described in any one of claims 1-7.

Images