Ship re-identification method, system and device based on global-local feature interactive learning

Abstract

The invention discloses a ship re-identification method, system and device based on global-local feature interactive learning. According to the method, a double-branch architecture is adopted, and global features and local features are extracted respectively. The global branch uses a pre-trained large model to extract global features with wide applicability and robustness. In order to utilize local detail features of a ship, a local branch is introduced into a local feature sensing module and focuses on capturing fine-grained local information in an image. On the basis, a feature interaction module is introduced, and unimportant image features are filtered through attention score evaluation. In order to promote collaborative learning and effective fusion of global and local features, a collaborative interaction learning module is introduced. The model adopts a double-branch joint training and single-branch reasoning strategy, so that the calculation complexity is reduced while the model performance is ensured, and the requirements of practical application are met.

Description

Technical Field

[0001] The present invention belongs to the field of image retrieval, and in particular relates to a ship re-identification method, system and device based on global-local feature interactive learning. Background Art

[0002] Re-identification (ReID) is a core task in computer vision, aiming to accurately identify the same entity (such as a person, vehicle, or object) across different time and camera viewpoints. In recent years, vessel Re-ID has become a research hotspot. In maritime surveillance systems, vessel Re-ID aims to accurately identify the presence of the same vessel at different times and locations by analyzing its appearance. This technology has significant research significance and broad application value. For example, in maritime security monitoring, vessel Re-ID can help surveillance systems track vessel activity and promptly detect abnormal behavior such as illegal intrusion, illegal fishing, or malicious navigation, thereby strengthening maritime safety supervision. Currently, the maritime industry primarily relies on Automatic Identification Systems (AIS) for vessel tracking and identification. However, the reliability of AIS data depends on the vessel's autonomous reporting. Vessels may evade monitoring by disabling AIS, thereby increasing the secrecy and uncertainty of maritime activities. Furthermore, the information provided by AIS is relatively limited, focusing primarily on location data. This information cannot meet the requirements for detailed monitoring of a vessel's specific characteristics and behavior, limiting its effective application in complex scenarios. To overcome the limitations of the AIS system, vision-based vessel Re-ID methods have emerged. Compared to AIS, it can perform real-time analysis independent of active vessel reporting and is unaffected by vessels shutting down their AIS, significantly improving monitoring reliability and continuity. It can also extract a vessel's appearance characteristics and other detailed visual information, providing the maritime industry with a more comprehensive and in-depth monitoring tool, addressing the shortcomings of AIS in monitoring vessel characteristics and behavior. However, vision-based ship re-identification is still in its developmental stages and faces numerous challenges.

[0003] 1) Incomplete ship images. Due to the large size of ships, in real-world surveillance scenarios, cameras often only capture a partial area of ​​the ship, resulting in an incomplete image of the ship's structure, consisting of only the bow, hull, or stern. This incompleteness makes it difficult for the model to fully capture the ship's overall features, thus affecting the accuracy of re-ID.

[0004] 2) Variations in vessel size. A vessel may exhibit different apparent dimensions under different loading conditions, resulting in changes in the shape of its visible area and unstable visual characteristics. For example, a fully loaded vessel has a lower waterline, while an empty vessel reveals more of the hull. Furthermore, different types of cargo, loading quantities, and distribution patterns can significantly alter the vessel's profile.

[0005] These characteristics make it difficult for existing re-ID methods to achieve satisfactory results when dealing with ships. Therefore, developing more robust visual feature extraction methods and introducing optimization strategies specifically tailored to ship characteristics have become key directions for promoting further development in this field. Summary of the Invention

[0006] The purpose of the present invention is to address the deficiencies of the existing technology and propose a ship re-identification method, system and device based on global-local feature interactive learning, which are used to effectively identify ships in open scenarios.

[0007] The object of the present invention is achieved through the following technical solution: The present invention proposes a ship re-identification method based on global-local feature interactive learning, which includes the following steps:

[0008] (1) Obtain a ship image and divide it into a sequence of image blocks;

[0009] (2) extracting the global features of the ship from the image block sequence obtained in step (1) based on the global attention mechanism;

[0010] (3) Grouping the image block sequences obtained in step (1) and adding a learnable local token to each group. Each local token only focuses on the image block sequence of interest within the group.

[0011] (4) Using the global attention score in step (2) as a guide, enhance the feature representation of the relevant image block sequence in step (3) to extract more accurate local features;

[0012] (5) The global features obtained in step (2) and the local features obtained in step (4) are collaboratively interactively learned to obtain more discriminative global features;

[0013] (6) Calculate the similarity between the global features of the query image obtained in step (5) and the features of the library image, sort them according to the similarity, and obtain the final re-identification result.

[0014] Furthermore, in step (2), the visual encoder of CLIP is used to encode the visual features of the image block sequence, and the information of the image block sequence is aggregated into class tokens using the global attention mechanism, and the obtained class tokens are used as global features.

[0015] Furthermore, in step (3), a mask mechanism is used to group the image block sequence, and each local token has a corresponding mask. The values ​​1 in the mask represent image blocks that are visible to the local token.

[0016] Furthermore, in step (4), the global class token's attention score for the image block sequence is used as an indicator to judge the importance of the image block. The image blocks are sorted according to the attention score, and some image blocks with higher importance are retained as needed. The mask mechanism is also used to control the visibility of the image blocks to the local token.

[0017] Furthermore, in step (5), collaborative interactive learning is specifically as follows: constructing a local feature positive and negative sample package, where the local feature with the same identity as the global feature is the positive sample, and the local feature with a different identity from the global feature is the negative sample, and finding the corresponding local feature positive and negative sample closest to the global feature from the positive and negative sample package, and using the triplet loss to shorten the distance between the global feature and the local feature positive sample, and to increase the distance between the global feature and the local feature negative sample.

[0018] Furthermore, during the training process, global and local features are optimized separately. In order to ensure that the global features have the ability to distinguish different ships, the cross-entropy loss is used to constrain the global features to learn the local feature extraction process, and the subtle differences between different ships are learned based on the triplet loss; the local features are also supervised using cross-entropy and triplet losses.

[0019] In a second aspect, the present invention further provides a ship re-identification system based on global-local feature interactive learning, the system comprising:

[0020] Image block sequence extraction module: used to obtain ship images and divide them into image block sequences;

[0021] Global feature extraction module: used to extract global features of ships from image block sequences based on the global attention mechanism;

[0022] Local feature perception module: used to group image block sequences, add learnable local tokens to each group, and extract local features of the ship using the local perception attention mechanism;

[0023] Feature Interaction Module: Used to enhance the feature representation of related image block sequences using the global attention score as a guide to extract more accurate local features;

[0024] Collaborative interactive learning module: used to collaboratively interactively learn global features and local features to obtain more discriminative global features.

[0025] Ship re-identification module: used to calculate the similarity between the query image and the library image features, sort them according to the similarity, and obtain the final re-identification result.

[0026] In a third aspect, the present invention also provides a ship re-identification device based on global-local feature interactive learning, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the ship re-identification method based on global-local feature interactive learning.

[0027] In a fourth aspect, the present invention further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the ship re-identification method based on global-local feature interactive learning.

[0028] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the ship re-identification method based on global-local feature interactive learning.

[0029] Beneficial effects of the present invention:

[0030] (1) Improve recognition accuracy: Through the collaborative learning of global and local features, local information can effectively supplement global information, which enhances the model's ability to capture local information. This method significantly improves the accuracy of ship re-identification when processing ship images with incompleteness and size changes.

[0031] (2) Reduce computational complexity: This method adopts a global and local feature joint training and global feature inference strategy. In the inference stage, there is no need for additional local feature extraction steps, which reduces computational overhead while ensuring performance.

[0032] (3) Strong model transferability: This method has strong transferability and can be applied to re-identification tasks in different fields. Through further optimization through fine-tuning, the model can flexibly adapt to the re-identification of pedestrians and vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 This is a flow chart of the ship re-identification method based on local feature interactive learning of the present invention.

[0035] Figure 2 Schematic diagram of the structure of the ship re-identification method based on local feature interactive learning of the present invention.

[0036] Figure 3 This is a structural diagram of the ship re-identification device based on local feature interactive learning provided by the present invention. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0039] This paper provides a ship re-identification method based on interactive learning of global and local features. This method uses a multi-branch, multi-feature representation to extract finer-grained features and uses local features to guide global feature learning to address incompleteness and size variation. During the training phase, joint training is performed using global and local branches. A CNN is used to partition the image into blocks, generating a sequence of blocks. In the global branch, the pre-trained CLIP model, leveraging its powerful generalization capabilities, is used to extract global features using its visual encoder. In the local branch, the block sequence is further divided into groups, each assigned a learnable local token that focuses solely on the block sequence within the group. Using the attention scores of the global class tokens in the global branch as guidance, less relevant blocks within the group are discarded to suppress interference from irrelevant information such as background. A local attention mechanism aggregates the retained block information within each group into local tokens, resulting in local feature representations. By integrating global and local features, the model learns richer representations, combining broad contextual understanding with detailed global features. During the inference phase, while maintaining accuracy, the model can choose to use only the global branch or only the local branch for ship recognition to avoid additional computational overhead.

[0040] The present invention proposes a ship re-identification method based on interactive learning of local features, which makes full use of global information and local information, extracts global and local features, suppresses interference from irrelevant information, and fuses global and local features, and only uses global features for ship re-identification. Figure 1 and Figure 2 As shown, the specific steps include:

[0041] (1) Image block sequence generation

[0042] First, the image of the ship is obtained and converted into a uniform image using image preprocessing technology. Where B represents the number of images in the batch, H and W represent the height and width of the image, which are set to 256 pixels by default, and C represents the number of color channels contained in each pixel, which is set to 3 by default. On this image, a convolutional neural network (CNN) is used for processing. The convolutional network converts image I into a series of image blocks. Specifically, the image is divided into several sub-regions, each of which is a block in the image. Through the convolution operation, image information is extracted and a sequence of image blocks {x i |i=1,2,…,N}, where N represents the total number of image patches, 256 in total, and the image patch size is 16×16. These image patches represent different regions of the image, and each patch captures the characteristics of a local area in the image, such as different parts of the ship, color, texture, and other information.

[0043] (2) Global feature extraction stage

[0044] To extract global features, a learnable class token x cls It is added to the front of the image sequence to represent the global features of the entire image. In addition, spatial information is added to the model through learnable position information embedding. Finally, the image token and class token together form a sequence and are input to the Transformer for feature extraction. This process can be expressed as:

[0045]

[0046] Among them, Z g represents the embedding representation of the input sequence, represents the position information encoding embedding, It is a linear mapping that maps the image block sequence to D g dimensional space. By embedding the feature Z g Apply the attention mechanism to calculate the global feature f g The attention mechanism is based on multi-head attention, and its core is to calculate the query Q cls , key K cls Sum V cls , as shown below:

[0047] Q cls =Z g W Q =[q cls ,q1,q2,...,q N ]

[0048] K cls =Z g W K =[k cls ,k1,k2,...,k N ]

[0049] V cls =Z g W V =[v cls ,v1,v2,...,v N ]

[0050] Among them, W Q 、W K and W V are different linear transformation matrices, q cls 、k cls and v cls x cls After W Q 、W K and W V The result after linear transformation, q N 、k N and v N They are After W Q 、W K and W V The result after linear transformation. Then the global attention is calculated by the following formula:

[0051]

[0052] The output of the global attention is then processed by a feedforward neural network. By repeating this process L times, the global feature f is finally obtained. g .

[0053] (3) Local feature extraction stage

[0054] (3-1) Configuring a Learnable Local Token

[0055] The image block sequence is further divided into M groups, each group represents a region of the ship, such as the bow, hull, and stern. For each group, a corresponding learnable local token {x pi |i=1,2,...,M}. Therefore, the input of the local branch can be expressed as:

[0056]

[0057] Among them, Z l represents the input feature sequence of the local branch, represents the position information encoding embedding, D l is the number of channels, is a linear projection that maps the image patch sequence to D l dimensional space.

[0058] (3-2) Feature Interaction Module

[0059] Assuming that the global branch contains K layers of self-attention, the attention weight A of each layer can be used k To express the importance of each image block sequence, it can be expressed as:

[0060]

[0061] in, is the weight corresponding to the class token, N is the number of image patch sequences, is the self-attention weight of the kth layer corresponding to the Nth image block. In order to comprehensively evaluate the importance of each image token, a comprehensive attention score A is obtained by multiplying the attention weights of each layer. score , this score reflects the importance of each image token in the multi-layer attention, which can be expressed as:

[0062]

[0063] Among them, A score Represents the comprehensive relationship between tokens. Then from A score Extract the token with class The relevant weights, K The final comprehensive attention score of the class token on the image token is obtained after multiplication. Finally, a mask is generated for each self-attention layer In the subsequent local attention calculation, the mask is sorted according to the attention scores, and the more important image block sequences are extracted and retained based on the class tokens, while suppressing the relatively less important image block sequences. This process can be expressed as:

[0064]

[0065] Among them, Mask means converting the selected image block into a mask form, Top s Retain the first s important image block sequences (s∈N + ).

[0066] (3-3) Partial Perception Feature Extraction

[0067] The global class token focuses on the global information of the entire image, while each local token Focus on different regions of interest (ROI) in the image and learn local features based on the image block sequence in the corresponding ROI. The local attention mechanism uses a multi-head attention structure similar to the global attention. The difference is that the local attention assigns the query Q, key K and value V to each local token, thereby focusing on the information of the corresponding ROI. p , value Vp and key K p Based on Z l Computed so that each local token Able to interact with information of related regions in the image. Partial perceptual attention can be expressed as:

[0068]

[0069] in, And {k i ,...,k i +m} represents the local token The associated sequence of m+1 image blocks. and Calculated in the same way. The importance mask, partial perception attention and M outputs corresponding to the number of groups generated in the previous feature interaction module step will pass through the feedforward network. By repeating this process L times, the local features can be obtained

[0070] (4) Collaborative interactive learning stage

[0071] For an input image within a mini-batch According to the local features of the input image x Construct a positive sample package P + and a negative sample bag P - The positive sample bag includes the local features of all images with the same ID as x in the same small batch, and each image has M local features. The negative sample bag consists of the local features of all images with different IDs from x. Next, find the global features f of x from the positive sample bag and the negative sample bag respectively. g The local feature d with the minimum L2 distance + and d + , they can be calculated as follows:

[0072]

[0073] Then, we need to make the distance d between the global feature and the nearest positive sample local feature + Closer, while making the distance d between the global feature and the nearest negative sample local feature - The calculation formula can be expressed as:

[0074]

[0075] (5) Dual-branch training stage

[0076] During the training process, global and local features are optimized separately. In order to ensure that the global feature fg It has the ability to distinguish different ships and uses cross entropy loss to constrain global features in the learning feature extraction stage.

[0077]

[0078] Among them, q k is the smoothed label, ε is the smoothing factor, usually set to 0.1; C is the total number of categories; y is the true label; p k is the probability predicted by the model. In addition to the cross-entropy loss, triplet loss is introduced to further optimize the model's ability to distinguish fine-grained features. In ship re-identification, different ships may appear very similar. Triplet loss helps the model better learn the subtle differences between different ships.

[0079]

[0080] where d p and d n Is the feature distance between the positive sample pair and the negative sample pair, α is a predefined distance threshold that helps to strengthen the distinction between similarity and difference. The local branch is also supervised by cross entropy and triple loss, and its loss function is and The calculation method of the global branch and Totally consistent.

[0081] (6) Single-branch reasoning stage

[0082] During the inference phase, a single-branch inference mechanism is used to maintain model accuracy while avoiding additional computational overhead. Specifically, during inference, one can choose to use only the global branch or only the local branch for vessel re-identification. This approach allows the model to flexibly select computing resources based on actual needs while maintaining high recognition accuracy. Of course, if resources permit, the model also supports dual-branch inference. By using both global and local branches, the model can fully utilize multi-scale information, resulting in more accurate recognition results.

[0083] Corresponding to the aforementioned embodiment of a ship re-identification method based on interactive learning of local features, the present invention also provides an embodiment of a ship re-identification system based on interactive learning of local features. The implementation process of each module in this system is described with reference to the specific steps of the aforementioned embodiment of a ship re-identification method based on interactive learning of local features. The system includes:

[0084] Image block sequence extraction module: used to obtain ship images, divide the images into blocks, and obtain image block sequences;

[0085] Global feature extraction module: used to input the image block sequence into the visual encoder and use the global attention mechanism to extract the global features of the ship.

[0086] Local feature perception module: It is used to group the image block sequence, add a learnable local token to each group, and use the local perception attention mechanism to extract the local features of the ship.

[0087] Feature Interaction Module: It is used to enhance the feature representation of related image patch sequences using the global attention score as a guide to better extract local features.

[0088] Collaborative interactive learning module: used to effectively collaboratively and interactively learn global features and local features to obtain more discriminative global features.

[0089] Ship re-identification module: used to calculate the similarity between the query image and the library image features, sort them according to the similarity, and obtain the final re-identification result.

[0090] Corresponding to the aforementioned embodiment of a ship re-identification method based on global-local feature interactive learning, the present invention also provides an embodiment of a ship re-identification device based on global-local feature interactive learning.

[0091] See also Figure 3 An embodiment of the present invention provides a ship re-identification device based on global-local feature interactive learning, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a ship re-identification method based on global-local feature interactive learning in the above embodiment.

[0092] The embodiment of the ship re-identification device based on global-local feature interactive learning provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 3 As shown in the figure, a hardware structure diagram of a ship re-identification device based on global-local feature interactive learning provided by the present invention is located in any device with data processing capability, except Figure 3In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0093] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0094] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0095] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the ship re-identification method based on global-local feature interactive learning in the above embodiment is implemented.

[0096] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0097] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the ship re-identification method based on global-local feature interactive learning.

[0098] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A ship re-identification method based on global-local feature interactive learning, characterized in that: The method comprises the following steps: (1) Obtain a ship image and divide it into a sequence of image blocks; (2) extracting the global features of the ship from the image block sequence obtained in step (1) based on the global attention mechanism; (3) Grouping the image block sequences obtained in step (1) and adding a learnable local token to each group. Each local token only focuses on the image block sequence of interest within the group. (4) Using the global attention score in step (2) as a guide, enhance the feature representation of the relevant image block sequence in step (3) to extract more accurate local features; (5) The global features obtained in step (2) and the local features obtained in step (4) are collaboratively interactively learned to obtain more discriminative global features; (6) Calculate the similarity between the global features of the query image obtained in step (5) and the features of the library image, sort them according to the similarity, and obtain the final re-identification result.

2. A ship re-identification method based on global-local feature interactive learning according to claim 1, characterized in that: In step (2), the CLIP visual encoder is used to encode the visual features of the image block sequence, and the global attention mechanism is used to aggregate the information of the image block sequence into class tokens, and the obtained class tokens are used as global features.

3. The ship re-identification method based on global-local feature interactive learning according to claim 1, characterized in that: In step (3), a mask mechanism is used to group the image block sequence. Each local token has a corresponding mask, and the value 1 in the mask represents the image block that is visible to the local token.

4. The ship re-identification method based on global-local feature interactive learning according to claim 2 is characterized in that: In step (4), the global class token’s attention score for the image block sequence is used as an indicator to judge the importance of the image block. The image blocks are sorted according to the attention score, and some image blocks with higher importance are retained as required. The mask mechanism is also used to control the visibility of the image blocks to the local token.

5. The ship re-identification method based on global-local feature interactive learning according to claim 1, characterized in that: In step (5), collaborative interactive learning is specifically as follows: constructing a local feature positive and negative sample package, where the local feature with the same identity as the global feature is the positive sample, and the local feature with a different identity from the global feature is the negative sample, and finding the corresponding local feature positive and negative sample closest to the global feature from the positive and negative sample package, and using the triplet loss to shorten the distance between the global feature and the local feature positive sample, and to increase the distance between the global feature and the local feature negative sample.

6. The ship re-identification method based on global-local feature interactive learning according to claim 1, characterized in that: During the training process, global and local features are optimized separately. To ensure that global features have the ability to distinguish different ships, cross-entropy loss is used to constrain global features to learn local feature extraction, and the subtle differences between different ships are learned based on triplet loss. Local features are also supervised using cross-entropy and triplet losses.

7. A ship re-identification system based on global-local feature interactive learning, characterized in that: The system includes: Image block sequence extraction module: used to obtain ship images and divide them into image block sequences; Global feature extraction module: used to extract global features of ships from image block sequences based on the global attention mechanism; Local feature perception module: used to group image block sequences, add learnable local tokens to each group, and extract local features of the ship using the local perception attention mechanism; Feature Interaction Module: Used to enhance the feature representation of related image block sequences using the global attention score as a guide to extract more accurate local features; Collaborative interactive learning module: used to collaboratively interactively learn global features and local features to obtain more discriminative global features. Ship re-identification module: used to calculate the similarity between the query image and the library image features, sort them according to the similarity, and obtain the final re-identification result.

8. A ship re-identification device based on global-local feature interactive learning, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the ship re-identification method based on global-local feature interactive learning.

9. A computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the ship re-identification method based on global-local feature interactive learning is implemented.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the ship re-identification method based on global-local feature interactive learning.

Images