A cross-modal individual recognition method for marine targets

Abstract

The present invention discloses a method for cross-modal individual recognition of marine targets, which belongs to the field of data recognition. The method includes: constructing a cross-modal individual recognition network, which has three image input terminals, and the image modes corresponding to the three image input terminals are different, and also includes a multi-scale feature extraction network corresponding to the image input terminals one by one, and three feature fusion modules and a fully connected layer; designing a regression-based loss calculation method and a triplet loss with an adaptive threshold; then, training the cross-modal individual recognition network; finally, using the trained cross-modal individual recognition network to identify the target identity. The present invention constructs an individual recognition network capable of processing three modal data based on a twin network, improves the effect of cross-modal individual recognition of ship targets, and does not need to simultaneously obtain the image data of the three modalities of the target, and can realize the judgment and recognition of whether the identity of any two modal targets in the three modalities is consistent.

Description

Technical Field

[0001] The invention belongs to the field of data identification, and in particular relates to a method for identifying individual targets at sea. Background Art

[0002] The existing image-based individual recognition methods often only consider data from a single modality and cannot perform cross-modal recognition. However, the target individual may appear in different data modalities (visible light, infrared, SAR, etc.), and sometimes even only appear once in one modality. Therefore, single-modal recognition suffers from insufficient data coverage and limited recognition opportunities. For example, a target of a certain identity only appears once in a visible light image. When it appears in an infrared image next time, the network model for the infrared image cannot recognize it due to the lack of corresponding image data. Summary of the invention

[0003] The present invention proposes a cross-modal individual recognition method for marine targets, the purpose of which is to solve the problem of insufficient coverage of single-modal data and too low recognition opportunity.

[0004] The technical solution of the present invention is as follows:

[0005] A cross-modal individual recognition method for marine targets, comprising the following steps:

[0006] Step 1: Construct a cross-modal individual recognition network;

[0007] The cross-modal individual recognition network has three image input terminals, and the image modalities corresponding to the three image input terminals are different;

[0008] The cross-modal individual recognition network also includes a multi-scale feature extraction network corresponding to the image input end one by one, and the multi-scale feature extraction network is used to extract multi-scale modal features of the input image;

[0009] The cross-modal individual recognition network also includes three feature fusion modules and a fully connected layer; the multi-scale modal features output by the three multi-scale feature extraction networks are combined in pairs and input into the corresponding feature fusion modules for fusion respectively, so as to obtain three groups of multi-scale fusion features; the fully connected layer obtains a binary classification result according to the splicing result of all the multi-scale fusion features, and the binary classification result is used to indicate the probability that the target individuals in two of the three input images are the same;

[0010] Step 2: training the cross-modal individual recognition network;

[0011] Step 3: Use the trained cross-modal individual recognition network to identify the target identity.

[0012] As a further improvement of the cross-modal individual recognition method for marine targets: the three image input ends are used to input visible light images, infrared images and SAR images respectively.

[0013] As a further improvement of the cross-modal individual recognition method for marine targets: the multi-scale feature extraction network includes a plurality of network access modules connected in series, wherein the last The network access module gets feature maps of different scales, Feature maps of different scales constitute the multi-scale modal features output by the multi-scale feature extraction network.

[0014] As a further improvement of the cross-modal individual recognition method for marine targets: the feature fusion module includes Different scale features with multi-scale modal features Figure 1 A corresponding processing branch, the feature maps of the same scale of the two multi-scale modal features input by the feature fusion module are input into the corresponding processing branch for the following processing: first, the feature maps are concatenated, then expanded into feature vectors through convolution, and then resized through a connection layer to obtain the fusion features of the scale;

[0015] The fusion features of different scales obtained by each processing branch are combined into multi-scale fusion features output by the corresponding feature fusion module.

[0016] As a further improvement of the cross-modal individual identification method for maritime targets, step 2 includes:

[0017] Step 2-1: Construct a first training set, which contains several initial sample pairs, each of which contains three images of different modalities and a label. , It means that the target identities in at least two images of the initial sample pair are the same. Indicates that the target identities in the three images of this initial sample pair are different;

[0018] Step 2-2: Filter out from the first training set And there are only two initial sample pairs with the same target identity in the three images, and then the same-identity images in each initial sample pair are superimposed and mixed with the same-modality images of other identities to obtain mixed sample pairs; all the mixed sample pairs are used to form the second training set;

[0019] Step 2-3, loop through steps 2-3-1 and 2-3-2 several times:

[0020] Step 2-3-1: Take e initial sample pairs from the first training set in batches and input them into the cross-modal individual recognition network to obtain the corresponding binary classification results. , and calculate the binary classification loss corresponding to each initial sample pair :

[0021] ;

[0022] According to e binary classification losses The average value of updates the parameters of the cross-modal individual recognition network;

[0023] Step 2-3-2: Take e mixed sample pairs from the second training set in batches and input them into the cross-modal individual recognition network respectively, and calculate the adaptive threshold triplet loss corresponding to each mixed sample pair respectively. , and then according to e adaptive threshold triplet loss The average value of updates the parameters of the cross-modal individual recognition network.

[0024] As a further improvement of the cross-modal individual recognition method for marine targets, the method of constructing mixed sample pairs in step 2-2 is as follows:

[0025] Suppose the same identity image in the initial sample pair is and , randomly selected with and Images with the same modality but different identities are and , then the two mixed images are:

[0026] ;

[0027] ;

[0028] in and They are all random numbers greater than or equal to 0.5 and less than or equal to 1;

[0029] Then, according to and Constructing labels for mixed sample pairs :

[0030] ;

[0031] Blend the images and , the original and and Images with different identities And tags Construct a mixed sample pair, denoted as .

[0032] As a further improvement of the cross-modal individual recognition method for maritime targets, the adaptive threshold triplet loss The calculation method is:

[0033] ;

[0034] in, Indicates the calculated distance value. is based on and The obtained multi-scale fusion features, is based on and The obtained multi-scale fusion features, is based on and The obtained multi-scale fusion features, is the distance difference threshold.

[0035] As a further improvement of the cross-modal individual identification method for marine targets, step 3 includes:

[0036] Step 3-1: Select several groups of existing images for the image to be identified, each group contains two existing images, and the existing images refer to images that are previously saved in the database and the identity of the target in the image is known. The following three conditions must be met when selecting:

[0037] Condition 1: The modalities of the two existing images and the image to be identified in the same group are different;

[0038] Condition 2: The identities of the targets in the two existing images in the same group are different;

[0039] Condition 3: If the number of individual identities in the database is even, the identity of the target in each group of images is not repeated in other groups; otherwise, only the last group is repeated, which contains the last individual identity and the first individual identity in the database;

[0040] Step 3-2: Combine the image to be identified and the existing images of all groups, and input them into the cross-modal individual identification network respectively to obtain the first and second classification results corresponding to each combination;

[0041] Step 3-3, if the maximum value of the first and second classification results is greater than the preset threshold, first find the maximum first and second classification result value, then select two existing images corresponding to the maximum value, and execute step 3-4;

[0042] Step 3-4, assuming that the two currently selected existing images are the first existing image and the second existing image, select a third existing image, and the identity of the target in the third existing image is different from the identity of the target in the first existing image and the second existing image; first input the image to be identified, the first existing image, and the third existing image into the cross-modal individual recognition network to obtain a second two-classification result, and then input the image to be identified, the second existing image, and the third existing image into the cross-modal individual recognition network to obtain a third two-classification result;

[0043] Step 3-5: If the second binary classification result is greater than the third binary classification result, the identity of the target in the image to be identified is the same as the identity of the target in the first existing image; otherwise, the identity of the target in the image to be identified is the same as the identity of the target in the second existing image.

[0044] As a further improvement of the cross-modal individual recognition method for maritime targets: in step 3-3, if the maximum value of all the first and second classification results is less than or equal to the preset threshold, the current image to be identified is treated as an unidentified image, the identity of its target is manually marked, and then it is added to the database.

[0045] As a further improvement of the cross-modal individual recognition method for maritime targets: the threshold is 0.5.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. The present invention constructs an individual recognition network capable of processing three modal data based on a twin network. The network first extracts multi-scale modal features of images of different modalities, then fuses them in pairs, and then judges whether the identities are consistent based on the fused results, thereby improving the effect of individual recognition of ship targets under different modalities. This method does not require simultaneous acquisition of image data of the three modalities of the target, and can realize the judgment and recognition of whether the identities of any two modal targets in the three modalities are consistent.

[0048] 2. The multi-scale feature extraction network of the present invention first extracts features of different sizes within the same scale through the Inception Block structure containing convolution kernels of different sizes, and then outputs features of three different scales of the last three Inception Blocks, thereby more comprehensively extracting features related to the identity of the ship target.

[0049] 3. The feature fusion module of the present invention can splice and fuse the features of three different scales, so as to better compare the feature information of the input targets under the two modes, and finally judge whether the identities of the above-mentioned ship targets are consistent through full connection, thereby improving the effect of subsequent classification tasks.

[0050] 4. The data enhancement method based on image mixing of the present invention modifies the predicted label value in combination with the mixing parameters, transforms the binary classification problem corresponding to the twin network into a label continuous regression problem, and modifies the threshold of the triplet loss in combination with the mixing parameters to make it adaptive to the image mixing situation. On the one hand, it increases the diversity of training samples, and on the other hand, it prompts the model to fully learn the discriminable features, thereby helping the algorithm model to find a better classification space. In addition, it can also stabilize the training process and reduce the loss jump changes caused by the threshold in the triplet loss.

[0051] 5. The present invention introduces triplet loss into the scenario of two-by-two modal fusion, and calculates the loss by the distance between the fused features of different identity combinations, which further improves the discriminability of targets with different identities, and improves the training effect and recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is the overall architecture diagram of the cross-modal individual recognition network;

[0053] Figure 2 This is the architecture diagram of the multi-scale feature extraction network;

[0054] Figure 3 This is the architecture diagram of the network access module;

[0055] Figure 4 This is the architecture diagram of the feature fusion module. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0057] A cross-modal individual recognition method for marine targets, comprising the following steps:

[0058] Step 1: Construct a cross-modal individual recognition network.

[0059] like Figure 1 The cross-modal individual recognition network has three image input terminals, and the three image input terminals correspond to different image modalities. In this embodiment, the three image input terminals are used to input visible light images, infrared images, and SAR images, respectively.

[0060] The cross-modal individual recognition network also includes a multi-scale feature extraction network corresponding one-to-one to the image input end, and the multi-scale feature extraction network is used to extract multi-scale modal features of the input image.

[0061] The cross-modal individual recognition network also includes three feature fusion modules and a fully connected layer. The multi-scale modal features output by the three multi-scale feature extraction networks are combined in pairs and input into the corresponding feature fusion modules for fusion, respectively, to obtain three sets of multi-scale fusion features. The fully connected layer obtains a binary classification result based on the splicing results of all multi-scale fusion features, and the binary classification result is used to indicate the probability that the target identity of two of the three input images is the same.

[0062] Specifically, Figure 2 The multi-scale feature extraction network includes several inception blocks connected in series, through which the last The network access module gets feature maps of different scales, Feature maps of different scales constitute the multi-scale modal features output by the multi-scale feature extraction network.

[0063] The structure of each Inception Block is as follows: Figure 3 As shown in the figure. The introduction of Inception Block can realize feature extraction at different spatial scales by combining multiple convolution kernels, and can improve the effect of the model with a smaller amount of calculation. By combining convolution kernels of different scales and different division methods, features at different levels can be accurately captured, so that the network can better adapt to different object shapes and structures.

[0064] like Figure 2 In this embodiment, the image input by the multi-scale feature extraction network first passes through three Inception Blocks with kn=16, 32, and 64 to obtain a 16×16×64 feature map, and then passes through three Inception Blocks in sequence (i.e. ), and concatenate the feature maps of the three Inception Blocks. The multi-scale modal features finally output include three sets of feature maps P3, P4, and P5 with sizes of 8×8×128, 4×4×256, and 2×2×512, respectively. The multi-scale modal features corresponding to visible light images, infrared images, and SAR images are recorded as , , .

[0065] Furthermore, if Figure 4 , the feature fusion module includes Different scale features with multi-scale modal features Figure 1In a corresponding processing branch, the feature maps of the same scale of the two multi-scale modal features input by the feature fusion module are input into the corresponding processing branch for the following processing: first, the feature maps are concatenated, then expanded into feature vectors through convolution, and then resized through a connection layer to obtain the fusion features of the scale. The fusion features of different scales obtained by each processing branch form the multi-scale fusion features output by the corresponding feature fusion module.

[0066] In this embodiment, after the two feature maps input into each processing branch are concatenated, they are first expanded into feature vectors after passing through a layer of convolution conv with k=3, s=1, and then connected to a fully connected layer and uniformly resized to 128 to obtain the fusion features FC3, FC4, and FC5 of the two modalities at three scales, all of which have a length of 128. , , The three multi-scale modal features are fused in pairs to obtain three groups of multi-scale fusion features. Each multi-scale fusion feature contains a group of fusion features FC3, FC4, and FC5. Finally, feature F with a length of 1152 can be obtained by splicing. The final fully connected layer obtains the binary classification result based on feature F.

[0067] Step 2: training the cross-modal individual recognition network, the specific steps are:

[0068] Step 2-1: Construct a first training set, which contains several initial sample pairs, each of which contains three images of different modalities. , , , also contains a label , the initial sample pair is represented as .in, It means that the target identities in at least two images of the initial sample pair are the same. It indicates that the target identities in the three images of this initial sample pair are different.

[0069] Step 2-2: Filter out from the first training set And there are only initial sample pairs with the same target identity in two images of the three images, and then the same-identity images in each initial sample pair are superimposed and mixed with the same-modal images of other identities to obtain mixed sample pairs; all the mixed sample pairs are used to form the second training set.

[0070] Specifically, let the same-identity images in the initial sample pair be and , randomly selected with and Images with the same modality but different identities are and , then the two mixed images are:

[0071] ;

[0072] ;

[0073] in and They are all random numbers greater than or equal to 0.5 and less than or equal to 1.

[0074] Then, according to and Constructing labels for mixed sample pairs :

[0075] .

[0076] In this way, the original binary labels can be transformed into continuous labels between [0.5, 1]. By training the twin network with mixed images and continuous labels, the diversity of training samples is increased on the one hand, and the model is prompted to fully learn discriminable features on the other hand, thereby helping the network find a better classification space and stabilize the training process.

[0077] Finally, the mixed image and , the original and and Images with different identities And tags Construct a mixed sample pair, denoted as .

[0078] Step 2-3, loop through steps 2-3-1 and 2-3-2 several times:

[0079] Step 2-3-1: Take e initial sample pairs from the first training set in batches and input them into the cross-modal individual recognition network to obtain the corresponding binary classification results. , and calculate the binary classification loss corresponding to each initial sample pair :

[0080] ;

[0081] According to e binary classification losses The average value of updates the parameters of the cross-modal individual recognition network.

[0082] Step 2-3-2: Take e mixed sample pairs from the second training set in batches and input them into the cross-modal individual recognition network respectively, and calculate the adaptive threshold triplet loss corresponding to each mixed sample pair respectively. :

[0083] ;

[0084] in, Indicates the calculated distance value. is based on and The obtained multi-scale fusion features, is based on and The obtained multi-scale fusion features, is based on and The obtained multi-scale fusion features, is the distance difference threshold.

[0085] Then according to e adaptive threshold triplet loss The average value of updates the parameters of the cross-modal individual recognition network.

[0086] It should be noted that the triplet loss in this method is essentially different from the conventional triplet loss. This method calculates the distance value between two multi-scale fusion features, and each fusion feature is obtained by the feature fusion module fusing two images, rather than the distance value of the two image features.

[0087] Introducing the image mixing-based data enhancement method into the improved adaptive threshold triplet loss can prompt the model to better learn discriminative features and stabilize the training process. At the same time, the present invention introduces triplet loss into the modal fusion scenario, takes the distance between the fusion features of different identity combinations as the loss value, prompting the model to learn a more discriminative feature space for each individual, thereby improving the training effect and recognition accuracy.

[0088] In this embodiment, Adam is used as the optimizer, the learning rate is set to 0.0002, the batch size is set to 64, and the number of cycles in steps 2-3 is 500.

[0089] Step 3: Use the trained cross-modal individual recognition network to identify the target identity. The specific process is as follows:

[0090] Step 3-1: Select several groups of existing images for the image to be identified, each group contains two existing images, and the existing images refer to images that are previously saved in the database and the identity of the target in the image is known. The following three conditions must be met when selecting:

[0091] Condition 1: The modalities of the two existing images and the image to be identified in the same group are different;

[0092] Condition 2: The identities of the targets in the two existing images in the same group are different;

[0093] Condition 3: If the number of individual identities in the database is even, the identity of the target in each group of images is not repeated in other groups; otherwise, only the last group is repeated, which contains the last individual identity and the first individual identity in the database.

[0094] Step 3-2: Combine the image to be identified and the existing images of all groups, and input them into the cross-modal individual recognition network respectively to obtain the first and second classification results corresponding to each combination.

[0095] Step 3-3: If the maximum value of the first binary classification result is greater than a preset threshold (preferably 0.5), first find the maximum first binary classification result value, then select two existing images corresponding to the maximum value, and execute step 3-4.

[0096] Step 3-4, assuming that the two currently selected existing images are the first existing image and the second existing image, select a third existing image, and the identity of the target in the third existing image is different from the identity of the target in the first existing image and the second existing image; first input the image to be identified, the first existing image and the third existing image into the cross-modal individual recognition network to obtain the second binary classification result, and then input the image to be identified, the second existing image and the third existing image into the cross-modal individual recognition network to obtain the third binary classification result.

[0097] Step 3-5: If the second binary classification result is greater than the third binary classification result, the identity of the target in the image to be identified is the same as the identity of the target in the first existing image; otherwise, the identity of the target in the image to be identified is the same as the identity of the target in the second existing image.

[0098] Furthermore, in step 3-3, if the maximum value of all the first and second classification results is less than or equal to a preset threshold, the current image to be identified is regarded as an unidentified image, the identity of the target is manually marked, and then it is added to the database.

[0099] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. The scope of the present invention is defined by the claims rather than the above description.

Claims

1. A cross-modal individual recognition method for marine targets, characterized by the following steps: include: Step 1: Construct a cross-modal individual recognition network; The cross-modal individual recognition network has three image input terminals, and the image modalities corresponding to the three image input terminals are different; The cross-modal individual recognition network also includes a multi-scale feature extraction network corresponding to the image input end one by one, and the multi-scale feature extraction network is used to extract multi-scale modal features of the input image; The cross-modal individual recognition network also includes three feature fusion modules and a fully connected layer; the multi-scale modal features output by the three multi-scale feature extraction networks are combined in pairs and input into the corresponding feature fusion modules for fusion respectively, so as to obtain three groups of multi-scale fusion features; the fully connected layer obtains a binary classification result according to the splicing result of all the multi-scale fusion features, and the binary classification result is used to indicate the probability that the target individuals in two of the three input images are the same; Step 2: training the cross-modal individual recognition network; Step 3: Use the trained cross-modal individual recognition network to identify the target identity.

2. The cross-modal individual identification method for marine targets according to claim 1, characterized in that: The three image input ports are used to input visible light images, infrared images and SAR images respectively.

3. The cross-modal individual identification method for marine targets according to claim 1, characterized in that: The multi-scale feature extraction network includes a plurality of network access modules connected in series, through which the last The network access module gets feature maps of different scales, Feature maps of different scales constitute the multi-scale modal features output by the multi-scale feature extraction network.

4. The cross-modal individual identification method for marine targets according to claim 3, characterized in that: The feature fusion module includes The feature fusion module inputs the feature maps of the same scale of the two multi-scale modal features into the corresponding processing branch for the following processing: first, the feature maps are concatenated, then expanded into feature vectors through convolution, and then resized through a connection layer to obtain the fusion features of the scale; The fusion features of different scales obtained by each processing branch are combined into multi-scale fusion features output by the corresponding feature fusion module.

5. The cross-modal individual identification method for marine targets according to claim 1, characterized in that: Step 2 includes: Step 2-1: Construct a first training set, which contains several initial sample pairs, each of which contains three images of different modalities and a label. , It means that the target identities in at least two images of the initial sample pair are the same. Indicates that the target identities in the three images of this initial sample pair are different; Step 2-2: Filter out from the first training set And there are only two initial sample pairs with the same target identity in the three images, and then the same-identity images in each initial sample pair are superimposed and mixed with the same-modality images of other identities to obtain mixed sample pairs; all the mixed sample pairs are used to form the second training set; Step 2-3, loop through steps 2-3-1 and 2-3-2 several times: Step 2-3-1: Take e initial sample pairs from the first training set in batches and input them into the cross-modal individual recognition network to obtain the corresponding binary classification results. , and calculate the binary classification loss corresponding to each initial sample pair : ； According to e binary classification losses The average value of updates the parameters of the cross-modal individual recognition network; Step 2-3-2: Take e mixed sample pairs from the second training set in batches and input them into the cross-modal individual recognition network respectively, and calculate the adaptive threshold triplet loss corresponding to each mixed sample pair respectively. , and then according to e adaptive threshold triplet loss The average value of updates the parameters of the cross-modal individual recognition network.

6. The cross-modal individual identification method for marine targets according to claim 5, characterized in that: The method of constructing mixed sample pairs in step 2-2 is: Suppose the same identity image in the initial sample pair is and , randomly selected with and Images with the same modality but different identities are and , then the two mixed images are: ； ； in and They are all random numbers greater than or equal to 0.5 and less than or equal to 1; Then, according to and Constructing labels for mixed sample pairs : ； Blend the images and , the original and and Images with different identities And tags Construct a mixed sample pair, denoted as .

7. The cross-modal individual identification method for marine targets according to claim 6, characterized in that: Adaptive Threshold Triplet Loss The calculation method is: ； in, Indicates the calculated distance value. is based on and The obtained multi-scale fusion features, is based on and The obtained multi-scale fusion features, is based on and The obtained multi-scale fusion features, is the distance difference threshold.

8. The cross-modal individual identification method for marine targets according to any one of claims 1 to 7, characterized in that: Step 3 includes: Step 3-1: Select several groups of existing images for the image to be identified, each group contains two existing images, and the existing images refer to images that are previously saved in the database and the identity of the target in the image is known. The following three conditions must be met when selecting: Condition 1: The modalities of the two existing images and the image to be identified in the same group are different; Condition 2: The identities of the targets in the two existing images in the same group are different; Condition 3: If the number of individual identities in the database is even, the identity of the target in each group of images is not repeated in other groups; otherwise, only the last group is repeated, which contains the last individual identity and the first individual identity in the database; Step 3-2: Combine the image to be identified and the existing images of all groups, and input them into the cross-modal individual identification network respectively to obtain the first and second classification results corresponding to each combination; Step 3-3, if the maximum value of the first and second classification results is greater than the preset threshold, first find the maximum first and second classification result value, then select two existing images corresponding to the maximum value, and execute step 3-4; Step 3-4, assuming that the two currently selected existing images are the first existing image and the second existing image, select a third existing image, and the identity of the target in the third existing image is different from the identity of the target in the first existing image and the second existing image; first input the image to be identified, the first existing image, and the third existing image into the cross-modal individual recognition network to obtain a second two-classification result, and then input the image to be identified, the second existing image, and the third existing image into the cross-modal individual recognition network to obtain a third two-classification result; Step 3-5: If the second binary classification result is greater than the third binary classification result, the identity of the target in the image to be identified is the same as the identity of the target in the first existing image; otherwise, the identity of the target in the image to be identified is the same as the identity of the target in the second existing image.

9. The cross-modal individual identification method for marine targets according to claim 8, characterized in that: In step 3-3, if the maximum value of all the first and second classification results is less than or equal to the preset threshold, the current image to be identified is regarded as an unidentified image, the identity of the target is manually marked, and then it is added to the database.

10. The cross-modal individual identification method for marine targets according to claim 8, characterized in that: The threshold is 0.5.

Images