Multispectral target type identification method based on deformable fusion

Abstract

The invention discloses a multispectral target type identification method based on deformable fusion. The method comprises the following steps: generating a first shared feature map and a first modal characteristic feature map based on a visible spectrum target image, and generating a second shared feature map and a second modal characteristic feature map based on an infrared spectrum target image; obtaining a cross-modal sharing fusion feature map; constructing a multi-view integration module based on variability fusion, and obtaining a first fusion feature map, a second fusion feature map and a third fusion feature map based on the multi-view integration module; and obtaining a final multi-scale representation graph, and inputting the final multi-scale representation graph to a target detector to obtain a multi-spectral target type recognition result. According to the method, sharing and modal characteristics of different spectral images can be obtained, key areas are explored, cross sampling and fusion are carried out, effective information of target characteristics is improved, the distinguishability of the target characteristics is enhanced, and then the accuracy of multi-spectral target type recognition is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of multi-spectral target type recognition, and in particular to a multi-spectral target type recognition method based on deformable fusion. Background Art

[0002] The existing multispectral target type recognition methods have obvious shortcomings, which are mainly reflected in the following aspects: First, these methods are not effective in fusing multispectral data, and often fail to fully utilize the information complementarity between different spectra. Secondly, the lack of effective identification and cross-fusion of key areas in each spectrum makes it difficult to fully extract and utilize important features. The fundamental reason for these shortcomings is that the processing mechanism of multispectral data in existing methods is relatively simple, which is difficult to adapt to complex target recognition needs. In addition, existing methods often rely on traditional feature extraction technology, resulting in limited processing capabilities for multidimensional data, making it difficult to capture the subtle differences and changes of the target. Therefore, the final recognition results are often not accurate enough, affecting the effectiveness and reliability of multispectral target type recognition in practical applications. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multi-spectral target type recognition method based on deformable fusion.

[0004] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0005] A multispectral target type recognition method based on deformable fusion includes the following steps:

[0006] S1. Generate a first shared feature map and a first modal characteristic feature map based on a visible spectrum target image, and generate a second shared feature map and a second modal characteristic feature map based on an infrared spectrum target image;

[0007] S2, integrating the first shared feature map and the second shared feature map in step S1 based on a probability distribution fusion strategy to obtain a cross-modal shared fusion feature map;

[0008] S3, constructing a multi-view integration module based on variability fusion, integrating the first shared feature map and the first modal characteristic feature map in step S1 based on the multi-view integration module to obtain a first fused feature map, integrating the second shared feature map and the second modal characteristic feature map in step S1 based on the multi-view integration module to obtain a second fused feature map, and integrating the first modal characteristic feature map and the second modal characteristic feature map in step S1 based on the multi-view integration module to obtain a third fused feature map;

[0009] S4, fusing the cross-modal shared fusion feature map in step S2 and the first fusion feature map, the second fusion feature map and the third fusion feature map in step S3 to obtain a final multi-scale representation map, and inputting the final multi-scale representation map into the target detector to obtain a multi-spectral target type recognition result.

[0010] Furthermore, in step S1, an orthogonal constraint loss minimization module is set between the first shared feature map and the first modal characteristic feature map, and between the second shared feature map and the second modal characteristic feature map; and a Jensen–Shannon divergence loss minimization module is set between the first shared feature map and the second shared feature map.

[0011] Furthermore, the data processing process of minimizing the Jensen–Shannon divergence loss module includes the following steps:

[0012] A1. Project the first shared feature map and the second shared feature map into the probability space, expressed as:

[0013]

[0014] Where: P is the first shared feature map after projection to the probability space, σ(·) is the standard distribution function that maps the feature to the probability space, is the first shared feature map, Q is the second shared feature map after projection to the probability space, is the second shared feature map, H, W, and C are the height, width, and number of channels of the feature map, respectively;

[0015] A2. Based on the first shared feature map and the second shared feature map projected to the probability space in step A1, the Jensen–Shannon divergence loss is calculated and minimized. The Jensen–Shannon divergence loss is calculated and expressed as:

[0016]

[0017] in: is the Jensen–Shannon divergence loss, D KL is the Kullback-Leibler divergence, and M is the average probability distribution of the first shared feature map and the second shared feature map after being projected into the probability space.

[0018] Further, in step S2, the first shared feature map and the second shared feature map in step S1 are integrated based on the probability distribution fusion strategy to obtain a cross-modal shared fusion feature map, which is expressed as:

[0019]

[0020] in: To share the fusion feature map across modalities, is the first shared feature map, is the second shared feature map.

[0021] Further, in step S3, the multi-view integration module includes a global feature branch, a local feature branch, and a concatenation dimension reduction layer that simultaneously connects the output end of the global feature branch and the output end of the local feature branch.

[0022] Furthermore, the global feature branch includes an input layer and a deformable cross sampling fusion submodule connected in sequence; the data processing process of the global feature branch is expressed as:

[0023]

[0024] in: is the global fusion feature output by the global feature branch, DFB(·) is the deformable cross sampling fusion submodule, is the first feature map of the input, is the second feature map of the input.

[0025] Furthermore, the local feature branch includes an input layer, a feature slicing layer, a deformable cross sampling fusion submodule and a feature reconstruction layer connected in sequence. The data processing process of the local feature branch is:

[0026] The feature segmentation layer segments the first feature map of the input and the second feature map of the input into feature blocks, expressed as:

[0027]

[0028] in: is the feature block of the first feature map of the input at the same spatial position of j, split(·) is the segmentation operation, is the first feature map of the input, is the feature block of the second feature map of the input at the same spatial position of j, is the second feature map of the input;

[0029] The deformable cross sampling fusion submodule performs local multispectral feature fusion on the feature blocks of the input first feature map and the feature blocks of the input second feature map to obtain the local fusion features at the same spatial position, which is expressed as:

[0030]

[0031] in: is the local fusion feature of the same spatial position of the jth node, DFB(·) is the deformable cross sampling fusion submodule;

[0032] The feature reconstruction layer reconstructs the local fusion features of the same spatial position according to the initial spatial position to obtain the local fusion features, which is expressed as:

[0033]

[0034] in: is the local fusion feature output by the local feature branch, and rebulid(·) is the feature map reconstruction operation.

[0035] Furthermore, the deformable cross sampling fusion submodule includes two deformable cross sampling branches and a fusion feature layer simultaneously connected to the output ends of the two deformable cross sampling branches; the deformable cross sampling branch includes an input layer, a reference point setting layer, a position offset sub-branch, an overlay layer, a cross sampling layer, a splicing layer and a self-attention layer; the input layers of the deformable cross sampling branches are all connected to the cross sampling layer in another deformable cross sampling branch.

[0036] Furthermore, the data processing process of the position offset sub-branch is expressed as:

[0037]

[0038] Where: Δp is the position offset, Conv1×1 is a 1×1 convolution operation, GELU(·) is the GELU activation function, DWConv(·) is the depth convolution layer, is the input feature map.

[0039] Furthermore, the data processing process of the self-attention layer is:

[0040]

[0041] in: is the deformable fusion attention enhanced feature map, M is the self-attention weight, v is the second deformable fusion attention map after learning, is the input feature map, LN(·) is the normalization layer, MLP(·) is the multi-layer perceptron, and v is the second deformable fusion attention map after learning.

[0042] The present invention has the following beneficial effects:

[0043] The present invention integrates the first shared feature map and the second shared feature map to obtain a cross-modal shared fusion feature map, then constructs a multi-view integration module based on variability fusion, and then integrates the first shared feature map and the first modal characteristic feature map to obtain a first fused feature map, integrates the second shared feature map and the second modal characteristic feature map to obtain a second fused feature map, integrates the first modal characteristic feature map and the second modal characteristic feature map to obtain a third fused feature map, and then fuses the cross-modal shared fusion feature map, the first fused feature map, the second fused feature map and the third fused feature map to obtain a final multi-scale representation map, and inputs the final multi-scale representation map to a target detector to obtain a multi-spectral target type recognition result. The whole process can obtain the shared and modal characteristic features of different spectral images, explore key areas and perform cross-sampling and fusion, improve the effective information of target features and enhance their distinguishability, thereby enhancing the accuracy of multi-spectral target type recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a multi-spectral target type recognition method based on deformable fusion;

[0045] Figure 2 This is a schematic diagram of the data processing flow of the multi-view fusion module;

[0046] Figure 3 Schematic diagram of the data processing flow of the deformable cross sampling fusion submodule. DETAILED DESCRIPTION

[0047] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0048] like Figure 1 As shown, a multi-spectral target type recognition method based on deformable fusion includes steps S1-S4, which are as follows:

[0049] S1. Generate a first shared feature map and a first modal characteristic feature map based on a visible spectrum target image, and generate a second shared feature map and a second modal characteristic feature map based on an infrared spectrum target image.

[0050] In an optional embodiment of the present invention, a module for minimizing orthogonal constraint loss is set between the first shared feature map and the first modal characteristic feature map, and between the second shared feature map and the second modal characteristic feature map; a module for minimizing Jensen–Shannon divergence loss is set between the first shared feature map and the second shared feature map.

[0051] The data processing process of minimizing the Jensen–Shannon divergence loss module includes the following steps:

[0052] A1. Project the first shared feature map and the second shared feature map into the probability space, expressed as:

[0053]

[0054] Where: P is the first shared feature map after projection to the probability space, σ(·) is the standard distribution function that maps the feature to the probability space, is the first shared feature map, Q is the second shared feature map after projection to the probability space, is the second shared feature map.

[0055] A2. Based on the first shared feature map and the second shared feature map projected to the probability space in step A1, the Jensen–Shannon divergence loss is calculated and minimized. The Jensen–Shannon divergence loss is calculated and expressed as:

[0056]

[0057] in: is the Jensen–Shannon divergence loss, D KL is the Kullback-Leibler divergence, and M is the average probability distribution of the first shared feature map and the second shared feature map after being projected into the probability space.

[0058] S2. Integrate the first shared feature map and the second shared feature map in step S1 based on a probability distribution fusion strategy to obtain a cross-modal shared fusion feature map.

[0059] In an optional embodiment of the present invention, the present invention integrates the first shared feature map and the second shared feature map in step S1 based on a probability distribution fusion strategy to obtain a cross-modal shared fusion feature map, which is expressed as:

[0060]

[0061] in: To share the fusion feature map across modalities, is the first shared feature map, is the second shared feature map.

[0062] S3. Construct a multi-view integration module based on variability fusion, integrate the first shared feature map and the first modal characteristic feature map in step S1 based on the multi-view integration module to obtain a first fused feature map, integrate the second shared feature map and the second modal characteristic feature map in step S1 based on the multi-view integration module to obtain a second fused feature map, and integrate the first modal characteristic feature map and the second modal characteristic feature map in step S1 based on the multi-view integration module to obtain a third fused feature map.

[0063] In an optional embodiment of the present invention, the multi-view integration module includes a global feature branch, a local feature branch, and a splicing dimension reduction layer that simultaneously connects the output end of the global feature branch and the output end of the local feature branch.

[0064] like Figure 2 As shown, input 1 is the input layer, DFB is the deformable cross sampling fusion submodule, and the global feature branch includes the input layer and the deformable cross sampling fusion submodule connected in sequence; the data processing process of the global feature branch is expressed as:

[0065]

[0066] in: is the global fusion feature output by the global feature branch, DFB(·) is the deformable cross sampling fusion submodule, is the first feature map of the input, is the second feature map of the input.

[0067] like Figure 2 As shown, input 2 is the input layer, and the local feature branch includes the input layer, feature slicing layer, deformable cross sampling fusion submodule and feature reconstruction layer connected in sequence. The data processing process of the local feature branch is:

[0068] The feature segmentation layer segments the first feature map of the input and the second feature map of the input into feature blocks, expressed as:

[0069]

[0070] in: is the feature block of the first feature map of the input at the same spatial position of j, split(·) is the segmentation operation, is the first feature map of the input, is the feature block of the second feature map of the input at the same spatial position of j, is the second feature map of the input.

[0071] The deformable cross sampling fusion submodule performs local multispectral feature fusion on the feature blocks of the input first feature map and the feature blocks of the input second feature map to obtain the local fusion features at the same spatial position, which is expressed as:

[0072]

[0073] in: is the local fusion feature of the jth same spatial position, DFB(·) is the deformable cross sampling fusion submodule,

[0074] The feature reconstruction layer reconstructs the local fusion features of the same spatial position according to the initial spatial position to obtain the local fusion features, which is expressed as:

[0075]

[0076] in: is the local fusion feature output by the local feature branch, and rebulid(·) is the feature map reconstruction operation.

[0077] like Figure 3 As shown, the deformable cross sampling fusion submodule includes two deformable cross sampling branches and a fusion feature layer that is simultaneously connected to the output ends of the two deformable cross sampling branches; the deformable cross sampling branch includes an input layer, a reference point setting layer, a position offset sub-branch, an overlay layer, a cross sampling layer, a splicing layer, and a self-attention layer; the input layers of the deformable cross sampling branches are all connected to the cross sampling layer in another deformable cross sampling branch.

[0078] The reference point setting layer generates a set of equidistant grid reference points based on the input feature map. Specifically, the grid size is determined by the coefficient α, where H G =H / α,W G =W / α.

[0079] The data processing process of the position offset sub-branch is expressed as:

[0080]

[0081] Where: Δp is the position offset, Conv1×1 is a 1×1 convolution operation, GELU(·) is the GELU activation function, DWConv(·) is the depth convolution layer, is the input feature map.

[0082] The data processing process of the overlay layer is expressed as:

[0083] p D =p+Δp

[0084] Where: p DThe deformable points output for the overlay.

[0085] The data processing process of the cross sampling layer is expressed as:

[0086]

[0087] in: It is the sampled features output by the cross sampling layer.

[0088] Specifically, in order to promote the interaction between the two features, the present invention uses deformable points in the branch of input 1 respectively. and From the input feature map Get the sampled features and In the input 2 branch, the sampled features are obtained in the same way and The above sampling operation is expressed as:

[0089]

[0090] in: The feature map input in input 1 And the sampled features of the deformable points output in input 1 through the cross sampling layer, is the feature map input in input 1, The feature map input in input 1 The deformable points output by the overlay layer, The feature map input in input 2 And the sampled features of the deformable points output from input 2 through the cross sampling layer, is the feature map input in input 2, The feature map input in input 2 The deformable points output by the overlay layer, The feature map input in input 1 And the sampled features of the deformable points output from input 2 through the cross sampling layer, The feature map input in input 2 And the sampled features of the deformable points output in input 1 through the cross sampling layer.

[0091] The concatenation layer concatenates the sampled features obtained by the above sampling operations to obtain the deformation fusion attention map. and It is expressed as:

[0092]

[0093] in: is the deformable fusion attention map obtained from input 1, is the deformable fused attention map obtained from input 2, and (·:·) is the concatenation operation.

[0094] The data processing process of the self-attention layer is:

[0095]

[0096]

[0097] Where: q is the feature map of the input after learning, is the input feature map, W q is the first learnable parameter, k is the first deformable fusion attention map after learning, is a deformable fusion attention map, including the deformable fusion attention map obtained in input 1 and the deformable fusion attention map obtained in input 2, W k is the second learnable parameter, v is the second deformable fusion attention map after learning, W v is the third learnable parameter, M is the self-attention weight, softmax(·) is the softmax activation function, T is the transposed sign, C is the number of channels, h is the number of self-attention heads, is the deformable fused attention enhanced feature map, LN(·) is the normalization layer, and MLP(·) is the multi-layer perceptron.

[0098] S4, fusing the cross-modal shared fusion feature map in step S2 and the first fusion feature map, the second fusion feature map and the third fusion feature map in step S3 to obtain a final multi-scale representation map, and inputting the final multi-scale representation map into the target detector to obtain a multi-spectral target type recognition result.

[0099] In an optional embodiment of the present invention, the target detector in the present invention adopts the detection head of the YOLOv5 model, and the present invention inputs the final multi-scale representation map into the detection head of the YOLOv5 model to obtain a multi-spectral target type recognition result.

[0100] In order to evaluate the performance of the proposed multispectral target type recognition method based on deformable fusion, a comparative experiment was carried out on the baseline model and the present model using a self-built data set. The experimental results are shown in Table 1. The present method is improved by 22.1% compared with the baseline model, which shows the effectiveness and advancement of the proposed method.

[0101] Table 1 Comparison of experimental results of this method on self-built virtual dataset

[0102]

[0103] In summary, the present invention obtains the shared and modal characteristics of different spectral images, explores key areas, and performs cross-sampling and fusion, thereby improving the effective information of target features and enhancing their distinguishability, thereby enhancing the accuracy of multi-spectral target type recognition.

[0104] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0107] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0108] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A multispectral target type recognition method based on deformable fusion, characterized in that: The following steps are involved: S1. Generate a first shared feature map and a first modal characteristic feature map based on a visible spectrum target image, and generate a second shared feature map and a second modal characteristic feature map based on an infrared spectrum target image; S2, integrating the first shared feature map and the second shared feature map in step S1 based on a probability distribution fusion strategy to obtain a cross-modal shared fusion feature map; S3, constructing a multi-view integration module based on variability fusion, integrating the first shared feature map and the first modal characteristic feature map in step S1 based on the multi-view integration module to obtain a first fused feature map, integrating the second shared feature map and the second modal characteristic feature map in step S1 based on the multi-view integration module to obtain a second fused feature map, and integrating the first modal characteristic feature map and the second modal characteristic feature map in step S1 based on the multi-view integration module to obtain a third fused feature map; S4, fusing the cross-modal shared fusion feature map in step S2 and the first fusion feature map, the second fusion feature map and the third fusion feature map in step S3 to obtain a final multi-scale representation map, and inputting the final multi-scale representation map into the target detector to obtain a multi-spectral target type recognition result.

2. The method for multispectral target type recognition based on deformable fusion according to claim 1 is characterized in that: In step S1, an orthogonal constraint loss minimization module is set between the first shared feature map and the first modal characteristic feature map, and between the second shared feature map and the second modal characteristic feature map; a Jensen–Shannon divergence loss minimization module is set between the first shared feature map and the second shared feature map.

3. The method for multi-spectral target type recognition based on deformable fusion according to claim 2 is characterized in that: The data processing process of minimizing the Jensen–Shannon divergence loss module includes the following steps: A1. Project the first shared feature map and the second shared feature map into the probability space, expressed as: Where: P is the first shared feature map after projection to the probability space, σ(·) is the standard distribution function that maps the feature to the probability space, is the first shared feature map, Q is the second shared feature map after projection to the probability space, is the second shared feature map; A2. Based on the first shared feature map and the second shared feature map projected to the probability space in step A1, the Jensen–Shannon divergence loss is calculated and minimized. The Jensen–Shannon divergence loss is calculated and expressed as: M=(P+Q) / 2 in: is the Jensen–Shannon divergence loss, D KL is the Kullback-Leibler divergence, and M is the average probability distribution of the first shared feature map and the second shared feature map after being projected into the probability space.

4. The method for multi-spectral target type recognition based on deformable fusion according to claim 1 is characterized in that: In step S2, the first shared feature map and the second shared feature map in step S1 are integrated based on the probability distribution fusion strategy to obtain a cross-modal shared fusion feature map, which is expressed as: in: To share the fusion feature map across modalities, is the first shared feature map, is the second shared feature map.

5. The method for multi-spectral target type recognition based on deformable fusion according to claim 1 is characterized in that: In step S3, the multi-view integration module includes a global feature branch, a local feature branch, and a concatenation dimension reduction layer that simultaneously connects the output end of the global feature branch and the output end of the local feature branch.

6. The method for multi-spectral target type recognition based on deformable fusion according to claim 5 is characterized in that: The global feature branch includes the input layer and the deformable cross sampling fusion submodule connected in sequence; the data processing process of the global feature branch is expressed as: in: is the global fusion feature output by the global feature branch, DFB(·) is the deformable cross sampling fusion submodule, is the first feature map of the input, is the second feature map of the input.

7. The method for multi-spectral target type recognition based on deformable fusion according to claim 5 is characterized in that: The local feature branch includes the input layer, feature slicing layer, deformable cross sampling fusion submodule and feature reconstruction layer connected in sequence. The data processing process of the local feature branch is as follows: The feature segmentation layer segments the first feature map of the input and the second feature map of the input into feature blocks, expressed as: in: is the feature block of the first feature map of the input at the same spatial position of j, split(·) is the segmentation operation, is the first feature map of the input, is the feature block of the second feature map of the input at the same spatial position of j, is the second feature map of the input; The deformable cross sampling fusion submodule performs local multispectral feature fusion on the feature blocks of the input first feature map and the feature blocks of the input second feature map to obtain the local fusion features at the same spatial position, which is expressed as: in: is the local fusion feature of the same spatial position of the jth node, DFB(·) is the deformable cross sampling fusion submodule; The feature reconstruction layer reconstructs the local fusion features of the same spatial position according to the initial spatial position to obtain the local fusion features, which is expressed as: in: is the local fusion feature output by the local feature branch, and rebulid(·) is the feature map reconstruction operation.

8. The method for multi-spectral target type recognition based on deformable fusion according to claim 6 is characterized in that: The deformable cross sampling fusion submodule includes two deformable cross sampling branches and a fusion feature layer simultaneously connecting the output ends of the two deformable cross sampling branches; The deformable cross sampling branch includes an input layer, a reference point setting layer, a position offset sub-branch, an overlay layer, a cross sampling layer, a splicing layer and a self-attention layer; the input layer of the deformable cross sampling branch is connected to the cross sampling layer in another deformable cross sampling branch.

9. The method for multi-spectral target type recognition based on deformable fusion according to claim 8 is characterized in that: The data processing process of the position offset sub-branch is expressed as: Where: Δp is the position offset, Conv1×1 is a 1×1 convolution operation, GELU(·) is the GELU activation function, DWConv(·) is the depth convolution layer, is the input feature map.

10. The method for multi-spectral target type recognition based on deformable fusion according to claim 8, characterized in that: The data processing process of the self-attention layer is: in: is the deformable fusion attention enhanced feature map, M is the self-attention weight, v is the second deformable fusion attention map after learning, is the input feature map, LN(·) is the normalization layer, and MLP(·) is the multi-layer perceptron.

Images