A multispectral target type recognition method based on deformable fusion

Abstract

The present invention discloses a method for multispectral target type recognition based on deformable fusion. The method comprises the following steps: generating a first shared feature map and a first modal characteristic feature map based on a visible spectrum target image, and generating a second shared feature map and a second modal characteristic feature map based on an infrared spectrum target image; obtaining a cross-modal shared fusion feature map; constructing a multi-view integration module based on variability fusion, and obtaining a first fusion feature map, a second fusion feature map, and a third fusion feature map based on the multi-view integration module; obtaining a final multi-scale representation map, and inputting the final multi-scale representation map into a target detector to obtain a multispectral target type recognition result. The present invention can obtain shared and modal characteristic features of different spectral images, explore key areas, and perform cross-sampling and fusion, thereby improving the effective information of target features and enhancing their distinguishability, thereby enhancing the accuracy of multispectral target type recognition.

Description

Technical Field

[0001] The present invention relates to the field of multispectral target type recognition, and in particular to a multispectral target type recognition method based on deformable fusion. Background Art

[0002] Existing multispectral target type recognition methods have obvious shortcomings, which are mainly reflected in the following aspects: First, these methods are not effective when fusing multispectral data, and often cannot fully utilize the information complementarity between different spectra. Secondly, the lack of effective identification and cross-fusion of key areas in each spectrum makes it difficult to fully extract and utilize important features. The fundamental reason for these shortcomings is that the existing methods have a relatively simple processing mechanism for multispectral data and are difficult to adapt to complex target recognition needs. In addition, existing methods often rely on traditional feature extraction technology, resulting in limited processing capabilities for multidimensional data and difficulty in capturing subtle differences and changes in targets. Therefore, the final recognition results are often not accurate enough, affecting the effectiveness and reliability of multispectral target type recognition in practical applications. Summary of the Invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multispectral target type recognition method based on deformable fusion.

[0004] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0005] A multispectral target type recognition method based on deformable fusion includes the following steps:

[0006] S1. Generate a first shared feature map and a first modal characteristic feature map based on a visible spectrum target image, and generate a second shared feature map and a second modal characteristic feature map based on an infrared spectrum target image;

[0007] S2. Integrate the first shared feature map and the second shared feature map in step S1 based on a probability distribution fusion strategy to obtain a cross-modal shared fusion feature map;

[0008] S3. Constructing a multi-view integration module based on variability fusion, integrating the first shared feature map and the first modal characteristic feature map in step S1 based on the multi-view integration module to obtain a first fused feature map, integrating the second shared feature map and the second modal characteristic feature map in step S1 based on the multi-view integration module to obtain a second fused feature map, and integrating the first modal characteristic feature map and the second modal characteristic feature map in step S1 based on the multi-view integration module to obtain a third fused feature map;

[0009] S4. Fuse the cross-modal shared fusion feature map in step S2 and the first fusion feature map, the second fusion feature map, and the third fusion feature map in step S3 to obtain a final multi-scale representation map, and input the final multi-scale representation map into the target detector to obtain a multispectral target type recognition result.

[0010] Furthermore, in step S1, an orthogonality constraint loss minimization module is set between the first shared feature map and the first modal characteristic feature map, and between the second shared feature map and the second modal characteristic feature map; a Jensen–Shannon divergence loss minimization module is set between the first shared feature map and the second shared feature map.

[0011] Furthermore, the data processing process of the Jensen–Shannon divergence loss minimization module includes the following steps:

[0012] A1. Project the first shared feature map and the second shared feature map into the probability space, expressed as:

[0013]

[0014] Where: P is the first shared feature map after projection to the probability space, σ(·) is the standard distribution function that maps the feature to the probability space, is the first shared feature map, Q is the second shared feature map after projection to the probability space, is the second shared feature map, H, W, and C are the height, width, and number of channels of the feature map, respectively;

[0015] A2. Based on the first shared feature map and the second shared feature map after being projected into the probability space in step A1, the Jensen–Shannon divergence loss is calculated and minimized. The Jensen–Shannon divergence loss is calculated and expressed as:

[0016]

[0017] in: is the Jensen–Shannon divergence loss, D KL is the Kullback-Leibler divergence, and M is the average probability distribution of the first shared feature map and the second shared feature map after projection into the probability space.

[0018] Furthermore, in step S2, the first shared feature map and the second shared feature map in step S1 are integrated based on the probability distribution fusion strategy to obtain a cross-modal shared fusion feature map, which is expressed as:

[0019]

[0020] in: To share the fusion feature map across modalities, is the first shared feature map, is the second shared feature map.

[0021] Furthermore, in step S3, the multi-view integration module includes a global feature branch, a local feature branch, and a splicing dimensionality reduction layer that simultaneously connects the output end of the global feature branch and the output end of the local feature branch.

[0022] Furthermore, the global feature branch includes an input layer and a deformable cross-sampling fusion submodule connected in sequence; the data processing process of the global feature branch is expressed as:

[0023]

[0024] in: is the global fusion feature output by the global feature branch, DFB(·) is the deformable cross sampling fusion submodule, is the first feature map of the input, is the second feature map of the input.

[0025] Furthermore, the local feature branch includes an input layer, a feature slicing layer, a deformable cross-sampling fusion submodule, and a feature reconstruction layer connected in sequence. The data processing process of the local feature branch is as follows:

[0026] The feature segmentation layer divides the first feature map of the input and the second feature map of the input into feature blocks, which are expressed as:

[0027]

[0028] in: is the feature block at the same spatial position j of the first feature map of the input, split(·) is the segmentation operation, is the first feature map of the input, is the feature block of the second feature map of the input at the same spatial position j, is the second feature map of the input;

[0029] The deformable cross sampling fusion submodule performs local multispectral feature fusion on the feature blocks of the input first feature map and the feature blocks of the input second feature map to obtain the local fusion features at the same spatial position, which is expressed as:

[0030]

[0031] in: is the local fusion feature of the jth same spatial position, DFB(·) is the deformable cross sampling fusion submodule;

[0032] The feature reconstruction layer reconstructs the local fusion features of the same spatial position according to the initial spatial position to obtain the local fusion features, which can be expressed as:

[0033]

[0034] in: is the local fusion feature output by the local feature branch, and rebulid(·) is the feature map reconstruction operation.

[0035] Furthermore, the deformable cross sampling fusion submodule includes two deformable cross sampling branches and a fusion feature layer that is simultaneously connected to the output ends of the two deformable cross sampling branches; the deformable cross sampling branch includes an input layer, a reference point setting layer, a position offset sub-branch, an overlay layer, a cross sampling layer, a splicing layer and a self-attention layer; the input layers of the deformable cross sampling branches are all connected to the cross sampling layer in another deformable cross sampling branch.

[0036] Furthermore, the data processing process of the position offset sub-branch is expressed as:

[0037]

[0038] Where: Δp is the position offset, Conv1×1 is a 1×1 convolution operation, GELU(·) is the GELU activation function, DWConv(·) is the depth convolution layer, is the input feature map.

[0039] Furthermore, the data processing process of the self-attention layer is:

[0040]

[0041] in: is the deformable fusion attention enhanced feature map, M is the self-attention weight, v is the second deformable fusion attention map after learning, is the input feature map, LN(·) is the normalization layer, MLP(·) is the multi-layer perceptron, and v is the second deformable fusion attention map after learning.

[0042] The present invention has the following beneficial effects:

[0043] The present invention integrates the first shared feature map and the second shared feature map to obtain a cross-modal shared fusion feature map, then constructs a multi-view integration module based on variability fusion, and then integrates the first shared feature map and the first modal characteristic feature map to obtain a first fusion feature map, integrates the second shared feature map and the second modal characteristic feature map to obtain a second fusion feature map, integrates the first modal characteristic feature map and the second modal characteristic feature map to obtain a third fusion feature map, and then fuses the cross-modal shared fusion feature map, the first fusion feature map, the second fusion feature map and the third fusion feature map to obtain a final multi-scale representation map, and inputs the final multi-scale representation map into the target detector to obtain a multispectral target type recognition result. The whole process can obtain the shared and modal characteristic features of different spectral images, explore key areas and perform cross-sampling and fusion, improve the effective information of target features and enhance their distinguishability, thereby enhancing the accuracy of multispectral target type recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of a multispectral target type recognition method based on deformable fusion;

[0045] Figure 2 This is a schematic diagram of the data processing flow of the multi-view fusion module;

[0046] Figure 3 Schematic diagram of the data processing flow of the deformable cross-sampling fusion sub-module. DETAILED DESCRIPTION

[0047] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0048] like Figure 1 As shown, a multispectral target type recognition method based on deformable fusion includes steps S1-S4, which are specifically as follows:

[0049] S1. Generate a first shared feature map and a first modal characteristic feature map based on a visible spectrum target image, and generate a second shared feature map and a second modal characteristic feature map based on an infrared spectrum target image.

[0050] In an optional embodiment of the present invention, an orthogonal constraint loss minimization module is set between the first shared feature map and the first modal characteristic feature map, and between the second shared feature map and the second modal characteristic feature map; and a Jensen–Shannon divergence loss minimization module is set between the first shared feature map and the second shared feature map.

[0051] The data processing process of the module minimizing the Jensen–Shannon divergence loss includes the following steps:

[0052] A1. Project the first shared feature map and the second shared feature map into the probability space, expressed as:

[0053]

[0054] Where: P is the first shared feature map after projection to the probability space, σ(·) is the standard distribution function that maps the feature to the probability space, is the first shared feature map, Q is the second shared feature map after projection to the probability space, is the second shared feature map.

[0055] A2. Based on the first shared feature map and the second shared feature map after being projected into the probability space in step A1, the Jensen–Shannon divergence loss is calculated and minimized. The Jensen–Shannon divergence loss is calculated and expressed as:

[0056]

[0057] in: is the Jensen–Shannon divergence loss, D KL is the Kullback-Leibler divergence, and M is the average probability distribution of the first shared feature map and the second shared feature map after projection into the probability space.

[0058] S2. Integrate the first shared feature map and the second shared feature map in step S1 based on a probability distribution fusion strategy to obtain a cross-modal shared fusion feature map.

[0059] In an optional embodiment of the present invention, the present invention integrates the first shared feature map and the second shared feature map in step S1 based on a probability distribution fusion strategy to obtain a cross-modal shared fusion feature map, which is expressed as:

[0060]

[0061] in: To share the fusion feature map across modalities, is the first shared feature map, is the second shared feature map.

[0062] S3. Construct a multi-view integration module based on variability fusion, integrate the first shared feature map and the first modal characteristic feature map in step S1 based on the multi-view integration module to obtain a first fused feature map, integrate the second shared feature map and the second modal characteristic feature map in step S1 based on the multi-view integration module to obtain a second fused feature map, and integrate the first modal characteristic feature map and the second modal characteristic feature map in step S1 based on the multi-view integration module to obtain a third fused feature map.

[0063] In an optional embodiment of the present invention, the multi-view integration module includes a global feature branch, a local feature branch, and a splicing dimensionality reduction layer that simultaneously connects the output end of the global feature branch and the output end of the local feature branch.

[0064] like Figure 2 As shown in Figure 1, input 1 is the input layer, DFB is the deformable cross sampling fusion submodule, and the global feature branch includes the input layer and the deformable cross sampling fusion submodule connected in sequence. The data processing process of the global feature branch is expressed as follows:

[0065]

[0066] in: is the global fusion feature output by the global feature branch, DFB(·) is the deformable cross sampling fusion submodule, is the first feature map of the input, is the second feature map of the input.

[0067] like Figure 2 As shown in Figure 2, input 2 is the input layer. The local feature branch includes the input layer, feature slicing layer, deformable cross sampling fusion submodule and feature reconstruction layer connected in sequence. The data processing process of the local feature branch is as follows:

[0068] The feature segmentation layer divides the first feature map of the input and the second feature map of the input into feature blocks, which are expressed as:

[0069]

[0070] in: is the feature block at the same spatial position j of the first feature map of the input, split(·) is the segmentation operation, is the first feature map of the input, is the feature block of the second feature map of the input at the same spatial position j, is the second feature map of the input.

[0071] The deformable cross sampling fusion submodule performs local multispectral feature fusion on the feature blocks of the input first feature map and the feature blocks of the input second feature map to obtain the local fusion features at the same spatial position, which is expressed as:

[0072]

[0073] in: is the local fusion feature of the jth same spatial position, DFB(·) is the deformable cross sampling fusion submodule,

[0074] The feature reconstruction layer reconstructs the local fusion features of the same spatial position according to the initial spatial position to obtain the local fusion features, which can be expressed as:

[0075]

[0076] in: is the local fusion feature output by the local feature branch, and rebulid(·) is the feature map reconstruction operation.

[0077] like Figure 3 As shown, the deformable cross sampling fusion submodule includes two deformable cross sampling branches and a fusion feature layer that is simultaneously connected to the output ends of the two deformable cross sampling branches; the deformable cross sampling branch includes an input layer, a reference point setting layer, a position offset sub-branch, an overlay layer, a cross sampling layer, a splicing layer, and a self-attention layer; the input layers of the deformable cross sampling branches are all connected to the cross sampling layer in another deformable cross sampling branch.

[0078] The reference point setting layer generates a set of equidistant grid reference points based on the input feature map Specifically, the grid size is determined by the coefficient α, where H G =H / α,W G =W / α.

[0079] The data processing process of the position offset sub-branch is expressed as:

[0080]

[0081] Where: Δp is the position offset, Conv1×1 is a 1×1 convolution operation, GELU(·) is the GELU activation function, DWConv(·) is the depth convolution layer, is the input feature map.

[0082] The data processing process of the overlay layer is expressed as:

[0083] p D =p+Δp

[0084] Where: p DOutputs the deformable points for the overlay layer.

[0085] The data processing process of the cross sampling layer is expressed as:

[0086]

[0087] in: It is the sampled feature output by the cross sampling layer.

[0088] Specifically, in order to promote the interaction between the two features, the present invention uses deformable points in the branch of input 1. and From the input feature map Get the sampled features and In the input 2 branch, the sampled features are obtained in the same way and The above sampling operation is expressed as:

[0089]

[0090] in: The feature map input in input 1 And the features of the deformable points output in input 1 after sampling by the cross sampling layer, is the feature map input in input 1, The feature map input in input 1 The deformable points output by the overlay layer, The feature map input in input 2 And the features of the deformable points output in input 2 after sampling by the cross sampling layer, is the feature map input in input 2, The feature map input in input 2 The deformable points output by the overlay layer, The feature map input in input 1 And the features of the deformable points output in input 2 after sampling by the cross sampling layer, The feature map input in input 2 And the features of the deformable points output in input 1 after sampling by the cross sampling layer.

[0091] The splicing layer splices the sample features obtained by the above sampling operation to obtain the deformation fusion attention map and Expressed as:

[0092]

[0093] in: is the deformable fusion attention map obtained from input 1, is the deformable fusion attention map obtained from input 2, and (·:·) is the concatenation operation.

[0094] The data processing process of the self-attention layer is:

[0095]

[0096]

[0097] Where: q is the feature map of the input after learning, is the input feature map, W q is the first learnable parameter, k is the first deformable fusion attention map after learning, is a deformable fusion attention map, including the deformable fusion attention map obtained in input 1 and the deformable fusion attention map obtained in input 2, W k is the second learnable parameter, v is the second deformable fusion attention map after learning, W v is the third learnable parameter, M is the self-attention weight, softmax(·) is the softmax activation function, T is the transposed sign, C is the number of channels, h is the number of self-attention heads, is the deformable fusion attention enhanced feature map, LN(·) is the normalization layer, and MLP(·) is the multi-layer perceptron.

[0098] S4. Fuse the cross-modal shared fusion feature map in step S2 and the first fusion feature map, the second fusion feature map, and the third fusion feature map in step S3 to obtain a final multi-scale representation map, and input the final multi-scale representation map into the target detector to obtain a multispectral target type recognition result.

[0099] In an optional embodiment of the present invention, the target detector in the present invention adopts the detection head of the YOLOv5 model. The present invention inputs the final multi-scale representation map into the detection head of the YOLOv5 model to obtain a multispectral target type recognition result.

[0100] In order to evaluate the performance of the proposed multispectral target type recognition method based on deformable fusion, a comparative experiment was conducted on the baseline model and the proposed model using a self-built dataset. The experimental results are shown in Table 1. The proposed method improves by 22.1% compared with the baseline model, demonstrating the effectiveness and advancement of the proposed method.

[0101] Table 1 Comparison of experimental results of this method on self-built virtual dataset

[0102]

[0103] In summary, the present invention obtains the shared and modal characteristics of different spectral images, explores key areas, and performs cross-sampling and fusion, thereby improving the effective information of target features and enhancing their distinguishability, thereby enhancing the accuracy of multispectral target type recognition.

[0104] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0107] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

[0108] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A multispectral target type recognition method based on deformable fusion, characterized in that: The following steps are involved: S1. Generate a first shared feature map and a first modal characteristic feature map based on a visible spectrum target image, and generate a second shared feature map and a second modal characteristic feature map based on an infrared spectrum target image; S2. Integrate the first shared feature map and the second shared feature map in step S1 based on a probability distribution fusion strategy to obtain a cross-modal shared fusion feature map; S3. Constructing a multi-view integration module based on variability fusion, integrating the first shared feature map and the first modal characteristic feature map in step S1 based on the multi-view integration module to obtain a first fused feature map, integrating the second shared feature map and the second modal characteristic feature map in step S1 based on the multi-view integration module to obtain a second fused feature map, and integrating the first modal characteristic feature map and the second modal characteristic feature map in step S1 based on the multi-view integration module to obtain a third fused feature map; S4. Fuse the cross-modal shared fusion feature map in step S2 and the first fusion feature map, the second fusion feature map, and the third fusion feature map in step S3 to obtain a final multi-scale representation map, and input the final multi-scale representation map into the target detector to obtain a multispectral target type recognition result.

2. The multispectral target type recognition method based on deformable fusion according to claim 1 is characterized in that: In step S1, an orthogonal constraint loss minimization module is set between the first shared feature map and the first modal characteristic feature map, and between the second shared feature map and the second modal characteristic feature map; a Jensen–Shannon divergence loss minimization module is set between the first shared feature map and the second shared feature map.

3. The multispectral target type recognition method based on deformable fusion according to claim 2 is characterized in that: The data processing process of the module minimizing the Jensen–Shannon divergence loss includes the following steps: A1. Project the first shared feature map and the second shared feature map into the probability space, expressed as: Where: P is the first shared feature map after projection to the probability space, σ(·) is the standard distribution function that maps the feature to the probability space, is the first shared feature map, Q is the second shared feature map after projection to the probability space, is the second shared feature map; A2. Based on the first shared feature map and the second shared feature map after being projected into the probability space in step A1, the Jensen–Shannon divergence loss is calculated and minimized. The Jensen–Shannon divergence loss is calculated and expressed as: M=(P+Q) / 2 in: is the Jensen–Shannon divergence loss, D KL is the Kullback-Leibler divergence, and M is the average probability distribution of the first shared feature map and the second shared feature map after projection into the probability space.

4. The multispectral target type recognition method based on deformable fusion according to claim 1 is characterized in that: In step S2, the first shared feature map and the second shared feature map in step S1 are integrated based on the probability distribution fusion strategy to obtain a cross-modal shared fusion feature map, which is expressed as: in: To share the fusion feature map across modalities, is the first shared feature map, is the second shared feature map.

5. The multispectral target type recognition method based on deformable fusion according to claim 1 is characterized in that: In step S3, the multi-view integration module includes a global feature branch, a local feature branch, and a splicing dimension reduction layer that simultaneously connects the output end of the global feature branch and the output end of the local feature branch.

6. The multispectral target type recognition method based on deformable fusion according to claim 5 is characterized in that: The global feature branch includes the sequentially connected input layer and the deformable cross-sampling fusion submodule; the data processing process of the global feature branch is expressed as: in: is the global fusion feature output by the global feature branch, DFB(·) is the deformable cross sampling fusion submodule, is the first feature map of the input, is the second feature map of the input.

7. The multispectral target type recognition method based on deformable fusion according to claim 5 is characterized in that: The local feature branch includes the sequentially connected input layer, feature slicing layer, deformable cross sampling fusion submodule and feature reconstruction layer. The data processing process of the local feature branch is as follows: The feature segmentation layer divides the first feature map of the input and the second feature map of the input into feature blocks, which are expressed as: in: is the feature block at the same spatial position j of the first feature map of the input, split(·) is the segmentation operation, is the first feature map of the input, is the feature block of the second feature map of the input at the same spatial position j, is the second feature map of the input; The deformable cross sampling fusion submodule performs local multispectral feature fusion on the feature blocks of the input first feature map and the feature blocks of the input second feature map to obtain the local fusion features at the same spatial position, which is expressed as: in: is the local fusion feature of the jth same spatial position, DFB(·) is the deformable cross sampling fusion submodule; The feature reconstruction layer reconstructs the local fusion features of the same spatial position according to the initial spatial position to obtain the local fusion features, which can be expressed as: in: is the local fusion feature output by the local feature branch, and rebulid(·) is the feature map reconstruction operation.

8. The multispectral target type recognition method based on deformable fusion according to claim 6 is characterized in that: The deformable cross sampling fusion submodule includes two deformable cross sampling branches and a fusion feature layer that simultaneously connects the output ends of the two deformable cross sampling branches; The deformable cross sampling branch includes an input layer, a reference point setting layer, a position offset sub-branch, an overlay layer, a cross sampling layer, a splicing layer, and a self-attention layer; the input layer of the deformable cross sampling branch is connected to the cross sampling layer in another deformable cross sampling branch.

9. The multispectral target type recognition method based on deformable fusion according to claim 8 is characterized in that: The data processing process of the position offset sub-branch is expressed as: Where: Δp is the position offset, Conv1×1 is a 1×1 convolution operation, GELU(·) is the GELU activation function, DWConv(·) is the depth convolution layer, is the input feature map.

10. The multispectral target type recognition method based on deformable fusion according to claim 8, characterized in that: The data processing process of the self-attention layer is: in: is the deformable fusion attention enhanced feature map, M is the self-attention weight, v is the second deformable fusion attention map after learning, is the input feature map, LN(·) is the normalization layer, and MLP(·) is the multi-layer perceptron.

Images