A multi-spectral target type recognition method based on multi-dimension fusion

Abstract

This invention relates to the field of target recognition technology and discloses a multispectral target type recognition method based on multidimensional fusion. The method includes: acquiring multimodal image data; constructing a multispectral target type recognition model; extracting features from the multimodal image data using a multimodal feature extraction module to obtain multi-scale feature maps representing different modalities; fusing and interacting the multi-scale feature maps representing different modalities at different depths using a cross-modal local and global fusion module to obtain a single-modal feature map that fuses local and global information; fusing the single-modal feature map that fuses local and global information using a cross-modal channel fusion module to obtain a multimodal channel fusion feature map; and fusing the multimodal channel fusion feature map using a detection module to obtain a multispectral target detection result. This invention significantly improves the effectiveness of target recognition and detection.

Description

Technical Field

[0001] This invention relates to the field of target recognition technology, and specifically to a multispectral target type recognition method based on multidimensional fusion. Background Technology

[0002] Because the target moves at high speed, it cannot be correctly identified by relying solely on ordinary optical images. Therefore, it is necessary to rely on multimodal information to detect the target from multiple dimensions and aspects. Summary of the Invention

[0003] To address the aforementioned shortcomings in the existing technology, this invention provides a multispectral target type identification method based on multidimensional fusion.

[0004] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0005] A multispectral target type identification method based on multidimensional fusion includes the following steps:

[0006] Acquire multimodal image data;

[0007] A multispectral target type recognition model is constructed; the multispectral target type recognition model includes a multimodal feature extraction module, a cross-modal local and global fusion module, a cross-modal channel fusion module, and a detection module;

[0008] The multimodal feature extraction module is used to extract features from the multimodal image data to obtain multi-scale feature maps representing different modalities.

[0009] By utilizing the cross-modal local and global fusion module, multi-scale feature maps representing different modalities are fused and interacted at different depths using spatial dimensions to obtain single-modal feature maps that fuse local and global information.

[0010] The cross-modal channel fusion module is used to perform feature fusion on the single-modal feature map that fuses local and global information to obtain a multimodal channel fused feature map;

[0011] The detection module is used to perform multi-scale feature map fusion on the multimodal channel fusion feature map to obtain multispectral target detection results.

[0012] Preferably, the multimodal image data includes:

[0013] RGB images, infrared images, and plasma images.

[0014] Preferably, the multimodal feature extraction module includes:

[0015] The backbone network of the YOLOv5 model.

[0016] Preferably, the cross-modal local and global fusion module includes:

[0017] Cross-modal local and global fusion sub-modules corresponding to each level of the multimodal feature extraction module;

[0018] The cross-modal local and global fusion submodule includes multiple cross-modal local fusion units and cross-modal global fusion units;

[0019] The cross-modal local fusion unit is used to extract cross-modal local fusion feature maps from the corresponding scale feature maps of the current modality representation and the corresponding scale feature maps of other modality representations;

[0020] The cross-modal global fusion unit is used to extract cross-modal global fusion feature maps from the corresponding scale feature maps of the current modality representation and the corresponding scale feature maps of other modality representations, and to concatenate them with the cross-modal local fusion feature maps extracted by the cross-modal local fusion unit to obtain a single-modal feature map that fuses local and global information.

[0021] Preferably, the cross-modal local fusion unit first segments the corresponding scale feature map of the current modality representation and the corresponding scale feature maps of other modal representations into patch marker sequences through a first local fusion layer. Then, each patch marker sequence is segmented into local regions through a local window. Next, a window-based multi-head cross-attention fusion of local regions with the same spatial position in the multimodal representation is performed on each window pair. Then, a second local fusion layer moves the local window by a set number of pixels and segments the image in the local region of the new window. Next, a window-based multi-head cross-attention fusion of local regions with the same spatial position in the multimodal representation is performed on each new window pair. Finally, the patch marker sequence after the second fusion is reconstructed to obtain the cross-modal local fusion feature map.

[0022] Preferably, the cross-modal global fusion unit first segments the corresponding scale feature map of the current modality representation and the corresponding scale feature maps of other modality representations into patch label sequences, and adds two learnable global embedding labels before each patch label sequence; then, after exchanging the learnable global embedding labels of each patch label sequence, it interacts with the global information between the fusion modalities through the global fusion attention layer; then, after exchanging the learnable global embedding labels of each patch label sequence, it maps the learned fusion representation into the label sequence of its original modality through the self-attention layer; finally, it reconstructs the cross-modal global fusion feature map from the obtained fusion feature map.

[0023] Preferably, the cross-modal channel fusion module first concatenates the single-modal feature map that fuses local and global information, and then performs channel fusion through a convolutional layer. Then, the channel fusion feature map is divided into multimodal channel fusion attention maps along the channel dimension, and the attention maps are weighted and summed with the original feature maps of the corresponding modalities before being concatenated to obtain the multimodal channel fusion feature map.

[0024] The present invention has the following beneficial effects:

[0025] This invention uses multispectral information to supplement RGB images. By fusing the multispectral images of the target, a more comprehensive spatial representation of the target can be obtained, greatly improving the effectiveness of target recognition and detection. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of a multispectral target type identification method based on multidimensional fusion;

[0027] Figure 2 A schematic diagram of the multispectral target type identification model structure;

[0028] Figure 3 A schematic diagram of the framework for a multispectral target type identification model;

[0029] Figure 4 This is a schematic diagram of a cross-modal local fusion unit structure;

[0030] Figure 5 This is a schematic diagram of the cross-modal global fusion unit structure;

[0031] Figure 6 This is a schematic diagram of the cross-modal channel fusion module structure;

[0032] Figure 7 A visual representation of the results of cone-shaped target recognition;

[0033] Figure 8 A visual representation of the results of cylindrical target recognition;

[0034] Figure 9 A visual representation of the recognition results for elliptical targets;

[0035] Figure 10 A visual diagram illustrating the results of conventional target recognition. Detailed Implementation

[0036] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0037] like Figure 1 As shown, this embodiment of the invention provides a multispectral target type identification method based on multidimensional fusion, including the following steps S1 to S6:

[0038] S1. Acquire multimodal image data;

[0039] S2. Construct a multispectral target type recognition model; the multispectral target type recognition model includes a multimodal feature extraction module, a cross-modal local and global fusion module, a cross-modal channel fusion module, and a detection module;

[0040] S3. Use the multimodal feature extraction module to extract features from the multimodal image data to obtain multi-scale feature maps representing different modalities;

[0041] S4. Utilize the cross-modal local and global fusion module to perform multi-dimensional cross-modal feature fusion and interaction of multi-scale feature maps representing different modalities at different depths, and obtain a single-modal feature map that fuses local and global information.

[0042] S5. Use the cross-modal channel fusion module to perform feature fusion on the single-modal feature map that fuses local and global information to obtain a multimodal channel fused feature map;

[0043] S6. Use the detection module to perform multi-scale feature map fusion on the multi-modal channel fusion feature map to obtain multispectral target detection results.

[0044] The multispectral target type recognition model constructed in this embodiment is as follows: Figure 2 As shown, the system includes a multimodal feature extraction module, a cross-modal local and global fusion module, a cross-modal channel fusion module, and a detection module. RGB, infrared, and plasma modalities are input into the feature extraction module for learning their respective modal representations. During feature extraction, the feature maps obtained at each stage of different modalities are input into the cross-modal local and global fusion module for spatial dimension feature fusion and interaction to obtain single-modal features with both local and global information. Subsequently, the single-modal features that have undergone cross-modal local and global interaction are input into the cross-modal channel fusion module to obtain the final multimodal fusion features at that resolution. Finally, the fusion feature maps at different resolutions obtained at each stage are input into the detection head for multi-scale feature map fusion to obtain the detection and recognition results.

[0045] like Figure 3 The diagram shows the architecture of the multispectral target type recognition model constructed in this embodiment, comprising five main parts: an RGB feature learning branch, a multispectral feature learning branch, a cross-modal local fusion module (CMLF), a cross-modal global fusion module (CMGF), and a cross-modal channel fusion module (CMCF). This embodiment uses the YOLOv5 model as the backbone of two branches. During feature extraction, features from multiple modalities are fused and interacted across different network depths in a multidimensional cross-modal manner to integrate information from different modalities and complement each other. Given a pair of RGB images I... R and multispectral images I I They are input into the RGB and multispectral image branches respectively to extract representations. Features from both modalities are input into the CMLF and CMGF modules to obtain local and global fused feature maps. Then, the two types of fused features are summed and input into the CMCF module to obtain channel-level fused features. [1,3]. The final fusion feature M calculated using different scale representations. i The detection head is sent to the YOLOv5 model.

[0046] To more effectively fuse multispectral images and enhance the recognition accuracy of multimodal target detectors, this invention proposes a multidimensional fusion target detection method. Multispectral information is used to supplement RGB images; by fusing the multispectral images of the target, a more comprehensive spatial representation of the target can be obtained, significantly improving target recognition and detection performance. This invention fuses and extracts relationships from local, global, and channel information of different spectral images, enriching the final features of the target.

[0047] In an optional embodiment of the present invention, the multimodal image data includes:

[0048] RGB images, infrared images, and plasma images.

[0049] In an optional embodiment of the present invention, the multimodal feature extraction module includes:

[0050] The backbone network of the YOLOv5 model.

[0051] In an optional embodiment of the present invention, the cross-modal local and global fusion module includes:

[0052] Cross-modal local and global fusion sub-modules corresponding to each level of the multimodal feature extraction module;

[0053] The cross-modal local and global fusion submodule includes multiple cross-modal local fusion units and cross-modal global fusion units;

[0054] The cross-modal local fusion unit is used to extract cross-modal local fusion feature maps from the corresponding scale feature maps of the current modality representation and the corresponding scale feature maps of other modality representations;

[0055] The cross-modal global fusion unit is used to extract cross-modal global fusion feature maps from the corresponding scale feature maps of the current modality representation and the corresponding scale feature maps of other modality representations, and to concatenate them with the cross-modal local fusion feature maps extracted by the cross-modal local fusion unit to obtain a single-modal feature map that fuses local and global information.

[0056] In an optional embodiment of the present invention, the cross-modal local fusion unit first segments the corresponding scale feature map of the current modality representation and the corresponding scale feature maps of other modal representations into patch marker sequences through a first local fusion layer. Then, each patch marker sequence is segmented into local regions through a local window. Then, for each window pair, a window-based multi-head cross-attention fusion is performed to fuse local regions with the same spatial position in the multimodal representation. Then, through a second local fusion layer, the local window is moved by a set number of pixels, and the image is segmented in the local region of the new window. Then, for each new window pair, a window-based multi-head cross-attention fusion is performed to fuse local regions with the same spatial position in the multimodal representation. Finally, the patch marker sequence after the second fusion is reconstructed to obtain the cross-modal local fusion feature map.

[0057] To more accurately capture and fuse details from multispectral images, this study proposes a cross-modal local fusion module. In the first local window fusion layer, each modality can be divided into non-overlapping image patches by a fixed-size rectangular local window. The fusion process is as follows: Figure 4 As shown, the representations of the two modes are segmented into a sequence of patch markers. Each sequence is segmented into M×M local windows. A local area, in which This refers to the number of sequences. Then, for each window pair, window-based multi-head cross-attention (W-MCA) fusion is performed. Local regions with the same spatial location in both modalities. The input to the first local fusion layer in the CMLF module is defined as:

[0058] Q1 = P2W q

[0059] K1 = P2W k

[0060] V1 = P1W v

[0061] Among them, W q W k , and W v This represents the learnable parameters. The W-MCA fusion attention is calculated from two local features and can be represented as:

[0062]

[0063] Where d represents the dimensions of Q1 and K1. The output of each window pair in the first local fusion layer is represented as:

[0064] F W =A W +P1+P2+MLP(LN(A F +P1+P2))

[0065] Where LN(·) and MLP(·) represent layer normalization and multilayer perceptron, respectively.

[0066] To establish correlations between multimodal local information and enhance the fusion effect, the local window is moved in the next fusion layer. The image is segmented into pixels and local regions within the new window. The input to the second fusion layer is a new shifted window pair from the first fusion feature and another modality feature. The fusion process is the same as the first fusion layer. Each new window pair is fused using shifted window-based multi-head cross-attention (SW-MCA). The formula for the second fusion layer is expressed as follows:

[0067] Q2 = P2 ′ W q

[0068] K2 = P2 ′ W k

[0069] V2 = P1 ′ W v

[0070]

[0071] F SW =A SW +P′1+P′2+MLP(LN(A SW +P′1+P′2))

[0072] Finally, cross-modal local fusion feature maps are obtained by reconstructing the sequence after secondary fusion labeling.

[0073] In an optional embodiment of the present invention, the cross-modal global fusion unit first segments the corresponding scale feature map of the current modality representation and the corresponding scale feature maps of other modality representations into patch label sequences, and adds two learnable global embedding labels before each patch label sequence; then, after exchanging the learnable global embedding labels of each patch label sequence, it interacts with the global information between the fusion modalities through the global fusion attention layer; then, after exchanging the learnable global embedding labels of each patch label sequence, it maps the learned fusion representation into the label sequence of its original modality through the self-attention layer; finally, it reconstructs the cross-modal global fusion feature map from the obtained fusion feature map.

[0074] In the spatial dimension, the global perspective is an important complement to the local perspective. Therefore, this embodiment proposes a cross-modal global fusion module to learn global multispectral interaction information, such as... Figure 5 As shown, the feature maps of the two modalities are divided into patch-marked sequences. To learn the global features of the two modalities and enable interaction, this embodiment adds two learnable global embeddings before each patch tag sequence. Since feature maps lose global spatial location information during the decomposition into a sequence of individual vectors, this embodiment employs location embedding to address this issue. This is to obtain the necessary spatial location information. Therefore, the input label sequence for each modality is as follows:

[0075]

[0076] Its output can be described as follows:

[0077]

[0078] Among them, X * SA(·) represents the output of the first self-attention layer.

[0079] Subsequently, the learnable global embeddings of each sequence are swapped and fed into a global fusion attention layer to interact with and fuse global information between modalities. To reduce computational complexity, this method uses only the learnable global embeddings as q, and the new sequences as k and v. The computation process is as follows:

[0080]

[0081]

[0082] k = X I ′ W k

[0083] v = X I′ W v

[0084]

[0085] Where, X′ R and X′ I It is the new sequence after token swapping, EC(·) denotes a learnable globally embedded token swapping operation, W q W k , and W v represents the learnable parameters, h represents the number of self-attention heads, and M represents the global fusion attention map.

[0086] Subsequently, layer normalization (LN) and multilayer perceptron (MLP) were used for self-attention, and the calculation was performed as follows:

[0087]

[0088] in, Z represents the currently learnable global embedding label, which incorporates global features from other modalities. I This represents the output of the global fusion attention layer, and (·:·) represents the concatenation operation.

[0089] Next, the two learnable global embedding labels are swapped again and fed into a self-attention layer to map the learned fusion representation. and In its original modality's labeled sequence, it is expressed as follows:

[0090]

[0091]

[0092] Among them, T * ′ represents the output of the second self-attention layer.

[0093] Finally, the feature map is reconstructed from one dimension to two dimensions.

[0094] In an optional embodiment of the present invention, the cross-modal channel fusion module first concatenates the single-modal feature map that fuses local and global information and then performs channel fusion through a convolutional layer. Then, the channel fusion feature map is divided into multimodal channel fusion attention maps along the channel dimension, and the attention maps are weighted and summed with the original feature maps of the corresponding modalities and then concatenated to obtain the multimodal channel fusion feature map.

[0095] For image data, channel information is also crucial. Therefore, this embodiment proposes a cross-modal channel fusion module to fuse channel cues from two modalities, such as... Figure 6As shown. Channel fusion is achieved by concatenating two feature maps along the channel dimension and using a 1×1 transformation layer. The formula is as follows:

[0096]

[0097] A C =Conv(R C )

[0098] Among them, R C A represents the feature map after channel splicing. C It is the representation after channel fusion, and Conv(·) represents a 1×1 convolutional layer.

[0099] Then, the fused representation is evenly divided into multimodal channel fusion attention maps along the channel dimension, and the original feature maps of the two modalities are weighted and summed to obtain channel-level fused features. The calculation process is as follows:

[0100]

[0101] in, and This represents the channel fusion attention map, Split(·) represents the splitting operation, and F C It is the final channel fusion feature.

[0102] To evaluate the performance of the proposed multispectral target detection method, a comparative experiment was conducted on the baseline model and the proposed model using a self-built dataset. The experimental results are shown in Table 1. The proposed method improves performance by 21.8% compared to the baseline model, demonstrating its effectiveness and advancement.

[0103] Table 1 Comparison of experimental results of the present invention on a self-built virtual dataset.

[0104]

[0105]

[0106] To make the detection results more intuitive, the detection and recognition results were visualized on a self-built dataset, such as... Figure 7 , Figure 8 , Figure 9 and Figure 10 As shown in the visualization results, this invention can effectively capture and identify targets under multispectral conditions.

[0107] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0111] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A multispectral target type identification method based on multidimensional fusion, characterized in that, Includes the following steps: Acquire multimodal image data; A multispectral target type recognition model is constructed; the multispectral target type recognition model includes a multimodal feature extraction module, a cross-modal local and global fusion module, a cross-modal channel fusion module, and a detection module; The multimodal feature extraction module is used to extract features from the multimodal image data to obtain multi-scale feature maps representing different modalities. By utilizing the cross-modal local and global fusion module, multi-scale feature maps representing different modalities are fused and interacted at different depths using spatial dimensions to obtain single-modal feature maps that fuse local and global information. The cross-modal local and global fusion module includes: Cross-modal local and global fusion sub-modules corresponding to each level of the multimodal feature extraction module; The cross-modal local and global fusion submodule includes multiple cross-modal local fusion units and cross-modal global fusion units; The cross-modal local fusion unit is used to extract cross-modal local fusion feature maps from the corresponding scale feature maps of the current modality representation and the corresponding scale feature maps of other modality representations; The cross-modal global fusion unit is used to extract cross-modal global fusion feature maps from the corresponding scale feature maps of the current modality representation and the corresponding scale feature maps of other modality representations, and to concatenate them with the cross-modal local fusion feature maps extracted by the cross-modal local fusion unit to obtain a single-modal feature map that fuses local and global information. The cross-modal channel fusion module is used to perform feature fusion on the single-modal feature map that fuses local and global information to obtain a multimodal channel fused feature map; The detection module is used to perform multi-scale feature map fusion on the multimodal channel fusion feature map to obtain multispectral target detection results.

2. The multispectral target type identification method based on multidimensional fusion according to claim 1, characterized in that, The multimodal image data includes: RGB images, infrared images, and plasma images.

3. The multispectral target type identification method based on multidimensional fusion according to claim 1, characterized in that, The multimodal feature extraction module includes: The backbone network of the YOLOv5 model.

4. The multispectral target type identification method based on multidimensional fusion according to claim 1, characterized in that, The cross-modal local fusion unit first segments the corresponding scale feature map of the current modality representation and the corresponding scale feature maps of other modal representations into patch marker sequences through the first local fusion layer. Then, each patch marker sequence is segmented into local regions through a local window. Next, window-based multi-head cross-attention fusion of local regions with the same spatial position in the multimodal representation is performed on each window pair. Then, the local window is moved by a set number of pixels through the second local fusion layer, and the image is segmented in the local region of the new window. Then, window-based multi-head cross-attention fusion of local regions with the same spatial position in the multimodal representation is performed on each new window pair. Finally, the patch marker sequence after the second fusion is reconstructed to obtain the cross-modal local fusion feature map.

5. The multispectral target type identification method based on multidimensional fusion according to claim 1, characterized in that, The cross-modal global fusion unit first divides the corresponding scale feature map of the current modality representation and the corresponding scale feature map of other modality representations into patch label sequences, and adds two learnable global embedding labels before each patch label sequence; Then, the learnable global embedding labels of each patch label sequence are swapped and the global information between the fusion attention layer and the fusion modality is interacted through. Then, the learnable global embedding labels of each patch label sequence are swapped and the fusion representation learned by the self-attention layer is mapped into the label sequence of its original modality. Finally, the obtained fusion feature map is reconstructed to obtain the cross-modality global fusion feature map.

6. The multispectral target type identification method based on multidimensional fusion according to claim 1, characterized in that, The cross-modal channel fusion module first concatenates the single-modal feature map that integrates local and global information, and then performs channel fusion through a convolutional layer. Next, the channel fusion feature map is divided into multimodal channel fusion attention maps along the channel dimension, and then weighted and summed with the original feature map of the corresponding modality before concatenation to obtain the multimodal channel fusion feature map.

Images