Multi-spectral target type identification method based on multi-dimensional fusion

Abstract

The invention relates to the technical field of target identification, and discloses a multispectral target type identification method based on multi-dimensional fusion, which comprises the following steps: acquiring multi-modal image data; constructing a multispectral target type identification model; performing feature extraction on the multi-modal image data by using a multi-modal feature extraction module to obtain multi-scale feature maps represented by different modalities; a cross-modal local and global fusion module is used for carrying out spatial dimension multi-dimensional cross-modal feature fusion and interaction on the multi-scale feature maps represented by different modals at different depths, and a single-modal feature map fusing local and global information is obtained; a cross-modal channel fusion module is used for carrying out feature fusion on the single-modal feature map fusing the local and global information, and a multi-modal channel fusion feature map is obtained; and performing multi-scale feature map fusion on the multi-modal channel fusion feature map by using a detection module to obtain a multi-spectral target detection result. According to the invention, the target identification and detection effects are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target recognition, and in particular to a multi-spectral target type recognition method based on multi-dimensional fusion. Background Art

[0002] Since the target moves very fast, it is impossible to correctly identify it simply by relying on ordinary optical images. Therefore, it is necessary to rely on multimodal information to detect the target from multiple dimensions and aspects. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multi-spectral target type recognition method based on multi-dimensional fusion.

[0004] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0005] A multi-spectral target type recognition method based on multi-dimensional fusion includes the following steps:

[0006] Acquire multimodal image data;

[0007] Constructing a multispectral target type recognition model; the multispectral target type recognition model includes a multimodal feature extraction module, a cross-modal local and global fusion module, a cross-modal channel fusion module and a detection module;

[0008] The multimodal feature extraction module is used to extract features from the multimodal image data to obtain multi-scale feature maps representing different modalities.

[0009] The cross-modal local and global fusion module is used to perform multi-dimensional cross-modal feature fusion and interaction at different depths on the multi-scale feature maps represented by different modalities to obtain a single-modal feature map that integrates local and global information.

[0010] The cross-modal channel fusion module is used to perform feature fusion on the single-modal feature map that fuses local and global information to obtain a multi-modal channel fusion feature map;

[0011] The detection module is used to perform multi-scale feature map fusion on the multimodal channel fusion feature map to obtain the multispectral target detection result.

[0012] Preferably, the multimodal image data includes:

[0013] RGB images, infrared images, and plasma images.

[0014] Preferably, the multimodal feature extraction module comprises:

[0015] The backbone network of the YOLOv5 model.

[0016] Preferably, the cross-modal local and global fusion module includes:

[0017] Cross-modal local and global fusion submodules corresponding to each level of the multimodal feature extraction module;

[0018] The cross-modal local and global fusion submodule includes a plurality of cross-modal local fusion units and a cross-modal global fusion unit;

[0019] The cross-modal local fusion unit is used to extract a cross-modal local fusion feature map from the corresponding scale feature map represented by the current modality and the corresponding scale feature map represented by other modalities;

[0020] The cross-modal global fusion unit is used to extract a cross-modal global fusion feature map from the corresponding scale feature map represented by the current modality and the corresponding scale feature map represented by other modalities, and splice it with the cross-modal local fusion feature map extracted by the cross-modal local fusion unit to obtain a single modal feature map that fuses local and global information.

[0021] Preferably, the cross-modal local fusion unit first divides the corresponding scale feature map represented by the current modality and the corresponding scale feature map represented by other modalities into a patch mark sequence through a first local fusion layer, and then segments each patch mark sequence into local regions through a local window, and then performs a window-based multi-head cross-attention fusion of local regions with the same spatial position in the multimodality for each window pair; then the local window is moved by a set number of pixels through a second local fusion layer, and the image is segmented in the local region of the new window, and then each new window pair performs a shifted window-based multi-head cross-attention fusion of local regions with the same spatial position in the multimodality, and finally reconstructs the patch mark sequence after secondary fusion to obtain a cross-modal local fusion feature map.

[0022] Preferably, the cross-modal global fusion unit first divides the corresponding scale feature map of the current modality representation and the corresponding scale feature map of other modal representations into patch mark sequences, and adds two learnable global embedding markers before each patch mark sequence; then exchanges the learnable global embedding markers of each patch mark sequence and interacts and fuses the global information between the modalities through the global fusion attention layer; then exchanges the learnable global embedding markers of each patch mark sequence and maps the learned fusion representation to the mark sequence of its original modality through the self-attention layer; finally, the obtained fused feature map is reconstructed to obtain a cross-modal global fusion feature map.

[0023] Preferably, the cross-modal channel fusion module first splices the unimodal feature maps that fuse local and global information and then performs channel fusion through a convolutional layer, then divides the channel fusion feature map into a multimodal channel fusion attention map along the channel dimension, and performs weighted summation with the original feature map of the corresponding modality and then splices them to obtain a multimodal channel fusion feature map.

[0024] The present invention has the following beneficial effects:

[0025] The present invention uses multispectral information as information to supplement RGB images. By fusing the multispectral images of the target, a more comprehensive space of the target can be obtained, greatly improving the effect of target recognition and detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of a multi-spectral target type recognition method based on multi-dimensional fusion;

[0027] Figure 2 This is a schematic diagram of the multi-spectral target type recognition model structure;

[0028] Figure 3 This is a schematic diagram of the multi-spectral target type recognition model framework;

[0029] Figure 4 Schematic diagram of the cross-modal local fusion unit structure;

[0030] Figure 5 Schematic diagram of the cross-modal global fusion unit structure;

[0031] Figure 6 Schematic diagram of the cross-modal channel fusion module structure;

[0032] Figure 7 A diagram showing the visualization of cone-shaped target recognition results;

[0033] Figure 8 This is a visualization diagram of the cylindrical target recognition results;

[0034] Fig. 9 This is a visualization diagram of the elliptical target recognition results;

[0035] Fig.10 A schematic diagram showing the visualization of conventional target recognition results. DETAILED DESCRIPTION

[0036] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0037] like Figure 1 As shown, the embodiment of the present invention provides a multi-spectral target type recognition method based on multi-dimensional fusion, comprising the following steps S1 to S6:

[0038] S1, acquiring multimodal image data;

[0039] S2. Construct a multispectral target type recognition model; the multispectral target type recognition model includes a multimodal feature extraction module, a cross-modal local and global fusion module, a cross-modal channel fusion module and a detection module;

[0040] S3, using a multimodal feature extraction module to perform feature extraction on the multimodal image data respectively, and obtain multi-scale feature maps representing different modalities;

[0041] S4. Using the cross-modal local and global fusion module, multi-dimensional cross-modal feature fusion and interaction are performed on the multi-scale feature maps represented by different modalities at different depths to obtain a single-modal feature map that fuses local and global information.

[0042] S5. Using a cross-modal channel fusion module to perform feature fusion on the single-modal feature map that fuses local and global information, to obtain a multi-modal channel fusion feature map;

[0043] S6. Use the detection module to perform multi-scale feature map fusion on the multi-modal channel fusion feature map to obtain the multi-spectral target detection result.

[0044] The multi-spectral target type recognition model constructed in this embodiment is as follows: Figure 2 As shown, it includes a multimodal feature extraction module, a cross-modal local and global fusion module, a cross-modal channel fusion module and a detection module. RGB, infrared and plasma modes are respectively input into the feature extraction module to learn the representation of each mode. In the process of feature extraction, the feature maps obtained by different modes in each stage are input into the cross-modal local and global fusion module for feature fusion and interaction in the spatial dimension to obtain single-modal features with local and global information respectively. Subsequently, the single-modal features that have undergone cross-modal local and global interaction are input into the cross-modal channel fusion module to obtain the final multimodal fusion features at this resolution. Finally, the fused feature maps of different resolutions obtained in each stage are input into the detection head for multi-scale feature map fusion and the detection and recognition results are obtained.

[0045] like Figure 3 As shown in FIG. 1 , a schematic diagram of the architecture of the multispectral target type recognition model constructed in this embodiment includes five main parts: RGB feature learning branch, multispectral feature learning branch, cross-modal local fusion module (CMLF), cross-modal global fusion module (CMGF) and cross-modal channel fusion module (CMCF). This embodiment uses the YOLOv5 model as the backbone of the two branches. In the process of feature extraction, the features of multiple modalities are multi-dimensionally cross-modally fused and interacted at different depths of the network to integrate the information of different modalities and complement each other. Given a pair of RGB images I R and multispectral image I I , which are input into the RGB and multispectral image branches respectively to extract the representation The features of both modalities are input into the CMLF and CMGF modules to obtain local and global fusion feature maps. Afterwards, the two types of fusion features are added and input into the CMCF module to obtain channel-level fusion features. [1,3]. The final fusion feature M calculated by different scale representations i Sent to the detection head of the YOLOv5 model.

[0046] In order to more effectively fuse multispectral images and enhance the recognition accuracy of multimodal target detectors, the present invention proposes a multidimensional fusion target detection method, which uses multispectral information as information to supplement RGB images. By fusing the multispectral images of the target, a more comprehensive space of the target can be obtained, greatly improving the effect of target recognition and detection. The present invention fuses the local, global and channel information of different spectral images and extracts the relationship to enrich the final features of the target.

[0047] In an optional embodiment of the present invention, the multimodal image data includes:

[0048] RGB images, infrared images, and plasma images.

[0049] In an optional embodiment of the present invention, the multimodal feature extraction module includes:

[0050] The backbone network of the YOLOv5 model.

[0051] In an optional embodiment of the present invention, the cross-modal local and global fusion module includes:

[0052] Cross-modal local and global fusion submodules corresponding to each level of the multimodal feature extraction module;

[0053] The cross-modal local and global fusion submodule includes a plurality of cross-modal local fusion units and a cross-modal global fusion unit;

[0054] The cross-modal local fusion unit is used to extract a cross-modal local fusion feature map from the corresponding scale feature map represented by the current modality and the corresponding scale feature map represented by other modalities;

[0055] The cross-modal global fusion unit is used to extract a cross-modal global fusion feature map from the corresponding scale feature map represented by the current modality and the corresponding scale feature map represented by other modalities, and splice it with the cross-modal local fusion feature map extracted by the cross-modal local fusion unit to obtain a single modal feature map that fuses local and global information.

[0056] In an optional embodiment of the present invention, the cross-modal local fusion unit first divides the corresponding scale feature map represented by the current modality and the corresponding scale feature map represented by other modalities into a patch mark sequence through a first local fusion layer, and then segments each patch mark sequence into local areas through a local window, and then performs a window-based multi-head cross-attention fusion of local areas with the same spatial position in the multimodality for each window pair; then the local window is moved by a set number of pixels through a second local fusion layer, and the image is segmented in the local area of ​​the new window, and then each new window pair performs a shifted window-based multi-head cross-attention fusion of local areas with the same spatial position in the multimodality, and finally reconstructs the patch mark sequence after secondary fusion to obtain a cross-modal local fusion feature map.

[0057] In order to more accurately capture and fuse the details of multispectral images, this study proposes a cross-modal local fusion module. In the first local window fusion layer, each modality can be divided into some non-overlapping image blocks by a rectangular local window of a fixed size. The fusion process is as follows Figure 4 The representations of the two modalities are split into sequences of patch labels Each sequence is segmented into M×M local windows of the local area, where is the number of sequences. Then, a window-based multi-head cross-attention (W-MCA) fusion is performed on each window pair. The local regions with the same spatial location in both modalities. The input of the first local fusion layer in the CMLF module is defined as:

[0058] Q1=P2W q

[0059] K1=P2W k

[0060] V1=P1W v

[0061] Among them, W q ,W k , and W v represents a learnable parameter. W-MCA fusion attention is calculated by two local features and can be expressed as:

[0062]

[0063] Where d represents the dimension of Q1 and K1. The output of each window pair in the first local fusion layer is expressed as:

[0064] F W =A W +P1+P2+MLP(LN(A F +P1+P2))

[0065] Here, LN(·) and MLP(·) represent layer normalization and multi-layer perceptron, respectively.

[0066] In order to associate the multimodal local information and enhance the fusion effect, in the next fusion layer, the local window is moved pixels and segment the image in the local area of ​​the new window. The input of the second fusion layer is the new shifted window pair from the first fusion feature and the other modality feature. The fusion process is the same as the first fusion layer. Each new window pair is fused using Shifted Window-based multi-head Cross-attention (SW-MCA) The formula of the second fusion layer is expressed as:

[0067] Q2=P2 ′ W q

[0068] K2=P2 ′ W k

[0069] V2=P1 ′ W v

[0070]

[0071] F SW =A SW +P′1+P′2+MLP(LN(A SW +P′1+P′2))

[0072] Finally, the cross-modal local fusion feature map is obtained by reconstructing the sequence labeled by secondary fusion.

[0073] In an optional embodiment of the present invention, the cross-modal global fusion unit first divides the corresponding scale feature map represented by the current modality and the corresponding scale feature map represented by other modalities into patch mark sequences, and adds two learnable global embedding marks before each patch mark sequence; then the learnable global embedding marks of each patch mark sequence are exchanged and the global information between the modalities is interacted and fused through the global fusion attention layer; then the learnable global embedding marks of each patch mark sequence are exchanged and the learned fusion representation is mapped to the mark sequence of its original modality through the self-attention layer; finally, the obtained fused feature map is reconstructed to obtain a cross-modal global fusion feature map.

[0074] In the spatial dimension, the global perspective is an important supplement to the local perspective. Therefore, this embodiment proposes a cross-modal global fusion module to learn global multi-spectral interaction information, such as Figure 5 The feature maps of the two modalities are divided into a sequence of patch labels In order to learn the global features of the two modalities and interact with each other, this embodiment adds two learnable global embeddings before each patch label sequence. Since the feature map loses its global spatial position information when it is decomposed into a single vector sequence, this embodiment adopts position embedding to solve this problem. To obtain the necessary spatial position information. Therefore, the input token sequence for each modality is as follows:

[0075]

[0076] Its output can be expressed as follows:

[0077]

[0078] Among them, X * represents the output of the first self-attention layer, and SA(·) represents the self-attention layer.

[0079] Subsequently, the learnable global embedding tags of each sequence are exchanged and input into the global fusion attention layer to interact and fuse the global information between the modalities. In order to reduce the computational complexity, this method only uses the learnable global embedding tag as q, and the new sequences are k and v. The calculation process is as follows:

[0080]

[0081]

[0082] k=X I ′ W k

[0083] v=X I′ W v

[0084]

[0085] Among them, X′ R and X′ I is the new sequence after token exchange, EC(·) represents the learnable global embedding token exchange operation, and W q ,W k , and W v represents a learnable parameter, h represents the number of self-attention heads, and M represents the global fusion attention map.

[0086] After that, layer normalization (LN) and multi-layer perceptron (MLP) are used as self-attention, which is calculated as follows:

[0087]

[0088] in, represents the currently learnable global embedding tag, which integrates the global features of other modalities, Z I represents the output of the global fusion attention layer, and (·:·) represents the concatenation operation.

[0089] Afterwards, the two learnable global embedding tokens are exchanged again and fed into the self-attention layer to map the learned fusion representation and to its original modality's token sequence, which is expressed as follows:

[0090]

[0091]

[0092] Among them, T * ′ represents the output of the second self-attention layer.

[0093] Finally, the feature map is reconstructed from one dimension to two dimensions.

[0094] In an optional embodiment of the present invention, the cross-modal channel fusion module first splices the unimodal feature maps that fuse local and global information and then performs channel fusion through a convolutional layer. The channel fusion feature map is then divided into a multimodal channel fusion attention map along the channel dimension, and the weighted sum is performed with the original feature map of the corresponding modality and then spliced ​​to obtain a multimodal channel fusion feature map.

[0095] For image data, channel information is also crucial. Therefore, this embodiment proposes a cross-modal channel fusion module to fuse the channel cues of the two modalities, such as Figure 6As shown. Channel fusion is achieved by concatenating two feature maps in the channel dimension and using a 1×1 conversion layer. The formula is as follows:

[0096]

[0097] A C =Conv(R C )

[0098] Among them, R C A represents the feature map after channel concatenation. C is the representation after channel fusion, and Conv(·) represents a 1×1 convolutional layer.

[0099] Then, the fused representation is equally divided into multimodal channel fusion attention maps along the channel dimension, and the weighted sum of the original feature maps of the two modalities is performed to obtain the channel-level fusion feature. The calculation process is as follows:

[0100]

[0101] in, and represents the channel fusion attention map, Split(·) represents the segmentation operation, and F C It is the final channel fusion feature.

[0102] In order to evaluate the performance of the proposed multispectral target detection method, a comparative experiment was carried out on the baseline model and the present model using a self-built dataset. The experimental results are shown in Table 1. The present invention is 21.8% better than the baseline model, which shows the effectiveness and advancement of the proposed method.

[0103] Table 1 Comparison of experimental results of the present invention on self-built virtual data sets

[0104]

[0105]

[0106] In order to make the detection results more intuitive, the detection and recognition results are visualized on the self-built data set, such as Figure 7 , Figure 8 , Fig. 9 and Fig.10 As shown in the visualization results, it can be found that the present invention can effectively capture and identify targets under multi-spectral conditions.

[0107] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0111] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A multi-spectral target type recognition method based on multi-dimensional fusion, characterized in that: The following steps are involved: Acquire multimodal image data; Constructing a multispectral target type recognition model; the multispectral target type recognition model includes a multimodal feature extraction module, a cross-modal local and global fusion module, a cross-modal channel fusion module and a detection module; The multimodal feature extraction module is used to extract features from the multimodal image data to obtain multi-scale feature maps representing different modalities. The cross-modal local and global fusion module is used to perform multi-dimensional cross-modal feature fusion and interaction at different depths on the multi-scale feature maps represented by different modalities to obtain a single-modal feature map that integrates local and global information. The cross-modal channel fusion module is used to perform feature fusion on the single-modal feature map that fuses local and global information to obtain a multi-modal channel fusion feature map; The detection module is used to perform multi-scale feature map fusion on the multimodal channel fusion feature map to obtain the multispectral target detection result.

2. The multispectral target type recognition method based on multi-dimensional fusion according to claim 1 is characterized in that: The multimodal image data includes: RGB images, infrared images, and plasma images.

3. The multispectral target type recognition method based on multi-dimensional fusion according to claim 1 is characterized in that: The multimodal feature extraction module comprises: The backbone network of the YOLOv5 model.

4. The multispectral target type recognition method based on multi-dimensional fusion according to claim 1 is characterized in that: The cross-modal local and global fusion module includes: Cross-modal local and global fusion submodules corresponding to each level of the multimodal feature extraction module; The cross-modal local and global fusion submodule includes a plurality of cross-modal local fusion units and a cross-modal global fusion unit; The cross-modal local fusion unit is used to extract a cross-modal local fusion feature map from the corresponding scale feature map represented by the current modality and the corresponding scale feature map represented by other modalities; The cross-modal global fusion unit is used to extract a cross-modal global fusion feature map from the corresponding scale feature map represented by the current modality and the corresponding scale feature map represented by other modalities, and splice it with the cross-modal local fusion feature map extracted by the cross-modal local fusion unit to obtain a single modal feature map that fuses local and global information.

5. The multi-spectral target type recognition method based on multi-dimensional fusion according to claim 4 is characterized in that: The cross-modal local fusion unit first divides the corresponding scale feature map represented by the current modality and the corresponding scale feature map represented by other modalities into a patch mark sequence through a first local fusion layer, and then segments each patch mark sequence into local regions through a local window, and then performs window-based multi-head cross-attention fusion of local regions with the same spatial position in the multi-modality for each window pair; then, the local window is moved by a set number of pixels through a second local fusion layer, and the image is segmented in the local region of the new window, and then, for each new window pair, multi-head cross-attention fusion of local regions with the same spatial position in the multi-modality based on the shifted window is performed, and finally, the patch mark sequence after secondary fusion is reconstructed to obtain a cross-modal local fusion feature map.

6. The multispectral target type recognition method based on multi-dimensional fusion according to claim 4 is characterized in that: The cross-modal global fusion unit first divides the corresponding scale feature map of the current modality representation and the corresponding scale feature map of the other modality representation into a patch tag sequence, and adds two learnable global embedding tags before each patch tag sequence; Then, the learnable global embedding tags of each patch tag sequence are exchanged and the global information between the modalities is interacted and fused through the global fusion attention layer; then, the learnable global embedding tags of each patch tag sequence are exchanged and the learned fusion representation is mapped to the tag sequence of its original modality through the self-attention layer; finally, the obtained fusion feature map is reconstructed to obtain a cross-modal global fusion feature map.

7. The multi-spectral target type recognition method based on multi-dimensional fusion according to claim 1 is characterized in that: The cross-modal channel fusion module first splices the unimodal feature maps that fuse local and global information and then performs channel fusion through a convolutional layer. It then divides the channel fusion feature map into a multimodal channel fusion attention map along the channel dimension, and performs weighted summation with the original feature map of the corresponding modality and then splices them to obtain a multimodal channel fusion feature map.

Images