Multimodal object detection method and system based on dual-branch asymmetric attention backbone and feature fusion pyramid network

Abstract

The multimodal target detection method and system based on the dual-branch asymmetric attention backbone and feature fusion pyramid network belong to the field of target detection technology. In order to solve the problem that the existing multimodal target detection method is difficult to supplement the features lacking in each mode, thus affecting the current detection effect of the multimodal. The present invention uses optical images and infrared images as inputs of the optical branch backbone network and the infrared branch backbone network. The branch backbone network includes a convolution layer and several residual modules. A detail information supplement module or a semantic information supplement module is set between each group of residual modules. The detail information supplement module and the semantic information supplement module are alternately set in the sequential processing direction of the residual module; the two feature pyramids obtained by the two branches are sent to the feature fusion unit, and the feature fusion unit uses the features of each layer of the feature fusion pyramid network to fuse, and the final fusion features of multiple scales are sent to the detection unit for detection.

Description

Technical Field

[0001] The present invention belongs to the field of target detection technology, and specifically relates to a multimodal target detection method and system based on computer vision. Background Art

[0002] At present, multimodal target detection methods based on deep learning have gradually become a research hotspot. Figure 1 As shown in Figure 2, according to the different fusion stages, the fusion strategies of multimodal target detection algorithms can generally be divided into image-level fusion, feature-level fusion, and decision-level fusion. Figure 1 In the figure, red blocks represent feature extraction, green blocks represent fusion operations, and blue blocks represent object detection. Studies have shown that fusing different modal features mid-network is more likely to achieve better multimodal detection results than image-level fusion and decision-level fusion strategies. Therefore, mid-network fusion is gradually becoming the default fusion strategy for multimodal object detection based on deep learning. Currently, attention mechanisms are becoming a hot topic of research. Zhang et al. proposed ResNeST, a backbone network based on the self-attention mechanism. Li et al. used the concatenated fused features to weight the modal features and then added the branch features to obtain the fused features. Yang et al. proposed a gated convolutional architecture to obtain fused features after weighting the modal features. However, effective information exists in different forms in infrared and optical images, such as temperature information in infrared images and global detail information in optical images. Different forms of effective information have different distributions across dimensions, and using the same attention structure can reduce the network's learning efficiency. Furthermore, information compensation during feature extraction can reduce the difficulty of the network in discovering target features. Both Li and Yang performed feature fusion after feature extraction.

[0003] Recently, the Transformer architecture has shown increasing competitiveness. Fang et al. introduced the Transformer architecture to the field of feature fusion, achieving significant success. This approach converts features into vectors and then performs matrix multiplication to generate fused features of self-attention and mutual-attention components. This fusion of information simultaneously performs attention operations on each modality. However, the information fusion structure in this method is symmetrical and does not take into account the imaging characteristics of infrared and RGB images, thus supplementing the information gap.

[0004] Multimodal data is different and complementary. On the one hand, many of the above fusion methods do not take differences into account and directly fuse multimodal features, which may lead to the introduction of features that are not conducive to detection, thereby reducing the performance of target detection. On the other hand, although some methods use attention structures to enhance target features, they only weight each modal feature separately without performing direct information complementation operations. In some methods, although the above two characteristics of multimodal data are considered, the attention operation is designed to be symmetrical. In other words, the information complementation structure between the optical branch and the infrared branch is the same, which makes it difficult to supplement the features lacking in each mode. The above are problems that need to be solved urgently in the current multimodal target detection task. Summary of the Invention

[0005] The present invention aims to solve the problem that the existing multimodal target detection method is difficult to supplement the features lacking in each mode, thereby affecting the current detection effect of the multimodal method.

[0006] A multimodal target detection method based on a dual-branch asymmetric attention backbone and a feature fusion pyramid network acquires optical and infrared images of the area to be detected and implements target detection using a multimodal target detection network model. The multimodal target detection network model includes a feature extraction unit, a feature fusion unit, and a detection unit. The processing of the multimodal target detection network model includes:

[0007] The feature extraction unit adopts a dual-branch asymmetric attention backbone network, which includes an optical branch backbone network, an infrared branch backbone network, a detail information supplement module and a semantic information supplement module; the detail information supplement module and the semantic information supplement module are denoted as the detail information supplement module and the semantic information supplement module;

[0008] The optical image and the infrared image are used as inputs of the optical branch backbone network and the infrared branch backbone network. The optical branch backbone network and the infrared branch backbone network each include a convolutional layer and several residual modules, and the number of residual modules of the two is equal. The residual modules corresponding to the optical branch backbone network and the infrared branch backbone network are called a group of residual modules. A detail information supplementation module or a semantic information supplementation module is set between each group of residual modules. The detail information supplementation modules and the semantic information supplementation modules are alternately set in the sequential processing direction of the residual modules.

[0009] The detail information supplementation module uses the outputs of the two residual modules in this group as input, extracts detail features for compensating the residual modules corresponding to the infrared branch backbone network, that is, the features obtained by the detail information supplementation module are used as the features output by the residual module of the current group corresponding to the infrared branch backbone network, and are used for processing the next residual module of the infrared branch backbone network;

[0010] The semantic information supplementation module uses the outputs of the two residual modules in this group as input, extracts speech features for compensating the residual modules corresponding to the optical branch backbone network, that is, the features processed by the semantic information supplementation module are used as the features output by the residual module of the current group corresponding to the optical branch backbone network, and are used for processing the next residual module of the optical branch backbone network;

[0011] Based on the optical branch backbone network and the infrared branch backbone network, the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network are obtained. The number of feature pyramids corresponding to the optical branch backbone network and the feature pyramids corresponding to the infrared branch backbone network are the same, and the features of each layer of the two feature pyramids correspond to each other;

[0012] The two feature pyramids are then sent to the feature fusion unit, which uses the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids. A FPFM fusion feature is obtained for each layer of the feature pyramid. Based on the FPFM fusion features of all layers, the feature pyramid processing method is used to obtain fusion features of multiple scales.

[0013] Finally, the final fusion features of multiple scales are sent to the detection unit for detection, thereby realizing the detection of the target.

[0014] Preferably, a detail information supplement module or a semantic information supplement module is provided between each group of residual modules as follows: a detail information supplement module is provided between odd-numbered residual modules, and a semantic information supplement module is provided between even-numbered residual modules.

[0015] Furthermore, the detailed information supplement module processing process includes:

[0016] First, the feature F output by the residual module of the optical branch backbone network rgb After a 3*3 convolution layer, feature F1 is obtained rgb , F1 rgb Perform maximum pooling and average pooling respectively, and splice the pooled features in the channel dimension. Then perform 1*1 convolution on the spliced ​​features in the channel dimension to obtain the spatial principal component sampling matrix W(F1 rgb );

[0017] At the same time, the feature F output by the residual module of the infrared branch backbone network inf After a 1*1 convolution layer, feature F1 is obtained inf ;

[0018] F1 inf With W(F1 rgb ) to perform splicing in the channel dimension, and then obtain the features through a 1*1 convolution layer Then Perform broadcast processing on the channel dimension to obtain features Will with F1 inf After dot product, The features processed by the detail information supplement module are obtained by fusion.

[0019] Furthermore, the semantic information supplementation module processing process includes:

[0020] First, the feature F output by the residual module of the infrared branch backbone network inf After an average pooling, the feature F2 is obtained inf , and at the same time, the feature F output by the residual module of the optical branch backbone network rgb After an average pooling, the feature F2 is obtained rgb ; then press F2 inf and F2 rgb Perform splicing on the channel dimension, perform 1*1 convolution on the features of the channel dimension splicing, and then process it through the FC layer to obtain the feature W′(F rgb ,F inf ), then W′(F rgb ,F inf ) and F rgb After dot product, add F rgb The features processed by the semantic information supplement module are obtained by fusion.

[0021] Furthermore, the feature fusion unit utilizes the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids, including the following steps:

[0022] For the corresponding layer features of the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network, the linear layer of the channel dimension is first used to obtain the linear mapping of the three subspaces, namely the query tensor, key tensor and value tensor; the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q rgb , K rgb 、V rgb , the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q inf , K inf 、V inf ;

[0023] Based on the query tensor, key tensor, and value tensor obtained, the following processing is performed to obtain the FPFM fusion features of the corresponding layer:

[0024]

[0025] in, and is the weighted feature tensor, Sig represents the sigmoid function; C 1×1 (·,·) means that the features of the two inputs are concatenated and then a 1*1 convolution is performed, i.e., concatenation and convolution operations.

[0026] A multimodal object detection system based on a dual-branch asymmetric attention backbone and feature fusion pyramid network, including:

[0027] Data acquisition module for the area to be inspected: used to acquire optical images and infrared images of the area to be inspected;

[0028] Target detection module: For optical images and infrared images, a multimodal target detection network model is used to perform target detection;

[0029] The multimodal target detection network model includes a feature extraction unit, a feature fusion unit and a detection unit. The processing process of the multimodal target detection network model includes:

[0030] The feature extraction unit adopts a dual-branch asymmetric attention backbone network, which includes an optical branch backbone network, an infrared branch backbone network, a detail information supplement module and a semantic information supplement module; the detail information supplement module and the semantic information supplement module are denoted as the detail information supplement module and the semantic information supplement module;

[0031] The optical image and the infrared image are used as inputs of the optical branch backbone network and the infrared branch backbone network. The optical branch backbone network and the infrared branch backbone network each include a convolutional layer and several residual modules, and the number of residual modules of the two is equal. The residual modules corresponding to the optical branch backbone network and the infrared branch backbone network are called a group of residual modules. A detail information supplementation module or a semantic information supplementation module is set between each group of residual modules. The detail information supplementation modules and the semantic information supplementation modules are alternately set in the sequential processing direction of the residual modules.

[0032] The detail information supplementation module uses the outputs of the two residual modules in this group as input, extracts detail features for compensating the residual modules corresponding to the infrared branch backbone network, that is, the features obtained by the detail information supplementation module are used as the features output by the residual module of the current group corresponding to the infrared branch backbone network, and are used for processing the next residual module of the infrared branch backbone network;

[0033] The semantic information supplementation module uses the outputs of the two residual modules in this group as input, extracts speech features for compensating the residual modules corresponding to the optical branch backbone network, that is, the features processed by the semantic information supplementation module are used as the features output by the residual module of the current group corresponding to the optical branch backbone network, and are used for processing the next residual module of the optical branch backbone network;

[0034] Based on the optical branch backbone network and the infrared branch backbone network, the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network are obtained. The number of feature pyramids corresponding to the optical branch backbone network and the feature pyramids corresponding to the infrared branch backbone network are the same, and the features of each layer of the two feature pyramids correspond to each other;

[0035] The two feature pyramids are then sent to the feature fusion unit, which uses the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids. A FPFM fusion feature is obtained for each layer of the feature pyramid. Based on the FPFM fusion features of all layers, the feature pyramid processing method is used to obtain fusion features of multiple scales.

[0036] Finally, the final fusion features of multiple scales are sent to the detection unit for detection, thereby realizing the detection of the target.

[0037] Preferably, a detail information supplement module or a semantic information supplement module is provided between each group of residual modules as follows: a detail information supplement module is provided between odd-numbered residual modules, and a semantic information supplement module is provided between even-numbered residual modules.

[0038] Furthermore, the detailed information supplement module processing process includes:

[0039] First, the feature F output by the residual module of the optical branch backbone network rgb After a 3*3 convolution layer, feature F1 is obtained rgb , F1 rgb Perform maximum pooling and average pooling respectively, and splice the pooled features in the channel dimension. Then perform 1*1 convolution on the spliced ​​features in the channel dimension to obtain the spatial principal component sampling matrix W(F1 rgb );

[0040] At the same time, the feature F output by the residual module of the infrared branch backbone network inf After a 1*1 convolution layer, feature F1 is obtained inf ;

[0041] F1 inf With W(F1 rgb ) to perform splicing in the channel dimension, and then obtain the features through a 1*1 convolution layer Then Perform broadcast processing on the channel dimension to obtain features Will with F1 inf After dot product, The features processed by the detail information supplement module are obtained by fusion.

[0042] Furthermore, the semantic information supplementation module processing process includes:

[0043] First, the feature F output by the residual module of the infrared branch backbone network inf After an average pooling, the feature F2 is obtained inf , and at the same time, the feature F output by the residual module of the optical branch backbone network rgb After an average pooling, the feature F2 is obtained rgb ; then press F2 inf and F2 rgb Perform splicing on the channel dimension, perform 1*1 convolution on the features of the channel dimension splicing, and then process it through the FC layer to obtain the feature W′(F rgb ,F inf ), then W′(F rgb ,F inf ) and F rgb After dot product, add F rgb The features processed by the semantic information supplement module are obtained by fusion.

[0044] Furthermore, the feature fusion unit utilizes the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids, including the following steps:

[0045] For the corresponding layer features of the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network, the linear layer of the channel dimension is first used to obtain the linear mapping of the three subspaces, namely the query tensor, key tensor and value tensor; the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q rgb , K rgb 、V rgb , the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q inf , K inf 、V inf ;

[0046] Based on the query tensor, key tensor, and value tensor obtained, the following processing is performed to obtain the FPFM fusion features of the corresponding layer:

[0047]

[0048] in, and is the weighted feature tensor, Sig represents the sigmoid function; C 1×1 (·,·) means that the features of the two inputs are concatenated and then a 1*1 convolution is performed, i.e., concatenation and convolution operations.

[0049] In view of the characteristics of remote sensing multimodal image data, this paper studies how to obtain high-quality fusion feature representation while suppressing interference information, and proposes a new fusion target detection method. First, considering the differences between different modal data, a new dual-branch asymmetric attention backbone network is proposed to improve the reliability of each modal information; specifically, the backbone supplements the target information for infrared and RGB images through the detail information supplement module and the semantic information supplement module. Secondly, in order to better utilize the complementarity of multimodal data, a feature pyramid fusion module is designed to enhance the weight of the information of one modality by another modality, thereby enhancing useful information that is conducive to fusion and suppressing redundant information that is not conducive to fusion. Finally, multi-level and multi-scale fusion features are obtained. Therefore, the present invention can effectively improve the feature extraction capability of each modality and the information supplement capability between each modality, thereby improving the effect of multimodal target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram of three types of information fusion methods.

[0051] Figure 2 This is the overall architecture diagram of the algorithm.

[0052] Figure 3 Schematic diagram of the dual-branch asymmetric attention backbone network.

[0053] Figure 4 Supplement the module diagram for detailed information.

[0054] Figure 5 Schematic diagram of the semantic information supplement module.

[0055] Figure 6 Schematic diagram of feature fusion pyramid network.

[0056] Figure 7 This is a diagram of the subjective detection results of the present invention on the drone dataset. DETAILED DESCRIPTION

[0057] Due to the significant modal differences between optical and infrared images, the present invention employs a dual-branch backbone to extract features from different modalities. Taking into account the differences between the modalities, the semantic information in the infrared branch is used to enhance the optical features, while the detail information in the optical branch is used to enhance the infrared features. To further obtain a good joint feature representation, a fusion structure based on the Transformer strategy is proposed to enhance the useful information in the fusion stage and suppress redundant information that is detrimental to fusion. Finally, the fused features are used for target detection. This will be described below in conjunction with specific implementation methods.

[0058] Specific implementation method 1: Combination Figure 2 To explain this embodiment,

[0059] This embodiment is a multimodal target detection method based on a dual-branch asymmetric attention backbone and a feature fusion pyramid network. The optical image and infrared image of the area to be detected are obtained, and then the target detection is realized using a multimodal target detection network model. The multimodal target detection network model is as follows: Figure 2 As shown in FIG, it includes three parts: feature extraction unit, feature fusion unit and detection unit.

[0060] The feature extraction unit adopts a dual-branch asymmetric attention backbone network. Optical images and infrared images each have their own advantages and disadvantages, and in some challenging scenarios, their information is complementary. The backbone network is the basic information extractor of the target detection network. In some architectures, feature fusion is only performed at the neck after feature extraction, and the focus of network training is placed on the fusion structure. However, compared with the fusion structure, this training process improves the backbone network much less. Therefore, the present invention designs a new dual-branch backbone to better extract optical and infrared image features. Due to the difference in imaging principles, optical images often have more detailed information; however, infrared images are sensitive to temperature, and the edges between the target and the background are often more obvious and have more semantic information. Therefore, a detail information supplementation module and a semantic information supplementation module are used in the backbone network to supplement the information of each mode image feature and improve the quality of each mode image feature.

[0061] The dual-branch asymmetric attention backbone network includes an optical branch backbone network, an infrared branch backbone network, a detail information supplement module DISM, and a semantic information supplement module SISM;

[0062] The optical image and infrared image are used as inputs to the optical branch backbone network and the infrared branch backbone network. Both the optical branch backbone network and the infrared branch backbone network include a convolutional layer and several residual modules, and the number of residual modules in the two networks is equal. The residual modules corresponding to the optical branch backbone network and the infrared branch backbone network are referred to as a group of residual modules. A DISM or SISM is set between each group of residual modules. The DISM and SISM are alternately set in the sequential processing direction of the residual modules. In this embodiment, the DISM is set between the residual modules in odd groups, and the SISM is set between the residual modules in even groups. After feature extraction, the optical branch backbone network and the infrared branch backbone network each form two feature pyramids corresponding to the two modes. The two feature pyramids are then sent to the feature fusion unit.

[0063] Generally speaking, details and texture information are extracted more in the shallow layers of the network, while semantic information is extracted more in the deep layers. In the design process of the method of the present invention, a shallower detail information supplement module DISM and a deeper semantic information supplement module SISM are used. In each stage, only one information supplement module (detail information supplement module or semantic information supplement module) is used to establish a one-way information path in this area to improve the convergence speed of the network. Therefore, the detail information supplement module and the semantic information supplement module can appear in pairs to form a loop structure, such as Figure 3 The loop structure shown in (in fact, the main structure of the dual-branch asymmetric attention backbone network can be formed Figure 3 The loop closure structure shown in the figure can be inserted at any stage. In the dual-branch asymmetric attention backbone network, to focus on the proposed dual-branch architecture and loop structure, the optical branch backbone network and the infrared branch backbone network of this embodiment use the same residual structure as ResNet50, and the network depth of the optical branch backbone network and the infrared branch backbone network is the same.

[0064] like Figure 4 As shown, the detailed information supplement module DISM is as follows:

[0065] In the detail information supplementation module, we improve upon spatial attention and utilize optical image features to supplement infrared image feature details. To more directly supplement details, we dimensionally broadcast the weight matrix after the spatial attention operation to align it with the input feature dimension and then add it to the enhanced infrared features. This operation directly incorporates information obtained from the optical branch, further enhancing the influence of the weight matrix on the features.

[0066] First, the feature F output by the residual module of the optical branch backbone network rgb After a 3*3 convolution layer, feature F1 is obtained rgb , F1 rgb Perform maximum pooling and average pooling respectively, and concatenate the pooled features in the channel dimension, such as Figure 4 The c in the figure represents the operation on the channel dimension, and then the features of the channel dimension are convolved by 1*1 to obtain the spatial principal component sampling matrix W(F1 rgb );

[0067] At the same time, the feature F output by the residual module of the infrared branch backbone network inf After a 1*1 convolution layer, feature F1 is obtained inf ;

[0068] F1 inf With W(F1 rgb) to perform splicing in the channel dimension, and then obtain the features through a 1*1 convolution layer Then Perform broadcast processing on the channel dimension to obtain features Will with F1 inf After dot product, The features processed by DISM are obtained by fusion, and the features processed by DISM are used as the features output by the residual module corresponding to the infrared branch backbone network, and are used for processing the next residual module of the infrared branch backbone network;

[0069] The DISM process can be expressed as

[0070]

[0071] DISM (F1 rgb ,F1 inf )=B c [C 1×1 [W(F1 rgb ),C 1×1 (F inf )]]+C 1×1 [W(F1 rgb ),C 1×1 (F inf )]·F1 inf (2)

[0072] Among them, Pool represents the pooling operation. After pooling, the channel dimension is 1; B c represents the broadcast of channel dimension, C 1×1 Represents a scale-invariant convolution operation with a kernel size of 1. The two internal parameters represent the concatenation before processing.

[0073] It is worth noting that the present invention uses pooling operations to obtain the spatial principal component sampling matrix of the detailed information source and convolution operations to obtain the spatial sampling matrix of the supplementary infrared branch features. This is because we hope that DISM can adapt to the degree of infrared feature loss. This design allows the network to more dynamically supplement details to varying degrees.

[0074] like Figure 5 As shown, the semantic information supplement module SISM is as follows:

[0075] In the semantic information supplementation module, we improve upon channel attention and utilize infrared images to supplement the semantic information of optical image features. Because different feature branches may not necessarily focus on similar target features (infrared images may focus on outlines, while optical images may focus on details), direct feature supplementation may interfere with the optical branch. Therefore, in this module, we employ a weighted approach to enhance the semantic information of the optical branch features.

[0076] First, the feature F output by the residual module of the infrared branch backbone network inf After an average pooling, the feature F2 is obtained inf , and at the same time, the feature F output by the residual module of the optical branch backbone network rgb After an average pooling, the feature F2 is obtained rgb ; then press F2 inf and F2 rgb Perform splicing on the channel dimension, perform 1*1 convolution on the features of the channel dimension splicing, and then process it through the FC layer to obtain the feature W′(F rgb ,F inf ), then W′(F rgb ,F inf ) and F rgb After dot product, add F rgb The SISM-processed features are obtained by fusion, and the SISM-processed features are used as the features output by the residual module corresponding to the optical branch backbone network, and are used for processing the next residual module of the optical branch backbone network;

[0077] The SISM process can be expressed as

[0078]

[0079] SISM(F rgb ,F inf )=F rgb +W′(F rgb ,F inf )·F rgb (4)

[0080] in, It represents the pooling operation along the high-dimensional and wide-dimensional dimensions. After the operation, the high-dimensional and wide-dimensional features of the output tensor are both 1; FC refers to the fully connected layer; W′ is the channel attention weight vector obtained based on the two modal features.

[0081] The output of the semantic information supplementation module is the sum of the optical branch features enhanced by multimodal features and their pre-enhancement features.

[0082] Based on the optical branch backbone network and the infrared branch backbone network, the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network are obtained. The number of feature pyramids corresponding to the optical branch backbone network and the feature pyramids corresponding to the infrared branch backbone network are the same, and the features of each layer of the two feature pyramids correspond to each other.

[0083] The feature fusion unit uses the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids to obtain a high-quality fused feature pyramid. After obtaining the feature information of the infrared image and the optical image, in order to obtain a high-quality joint feature expression, the present invention designs a fusion symmetric decoder structure based on the Transformer strategy, which enhances the features of each modal scale of the multimodal information point by point based on the feature fusion pyramid network FFPN, thereby enhancing useful information and suppressing interference information that is not conducive to fusion. The detailed structure of FFPN is shown in the figure. Figure 6 To integrate multimodal features, taking the optical branch as an example, for each layer of the feature pyramid corresponding to the optical branch backbone network, a weight matrix is ​​calculated using the query tensor of the infrared branch features and the bond tensor of the optical branch, and then mapped to the optical features. Through cross-modal feature enhancement, high-quality co-representation can be obtained, further improving detection performance.

[0084] For the corresponding layer features of the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network, the linear layer of the channel dimension is first used to obtain the linear mapping of the three subspaces, namely the query tensor, key tensor and value tensor, that is, Q, K, V are all derived from the feature mapping, Q rgb and Q inf The calculation method is the same;

[0085] Taking the optical branch as an example, the query tensor, key tensor, and value tensor obtained are calculated as follows:

[0086]

[0087] in, All are 1*1 convolution processing, and different superscript convolution symbols represent different convolutions.

[0088] The same process is performed for the infrared branch.

[0089] Based on the query tensor, key tensor, and value tensor obtained, the following processing is performed to obtain the FPFM fusion features of the corresponding layer:

[0090]

[0091] in, and is the weighted feature tensor, Sig represents the sigmoid function, which can map the eigenvalue to between 0 and 1 as the enhanced weight of the feature point; C 1×1 (·,·) means that two features are connected in series and then 1*1 convolution is performed, i.e., concatenation and convolution operations. Figure 6 Indicated by C&C. Figure 6E-Pro and E-Plus represent the product of elements and the addition of elements.

[0092] A FPFM fusion feature is obtained for each layer of the feature pyramid. Based on the FPFM fusion features of all layers, the feature pyramid processing method is used to obtain fusion features of multiple scales.

[0093] Finally, the final fusion features of multiple scales are sent to the detection unit for detection, thereby realizing the detection of the target.

[0094] Example

[0095] The experimental equipment used an Intel(R) Xeon(R) Silver 4210R CPU and the operating system was Ubuntu 18.04. This method was trained on an NVIDIA RTX 3090 GPU and implemented in the PyTorch framework. We used an input image size of 1024, a batch size of 4, and an initial learning rate of 0.008. The optimizer used the stochastic gradient descent (SGD) algorithm with a decay weight of 0.0001 and a momentum of 0.937. We evaluated each model using the mAP50 score.

[0096] To further verify the advancedness of the proposed method, we conducted comparative experiments with other advanced algorithms on the public dataset DroneVehicle from the perspective of drones.

[0097] Table 1. Performance of various methods on the DroneVehicle dataset

[0098]

[0099] The “Modality” column in Table 1 indicates whether optical or infrared modality is used to achieve target detection.

[0100] The subjective detection results of the method of the present invention on the UAV dataset are as follows: Figure 7 As shown. The first line is an optical image, the second line is an infrared image of the marked target position, and the third line is a visualization of the detection results of the method of the present invention. Figure 7 In the first and second columns, some objects are under-detected in the optical image, making them difficult for target detection algorithms to detect and even difficult for the human eye to distinguish. In the third and fourth columns, some targets in the infrared image are difficult to discern with the naked eye, requiring additional color information in the optical image to confirm the target's location. The proposed method maintains good detection performance in these situations.

[0101] Specific implementation method 2: Combination Figure 2 To explain this embodiment,

[0102] This embodiment is a multimodal target detection system based on a dual-branch asymmetric attention backbone and a feature fusion pyramid network. It is at least a software system corresponding to the program of a multimodal target detection method based on a dual-branch asymmetric attention backbone and a feature fusion pyramid network described in the first embodiment. It may also include a processor and a memory, etc. It should be understood that including any device including a processor and a memory described in the present invention, the device may also include other units and modules that perform display, interaction, processing, control, etc. and other functions through signals or instructions.

[0103] The multimodal object detection system based on a dual-branch asymmetric attention backbone and a feature fusion pyramid network described in this embodiment includes:

[0104] Data acquisition module for the area to be inspected: used to acquire optical images and infrared images of the area to be inspected;

[0105] Target detection module: For optical images and infrared images, a multimodal target detection network model is used to perform target detection;

[0106] The multimodal target detection network model includes a feature extraction unit, a feature fusion unit and a detection unit. The processing process of the multimodal target detection network model includes:

[0107] The feature extraction unit adopts a dual-branch asymmetric attention backbone network, which includes an optical branch backbone network, an infrared branch backbone network, a detail information supplement module and a semantic information supplement module; the detail information supplement module and the semantic information supplement module are denoted as the detail information supplement module and the semantic information supplement module;

[0108] The optical image and the infrared image are used as the input of the optical branch backbone network and the infrared branch backbone network. The optical branch backbone network and the infrared branch backbone network both include a convolutional layer and several residual modules, and the number of residual modules of the two is equal; the residual modules corresponding to the optical branch backbone network and the infrared branch backbone network are called a group of residual modules, and a detail information supplement module or a semantic information supplement module is set between each group of residual modules. The detail information supplement module and the semantic information supplement module are alternately set in the sequential processing direction of the residual module; preferably, the method of setting a detail information supplement module or a semantic information supplement module between each group of residual modules is: setting a detail information supplement module between odd-group residual modules, and setting a semantic information supplement module between even-group residual modules.

[0109] The detail information supplementation module uses the outputs of the two residual modules in this group as input, extracts detail features for compensating the residual modules corresponding to the infrared branch backbone network, that is, the features obtained by the detail information supplementation module are used as the features output by the residual module of the current group corresponding to the infrared branch backbone network, and are used for processing the next residual module of the infrared branch backbone network;

[0110] The semantic information supplementation module uses the outputs of the two residual modules in this group as input, extracts speech features for compensating the residual modules corresponding to the optical branch backbone network, that is, the features processed by the semantic information supplementation module are used as the features output by the residual module of the current group corresponding to the optical branch backbone network, and are used for processing the next residual module of the optical branch backbone network;

[0111] Based on the optical branch backbone network and the infrared branch backbone network, the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network are obtained. The number of feature pyramids corresponding to the optical branch backbone network and the feature pyramids corresponding to the infrared branch backbone network are the same, and the features of each layer of the two feature pyramids correspond to each other;

[0112] The two feature pyramids are then sent to the feature fusion unit, which uses the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids. A FPFM fusion feature is obtained for each layer of the feature pyramid. Based on the FPFM fusion features of all layers, the feature pyramid processing method is used to obtain fusion features of multiple scales.

[0113] Finally, the final fusion features of multiple scales are sent to the detection unit for detection, thereby realizing the detection of the target.

[0114] The detailed information supplement module processing process includes:

[0115] First, the feature F output by the residual module of the optical branch backbone network rgb After a 3*3 convolution layer, feature F1 is obtained rgb , F1 rgb Perform maximum pooling and average pooling respectively, and splice the pooled features in the channel dimension. Then perform 1*1 convolution on the spliced ​​features in the channel dimension to obtain the spatial principal component sampling matrix W(F1 rgb );

[0116] At the same time, the feature F output by the residual module of the infrared branch backbone network inf After a 1*1 convolution layer, feature F1 is obtained inf ;

[0117] F1 inf With W(F1rgb ) to perform splicing in the channel dimension, and then obtain the features through a 1*1 convolution layer Then Perform broadcast processing on the channel dimension to obtain features Will with F1 inf After dot product, The features processed by the detail information supplement module are obtained by fusion.

[0118] The semantic information supplement module processing process includes:

[0119] First, the feature F output by the residual module of the infrared branch backbone network inf After an average pooling, the feature F2 is obtained inf , and at the same time, the feature F output by the residual module of the optical branch backbone network rgb After an average pooling, the feature F2 is obtained rgb ; then press F2 inf and F2 rgb Perform splicing on the channel dimension, perform 1*1 convolution on the features of the channel dimension splicing, and then process it through the FC layer to obtain the feature W′(F rgb ,F inf ), then W′(F rgb ,F inf ) and F rgb After dot product, add F rgb The features processed by the semantic information supplement module are obtained by fusion.

[0120] The feature fusion unit uses the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids. The process includes the following steps:

[0121] For the corresponding layer features of the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network, the linear layer of the channel dimension is first used to obtain the linear mapping of the three subspaces, namely the query tensor, key tensor and value tensor; the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q rgb , K rgb 、V rgb , the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q inf , K inf 、V inf ;

[0122] Based on the query tensor, key tensor, and value tensor obtained, the following processing is performed to obtain the FPFM fusion features of the corresponding layer:

[0123]

[0124] in, and is the weighted feature tensor, Sig represents the sigmoid function; C 1×1 (·,·) means that the features of the two inputs are concatenated and then a 1*1 convolution is performed, i.e., concatenation and convolution operations.

[0125] The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. A multimodal target detection method based on a dual-branch asymmetric attention backbone and a feature fusion pyramid network acquires optical and infrared images of the area to be detected and implements target detection using a multimodal target detection network model. The method is characterized by: The multimodal target detection network model includes a feature extraction unit, a feature fusion unit and a detection unit. The processing process of the multimodal target detection network model includes: The feature extraction unit adopts a dual-branch asymmetric attention backbone network, which includes an optical branch backbone network, an infrared branch backbone network, a detail information supplement module and a semantic information supplement module; the detail information supplement module and the semantic information supplement module are denoted as the detail information supplement module and the semantic information supplement module; The optical image and the infrared image are used as inputs of the optical branch backbone network and the infrared branch backbone network. The optical branch backbone network and the infrared branch backbone network each include a convolutional layer and several residual modules, and the number of residual modules of the two is equal. The residual modules corresponding to the optical branch backbone network and the infrared branch backbone network are called a group of residual modules. A detail information supplementation module or a semantic information supplementation module is set between each group of residual modules. The detail information supplementation modules and the semantic information supplementation modules are alternately set in the sequential processing direction of the residual modules. The detail information supplementation module uses the outputs of the two residual modules in this group as input, extracts detail features for compensating the residual modules corresponding to the infrared branch backbone network, that is, the features obtained by the detail information supplementation module are used as the features output by the residual module of the current group corresponding to the infrared branch backbone network, and are used for processing the next residual module of the infrared branch backbone network; The semantic information supplementation module uses the outputs of the two residual modules in this group as input, extracts speech features for compensating the residual modules corresponding to the optical branch backbone network, that is, the features processed by the semantic information supplementation module are used as the features output by the residual module of the current group corresponding to the optical branch backbone network, and are used for processing the next residual module of the optical branch backbone network; Based on the optical branch backbone network and the infrared branch backbone network, the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network are obtained. The number of feature pyramids corresponding to the optical branch backbone network and the feature pyramids corresponding to the infrared branch backbone network are the same, and the features of each layer of the two feature pyramids correspond to each other; The two feature pyramids are then sent to the feature fusion unit, which uses the feature fusion pyramid network (FPFM) to fuse the features corresponding to each layer of the two feature pyramids. A FPFM fusion feature is obtained for each layer of the feature pyramid. Based on the FPFM fusion features of all layers, the feature pyramid processing method is used to obtain fusion features of multiple scales. Finally, the final fusion features of multiple scales are sent to the detection unit for detection, thereby realizing the detection of the target.

2. The multimodal target detection method based on a dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 1, characterized in that: A detailed information supplement module or a semantic information supplement module is set between each group of residual modules as follows: a detailed information supplement module is set between odd-numbered residual modules, and a semantic information supplement module is set between even-numbered residual modules.

3. The multimodal target detection method based on a dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 1 or 2, characterized in that: The detailed information supplement module processing process includes: First, the feature F output by the residual module of the optical branch backbone network rgb After a 3*3 convolution layer, the features are obtained Will Perform maximum pooling and average pooling respectively, and splice the pooled features in the channel dimension. Then perform 1*1 convolution on the spliced ​​features in the channel dimension to obtain the spatial principal component sampling matrix of the optical branch. At the same time, the feature F output by the residual module of the infrared branch backbone network inf After a 1*1 convolution layer, the features are obtained Will and Perform splicing in the channel dimension and then obtain features through a 1*1 convolution layer Then Perform broadcast processing on the channel dimension to obtain features Will and After dot product, The features processed by the detail information supplement module are obtained by fusion.

4. The multimodal target detection method based on a dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 1 or 2, characterized in that: The semantic information supplement module processing process includes: First, the feature F output by the residual module of the infrared branch backbone network inf After an average pooling, the features are obtained At the same time, the feature F output by the residual module of the optical branch backbone network rgb After an average pooling, the features are obtained Then and Perform splicing on the channel dimension, perform 1*1 convolution on the features of the channel dimension splicing, and then process it through the FC layer to obtain the feature W′(F rgb ,F inf ), then W′(F rgb ,F inf ) and F rgb After dot product, add F rgb The features processed by the semantic information supplement module are obtained by fusion.

5. The multimodal target detection method based on a dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 1 or 2, characterized in that: The feature fusion unit uses the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids. The process includes the following steps: For the corresponding layer features of the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network, the linear layer of the channel dimension is first used to obtain the linear mapping of the three subspaces, namely the query tensor, key tensor and value tensor; the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q rgb , K rgb 、V rgb , the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q inf , K inf 、V inf ; Based on the query tensor, key tensor, and value tensor obtained, the following processing is performed to obtain the FPFM fusion features of the corresponding layer: in, and is the weighted feature tensor, Sig represents the sigmoid function; C 1×1 (·,·) means that the features of the two inputs are concatenated and then a 1*1 convolution is performed, i.e., concatenation and convolution operations.

6. A multimodal object detection system based on a dual-branch asymmetric attention backbone and feature fusion pyramid network, characterized by: include: Data acquisition module for the area to be inspected: used to acquire optical images and infrared images of the area to be inspected; Target detection module: For optical images and infrared images, a multimodal target detection network model is used to perform target detection; The multimodal target detection network model includes a feature extraction unit, a feature fusion unit and a detection unit. The processing process of the multimodal target detection network model includes: The feature extraction unit adopts a dual-branch asymmetric attention backbone network, which includes an optical branch backbone network, an infrared branch backbone network, a detail information supplement module and a semantic information supplement module; the detail information supplement module and the semantic information supplement module are denoted as the detail information supplement module and the semantic information supplement module; The optical image and the infrared image are used as inputs of the optical branch backbone network and the infrared branch backbone network. The optical branch backbone network and the infrared branch backbone network each include a convolutional layer and several residual modules, and the number of residual modules of the two is equal. The residual modules corresponding to the optical branch backbone network and the infrared branch backbone network are called a group of residual modules. A detail information supplementation module or a semantic information supplementation module is set between each group of residual modules. The detail information supplementation modules and the semantic information supplementation modules are alternately set in the sequential processing direction of the residual modules. The detail information supplementation module uses the outputs of the two residual modules in this group as input, extracts detail features for compensating the residual modules corresponding to the infrared branch backbone network, that is, the features obtained by the detail information supplementation module are used as the features output by the residual module of the current group corresponding to the infrared branch backbone network, and are used for processing the next residual module of the infrared branch backbone network; The semantic information supplementation module uses the outputs of the two residual modules in this group as input, extracts speech features for compensating the residual modules corresponding to the optical branch backbone network, that is, the features processed by the semantic information supplementation module are used as the features output by the residual module of the current group corresponding to the optical branch backbone network, and are used for processing the next residual module of the optical branch backbone network; Based on the optical branch backbone network and the infrared branch backbone network, the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network are obtained. The number of feature pyramids corresponding to the optical branch backbone network and the feature pyramids corresponding to the infrared branch backbone network are the same, and the features of each layer of the two feature pyramids correspond to each other; The two feature pyramids are then sent to the feature fusion unit, which uses the feature fusion pyramid network (FPFM) to fuse the features corresponding to each layer of the two feature pyramids. A FPFM fusion feature is obtained for each layer of the feature pyramid. Based on the FPFM fusion features of all layers, the feature pyramid processing method is used to obtain fusion features of multiple scales. Finally, the final fusion features of multiple scales are sent to the detection unit for detection, thereby realizing the detection of the target.

7. The multimodal object detection system based on a dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 6, characterized in that: A detailed information supplement module or a semantic information supplement module is set between each group of residual modules as follows: a detailed information supplement module is set between odd-numbered residual modules, and a semantic information supplement module is set between even-numbered residual modules.

8. The multimodal object detection system based on a dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 6 or 7, characterized in that: The detailed information supplement module processing process includes: First, the feature F output by the residual module of the optical branch backbone network rgb After a 3*3 convolution layer, the features are obtained Will Perform maximum pooling and average pooling respectively, and splice the pooled features in the channel dimension. Then perform 1*1 convolution on the spliced ​​features in the channel dimension to obtain the spatial principal component sampling matrix of the optical branch. At the same time, the feature F output by the residual module of the infrared branch backbone network inf After a 1*1 convolution layer, the features are obtained Will and Perform splicing in the channel dimension and then obtain features through a 1*1 convolution layer Then Perform broadcast processing on the channel dimension to obtain features Will and After dot product, The features processed by the detail information supplement module are obtained by fusion.

9. The multimodal object detection system based on a dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 6 or 7, characterized in that: The semantic information supplement module processing process includes: First, the feature F output by the residual module of the infrared branch backbone network inf After an average pooling, the features are obtained At the same time, the feature F output by the residual module of the optical branch backbone network rgb After an average pooling, the features are obtained Then and Perform splicing on the channel dimension, perform 1*1 convolution on the features of the channel dimension splicing, and then process it through the FC layer to obtain the feature W′(F rgb ,F inf ), then W′(F rgb ,F inf ) and F rgb After dot product, add F rgb The features processed by the semantic information supplement module are obtained by fusion.

10. The multimodal object detection system based on a dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 6 or 7, characterized in that: The feature fusion unit uses the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids. The process includes the following steps: For the corresponding layer features of the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network, the linear layer of the channel dimension is first used to obtain the linear mapping of the three subspaces, namely the query tensor, key tensor and value tensor; the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q rgb , K rgb 、V rgb , the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q inf , K inf 、V inf ; Based on the query tensor, key tensor, and value tensor obtained, the following processing is performed to obtain the FPFM fusion features of the corresponding layer: in, and is the weighted feature tensor, Sig represents the sigmoid function; C 1×1 (·,·) means that the features of the two inputs are concatenated and then a 1*1 convolution is performed, i.e., concatenation and convolution operations.

Images