Multi-modal target detection method and system based on double-branch asymmetric attention backbone and feature fusion pyramid network

Abstract

The invention discloses a multi-modal target detection method and system based on a double-branch asymmetric attention backbone and a feature fusion pyramid network, and belongs to the technical field of target detection. The objective of the invention is to solve the problem that the existing multi-modal target detection method is difficult to supplement features lacked by each mode, so that the multi-modal current detection effect is affected. An optical image and an infrared image are used as inputs of an optical branch backbone network and an infrared branch backbone network, each branch backbone network comprises a convolutional layer and a plurality of residual error modules, and a detail information supplementing module or a semantic information supplementing module is arranged between each group of residual error modules. The detail information supplement module and the semantic information supplement module are alternately arranged in the sequential processing direction of the residual module; and two feature pyramids obtained by the two branches are sent to a feature fusion unit, the feature fusion unit performs fusion by using features of each layer of a feature fusion pyramid network, and final multi-scale fusion features are sent to a detection unit for detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a multimodal target detection method and system based on computer vision. Background Art

[0002] At present, multimodal target detection methods based on deep learning have gradually become a research hotspot. Figure 1 As shown in the figure, according to the different fusion stages, the fusion strategies of multimodal target detection algorithms can generally be divided into image-level fusion, feature-level fusion and decision-level fusion. Figure 1 In the figure, the red blocks represent feature extraction, the green blocks represent fusion operations, and the blue blocks represent target detection. Studies have shown that compared with the two strategies of image-level fusion and decision-level fusion, fusing different modal features in the middle of the network is more likely to obtain better multimodal detection results. Therefore, the mid-way fusion method has gradually become the default fusion strategy for multimodal target detection based on deep learning. At present, the attention mechanism is becoming a hot topic of research. Zhang et al. proposed ResNeST, a backbone network based on the self-attention mechanism. Li et al. used the fusion features obtained by concatenation to weight the modal features, and then added the branch features to obtain the fusion features. Yang et al. proposed a gated convolution structure to obtain the fusion features after weighting the modal features. However, effective information exists in different forms in infrared images and optical images, such as temperature information in infrared images and global detail information in optical images. The distribution of different forms of effective information in each dimension is different, and using the same attention structure will reduce the learning efficiency of the network. In addition, information compensation during feature extraction can reduce the difficulty of the network to mine target features. Both Li and Yang performed feature fusion after feature extraction.

[0003] Recently, the Transformer structure has shown increasing competitiveness. Fang et al. introduced the Transformer architecture into the field of feature fusion and achieved great success. The features are converted into vector form, and then matrix multiplication is performed to obtain the fusion features of the self-attention component and the mutual attention component. While performing information fusion, attention operations are performed on each modal component. However, the structure of information fusion in this method is symmetrical, and the imaging characteristics of infrared and RGB images are not taken into account, so the lack of information is supplemented.

[0004] Multimodal data are different and complementary. On the one hand, many of the above fusion methods do not consider the differences and directly fuse multimodal features, which may lead to the introduction of features that are not conducive to detection, thereby reducing the performance of target detection. On the other hand, although some methods use attention structures to enhance target features, they only weight each modal feature separately without performing direct information complementation operations. In some methods, although the above two characteristics of multimodal data are considered, the attention operation is designed to be symmetrical. In other words, the information complementation structure between the optical branch and the infrared branch is the same, which makes it difficult to supplement the features lacking in each mode. The above are the problems that need to be solved urgently in the current multimodal target detection tasks. Summary of the invention

[0005] The present invention aims to solve the problem that the existing multimodal target detection method is difficult to supplement the features lacking in each mode, thereby affecting the current detection effect of the multimodal.

[0006] A multimodal target detection method based on a dual-branch asymmetric attention backbone and a feature fusion pyramid network obtains an optical image and an infrared image of a region to be detected, and implements target detection using a multimodal target detection network model. The multimodal target detection network model includes a feature extraction unit, a feature fusion unit, and a detection unit. The processing process of the multimodal target detection network model includes:

[0007] The feature extraction unit adopts a dual-branch asymmetric attention backbone network, which includes an optical branch backbone network, an infrared branch backbone network, and a detail information supplement module and a semantic information supplement module; the detail information supplement module and the semantic information supplement module are recorded as a detail information supplement module and a semantic information supplement module;

[0008] The optical image and the infrared image are used as the input of the optical branch backbone network and the infrared branch backbone network. The optical branch backbone network and the infrared branch backbone network both include a convolution layer and a plurality of residual modules, and the number of residual modules of the two is equal. The residual modules corresponding to the optical branch backbone network and the infrared branch backbone network are called a group of residual modules. A detail information supplement module or a semantic information supplement module is set between each group of residual modules. The detail information supplement module and the semantic information supplement module are alternately set in the sequential processing direction of the residual module.

[0009] The detail information supplement module uses the outputs of the two residual modules of this group as input, extracts detail features for compensating the residual modules corresponding to the infrared branch backbone network, that is, the features obtained by the detail information supplement module are used as the features of the output of the residual module of the current group corresponding to the infrared branch backbone network, and are used for processing the next residual module of the infrared branch backbone network;

[0010] The semantic information supplement module uses the outputs of the two residual modules of this group as input, extracts speech features for compensating the residual modules corresponding to the optical branch backbone network, that is, the features processed by the semantic information supplement module are used as the features output by the residual modules of the current group corresponding to the optical branch backbone network, and are used for processing the next residual module of the optical branch backbone network;

[0011] Based on the optical branch backbone network and the infrared branch backbone network, a feature pyramid corresponding to the optical branch backbone network and a feature pyramid corresponding to the infrared branch backbone network are obtained. The number of feature pyramids corresponding to the optical branch backbone network and the feature pyramids corresponding to the infrared branch backbone network are the same, and the features of each layer of the two feature pyramids correspond to each other.

[0012] Then the two feature pyramids are sent to the feature fusion unit, and the feature fusion unit uses the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids. A FPFM fusion feature is obtained for each layer of the feature pyramid. Based on the FPFM fusion features of all layers, the feature pyramid processing method is used to obtain fusion features of multiple scales.

[0013] Finally, the final fusion features of multiple scales are sent to the detection unit for detection, thereby realizing the detection of the target.

[0014] Preferably, a detail information supplement module or a semantic information supplement module is provided between each group of residual modules as follows: a detail information supplement module is provided between odd-numbered groups of residual modules, and a semantic information supplement module is provided between even-numbered groups of residual modules.

[0015] Furthermore, the detailed information supplementation module processing process includes:

[0016] First, the feature F output by the residual module of the optical branch backbone network rgb After a 3*3 convolution layer, feature F1 is obtained rgb , F1 rgb Perform maximum pooling and average pooling respectively, and concatenate the pooled features in the channel dimension. Then perform 1*1 convolution on the concatenated features in the channel dimension to obtain the spatial principal component sampling matrix W(F1 rgb );

[0017] At the same time, the feature F output by the residual module of the infrared branch backbone network inf After a 1*1 convolution layer, feature F1 is obtained inf ;

[0018] F1 inf With W(F1 rgb ) to perform splicing in the channel dimension, and then obtain the features through a 1*1 convolution layer Then Perform broadcast processing on the channel dimension to obtain the feature Will With F1 inf After dot multiplication, The features processed by the detail information supplement module are obtained by fusion.

[0019] Furthermore, the processing of the semantic information supplementation module includes:

[0020] First, the feature F output by the residual module of the infrared branch backbone network inf After an average pooling, feature F2 is obtained inf , and the feature F output by the residual module of the optical branch backbone network rgb After an average pooling, feature F2 is obtained rgb ; then press F2 inf and F2 rgb Perform splicing on the channel dimension, perform 1*1 convolution on the features of the channel dimension splicing, and then process them through the FC layer to obtain the feature W′(F rgb ,F inf ), then W′(F rgb ,F inf ) and F rgb After dot product, add F rgb The features processed by the semantic information supplement module are obtained by fusion.

[0021] Furthermore, the process in which the feature fusion unit fuses the features corresponding to each layer of the two feature pyramids using the feature fusion pyramid network FPFM includes the following steps:

[0022] For the corresponding layer features of the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network, the linear layer of the channel dimension is first used to obtain the linear mapping of the three subspaces, namely the query tensor, key tensor and value tensor; the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q rgb , K rgb 、V rgb , the query tensor, key tensor, and value tensor corresponding to the optical branch are denoted as Q inf , K inf 、V inf ;

[0023] Based on the obtained query tensor, key tensor, and value tensor, the following processing is performed to obtain the FPFM fusion features of the corresponding layer:

[0024]

[0025] in, and is the weighted feature tensor, Sig represents the sigmoid function; C 1×1 (·,·) means that the features of the two inputs are concatenated and then subjected to 1*1 convolution, i.e., concatenation and convolution operations.

[0026] A multimodal object detection system based on a dual-branch asymmetric attention backbone and feature fusion pyramid network, including:

[0027] Data acquisition module for the area to be detected: used to obtain optical images and infrared images of the area to be detected;

[0028] Target detection module: For optical images and infrared images, a multimodal target detection network model is used to perform target detection;

[0029] The multimodal target detection network model includes a feature extraction unit, a feature fusion unit and a detection unit. The processing process of the multimodal target detection network model includes:

[0030] The feature extraction unit adopts a dual-branch asymmetric attention backbone network, which includes an optical branch backbone network, an infrared branch backbone network, and a detail information supplement module and a semantic information supplement module; the detail information supplement module and the semantic information supplement module are recorded as a detail information supplement module and a semantic information supplement module;

[0031] The optical image and the infrared image are used as the input of the optical branch backbone network and the infrared branch backbone network. The optical branch backbone network and the infrared branch backbone network both include a convolution layer and a plurality of residual modules, and the number of residual modules of the two is equal. The residual modules corresponding to the optical branch backbone network and the infrared branch backbone network are called a group of residual modules. A detail information supplement module or a semantic information supplement module is set between each group of residual modules. The detail information supplement module and the semantic information supplement module are alternately set in the sequential processing direction of the residual module.

[0032] The detail information supplement module uses the outputs of the two residual modules of this group as input, extracts detail features for compensating the residual modules corresponding to the infrared branch backbone network, that is, the features obtained by the detail information supplement module are used as the features of the output of the residual module of the current group corresponding to the infrared branch backbone network, and are used for processing the next residual module of the infrared branch backbone network;

[0033] The semantic information supplement module uses the outputs of the two residual modules of this group as input, extracts speech features for compensating the residual modules corresponding to the optical branch backbone network, that is, the features processed by the semantic information supplement module are used as the features output by the residual modules of the current group corresponding to the optical branch backbone network, and are used for processing the next residual module of the optical branch backbone network;

[0034] Based on the optical branch backbone network and the infrared branch backbone network, a feature pyramid corresponding to the optical branch backbone network and a feature pyramid corresponding to the infrared branch backbone network are obtained. The number of feature pyramids corresponding to the optical branch backbone network and the feature pyramids corresponding to the infrared branch backbone network are the same, and the features of each layer of the two feature pyramids correspond to each other.

[0035] Then the two feature pyramids are sent to the feature fusion unit, and the feature fusion unit uses the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids. A FPFM fusion feature is obtained for each layer of the feature pyramid. Based on the FPFM fusion features of all layers, the feature pyramid processing method is used to obtain fusion features of multiple scales.

[0036] Finally, the final fusion features of multiple scales are sent to the detection unit for detection, thereby realizing the detection of the target.

[0037] Preferably, a detail information supplement module or a semantic information supplement module is provided between each group of residual modules as follows: a detail information supplement module is provided between odd-numbered groups of residual modules, and a semantic information supplement module is provided between even-numbered groups of residual modules.

[0038] Furthermore, the detailed information supplementation module processing process includes:

[0039] First, the feature F output by the residual module of the optical branch backbone network rgb After a 3*3 convolution layer, feature F1 is obtained rgb , F1 rgb Perform maximum pooling and average pooling respectively, and concatenate the pooled features in the channel dimension. Then perform 1*1 convolution on the concatenated features in the channel dimension to obtain the spatial principal component sampling matrix W(F1 rgb );

[0040] At the same time, the feature F output by the residual module of the infrared branch backbone network inf After a 1*1 convolution layer, feature F1 is obtained inf ;

[0041] F1 inf With W(F1 rgb ) to perform splicing in the channel dimension, and then obtain the features through a 1*1 convolution layer Then Perform broadcast processing on the channel dimension to obtain the feature Will With F1 inf After dot multiplication, The features processed by the detail information supplement module are obtained by fusion.

[0042] Furthermore, the processing of the semantic information supplementation module includes:

[0043] First, the feature F output by the residual module of the infrared branch backbone network inf After an average pooling, feature F2 is obtained inf , and the feature F output by the residual module of the optical branch backbone network rgb After an average pooling, feature F2 is obtained rgb ; then press F2 inf and F2 rgb Perform splicing on the channel dimension, perform 1*1 convolution on the features of the channel dimension splicing, and then process them through the FC layer to obtain the feature W′(F rgb ,F inf ), then W′(F rgb ,F inf ) and F rgb After dot product, add F rgb The features processed by the semantic information supplement module are obtained by fusion.

[0044] Furthermore, the process in which the feature fusion unit fuses the features corresponding to each layer of the two feature pyramids using the feature fusion pyramid network FPFM includes the following steps:

[0045] For the corresponding layer features of the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network, the linear layer of the channel dimension is first used to obtain the linear mapping of the three subspaces, namely the query tensor, key tensor and value tensor; the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q rgb , K rgb 、V rgb , the query tensor, key tensor, and value tensor corresponding to the optical branch are denoted as Q inf , K inf 、V inf ;

[0046] Based on the obtained query tensor, key tensor, and value tensor, the following processing is performed to obtain the FPFM fusion features of the corresponding layer:

[0047]

[0048] in, and is the weighted feature tensor, Sig represents the sigmoid function; C 1×1 (·,·) means that the features of the two inputs are concatenated and then subjected to 1*1 convolution, i.e., concatenation and convolution operations.

[0049] Aiming at the characteristics of remote sensing multimodal image data, this paper studies how to obtain high-quality fusion feature representation while suppressing interference information, and proposes a new fusion target detection method. First, considering the differences in different modal data, a new dual-branch asymmetric attention backbone network is proposed to improve the reliability of each modal information; specifically, the backbone supplements the target information for infrared and RGB images through the detail information supplement module and the semantic information supplement module. Secondly, in order to better utilize the complementarity of multimodal data, a feature pyramid fusion module is designed to enhance the weight of another modal information through one modality, thereby enhancing useful information that is conducive to fusion and suppressing redundant information that is not conducive to fusion. Finally, multi-level and multi-scale fusion features are obtained. Therefore, the present invention can effectively improve the feature extraction capability of each modality and the information supplement capability between each modality, thereby improving the effect of multimodal target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram of three types of information fusion methods.

[0051] Figure 2 This is the overall architecture diagram of the algorithm.

[0052] Figure 3 Schematic diagram of the dual-branch asymmetric attention backbone network.

[0053] Figure 4 Supplement the module schematic for detailed information.

[0054] Figure 5 Schematic diagram of the semantic information supplement module.

[0055] Figure 6 Schematic diagram of feature fusion pyramid network.

[0056] Figure 7 This is a diagram of the subjective detection results of the present invention on the drone dataset. DETAILED DESCRIPTION

[0057] Since there are obvious modal differences between optical images and infrared images, the present invention adopts a dual-branch trunk to extract different modal features respectively. Considering the differences between the modes, the semantic information in the infrared branch is used to enhance the optical features, and the detail information in the optical branch is used to enhance the infrared features. In order to further obtain a good joint feature representation, a fusion structure of the Transformer strategy is proposed to enhance the useful information in the fusion stage and suppress the redundant information that is not conducive to fusion. Finally, the fused features are used for target detection. The following is an explanation in conjunction with a specific implementation method.

[0058] Specific implementation method 1: Combination Figure 2 To explain this embodiment,

[0059] This embodiment is a multimodal target detection method based on a dual-branch asymmetric attention backbone and a feature fusion pyramid network, which obtains an optical image and an infrared image of the area to be detected, and then uses a multimodal target detection network model to achieve target detection. The multimodal target detection network model is as follows: Figure 2 As shown, it includes three parts: feature extraction unit, feature fusion unit and detection unit.

[0060] The feature extraction unit adopts a dual-branch asymmetric attention backbone network. Optical images and infrared images have their own advantages and disadvantages, and their information is complementary in some challenging scenarios. The backbone network is the basic information extractor of the target detection network. In some architectures, feature fusion is only performed at the neck after feature extraction, and the focus of network training is on the fusion structure. However, compared with the fusion structure, this training process improves the backbone network much less. Therefore, the present invention designs a new dual-branch backbone to better extract optical and infrared image features. Due to the difference in imaging principles, optical images tend to have more detailed information; however, infrared images are sensitive to temperature, and the edges between the target and the background are often more obvious and have more semantic information. Therefore, a detail information supplementation module and a semantic information supplementation module are used in the backbone network to supplement the information of each mode image feature and improve the quality of each mode image feature.

[0061] The dual-branch asymmetric attention backbone network includes an optical branch backbone network, an infrared branch backbone network, a detail information supplementation module DISM, and a semantic information supplementation module SISM;

[0062] The optical image and the infrared image are used as the input of the optical branch backbone network and the infrared branch backbone network. The optical branch backbone network and the infrared branch backbone network both include a convolution layer and a number of residual modules, and the number of residual modules of the two is equal; the residual modules corresponding to the optical branch backbone network and the infrared branch backbone network are called a group of residual modules, and a DISM or SISM is set between each group of residual modules, and the DISM and SISM are alternately set in the sequential processing direction of the residual modules; in this embodiment, the DISM is set between the residual modules of the odd group, and the SISM is set between the residual modules of the even group. After the optical branch backbone network and the infrared branch backbone network perform feature extraction, two feature pyramids corresponding to the two modes are formed respectively; then the two feature pyramids are sent to the feature fusion unit.

[0063] Generally speaking, details and texture information are extracted more in the shallow layers of the network, while semantic information is extracted more in the deep layers. In the design process of the method of the present invention, a shallower detail information supplement module DISM and a deeper semantic information supplement module SISM are used. In each stage, only one information supplement module (detail information supplement module or semantic information supplement module) is used to establish a unidirectional information path in this area to improve the convergence speed of the network. Therefore, the detail information supplement module and the semantic information supplement module can appear in pairs to form a loop structure, such as Figure 3 The loop structure shown in (in fact, the main structure of the dual-branch asymmetric attention backbone network can be formed Figure 3 The loop structure shown in the figure) can be inserted at any stage. In the dual-branch asymmetric attention backbone network, in order to focus on the proposed dual-branch architecture and loop structure, the optical branch backbone network and the infrared branch backbone network of this embodiment use the same residual structure as ResNet50, and the network depth of the optical branch backbone network and the infrared branch backbone network is the same.

[0064] like Figure 4 As shown, the detailed information supplement module DISM is as follows:

[0065] In the detail information supplementation module, we improve on the spatial attention and use the optical image features to supplement the details of the infrared image features. In order to supplement the details more directly, we broadcast the dimension of the weight matrix after the spatial attention operation to make it consistent with the input feature dimension, and then add it to the enhanced infrared features. Such an operation can directly introduce the information obtained from the optical branch and further enhance the influence of the weight matrix on the features.

[0066] First, the feature F output by the residual module of the optical branch backbone network rgb After a 3*3 convolution layer, feature F1 is obtained rgb , F1 rgb Perform maximum pooling and average pooling respectively, and concatenate the pooled features in the channel dimension, such as Figure 4 The c in the figure represents the operation on the channel dimension. Then the concatenated features of the channel dimension are convolved by 1*1 to obtain the spatial principal component sampling matrix W(F1 rgb );

[0067] At the same time, the feature F output by the residual module of the infrared branch backbone network inf After a 1*1 convolution layer, feature F1 is obtained inf ;

[0068] F1 inf With W(F1 rgb) to perform splicing in the channel dimension, and then obtain the features through a 1*1 convolution layer Then Perform broadcast processing on the channel dimension to obtain the feature Will With F1 inf After dot multiplication, The features processed by DISM are obtained by fusion, and the features processed by DISM are used as the features output by the residual module corresponding to the infrared branch backbone network, and used for processing the next residual module of the infrared branch backbone network;

[0069] The DISM process can be expressed as

[0070]

[0071] DISM (F1 rgb ,F1 inf )=B c [C 1×1 [W(F1 rgb ),C 1×1 (F inf )]]+C 1×1 [W(F1 rgb ),C 1×1 (F inf )]·F1 inf (2)

[0072] Among them, Pool represents the pooling operation. After pooling, the channel dimension is 1; B c represents the broadcast of the channel dimension, C 1×1 Represents a scale-invariant convolution operation with a kernel size of 1. The two internal parameters represent the concatenation before processing.

[0073] It is worth noting that the present invention uses pooling operations to obtain the spatial principal component sampling matrix of the detailed information source, and uses convolution operations to obtain the spatial sampling matrix of the supplementary infrared branch features. This is because we hope that DISM can adapt to the degree of missing infrared features. This design allows the network to more dynamically supplement details to varying degrees.

[0074] like Figure 5 As shown, the semantic information supplement module SISM is as follows:

[0075] In the semantic information supplementation module, we improve on channel attention and use infrared images to supplement the semantic information of optical image features. Since the target features focused on by different feature branches are not necessarily similar (infrared images may focus on contours, while optical images may focus on details), direct feature supplementation may interfere with the optical branch. Therefore, in this module, we use a weighted approach to enhance the semantic information of optical branch features.

[0076] First, the feature F output by the residual module of the infrared branch backbone network inf After an average pooling, feature F2 is obtained inf , and the feature F output by the residual module of the optical branch backbone network rgb After an average pooling, feature F2 is obtained rgb ; then press F2 inf and F2 rgb Perform splicing on the channel dimension, perform 1*1 convolution on the features of the channel dimension splicing, and then process them through the FC layer to obtain the feature W′(F rgb ,F inf ), then W′(F rgb ,F inf ) and F rgb After dot product, add F rgb The features after SISM processing are obtained by fusion, and the features after SISM processing are used as the features output by the residual module corresponding to the optical branch backbone network, and are used for processing the next residual module of the optical branch backbone network;

[0077] The SISM process can be expressed as

[0078]

[0079] SISM(F rgb ,F inf )=F rgb +W′(F rgb ,F inf )·F rgb (4)

[0080] in, It indicates that the pooling operation is performed along the high dimension and the wide dimension. After the operation, the high dimension and the wide dimension of the feature tensor output are both 1; FC refers to the fully connected layer; W′ is the channel attention weight vector obtained based on the two modal features.

[0081] The output of the semantic information supplementation module is the sum of the light branch features enhanced by multimodal features and their pre-enhancement features.

[0082] Based on the optical branch backbone network and the infrared branch backbone network, the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network are obtained. The number of feature pyramids corresponding to the optical branch backbone network and the feature pyramids corresponding to the infrared branch backbone network are the same, and the features of each layer of the two feature pyramids correspond to each other.

[0083] The feature fusion unit uses the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids to obtain a high-quality fused feature pyramid. After obtaining the feature information of the infrared image and the optical image, in order to obtain a high-quality joint feature expression, the present invention designs a fusion symmetric decoder structure based on the Transformer strategy, which enhances the features of each modal scale point by point based on the feature fusion pyramid network FFPN for multimodal information, enhances useful information, and suppresses interference information that is not conducive to fusion. The detailed structure of FFPN is shown in Figure 6 As shown. In order to integrate multimodal features, taking the optical branch as an example, for each layer of the feature pyramid corresponding to the optical branch backbone network, the weight matrix is ​​calculated using the query tensor of the infrared branch feature and the bond tensor of the optical branch, and mapped to the optical feature. Through cross-modal feature enhancement, high-quality co-expression can be obtained, further improving the detection performance.

[0084] For the corresponding layer features of the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network, the linear layer of the channel dimension is first used to obtain the linear mapping of the three subspaces, namely the query tensor, key tensor and value tensor, that is, Q, K, and V are all derived from the feature mapping, Q rgb and Q inf The calculation method is the same as;

[0085] Taking the optical branch as an example, the query tensor, key tensor, and value tensor obtained are calculated as follows:

[0086]

[0087] in, All are 1*1 convolution processing, and convolution symbols with different superscripts represent different convolutions.

[0088] The same process is performed for the infrared branch.

[0089] Based on the obtained query tensor, key tensor, and value tensor, the following processing is performed to obtain the FPFM fusion features of the corresponding layer:

[0090]

[0091] in, and is the weighted feature tensor, Sig represents the sigmoid function, which can map the feature value to between 0 and 1 as the enhanced weight of the feature point; C 1×1 (·,·) means that two features are connected in series and then 1*1 convolution is performed, that is, concatenation and convolution operations. Figure 6 Indicated by C&C. Figure 6E-Pro and E-Plus represent the product of elements and the addition of elements.

[0092] A FPFM fusion feature is obtained for each layer of the feature pyramid. Based on the FPFM fusion features of all layers, the feature pyramid processing method is used to obtain fusion features of multiple scales.

[0093] Finally, the final fusion features of multiple scales are sent to the detection unit for detection, thereby realizing the detection of the target.

[0094] Example

[0095] The experimental equipment uses Intel(R) Xeon(R) Silver 4210R CPU and the operating system is Ubuntu 18.04. The method was trained on NVIDIA RTX 3090 GPU and implemented in Pytorch framework. We use 1024 as the input image size, batch size is 4, and initial learning rate is 0.008. The optimizer uses stochastic gradient descent (SGD) algorithm, the decay weight is 0.0001, and the momentum is 0.937. The present invention uses mAP50 to evaluate each model.

[0096] To further verify the advancedness of the proposed method, we conducted comparative experiments with other advanced algorithms on the public dataset DroneVehicle from the perspective of drones.

[0097] Table 1. Performance of various methods on the DroneVehicle dataset

[0098]

[0099] The “Modality” column in Table 1 indicates which modality, optical or infrared, is used to achieve target detection.

[0100] The subjective detection results of the method of the present invention on the drone dataset are as follows: Figure 7 As shown. The first line is an optical image, the second line is an infrared image of the marked target position, and the third line is a visualization of the detection results of the method of the present invention. Figure 7 In the first and second columns, some objects are in a low detection state in the optical image, which is difficult for the target detection algorithm to detect, and even the human eye has difficulty distinguishing them. In the third and fourth columns, some targets in the infrared image are difficult to identify with the naked eye, and the optical image needs to be supplemented with color information to confirm the target position. The method proposed in the present invention maintains good detection performance in the above cases.

[0101] Specific implementation method 2: Combination Figure 2 To explain this embodiment,

[0102] This embodiment is a multimodal target detection system based on a dual-branch asymmetric attention backbone and a feature fusion pyramid network, which is at least a software system corresponding to the program of a multimodal target detection method based on a dual-branch asymmetric attention backbone and a feature fusion pyramid network described in the specific embodiment 1, and may also include a processor and a memory, etc. It should be understood that including any device including a processor and a memory described in the present invention, the device may also include other units and modules that perform display, interaction, processing, control, etc. and other functions through signals or instructions.

[0103] A multimodal target detection system based on a dual-branch asymmetric attention backbone and a feature fusion pyramid network described in this embodiment includes:

[0104] Data acquisition module for the area to be detected: used to obtain optical images and infrared images of the area to be detected;

[0105] Target detection module: For optical images and infrared images, a multimodal target detection network model is used to perform target detection;

[0106] The multimodal target detection network model includes a feature extraction unit, a feature fusion unit and a detection unit. The processing process of the multimodal target detection network model includes:

[0107] The feature extraction unit adopts a dual-branch asymmetric attention backbone network, which includes an optical branch backbone network, an infrared branch backbone network, and a detail information supplement module and a semantic information supplement module; the detail information supplement module and the semantic information supplement module are recorded as a detail information supplement module and a semantic information supplement module;

[0108] The optical image and the infrared image are used as the input of the optical branch backbone network and the infrared branch backbone network. The optical branch backbone network and the infrared branch backbone network both include a convolution layer and a plurality of residual modules, and the number of residual modules of the two is equal. The residual modules corresponding to the optical branch backbone network and the infrared branch backbone network are called a group of residual modules. A detail information supplement module or a semantic information supplement module is arranged between each group of residual modules. The detail information supplement module and the semantic information supplement module are arranged alternately in the sequential processing direction of the residual modules. Preferably, the manner of arranging a detail information supplement module or a semantic information supplement module between each group of residual modules is as follows: a detail information supplement module is arranged between odd-group residual modules, and a semantic information supplement module is arranged between even-group residual modules.

[0109] The detail information supplement module uses the outputs of the two residual modules of this group as input, extracts detail features for compensating the residual modules corresponding to the infrared branch backbone network, that is, the features obtained by the detail information supplement module are used as the features of the output of the residual module of the current group corresponding to the infrared branch backbone network, and are used for processing the next residual module of the infrared branch backbone network;

[0110] The semantic information supplement module uses the outputs of the two residual modules of this group as input, extracts speech features for compensating the residual modules corresponding to the optical branch backbone network, that is, the features processed by the semantic information supplement module are used as the features output by the residual modules of the current group corresponding to the optical branch backbone network, and are used for processing the next residual module of the optical branch backbone network;

[0111] Based on the optical branch backbone network and the infrared branch backbone network, a feature pyramid corresponding to the optical branch backbone network and a feature pyramid corresponding to the infrared branch backbone network are obtained. The number of feature pyramids corresponding to the optical branch backbone network and the feature pyramids corresponding to the infrared branch backbone network are the same, and the features of each layer of the two feature pyramids correspond to each other.

[0112] Then the two feature pyramids are sent to the feature fusion unit, and the feature fusion unit uses the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids. A FPFM fusion feature is obtained for each layer of the feature pyramid. Based on the FPFM fusion features of all layers, the feature pyramid processing method is used to obtain fusion features of multiple scales.

[0113] Finally, the final fusion features of multiple scales are sent to the detection unit for detection, thereby realizing the detection of the target.

[0114] The detailed information supplementation module processing process includes:

[0115] First, the feature F output by the residual module of the optical branch backbone network rgb After a 3*3 convolution layer, feature F1 is obtained rgb , F1 rgb Perform maximum pooling and average pooling respectively, and concatenate the pooled features in the channel dimension. Then perform 1*1 convolution on the concatenated features in the channel dimension to obtain the spatial principal component sampling matrix W(F1 rgb );

[0116] At the same time, the feature F output by the residual module of the infrared branch backbone network inf After a 1*1 convolution layer, feature F1 is obtained inf ;

[0117] F1 inf With W(F1rgb ) to perform splicing in the channel dimension, and then obtain the features through a 1*1 convolution layer Then Perform broadcast processing on the channel dimension to obtain the feature Will With F1 inf After dot multiplication, The features processed by the detail information supplement module are obtained by fusion.

[0118] The semantic information supplementation module processing process includes:

[0119] First, the feature F output by the residual module of the infrared branch backbone network inf After an average pooling, feature F2 is obtained inf , and the feature F output by the residual module of the optical branch backbone network rgb After an average pooling, feature F2 is obtained rgb ; then press F2 inf and F2 rgb Perform splicing on the channel dimension, perform 1*1 convolution on the features of the channel dimension splicing, and then process them through the FC layer to obtain the feature W′(F rgb ,F inf ), then W′(F rgb ,F inf ) and F rgb After dot product, add F rgb The features processed by the semantic information supplement module are obtained by fusion.

[0120] The process of the feature fusion unit fusing the features corresponding to each layer of the two feature pyramids using the feature fusion pyramid network FPFM includes the following steps:

[0121] For the corresponding layer features of the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network, the linear layer of the channel dimension is first used to obtain the linear mapping of the three subspaces, namely the query tensor, key tensor and value tensor; the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q rgb , K rgb 、V rgb , the query tensor, key tensor, and value tensor corresponding to the optical branch are denoted as Q inf , K inf 、V inf ;

[0122] Based on the obtained query tensor, key tensor, and value tensor, the following processing is performed to obtain the FPFM fusion features of the corresponding layer:

[0123]

[0124] in, and is the weighted feature tensor, Sig represents the sigmoid function; C 1×1 (·,·) means that the features of the two inputs are concatenated and then subjected to 1*1 convolution, i.e., concatenation and convolution operations.

[0125] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A multimodal target detection method based on a dual-branch asymmetric attention backbone and a feature fusion pyramid network obtains optical images and infrared images of the area to be detected, and uses a multimodal target detection network model to realize target detection, which is characterized by: The multimodal target detection network model includes a feature extraction unit, a feature fusion unit and a detection unit. The processing process of the multimodal target detection network model includes: The feature extraction unit adopts a dual-branch asymmetric attention backbone network, which includes an optical branch backbone network, an infrared branch backbone network, and a detail information supplement module and a semantic information supplement module; the detail information supplement module and the semantic information supplement module are recorded as a detail information supplement module and a semantic information supplement module; The optical image and the infrared image are used as the input of the optical branch backbone network and the infrared branch backbone network. The optical branch backbone network and the infrared branch backbone network both include a convolution layer and a plurality of residual modules, and the number of residual modules of the two is equal. The residual modules corresponding to the optical branch backbone network and the infrared branch backbone network are called a group of residual modules. A detail information supplement module or a semantic information supplement module is set between each group of residual modules. The detail information supplement module and the semantic information supplement module are alternately set in the sequential processing direction of the residual module. The detail information supplement module uses the outputs of the two residual modules of this group as input, extracts detail features for compensating the residual modules corresponding to the infrared branch backbone network, that is, the features obtained by the detail information supplement module are used as the features of the output of the residual module of the current group corresponding to the infrared branch backbone network, and are used for processing the next residual module of the infrared branch backbone network; The semantic information supplement module uses the outputs of the two residual modules of this group as input, extracts speech features for compensating the residual modules corresponding to the optical branch backbone network, that is, the features processed by the semantic information supplement module are used as the features output by the residual modules of the current group corresponding to the optical branch backbone network, and are used for processing the next residual module of the optical branch backbone network; Based on the optical branch backbone network and the infrared branch backbone network, a feature pyramid corresponding to the optical branch backbone network and a feature pyramid corresponding to the infrared branch backbone network are obtained. The number of feature pyramids corresponding to the optical branch backbone network and the feature pyramids corresponding to the infrared branch backbone network are the same, and the features of each layer of the two feature pyramids correspond to each other. Then the two feature pyramids are sent to the feature fusion unit, and the feature fusion unit uses the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids. A FPFM fusion feature is obtained for each layer of the feature pyramid. Based on the FPFM fusion features of all layers, the feature pyramid processing method is used to obtain fusion features of multiple scales. Finally, the final fusion features of multiple scales are sent to the detection unit for detection, thereby realizing the detection of the target.

2. The multimodal target detection method based on a dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 1, characterized in that: The manner of setting a detail information supplement module or a semantic information supplement module between each group of residual modules is as follows: setting a detail information supplement module between odd-numbered residual modules and setting a semantic information supplement module between even-numbered residual modules.

3. The multimodal target detection method based on dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 1 or 2, characterized in that: The detailed information supplement module processing process includes: First, the feature F output by the residual module of the optical branch backbone network rgb After a 3*3 convolution layer, the features are obtained Will Perform maximum pooling and average pooling respectively, and concatenate the pooled features in the channel dimension. Then perform 1*1 convolution on the concatenated features in the channel dimension to obtain the spatial principal component sampling matrix of the optical branch. At the same time, the feature F output by the residual module of the infrared branch backbone network inf After a 1*1 convolution layer, the features are obtained Will and Perform splicing in the channel dimension, and then obtain the features through a 1*1 convolution layer Then Perform broadcast processing on the channel dimension to obtain the feature Will and After dot multiplication, The features processed by the detail information supplement module are obtained by fusion.

4. The multimodal target detection method based on dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 1 or 2, characterized in that: The semantic information supplementation module processing process includes: First, the feature F output by the residual module of the infrared branch backbone network inf After an average pooling, the features are obtained At the same time, the feature F output by the residual module of the optical branch backbone network rgb After an average pooling, the features are obtained Then and Perform splicing on the channel dimension, perform 1*1 convolution on the features of the channel dimension splicing, and then process them through the FC layer to obtain the feature W′(F rgb ,F inf ), then W′(F rgb ,F inf ) and F rgb After dot product, add rgb The features processed by the semantic information supplement module are obtained by fusion.

5. The multimodal target detection method based on dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 1 or 2, characterized in that: The process of the feature fusion unit fusing the features corresponding to each layer of the two feature pyramids using the feature fusion pyramid network FPFM includes the following steps: For the corresponding layer features of the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network, the linear layer of the channel dimension is first used to obtain the linear mapping of the three subspaces, namely the query tensor, key tensor and value tensor; the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q rgb , K rgb 、V rgb , the query tensor, key tensor, and value tensor corresponding to the optical branch are denoted as Q inf , K inf 、V inf ; Based on the obtained query tensor, key tensor, and value tensor, the following processing is performed to obtain the FPFM fusion features of the corresponding layer: in, and is the weighted feature tensor, Sig represents the sigmoid function; C 1×1 (·,·) means that the features of the two inputs are concatenated and then subjected to 1*1 convolution, i.e., concatenation and convolution operations.

6. A multimodal object detection system based on a dual-branch asymmetric attention backbone and feature fusion pyramid network, characterized by: include: Data acquisition module for the area to be detected: used to obtain optical images and infrared images of the area to be detected; Target detection module: For optical images and infrared images, a multimodal target detection network model is used to perform target detection; The multimodal target detection network model includes a feature extraction unit, a feature fusion unit and a detection unit. The processing process of the multimodal target detection network model includes: The feature extraction unit adopts a dual-branch asymmetric attention backbone network, which includes an optical branch backbone network, an infrared branch backbone network, and a detail information supplement module and a semantic information supplement module; the detail information supplement module and the semantic information supplement module are recorded as a detail information supplement module and a semantic information supplement module; The optical image and the infrared image are used as the input of the optical branch backbone network and the infrared branch backbone network. The optical branch backbone network and the infrared branch backbone network both include a convolution layer and a plurality of residual modules, and the number of residual modules of the two is equal. The residual modules corresponding to the optical branch backbone network and the infrared branch backbone network are called a group of residual modules. A detail information supplement module or a semantic information supplement module is set between each group of residual modules. The detail information supplement module and the semantic information supplement module are alternately set in the sequential processing direction of the residual module. The detail information supplement module uses the outputs of the two residual modules of this group as input, extracts detail features for compensating the residual modules corresponding to the infrared branch backbone network, that is, the features obtained by the detail information supplement module are used as the features of the output of the residual module of the current group corresponding to the infrared branch backbone network, and are used for processing the next residual module of the infrared branch backbone network; The semantic information supplement module uses the outputs of the two residual modules of this group as input, extracts speech features for compensating the residual modules corresponding to the optical branch backbone network, that is, the features processed by the semantic information supplement module are used as the features output by the residual modules of the current group corresponding to the optical branch backbone network, and are used for processing the next residual module of the optical branch backbone network; Based on the optical branch backbone network and the infrared branch backbone network, a feature pyramid corresponding to the optical branch backbone network and a feature pyramid corresponding to the infrared branch backbone network are obtained. The number of feature pyramids corresponding to the optical branch backbone network and the feature pyramids corresponding to the infrared branch backbone network are the same, and the features of each layer of the two feature pyramids correspond to each other. Then the two feature pyramids are sent to the feature fusion unit, and the feature fusion unit uses the feature fusion pyramid network FPFM to fuse the features corresponding to each layer of the two feature pyramids. A FPFM fusion feature is obtained for each layer of the feature pyramid. Based on the FPFM fusion features of all layers, the feature pyramid processing method is used to obtain fusion features of multiple scales. Finally, the final fusion features of multiple scales are sent to the detection unit for detection, thereby realizing the detection of the target.

7. The multimodal target detection system based on dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 6, characterized in that: The manner of setting a detail information supplement module or a semantic information supplement module between each group of residual modules is as follows: setting a detail information supplement module between odd-numbered residual modules and setting a semantic information supplement module between even-numbered residual modules.

8. The multimodal target detection system based on dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 6 or 7, characterized in that: The detailed information supplement module processing process includes: First, the feature F output by the residual module of the optical branch backbone network rgb After a 3*3 convolution layer, the features are obtained Will Perform maximum pooling and average pooling respectively, and concatenate the pooled features in the channel dimension. Then perform 1*1 convolution on the concatenated features in the channel dimension to obtain the spatial principal component sampling matrix of the optical branch. At the same time, the feature F output by the residual module of the infrared branch backbone network inf After a 1*1 convolution layer, the features are obtained Will and Perform splicing in the channel dimension, and then obtain the features through a 1*1 convolution layer Then Perform broadcast processing on the channel dimension to obtain the feature Will and After dot multiplication, The features processed by the detail information supplement module are obtained by fusion.

9. The multimodal target detection system based on dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 6 or 7, characterized in that: The semantic information supplementation module processing process includes: First, the feature F output by the residual module of the infrared branch backbone network inf After an average pooling, the features are obtained At the same time, the feature F output by the residual module of the optical branch backbone network rgb After an average pooling, the features are obtained Then and Perform splicing on the channel dimension, perform 1*1 convolution on the features of the channel dimension splicing, and then process them through the FC layer to obtain the feature W′(F rgb ,F inf ), then W′(F rgb ,F inf ) and F rgb After dot product, add F rgb The features processed by the semantic information supplement module are obtained by fusion.

10. The multimodal target detection system based on dual-branch asymmetric attention backbone and feature fusion pyramid network according to claim 6 or 7, characterized in that: The process of the feature fusion unit fusing the features corresponding to each layer of the two feature pyramids using the feature fusion pyramid network FPFM includes the following steps: For the corresponding layer features of the feature pyramid corresponding to the optical branch backbone network and the feature pyramid corresponding to the infrared branch backbone network, the linear layer of the channel dimension is first used to obtain the linear mapping of the three subspaces, namely the query tensor, key tensor and value tensor; the query tensor, key tensor and value tensor corresponding to the optical branch are denoted as Q rgb , K rgb 、V rgb , the query tensor, key tensor, and value tensor corresponding to the optical branch are denoted as Q inf , K inf 、V inf ; Based on the obtained query tensor, key tensor, and value tensor, the following processing is performed to obtain the FPFM fusion features of the corresponding layer: in, and is the weighted feature tensor, Sig represents the sigmoid function; C 1×1 (·,·) means that the features of the two inputs are concatenated and then subjected to 1*1 convolution, i.e., concatenation and convolution operations.

Images