Construction method of cross-modal context attention backbone network for multi-modal pedestrian detection YOLO model

Abstract

The application discloses a kind of multi-modal pedestrian detection YOLO model's construction method of cross-modal context attention backbone network.Four parts are composed of the backbone network: RGB, FIR branch network (CSPDarknet), cross-modal context attention model (CCAM), branch attention model (CBAM) and feature fusion model are composed.RGB, FIR each channel feature fusion result of lower layer is used to the feature of RGB, FIR each channel of upper layer Weight optimization in space, realize the context cross-modal complement of adjacent different scale features.The RGB-FIR fusion features outputted by the backbone network of the existing RGB-FIR multi-modal YOLO model in each scale can be effectively optimized by using the method of the application, and the accuracy and robustness of the pedestrian detection of the multi-modal YOLO in the insufficient light environment are enhanced.

Description

Technical Field

[0001] This invention belongs to the field of deep learning object detection, specifically a method for constructing a cross-modal contextual attention backbone network for a YOLO model for multimodal pedestrian detection. Technical Background

[0002] Object detection is fundamental to other tasks such as object tracking, recognition, and classification, and holds a crucial position in the field of computer vision. Pedestrian detection, as a major branch, has attracted increasing attention from scholars, especially with the rapid development of the autonomous driving industry in recent years, making the improvement of detection accuracy a hot research topic. Changes in climate and lighting conditions can severely affect detection accuracy, and improving the robustness of detection models under different lighting conditions is a pressing issue. Far-infrared images have strong penetration, long range, and are unaffected by lighting conditions, but their drawbacks include image blur and low spatial resolution. Visible light images can complement far-infrared images in these aspects. Therefore, many object detection algorithms based on the fusion of far-infrared and visible light multimodal technologies have emerged in recent years. In 2021, Xue et al. proposed MAF-YOLO in the journal *Infrared Physics and Technology*, which uses the MAF model to fuse the output features of three nodes in the Darknet53 backbone network (RGB and FIR branches). Similarly, in his 2021 patent application for "A Multimodal Pedestrian Detection Method Based on an Improved YOLO Model", Sun Jun proposed a weighted feature fusion layer and a CBAM attention mechanism (MAM+CBAM) to fuse the output features of three nodes in the RGB and FIR branches of the Darknet53 backbone network.

[0003] The above analysis shows that existing RGB-FIR multimodal YOLO models for pedestrian detection focus on how to fuse the output features of the RGB and FIR branch backbone networks at various scales. They do not optimize the backbone network structure across modes and ignore the contextual relationships between features at different scales when using the backbone network to extract basic features. Summary of the Invention

[0004] Objective: To address the aforementioned problems, this invention introduces a method for constructing a cross-modal contextual attention backbone network for YOLO models used in multimodal pedestrian detection. To overcome the shortcomings of existing methods, this invention utilizes the fusion results of RGB-FIR features in each channel of the lower layer (Resblockbady(i)) of the RGB and FIR branch backbone networks to perform spatial weight optimization on the features of each channel of the upper layer (Resblockbady(i+1)). This effectively enhances the influence of the spatial importance of the receptive fields of lower-layer features on the receptive fields of upper-layer features during pedestrian detection, and strengthens the contextual relationships between features of adjacent scales.

[0005] This invention effectively utilizes machine learning and deep learning methods to optimize the backbone network structure of the existing RGB-FIR multimodal YOLO model. This enables the RGB and FIR branch backbone networks to fuse cross-modal contextual relationships before outputting features at different scales, thereby improving the effectiveness of RGB-FIR fused features and enhancing the accuracy and robustness of RGB-FIR multimodal YOLO in pedestrian detection.

[0006] Technical solution: A method for constructing a cross-modal contextual attention backbone network for a YOLO model for multimodal pedestrian detection includes the following steps:

[0007] Step 1) Using the multimodal features output by the Resblockbady(i) nodes in each branch of the CSPDarknet network (RGB and FIR) as input, construct a cross-modal contextual attention model (CCAM). For the output features of the Resblockbady(i+1) nodes in each branch, assign spatial attention weight matrices to the CCAM outputs by channel. Obtain the first-order optimized features of the Resblockbady(i+1) nodes after contextual attention optimization by multiplying the matrix elements.

[0008] Step 2) Using the CBAM attention model, the first-order optimized features of each branch Resblockbady(i+1) obtained in Step 1) are optimized again in terms of channels and space to obtain the second-order optimized features of each branch Resblockbady(i+1).

[0009] Step 3) Take the second-order optimized features of each branch Resblockbady(i+1) obtained in Step 2) as input to construct a feature fusion model and obtain the RGB-FIR multimodal fusion features of the Resblockbady(i+1) node.

[0010] Step 4) Following the structure and operations defined in Steps 1), 2), and 3), the RGB-FIR cross-modal context attention backbone network model CCAM is constructed by connecting the RGB and FIR branches of the CSPDarknet network using the cross-modal context attention model. RGB-FIR multimodal fusion features at different scales are obtained at the Resblockbady2, Resblockbady3, and Resblockbady4 nodes respectively.

[0011] Further, in step 1), the multimodal features output by the Resblockbady(i) nodes in each branch of the CSPDarknet network (RGB and FIR) are used as input to construct a cross-modal contextual attention model (CCAM). For the output features of each branch's Resblockbady(i+1) nodes, spatial attention weight matrices are assigned to the CCAM outputs by channel. The first-order optimized features of the Resblockbady(i+1) nodes after contextual attention optimization are obtained through element-wise multiplication of the matrices. The specific steps are as follows:

[0012] The cross-modal contextual attention model (CCAM) proposed in this invention consists of four modules: feature channel expansion module, channel feature fusion module, feature downsampling module, and spatial weight generation module.

[0013] Step 1-1: Use the feature channel expansion module to expand the original output features of the Resblockbady(i) node in the RGB and FIR branch CSPDarknet network to the same number of channels as the original output features of the Resblockbady(i+l) node. The specific method is as follows:

[0014] Let the original output features of the Resblockbady(i) node in the RGB and FIR branches of the CSPDarknet network be as follows: and Where c, h, and w are the number of feature channels, the height of the feature, and the width of the feature, respectively. Construct the 2D convolution operator F. 2Dconv , respectively and The number of channels is doubled, resulting in expanded features. and The calculation formula is as follows:

[0015]

[0016]

[0017] In the formula, This represents the visible light RGB feature matrix after channel expansion. The far-infrared FIR feature matrix represents the expanded channel.

[0018] Step 1-2: Use the channel feature fusion module to fuse the branch features obtained from the channel expansion in Step 1-1. The steps are as follows:

[0019] Step 1-2-1: Utilize the sigmoid activation function F sigmoid Extend the far-infrared channel feature Normalize each channel individually to obtain spatial importance weight values. The calculation formula is as follows:

[0020]

[0021] In the formula, The spatial importance weight matrix represents channel j.

[0022] Step 1-2-2: Use the Hadamard product operation to assign importance weights to the FIR feature space. With RGB channel extension features Element-wise multiplication by channel yields the fused features. The calculation formula is as follows:

[0023]

[0024] In the formula, The fusion feature matrix representing channel j, This is for the Hadamard product operation.

[0025] Steps 1-2-3: In the feature downsampling module, construct a 2×2 max pooling operator F. MP , for fusion features Perform local max pooling to improve the output feature map The resolution is reduced to half of the original feature map resolution. Features with the same resolution as the original output feature map of the Resblockbady(i+l) node are obtained, calculated using the following formula:

[0026]

[0027] In the formula, This represents the fused feature matrix after downsampling of channel j.

[0028] Steps 1-2-4: In the spatial weight generation module, the sigmoid activation function F is used. sigmoid , downsample and fuse features Normalization is performed by channel to generate a cross-modal context fusion feature space importance weight matrix. The specific calculation formula is as follows:

[0029]

[0030] In the formula, The cross-modal context fusion feature space importance weight matrix represents channel j.

[0031] Steps 1-2-5: Fuse the feature space importance weight matrix using cross-modal context. Raw output features of Resblockbady(i+1) nodes for each branch of RGB and FIR Spatial importance optimization is performed based on channels to obtain first-order optimized features. The specific calculation formula is as follows:

[0032]

[0033]

[0034] In the formula, The first-order optimized feature matrix represents the channel j of the Resblockbady(i+1) node in the RGB branch CSPDarknet network. The first-order optimized feature matrix represents channel j of the Resblockbady(i+1) node in the FIR branch CSPDarknet network. This is for the Hadamard product operation.

[0035] Furthermore, in step 2), the CBAM attention model F is used. CBAM The first-order optimized features of Resblockbady(i+1) for each branch of RGB and FIR obtained in step 1) are... and Further optimizations are performed on the channels and space to obtain the second-order optimization features of Resblockbady(i+1) for each branch. The specific calculation formula is as follows:

[0036]

[0037]

[0038] In the formula, The quadratic optimized feature matrix represents the channel j of the Resblockbady(i+1) node in the RGB branch CSPDarknet network. F represents the quadratic optimized feature matrix of channel j of node Resblockbady(i+1) in the FIR branch CSPDarknet network. CBAM This represents the CBAM (channel-space) attention operator.

[0039] Furthermore, in step 3), the second-order optimized features of each branch Resblockbady(i+1) obtained in step 2) are used as input to construct a feature fusion model and obtain the RGB-FIR multimodal fusion features of the Resblockbady(i+1) node. The specific steps are as follows:

[0040] Step 3-1: Optimize the features of Resblockbady(i+1) for each branch of RGB and FIR in Step 2). and As input, a feature fusion model is constructed. The feature fusion model proposed in this invention consists of a channel concatenation module (Concat), a convolution module (Conv), and a self-attention module. The specific steps are as follows:

[0041] Step 3-1-1: In the channel splicing module, combine the second-order optimized features of Resblockbady(i+1) for each branch of RGB and FIR. and Perform splicing operation by channel F Cat The calculation formula is as follows:

[0042]

[0043] In the formula, Channel splicing features representing the second-order optimized features of RGB and FIR branches Resblockbady(i+1). F Cat This represents the channel splicing operator.

[0044] Step 3-1-2: In the convolution module, use the convolution operator F conv After compression and splicing The feature channels are calculated using the following formula:

[0045]

[0046] Step 3-1-3: In the self-attention module, utilize the self-attention operator F self The result obtained in step 3-2-2 After adjusting the importance weights, the final RGB-FIR fusion features of the Resblockbady(i+1) node are obtained. The calculation formula is as follows:

[0047]

[0048] In the formula, T represents the transpose operation. · represents matrix multiplication.

[0049] Furthermore, in step 4), following the structure and operations defined in steps 1), 2), and 3), the RGB-FIR cross-modal contextual attention backbone network model is constructed by connecting the RGB and FIR branches of the CSPDarknet network using a cross-modal contextual attention model. Different scales of RGB-FIR multimodal fusion features are then obtained at the Resblockbady2, Resblockbady3, and Resblockbady4 nodes. The specific steps are as follows:

[0050] Step 4-1: Obtain the RGB-FIR multimodal fusion features of the Resblockbady2 node using steps 1)-3).

[0051] Step 4-2: Obtain the RGB-FIR multimodal fusion features of the Resblockbady3 node using steps 1)-3).

[0052] Step 4-3: Obtain the RGB-FIR multimodal fusion features of the Resblockbady4 node using steps 1)-3).

[0053] Beneficial Effects: The method of this invention can effectively optimize the backbone network structure of the RGB-FIR multimodal YOLO. By utilizing the fusion results of RGB-FIR features in each channel of the lower layer (Resblockbady(i)) of the RGB and FIR branch backbone networks, spatial weight optimization is performed on the features of each channel of the upper layer (Resblockbady(i+1)), enhancing the contextual relationship between features of adjacent scales and achieving cross-modal contextual fusion. This improves the effectiveness of RGB-FIR fused features and enhances the accuracy and robustness of pedestrian detection in the RGB-FIR multimodal YOLO. Attached Figure Description

[0054] Figure 1 This is a schematic diagram illustrating the steps of constructing a cross-modal contextual attention backbone network for a YOLO model for multimodal pedestrian detection according to the present invention.

[0055] Figure 2 This is a schematic diagram of the cross-modal contextual attention model (CCAM) of the present invention, which is a method for constructing a cross-modal contextual attention backbone network for YOLO model for multimodal pedestrian detection.

[0056] Figure 3 This is a schematic diagram of the branch attention model (CBAM) structure of a method for constructing a cross-modal contextual attention backbone network for YOLO model for multimodal pedestrian detection according to the present invention.

[0057] Figure 4 This is a schematic diagram of the feature fusion model of the method for constructing a cross-modal contextual attention backbone network for YOLO model for multimodal pedestrian detection according to the present invention;

[0058] Figure 5 This is a schematic diagram of the overall network structure of a method for constructing a cross-modal contextual attention backbone network for a YOLO model for multimodal pedestrian detection according to the present invention.

[0059] Figure 6 This is a training loss iteration diagram of a method for constructing a cross-modal contextual attention backbone network for a YOLO model for multimodal pedestrian detection according to the present invention.

[0060] Figure 7 This is a comparison of the detection accuracy of the cross-modal contextual attention backbone network of the YOLO model for multimodal pedestrian detection proposed in this invention with that of the YOLOv5 unimodal, MAF-YOLO and MAM-CBAM models on the KAIST and LLVIP datasets.

[0061] Figure 8 This figure compares the detection results of the proposed method for constructing a cross-modal contextual attention backbone network for multimodal pedestrian detection YOLO models with those of the YOLOv5 unimodal and MAF-YOLO models on the KAIST dataset. The first row shows the results of the YOLOv5 unimodal model, the second row shows the results of the MAF-YOLO model, and the third row shows the detection results of the method proposed in this invention. Rectangles represent correctly detected pedestrians, solid ellipses represent missed detections, and dashed ellipses represent false detections.

[0062] Figure 9 This figure compares the detection results of the proposed method for constructing a cross-modal contextual attention backbone network for multimodal pedestrian detection YOLO models with those of the YOLOv5 unimodal and MAM+CBAM models on the LLVIP dataset. The first row shows the results of the YOLOv5 unimodal model, the second row shows the results of the MAM+CBAM model, and the third row shows the detection results of the proposed method. Rectangles represent correctly detected pedestrians, solid ellipses represent missed detections, and dashed ellipses represent false detections.

[0063] Specific implementation steps

[0064] The technical method of the present invention will now be described in further detail with reference to the accompanying drawings.

[0065] like Figure 1 As shown, a method for constructing a cross-modal contextual attention backbone network for a YOLO model for multimodal pedestrian detection includes the following steps:

[0066] Step 1) Using the multimodal features output by the Resblockbady(i) node in each branch of the CSPDarknet network (RGB and FIR) as input, construct a cross-modal contextual attention model (CCAM). The network structure is as follows: Figure 2As shown, for the output features of Resblockbady(i+1) nodes in each branch, the spatial attention weight matrix of the CCAM output is assigned according to the channel. By multiplying the matrix elements, the first-order optimized features of the Resblockbady(i+1) nodes after context attention optimization are obtained.

[0067] In step 1), the multimodal features output by the Resblockbady(i) nodes in each branch of the CSPDarknet network (RGB and FIR) are used as input to construct a cross-modal contextual attention model (CCAM). For the output features of the Resblockbady(i+1) nodes in each branch, spatial attention weight matrices are assigned to the CCAM outputs by channel. The first-order optimized features of the Resblockbady(i+1) nodes after contextual attention optimization are obtained through element-wise multiplication of the matrices. The specific steps are as follows:

[0068] The cross-modal contextual attention model (CCAM) proposed in this invention consists of four modules: feature channel expansion module, channel feature fusion module, feature downsampling module, and spatial weight generation module.

[0069] Step 1-1: Use the feature channel expansion module to expand the original output features of the Resblockbady(i) node in the RGB and FIR branch CSPDarknet network to the same number of channels as the original output features of the Resblockbady(i+1) node. The specific method is as follows:

[0070] Let the original output features of the Resblockbady(i) node in the RGB and FIR branches of the CSPDarknet network be as follows: and Where c, h, and w are the number of feature channels, the height of the feature, and the width of the feature, respectively. Construct the 2D convolution operator F. 2Dconv , respectively and The number of channels is doubled, resulting in expanded features. and The calculation formula is as follows:

[0071]

[0072]

[0073] In the formula, This represents the visible light RGB feature matrix after channel expansion. The far-infrared FIR feature matrix represents the expanded channel.

[0074] Step 1-2: Use the channel feature fusion module to fuse the branch features obtained from the channel expansion in Step 1-1. The steps are as follows:

[0075] Step 1-2-1: Utilize the sigmoid activation function F sigmoid Extend the far-infrared channel feature Normalize each channel individually to obtain spatial importance weight values. The calculation formula is as follows:

[0076]

[0077] In the formula, The spatial importance weight matrix represents channel j.

[0078] Step 1-2-2: Use the Hadamard product operation to assign importance weights to the FIR feature space. With RGB channel extension features Element-wise multiplication by channel yields the fused features. The calculation formula is as follows:

[0079]

[0080] In the formula, The fusion feature matrix representing channel j, This is for the Hadamard product operation.

[0081] Steps 1-2-3: In the feature downsampling module, construct a 2×2 max pooling operator F. MP , for fusion features Perform local max pooling to improve the output feature map The resolution is reduced to half of the original feature map resolution. Features with the same resolution as the original output features of the Resblockbady(i+1) node are obtained, calculated using the following formula:

[0082]

[0083] In the formula, This represents the fused feature matrix after downsampling of channel j.

[0084] Steps 1-2-4: In the spatial weight generation module, the sigmoid activation function F is used. sigmoid , downsample and fuse features Normalization is performed by channel to generate a cross-modal context fusion feature space importance weight matrix. The specific calculation formula is as follows:

[0085]

[0086] In the formula, The cross-modal context fusion feature space importance weight matrix represents channel j.

[0087] Steps 1-2-5: Fuse the feature space importance weight matrix using cross-modal context. Raw output features of Resblockbady(i+1) nodes for each branch of RGB and FIR Spatial importance optimization is performed based on channels to obtain first-order optimized features. The specific calculation formula is as follows:

[0088]

[0089]

[0090] In the formula, The first-order optimized feature matrix represents the channel j of the Resblockbady(i+1) node in the RGB branch CSPDarknet network. The first-order optimized feature matrix represents channel j of the Resblockbady(i+1) node in the FIR branch CSPDarknet network. This is for the Hadamard product operation.

[0091] Step 2) Using the CBAM attention model, the network structure is as follows: Figure 3 As shown, the first-order optimization features of each branch Resblockbady(i+1) obtained in step 1) are further optimized in terms of channels and space to obtain the second-order optimization features of each branch Resblockbady(i+1).

[0092] In step 2), the CBAM attention model F is used. CBAM The first-order optimized features of Resblockbady(i+1) for each branch of RGB and FIR obtained in step 1) are... and Further optimizations are performed on the channels and space to obtain the second-order optimization features of Resblockbady(i+1) for each branch. The specific calculation formula is as follows:

[0093]

[0094]

[0095] In the formula, The quadratic optimized feature matrix represents the channel j of the Resblockbady(i+1) node in the RGB branch CSPDarknet network. F represents the quadratic optimized feature matrix of channel j of node Resblockbady(i+1) in the FIR branch CSPDarknet network. CBAM This represents the CBAM (channel-space) attention operator.

[0096] Step 3) Using the second-order optimized features of each branch Resblockbady(i+1) obtained in Step 2) as input, construct a feature fusion model. The network structure is as follows: Figure 4 As shown, the RGB-FIR multimodal fusion features of the Resblockbady(i+1) node are obtained.

[0097] In step 3), the second-order optimized features of each branch Resblockbady(i+1) obtained in step 2) are used as input to construct a feature fusion model and obtain the RGB-FIR multimodal fusion features of the Resblockbady(i+1) node. The specific steps are as follows:

[0098] Step 3-1: Optimize the features of Resblockbady(i+1) for each branch of RGB and FIR in Step 2). and As input, a feature fusion model is constructed. The feature fusion model proposed in this invention consists of a channel concatenation module (Concat), a convolution module (Conv), and a self-attention module. The specific steps are as follows:

[0099] Step 3-1-1: In the channel splicing module, combine the second-order optimized features of Resblockbady(i+1) for each branch of RGB and FIR. and Perform splicing operation by channel F Cat The calculation formula is as follows:

[0100]

[0101] In the formula, Channel splicing features representing the second-order optimized features of RGB and FIR branches Resblockbady(i+1). F Cat This represents the channel splicing operator.

[0102] Step 3-1-2: In the convolution module, use the convolution operator F conv After compression and splicing The feature channels are calculated using the following formula:

[0103]

[0104] Step 3-1-3: In the self-attention module, utilize the self-attention operator Fself The result obtained in step 3-2-2 After adjusting the importance weights, the final RGB-FIR fusion features of the Resblockbady(i+1) node are obtained. The calculation formula is as follows:

[0105]

[0106] In the formula, T represents the transpose operation. · represents matrix multiplication.

[0107] Step 4) Following the structure and operations defined in Steps 1), 2), and 3), the RGB-FIR cross-modal contextual attention backbone network model CCAM is constructed by connecting the RGB and FIR branches of the CSPDarknet network using a cross-modal contextual attention model. Different scales of RGB-FIR multimodal fusion features are obtained at the Resblockbady2, Resblockbady3, and Resblockbady4 nodes. The overall network structure is as follows: Figure 5 As shown.

[0108] In step 4), following the structure and operations defined in steps 1), 2), and 3), the RGB-FIR cross-modal contextual attention backbone network model is constructed by connecting the RGB and FIR branches of the CSPDarknet network using a cross-modal contextual attention model. Different scales of RGB-FIR multimodal fusion features are then obtained at the Resblockbady2, Resblockbady3, and Resblockbady4 nodes. The specific steps are as follows:

[0109] Step 4-1: Obtain the RGB-FIR multimodal fusion features of the Resblockbady2 node using steps 1)-3).

[0110] Step 4-2: Obtain the RGB-FIR multimodal fusion features of the Resblockbady3 node using steps 1)-3).

[0111] Step 4-3: Obtain the RGB-FIR multimodal fusion features of the Resblockbady4 node using steps 1)-3).

[0112] like Figure 6The graph shows the training loss iteration on the OSU Color-Thermal dataset, illustrating the construction method of the cross-modal contextual attention backbone network for multimodal pedestrian detection YOLO models according to the present invention. The horizontal axis represents the number of training epochs, and the vertical axis represents the loss value. The ratio of the training set to the test set is 7:3.

[0113] like Figure 7 The figure shows a comparison of the detection accuracy of the proposed method with the YOLOv5 single-modal model, the Multi-modal Attention Fusion YOLO (MAF-YOLO: Multi-modal attention fusionbased YOLO for pedestrian detection, Infrared Physics and Technology, 2021), and the MAM+CBAM model (patent: Sun Jun, 202010639950.4) on two different datasets, KAIST and LLVIP. The experimental results show that the proposed method has the highest detection accuracy and best performance on both datasets.

[0114] like Figure 8 This figure compares the detection results of the proposed method for constructing a cross-modal contextual attention backbone network for multimodal pedestrian detection YOLO models with those of the YOLOv5 unimodal and MAF-YOLO models on the KAIST dataset. The first row shows the results of the YOLOv5 unimodal model, the second row shows the results of the MAF-YOLO model, and the third row shows the detection results of the proposed method. Rectangles represent correctly detected pedestrians, solid ellipses represent missed detections, and dashed ellipses represent false detections. This demonstrates that the proposed method effectively enhances the accuracy and robustness of multimodal YOLO pedestrian detection.

[0115] like Figure 9 This figure compares the detection results of the proposed method for constructing a cross-modal contextual attention backbone network for multimodal pedestrian detection YOLO models with those of the YOLOv5 unimodal and MAM+CBAM models on the LLVIP dataset. The first row shows the results of the YOLOv5 unimodal model, the second row shows the results of the MAM+CBAM model, and the third row shows the detection results of the proposed method. Rectangles represent correctly detected pedestrians, solid ellipses represent missed detections, and dashed ellipses represent false detections. It can be seen that the proposed method achieves the best detection performance and the highest accuracy.

[0116] This method addresses the poor detection performance of existing RGB-FIR multimodal YOLO models by introducing a method for constructing a cross-modal contextual attention backbone network for multimodal pedestrian detection YOLO models. This backbone network consists of four parts: an RGB and FIR branch network (CSPDarknet), a cross-modal contextual attention model (CCAM), a branch attention model (CBAM), and a feature fusion model. The CCAM utilizes the feature fusion results of the lower-layer RGB and FIR channels of the backbone network to spatially optimize the features of the upper-layer RGB and FIR channels, achieving contextual cross-modal complementarity between features at adjacent scales in the backbone network. Using this method, the RGB-FIR fusion features output by the backbone network of existing RGB-FIR multimodal YOLO models at various scales can be effectively optimized, enhancing the accuracy and robustness of RGB-FIR multimodal YOLO pedestrian detection.

[0117] The technical content and features of the present invention have been disclosed above. The scope of protection of the present invention is not limited to the above-described embodiments. Any equivalent modifications and other alterations made by those skilled in the art based on the content disclosed in the present invention should be included in the scope of protection set forth in the claims.

Claims

1. A method for constructing a cross-modal context attention backbone network of a multi-modal pedestrian detection YOLO model, characterized in that: Comprise the following steps: Step 1) the multi-modal features output by the Resblockbody(i) node in the RGB, FIR branch CSPDarknet network are input to build a cross-modal context attention model CCAM, and the 1st optimized features of the Resblockbody(i+1) node are obtained by multiplying the spatial attention weight matrix output by the CCAM and facing the output features of each branch Resblockbody(i+1) node; Step 2) using the CBAM attention model, the 1st optimized features of each branch Resblockbody(i+1) obtained in step 1) are further optimized in channel and space to obtain the 2nd optimized features of each branch Resblockbody(i+1); Step 3) the 2nd optimized features of each branch Resblockbody(i+1) obtained in step 2) are input to build a feature fusion model to obtain the RGB-FIR multi-modal fusion features of the Resblockbody(i+1) node; Step 4) according to the structure and operation defined in steps 1), 2) and 3), the cross-modal context attention model is used to connect the RGB and FIR branch CSPDarknet networks to build an RGB-FIR cross-modal context attention backbone network model CCAM, and different scale RGB-FIR multi-modal fusion features are obtained at the Resblockbody2, Resblockbody3 and Resblockbody4 nodes; In step 1), the multi-modal features output by the Resblockbody(i) node in the RGB, FIR branch CSPDarknet network are input to build a cross-modal context attention model CCAM, and the 1st optimized features of the Resblockbody(i+1) node are obtained by multiplying the spatial attention weight matrix output by the CCAM and facing the output features of each branch Resblockbody(i+1) node, and the specific steps are as follows: The cross-modal context attention model (CCAM) is composed of four modules: a feature channel expansion module, a channel feature fusion module, a feature down-sampling module, and a spatial weight generation module; Step 1-1: the feature channel expansion module is used to expand the channel of the original output features of the Resblockbody(i) node in the RGB and FIR branch CSPDarknet network to the same number of channels as the original output features of the Resblockbody(i+1) node, and the specific method is as follows: Let the original output features of the Resblockbody(i) nodes in the RGB and FIR branch CSPDarknet network be respectively: and where c, h, w are the feature channel number, height and width of the feature respectively, and a 2D convolution operator F 2Dconv is constructed and to expand the channel number to twice the original, to obtain the expanded features and The calculation formula is as follows: In the formula, a visible light RGB feature matrix after channel expansion, a far infrared FIR feature matrix after channel expansion; Step 1-2: the channel feature fusion module is used to fuse the branch features expanded in step 1-1, and the steps are as follows: Step 1-2-1: Use the sigmoid activation function F sigmoid Extending features of the far-infrared channel Normalize one by one by channel, get spatial importance weight value The calculation formula is as follows: wherein a spatial importance weight matrix representing the channel j; Step 1-2-2: Use Hadamard product operation to obtain the FIR feature space importance weight value RGB channel expansion features Element multiplication is performed on the channels to obtain the fusion features The calculation formula is as follows: wherein represents the fusion feature matrix of channel j, is a Hadamard product operation; Step 1-2-3: In the feature downsampling module, a 2x2 max pooling operator F is constructed MP , and local max pooling is performed on the fused feature , so that the output feature map is reduced to 1 / 2 of the original feature map resolution, and features with the same resolution as the original output features of the Resblockbody(i+1) node are obtained. The specific calculation formula is as follows: In the formula, represents the down-sampled fusion feature matrix of channel j; Step 1-2-4: In the spatial weight generation module, a sigmoid activation function F is used sigmoid Down-sampling the fused features Normalizing by channel to generate a cross-modal context fusion feature space importance weight matrix The specific calculation formula is as follows: In the formula, represents the cross-modal context fusion feature space importance weight matrix of channel j; Step 1-2-5: Utilize cross-modal context fusion feature space importance weight matrix The original output features of the Resblockbody(i+1) nodes of the RGB and FIR branches Optimize the spatial importance by channel to obtain the first optimized features, and the specific calculation formula is as follows: wherein represents the first-order optimized feature matrix of the Resblockbody(i+1) node channel j in the RGB branch CSPDarknet network, represents the first-order optimized feature matrix of the Resblockbody(i+1) node channel j in the FIR branch CSPDarknet network, is a Hadamard product operation; In the step 2), the CBAM attention model F is used CBAM In the step 1), the first-order optimized features of the Resblockbody(i+1) of each branch of RGB and FIR obtained in the step 1) are obtained And The channel and space are optimized again, and the second-order optimized features of the Resblockbody(i+1) of each branch are obtained, and the specific calculation formula is as follows: wherein, represents the 2nd order optimized feature matrix of Resblockbody(i+1) node channel j in the RGB branch CSPDarknet network, represents the 2nd order optimized feature matrix of Resblockbody(i+1) node channel j in the FIR branch CSPDarknet network, F CBAM represents the CBAM (Channel-Spatial) attention operator; In the step 3), the second optimization features of each branch Resblockbody(i+1) obtained in the step 2) are taken as inputs to construct a feature fusion model, and the RGB-FIR multi-modal fusion features of the Resblockbody(i+1) node are obtained, and the specific steps are as follows: Step 3-1: the second optimization feature of each branch Resblockbody(i+1) of RGB, FIR in step 2) and As input, a feature fusion model is constructed, which is composed of a channel splicing module (Concat), a convolution module (Conv) and a self-attention module (Self-attention), and the specific steps are as follows: Step 3-1-1: In the channel concatenation module, the 2nd optimized features of each branch Resblockbody(i+1) of RGB and FIR are concatenated and The channel concatenation operation F Cat is calculated as follows: In the formula, Channel concatenation features representing the 2nd optimization features of the RGB and FIR branch Resblockbody(i+1), F Cat represents a channel concatenation operator; Step 3-1-2: In the convolution module, use the convolution operator F conv After compression and splicing The feature channel is calculated as follows: Step 3-1-3: In the self-attention module, the self-attention operator F is used self , obtained in step 3-1-2 is subjected to importance weight adjustment to obtain the final RGB-FIR fusion features of the Resblockbody(i+1) node The calculation formula is as follows:​ In the formula, M represents the number of channels, T is a transpose operation, and • is matrix multiplication.

2. The method of claim 1, wherein the method is a method for constructing a cross-modal context attention backbone network of a multi-modal pedestrian detection YOLO model. In the step 4), according to the structure and operation defined in the steps 1), 2) and 3), the RGB-FIR cross-modal context attention backbone network model is constructed by connecting the RGB and FIR branch CSPDarknet networks using the cross-modal context attention model, and the RGB-FIR multi-modal fusion features of different scales are obtained at the Resblockbody2, Resblockbody3 and Resblockbody4 nodes, and the specific steps are as follows: Step 4-1: Obtain the RGB-FIR multi-modal fusion features of the Resblockbody2 node by using steps 1) to step 3) Step 4-2: Obtain the RGB-FIR multi-modal fusion feature of the Resblockbody3 node by using step 1) - step 3) Step 4-3: Obtain the RGB-FIR multi-modal fusion feature of the Resblockbody4 node by using step 1) - step 3)

Images