A Multimodal Object Detection Method Based on Dynamically Light-Aware Random Masking

Abstract

The present invention discloses a multimodal target detection method based on dynamic illumination perception random masking, which relates to the field of multimodal target detection. A dynamic illumination perception random masking method is proposed, which performs masking on a portion of the images in the training set, and takes into account the influence of exposure and strong light environments, so that the network can learn multimodal effective complementary features in an unbiased manner; constructs a basic network for dual-modal feature-level fusion target detection; introduces a cross-modal differential feature interaction module, which increases the complementary information interaction between the two modalities by extracting the difference features between visible light and infrared modalities and dynamically allocating attention weights in the channel-space dimension; and conducts model training and testing. The multimodal target detection method provided by the present invention improves the robustness of the multimodal target detection model to complex lighting environments such as exposure and strong light, and improves the multimodal remote sensing target detection capability of unmanned aerial vehicle platforms / satellite platforms and the target recognition accuracy under complex lighting conditions.

Description

Technical Field

[0001] The present invention relates to the field of multimodal target detection, and in particular to a multimodal target detection method based on dynamic illumination perception random mask. Background Art

[0002] With the remarkable success of deep learning technology in the field of natural images, deep learning target detection algorithms have been applied in various fields. However, most deep learning algorithms are designed and applied specifically for a single mode. Although visible light images contain rich target texture and color information, they cannot effectively monitor targets when used alone in low-light or no-light scenes at night. Thermal infrared images have the advantage of being less susceptible to illumination, but infrared images are often accompanied by low texture information, low resolution, and high noise levels. These single-mode limitations make it difficult to achieve satisfactory target detection accuracy and cannot operate in multiple working conditions at all times, limiting their scope of application.

[0003] Therefore, the fusion of complementary information between visible and infrared images for target detection has begun to attract research attention. However, existing methods lack attention to the complementary information between the two modalities and insufficiently exploit cross-modal complementary information, which can easily lead to the learning of redundant information. Furthermore, current research focuses on the design of fusion methods, neglecting the design of methods for information interaction between modalities. Detection accuracy for complex targets, such as those in bright and exposed light, densely packed objects, and those with limited background distinction, needs to be improved.

[0004] To address the above problems, the present invention proposes a multimodal target detection method based on illumination-aware random masks. Based on the YOLOv8n dual-stream backbone network, the present invention designs a cross-modal differential feature interaction module (CDFIM) to deeply mine and learn the complementary features of visible light and infrared images, enhance the interaction and feature representation between modalities, and improve the ability to learn the complementary features of the two modalities. In addition, the present invention designs an illumination-aware random mask training strategy, which adds dynamic complementary masks to the bimodal images based on the local illumination information of visible light, forcing the model to learn the complementary features between modalities in a balanced manner during training, thereby improving the robustness to different lighting environments. Summary of the Invention

[0005] To solve the above problems, this application proposes a multimodal target detection method based on dynamic illumination perception random mask, which includes the following steps:

[0006] S1. Obtain an image to be detected and preprocess the image to be detected to obtain an initial image;

[0007] S2. Based on YOLOv8n and the improved cross-modal differential feature interaction module, an improved multimodal detection model is constructed, a multimodal target detection dataset is obtained, and the improved multimodal detection model is trained using a dual-modal model training strategy that combines the multimodal target detection dataset with dynamic illumination-aware random masking.

[0008] The improved multimodal detection model includes a two-stream backbone network, a cross-modal differential feature interaction enhancement module and a YOLO detection head;

[0009] S3. Use the improved multimodal detection model to process the initial image to obtain the target detection result.

[0010] Preferably, the dual-stream backbone network is used to extract multi-scale features from the initial image, thereby performing subsequent feature fusion. The initial image includes a visible light image and an infrared image, and the image features include visible light features. and infrared characteristics .

[0011] Preferably, the cross-modal differential feature interaction enhancement module is used to utilize the differential complementary information between the initial images to enhance the inter-modal features from both channel and spatial aspects, and fuse the multi-scale features using a channel cascade method, feed them into a feature pyramid structure, and finally input them into the YOLO detection head;

[0012] The YOLO detection head detects the information to be detected to obtain a target detection result, where the target detection result includes a target position, a target category, and a predicted probability.

[0013] Preferably, the differential feature interaction enhancement module includes two parts: channel attention and spatial attention, and the channel attention and spatial attention are placed in series.

[0014] Preferably, the channel attention enhancement part processing steps are as follows:

[0015] S2101, differential feature enhancement, visible light feature and infrared characteristics Take them as input together and subtract them to get the differential features;

[0016] S2102, after 1×1 convolution and GLUE activation function, the enhanced differential feature map is obtained ;

[0017] S2103, the spatial information of the differential feature map is average pooled and max pooled to obtain a global receptive field and perform spatial aggregation;

[0018] S2104, the spatial information is then processed by the shared MLP and a channel attention map is generated , through the channel attention map and the differential features of the input Element-wise multiplication obtains channel attention enhanced features;

[0019] S2105, introduce the idea of residual addition, and combine the channel attention enhancement features with the visible light features and infrared characteristics Add together and get and ;

[0020] S2106, will and Add together to get the output of the channel attention part .

[0021] Preferably, the enhanced expression of channel attention is:

[0022] ;

[0023] in, ; ; ; ;

[0024] in, Represents element-wise multiplication; Represents the sigmoid function; Represents a 1×1 convolution operation; and denotes global average pooling and global maximum pooling; MLP denotes a shared multilayer perceptron.

[0025] Preferably, after calculating the channel attention map, a depthwise separable convolution module is used to extract important spatial regions for each channel, and finally a dynamically allocated spatial attention map is obtained;

[0026] The enhancement part of the channel attention adopts the Inception style.

[0027] Preferably, the spatial attention enhancement part processing steps are as follows:

[0028] S2107, use a 5×5 small kernel convolution to obtain the output result of the channel attention part The local feature information of express;

[0029] S2108, secondly, using multiple sets of parallel depth strip convolutions to capture local context information across multiple scales;

[0030] S2109, the final output feature is obtained by element-wise multiplication of the channel mixing result of the spatial attention mechanism and the channel prior feature ;

[0031] Among them, using 1×1 convolution to achieve channel mixing helps to obtain a more refined spatial attention map.

[0032] Preferably, the enhanced part of the spatial attention is expressed as:

[0033] ;

[0034] ;

[0035] Among them, DwConv represents depth-wise separable convolution; represents the i-th branch, is the original input, the rest Both consist of stripe convolution pairs; Represents a 1×1 convolution operation.

[0036] The multimodal target detection dataset includes visible light and infrared images, and the multimodal target detection dataset is divided into a training set, a validation set, and a test set;

[0037] The specific steps of the random mask training strategy using dynamic illumination perception are as follows:

[0038] Step 1: Divide a visible light image into N×N regions, calculate the average illumination value of each region, and sort these regions in ascending order of illumination value. The top K regions with the lowest brightness are considered dark regions (called Top-K regions), and the bottom K regions are considered bright regions (called Bottom-K regions).

[0039] Step 2: Divide the infrared image that matches the visible light image into N×N regions;

[0040] Step 3: For dark areas, add a mask to all dark areas of the visible light image, that is, set the pixel value to 0;

[0041] Step 4: For the bright light area, randomly select K / 2 locations of the bright light area and add a mask in the infrared mode, that is, set the pixel value to 0, and add a mask in the visible light mode to the remaining K / 2 locations, that is, set the pixel value to 0;

[0042] Step 5: Select a visible-infrared image of the target ratio from the training set of the multimodal dataset and repeat steps 1 to 4 above to perform masking.

[0043] Step 6: Masking is performed only on the training set, leaving the validation and test sets unchanged. The multimodal object detection model is trained and tested using the masked training set and the original validation and test sets.

[0044] In summary, the present invention provides a multimodal target detection method based on dynamic illumination perception random mask. Compared with traditional technologies, the present invention proposes a dynamic illumination perception random mask method, performs mask processing on a portion of the images in the training set, and considers the influence of exposure and strong light environment, so that the network can learn multimodal effective complementary features in an unbiased manner; constructs a dual-modal feature-level fusion target detection basic network; introduces a cross-modal differential feature interaction module, improves the multimodal remote sensing target detection capability of the UAV platform / satellite platform, and improves the target recognition accuracy under complex lighting conditions.

[0045] The technical method of the present invention is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of the principle flow of the present invention;

[0047] Figure 2 This is a diagram of the improved multimodal detection model of the present invention;

[0048] Figure 3 This is a schematic diagram of the cross-modal differential feature interaction module of the present invention;

[0049] Figure 4 This is a comparison chart of target detection results of different models under the VEDAI dataset of the present invention. DETAILED DESCRIPTION

[0050] The technical method of the present invention is further described below through the accompanying drawings and embodiments. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and values described in these embodiments do not limit the scope of this application.

[0051] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0052] Technologies, systems, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0053] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0054] Unless otherwise defined, technical or scientific terms used in the present invention shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0055] Based on the YOLOv8n model, the present invention designs a dual-stream feature extraction backbone network model to extract deep features from visible light and infrared images respectively. Secondly, the present invention designs a CDFIM differential feature interaction enhancement module, which uses the differential complementary information between multimodal images to enhance inter-modal features from both channel and spatial aspects. The multi-scale features are fused using channel cascade means, sent into the feature pyramid structure, and finally input into the YOLO detection head to obtain the fused recognition result. For model training, the present invention proposes a random masking strategy based on illumination perception. Based on the illumination prior information of visible light images, while balancing the model's attention to the two modalities, it improves the learning of effective modal information, and takes into account the effects of exposure and strong light, so that the network can focus on the complementary effective information of different modalities without bias.

[0056] The present invention provides a multimodal target detection method based on dynamic illumination perception random mask, such as Figure 1 and Figure 2 As shown, S1, obtain the image to be detected, and preprocess the image to be detected to obtain an initial image.

[0057] S2. Based on YOLOv8n and the improved cross-modal differential feature interaction module, an improved multimodal detection model is constructed, a multimodal target detection dataset is obtained, and the improved multimodal detection model is trained using a dual-modal model training strategy of multimodal target detection dataset combined with dynamic illumination-aware random mask.

[0058] The improved multimodal detection model includes a two-stream backbone network, a cross-modal differential feature interaction enhancement module and a YOLO detection head.

[0059] S3. Use the improved multimodal detection model to process the initial image to obtain the target detection result.

[0060] Furthermore, the dual-stream backbone network is used to extract multi-scale features from the initial image, thereby performing subsequent feature fusion. The initial image includes a visible light image and an infrared image, and the image features include visible light features. and infrared characteristics .

[0061] Furthermore, the cross-modal differential feature interaction enhancement module is used to utilize the differential complementary information between the initial images to enhance the inter-modal features from both channel and spatial aspects, and to fuse the multi-scale features using channel cascade means, feed them into the feature pyramid structure, and finally input them into the YOLO detection head.

[0062] The YOLO detection head detects the information to be detected to obtain a target detection result, where the target detection result includes a target position, a target category, and a predicted probability.

[0063] The present invention designs a CDFIM module as a bridge between the visible light and infrared branches. As shown in the figure, for bimodal information input, traditional processing methods typically use direct addition or channel cascading, which increases both differential and redundant information. Therefore, the CDFIM designed in the present invention focuses on differential features to enhance cross-modal complementary features. The CDFIM module consists of two parts: channel attention and spatial attention, arranged in series. A design that dynamically distributes attention weights across both channel and spatial dimensions allows for more accurate focus on important target features.

[0064] Further, such as Figure 3 As shown, the differential feature interaction enhancement module includes two parts: channel attention and spatial attention, and the channel attention and spatial attention are placed in series.

[0065] Furthermore, the channel attention enhancement part is processed as follows:

[0066] S2101, differential feature enhancement, visible light feature and infrared characteristics They are taken as input together and the differential features are obtained by subtracting the two.

[0067] S2102, after 1×1 convolution and GLUE activation function, the enhanced differential feature map is obtained .

[0068] S2103. The spatial information of the differential feature map is aggregated spatially by obtaining a global receptive field through average pooling and maximum pooling operations.

[0069] S2104, the spatial information is then processed by the shared MLP and a channel attention map is generated , through the channel attention map and the differential features of the input Element-wise multiplication is performed to obtain channel-wise attention-enhanced features.

[0070] S2105, introduce the idea of residual addition, and combine the channel attention enhancement features with the visible light features and infrared characteristics Add together and get and .

[0071] S2106, will and Add together to get the output of the channel attention part .

[0072] Furthermore, the enhanced expression of channel attention is:

[0073] .

[0074] in, . . . .

[0075] in, Represents element-wise multiplication. Represents the sigmoid function. Represents a 1×1 convolution operation. and denotes global average pooling and global maximum pooling. MLP denotes a shared multilayer perceptron.

[0076] Furthermore, after calculating the channel attention map, the depthwise separable convolution module is used to extract important spatial regions for each channel, and finally a dynamically allocated spatial attention map is obtained.

[0077] The enhancement part of the channel attention adopts the Inception style.

[0078] Furthermore, the steps of enhancing the spatial attention are as follows:

[0079] S2107, use a 5×5 small kernel convolution to obtain the output result of the channel attention part The local feature information of express.

[0080] S2108. Secondly, multiple sets of parallel depth strip convolutions are used to capture local context information across multiple scales.

[0081] A set of strip convolutions can be equivalent to the receptive field of a standard convolution, and the computational complexity of depthwise strip convolution is smaller than that of 2D convolution. In addition, strip convolution is convenient for extracting features of slender objects such as vehicles, ships, and aircraft, making it more suitable for remote sensing targets.

[0082] S2109, the final output feature is obtained by element-wise multiplication of the channel mixing result of the spatial attention mechanism and the channel prior feature .

[0083] Among them, using 1×1 convolution to achieve channel mixing helps to obtain a more refined spatial attention map.

[0084] Furthermore, the enhanced expression of the spatial attention is:

[0085] .

[0086] .

[0087] Among them, DwConv represents depth-wise separable convolution. represents the i-th branch, is the original input, the rest Both are composed of stripe convolution pairs. Represents a 1×1 convolution operation.

[0088] When using intelligent models to learn bimodal features, the backbone network usually tends to extract visible light modal features and ignores infrared modal features due to the rich texture and color information of the visible light modality. The idea of asymmetric masking can force the network to learn more effective features from complementary modes and avoid model bias learning. However, the typical random masking strategy does not consider the importance of different modalities in different areas of the image. To this end, the present invention proposes a dynamic perception random mask training strategy. Unlike the conventional random masking method, the strategy of the present invention adds a dynamic mask to each set of dual-modal images based on the local illumination perception information of the visible light image. In addition, the present invention takes into account the modal feature learning methods under strong light and exposure conditions, thereby improving the model's learning ability for complex environments.

[0089] The multimodal target detection dataset includes visible light and infrared images, and the multimodal target detection dataset is divided into a training set, a validation set, and a test set.

[0090] The specific steps of the random mask training strategy using dynamic illumination perception are as follows:

[0091] Step 1: Divide a visible light image into N×N regions evenly, calculate the average illumination value of each region, and sort these regions in ascending order of illumination value. The top K regions with the lowest brightness are dark regions (called Top-K regions), and the bottom K regions are bright regions (called Bottom-K regions).

[0092] Step 2: Divide the infrared image that matches the visible light image into N×N regions.

[0093] Step 3: For dark areas, add a mask to all dark area positions in the visible light image, that is, set the pixel value to 0.

[0094] Step 4: For the bright light area, randomly select K / 2 locations in the bright light area and add masks in the infrared mode, that is, set the pixel values to 0. Add masks to the remaining K / 2 locations in the visible light mode, that is, set the pixel values to 0.

[0095] Step 5: Select a visible light-infrared image of the target ratio in the training set of the multimodal dataset and repeat steps 1 to 4 above to perform masking.

[0096] final: .

[0097] .

[0098] Step 6: Masking is performed only on the training set, leaving the validation and test sets unchanged. The multimodal object detection model is trained and tested using the masked training set and the original validation and test sets.

[0099] To demonstrate the effectiveness of the multimodal detection model and dynamic illumination-aware random mask training strategy designed by the present invention, the present invention conducts target detection tests on the VEDAI dataset. VEDAI is a public dataset for small target detection in aerial images, which includes challenges such as lighting / shadow changes, strong light reflections and occlusions, and covers various scenes such as rural areas, urban areas and mountainous areas. The dataset contains 1,200 sets of RGB-IR image pairs, more than 3,700 annotated targets, and an average of 5.5 targets per image, which account for about 0.7% of the total pixels in the image. 9 types of ground targets are provided. In this article, the present invention uses the 1024×1024 pixel version for experiments, selects 1,089 pairs of image pairs as training sets, and 121 pairs as validation sets and test sets.

[0100] During the experiments, the Windows 11 operating system was used, and programming was performed using the PyCharm integrated development environment (IDE). This development environment was configured with Python 3.8, the PyTorch 1.8 deep learning framework, and CUDA 11.1. The hardware resources used in the experiments included an Intel Core i7-13700KF CPU and an NVIDIA RTX 4090 GPU. During the model training phase, 300 cycles were executed, with each cycle processing a batch size of 16 samples.

[0101] Table 1 shows the ablation experiments of each module on the VEDAI dataset. The base model of the present invention uses a bimodal YOLOv8n intermediate layer concat fusion approach. It can be found that on the VEDAI dataset, when the CDFIM module is added to the base model, the model's mAP50 increases by 1.5%, indicating that the addition of the CDFIM module of the present invention can effectively improve the complementary learning ability and feature interaction ability of bimodal features. In addition, the number of parameters increases by only 0.26M, and the computational cost increases by only 0.7GFLOPs. This is attributed to the cross-modal differential features extracted by the CDFIM module of the present invention, which avoids the learning of redundant information and introduces deep strip convolution to achieve the purpose of feature enhancement with extremely low computational cost. Combined with the dynamic illumination-aware random mask training strategy, the recognition accuracy on this dataset is improved to 79.5% without adding any computational burden to the model. These results demonstrate the effectiveness of the cross-modal differential feature extraction module and dynamic illumination-aware random mask training strategy proposed in the present invention.

[0102] Table 1 Ablation experiments of each module on the VEDAI dataset

[0103] ;

[0104] As shown in Table 2, the model of the present invention is 7.4% higher than the multimodal baseline model of the present invention in mAP50, which shows the advancement and effectiveness of the model proposed by the present invention. Compared with the YOLOv8n visible light single-mode model, the recognition accuracy mAP50 of the present invention is improved by 8.5%, and compared with the YOLOv8n infrared single-mode model, the recognition accuracy mAP50 of the present invention is improved by 8.1%. Compared with the MMYFnet model with the highest recognition accuracy among multimodal models. Although the model of the present invention is slightly lower by 0.5% in mAP, the number of parameters of the present invention is reduced by 72.8% and the amount of calculation is reduced by 71.6%. The lightweight model will be more conducive to the actual deployment and rapid reasoning of satellite-borne and airborne ends. Therefore, the multimodal target detection method proposed by the present invention shows a good accuracy-computation trade-off.

[0105] Table 2 Comparison of recognition performance of different target detection models under VEDAI dataset

[0106] ;

[0107] In order to more intuitively demonstrate the detection performance of this method, the detection results of the model of the present invention are compared with those of the other two models. Figure 4As shown, the YOLOv8n visible light single-mode model redundantly locates the truck target in the middle area because it is small and closely matches the background color. Due to the blurred infrared target outline, the YOLOv8n infrared single-mode model misclassifies and misses the target. While the baseline model also makes classification errors, the multimodal detection model of our invention correctly detects and locates all truck targets. This demonstrates the robustness of our model in detecting targets in complex scenes.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical method of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical method to deviate from the spirit and scope of the technical method of the present invention.

Claims

1. A multimodal target detection method based on dynamic illumination perception random mask, characterized in that: The following steps are involved: S1. Obtain an image to be detected and preprocess the image to be detected to obtain an initial image; S2. Based on YOLOv8n and the improved cross-modal differential feature interaction module, an improved multimodal detection model is constructed, a multimodal target detection dataset is obtained, and the improved multimodal detection model is trained using a dual-modal model training strategy that combines the multimodal target detection dataset with dynamic illumination-aware random masking. The improved multimodal detection model includes a two-stream backbone network, a cross-modal differential feature interaction enhancement module and a YOLO detection head; S3, using the improved multimodal detection model to process the initial image to obtain the target detection result; The dual-stream backbone network is used to extract multi-scale features from the initial image, thereby performing subsequent feature fusion; The initial image includes a visible light image and an infrared image, and the image features include visible light features. and infrared characteristics ; The cross-modal differential feature interaction module is used to utilize the differential complementary information between the initial images to enhance the inter-modal features from both channel and spatial aspects, and to fuse the multi-scale features using channel cascade means, and send them into the feature pyramid structure, and finally input them to the YOLO detection head; The YOLO detection head detects the information to be detected to obtain a target detection result, wherein the target detection result includes the target position, target category and prediction probability; The multimodal target detection dataset includes visible light and infrared images, and the multimodal target detection dataset is divided into a training set, a validation set, and a test set; The specific steps of the random mask training strategy using dynamic illumination perception are as follows: Step 1: Divide a visible light image into N×N regions, calculate the average illumination value of each region, and sort these regions in ascending order of illumination value. The top K regions with the lowest brightness are called the dark light regions, and the bottom K regions are called the bright light regions, and are called the bottom K regions. Step 2: Divide the infrared image that matches the visible light image into N×N regions; Step 3: For dark areas, add a mask to all dark areas of the visible light image, that is, set the pixel value to 0; Step 4: For the bright light area, randomly select K / 2 locations of the bright light area and add a mask in the infrared mode, that is, set the pixel value to 0, and add a mask in the visible light mode to the remaining K / 2 locations, that is, set the pixel value to 0; Step 5: Select a visible-infrared image of the target ratio from the training set of the multimodal dataset and repeat steps 1 to 4 above to perform masking. Step 6: Mask processing is performed only on the training set, while the validation set and test set remain unchanged. The multimodal object detection model is trained and tested using the masked training set and the original validation set and test set.

2. The multimodal target detection method based on dynamic illumination perception random mask according to claim 1, characterized in that: The differential feature interaction enhancement module includes two parts: channel attention and spatial attention, which are placed in series.

3. The multimodal target detection method based on dynamic illumination perception random mask according to claim 2, characterized in that: The enhanced processing steps of the channel attention are as follows: S2101, differential feature enhancement, visible light feature and infrared characteristics Take them as input together and subtract them to get the differential features; S2102, after 1×1 convolution and GLUE activation function, the enhanced differential feature map is obtained ; S2103, the spatial information of the differential feature map is average pooled and max pooled to obtain a global receptive field and perform spatial aggregation; S2104, the spatial information is then processed by the shared MLP and a channel attention map is generated , through the channel attention map and the differential features of the input Element-wise multiplication obtains channel attention enhanced features; S2105, introduce the idea of residual addition, and combine the channel attention enhancement features with the visible light features and infrared characteristics Add together and get and ; S2106, will and Add together to get the output of the channel attention part .

4. The multimodal target detection method based on dynamic illumination perception random mask according to claim 3, characterized in that: The enhanced expression of channel attention is: ； in, ; ; ; ; in, Represents element-wise multiplication; Represents the sigmoid function; Represents a 1×1 convolution operation; and denotes global average pooling and global maximum pooling; MLP denotes a shared multilayer perceptron.

5. The multimodal target detection method based on dynamic illumination perception random mask according to claim 4, characterized in that: After calculating the channel attention map, the depth-wise separable convolution module is used to extract the important spatial regions for each channel, and finally a dynamically allocated spatial attention map is obtained; The enhancement part of the channel attention adopts the Inception style.

6. The multimodal target detection method based on dynamic illumination perception random mask according to claim 5, characterized in that: The steps of the spatial attention enhancement part are as follows: S2107, use a 5×5 small kernel convolution to obtain the output result of the channel attention part The local feature information of express; S2108, secondly, using multiple sets of parallel depth strip convolutions to capture local context information across multiple scales; S2109, the final output feature is obtained by element-wise multiplication of the channel mixing result of the spatial attention mechanism and the channel prior feature ; Among them, using 1×1 convolution to achieve channel mixing helps to obtain a more refined spatial attention map.

7. The multimodal target detection method based on dynamic illumination perception random mask according to claim 6, characterized in that: The enhanced expression of the spatial attention is: ； ； Among them, DwConv represents depth-wise separable convolution; represents the i-th branch, is the original input, the rest Both consist of stripe convolution pairs; Represents a 1×1 convolution operation.

Images