A lightweight target detection method and system based on multimodal images

Abstract

The present invention provides a lightweight target detection method and system based on multimodal images, relating to the field of computer vision. The method comprises: performing convolution, batch normalization, and ReLU activation operations on infrared and visible light images respectively to obtain feature maps; then using convolution operations and Sigmoid functions to generate feature weight masks; performing pixel-by-pixel weighting on the feature maps based on the feature weight masks to generate optimized feature maps; extracting illumination perception information through a local illumination perception module to perform weighted calculations on the optimized feature maps to obtain weighted feature maps; splicing the visible light and infrared weighted feature maps in the channel dimension to generate a spliced ​​feature map; performing subsequent feature extraction operations through residual connections and a parameter-free channel attention module, and inputting the features into a target detection network for target detection, thereby outputting detection results. The present invention can achieve higher-precision target detection without introducing a larger number of parameters and computational complexity.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a lightweight target detection method and system based on multimodal images. Background Art

[0002] Object detection technology plays a crucial role in computer vision and is widely used in numerous fields, including autonomous driving, video surveillance, and intelligent security. In recent years, with the advancement of deep learning techniques, object detection algorithms based on convolutional neural networks (CNNs) have made significant progress. However, these methods still face challenges in complex scenarios, particularly in low-light environments, inclement weather, and under occlusion, where performance often degrades significantly. Traditional single-modal object detection, such as that based on visible light images, relies heavily on image illumination, resulting in suboptimal detection accuracy at night or in low-light environments. To address the shortcomings of single-modal detection, multimodal fusion detection has emerged as an effective approach. By combining data sources from different modalities (such as visible light and infrared images), object detectability can be significantly improved, especially in complex lighting conditions. This multimodal information fusion approach can supplement infrared thermal images when visible light information is insufficient, thereby enhancing detection robustness and accuracy. Currently, multimodal object detection can be categorized into three types based on fusion strategies: early fusion, mid-term fusion, and late fusion. Among them, early fusion is the most intuitive fusion method. It splices the infrared image and the visible light image to generate a four-channel image, which is then input into the conventional target detection architecture. Compared with the other two methods, the early fusion model is simpler, easier to implement and integrate. In the process of realizing the present invention, the applicant found that the existing single-modal target detection usually faces the problem of information loss under low-light conditions, making it difficult to accurately identify the target, and thus significantly reducing the performance. In addition, although fusion target detection can make up for the shortcomings of single modality to a certain extent, the existing early fusion method often faces the problem of not being able to fully explore the complementarity between modalities, resulting in the loss of key features, limiting the model's deep learning of modal relationships, and thus affecting performance in complex scenes. Secondly, there is the problem of information redundancy. After the repeated features of different modalities are fused, redundant information will be generated, which increases the computational burden and interferes with the model's extraction of effective information. Moreover, when there are lighting differences in the image, the model cannot accurately capture the image details, thereby affecting the detection accuracy. Therefore, how to improve the detection accuracy of the target detection task has become a technical problem that needs to be solved. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art or related art, and discloses a lightweight target detection method and system based on multimodal images, which can effectively fuse the features of infrared images and visible light images to achieve higher-precision target detection.

[0004] The first aspect of the present invention discloses a lightweight target detection method based on multimodal images, comprising: receiving an image to be detected: receiving a paired infrared image and a visible light image; weight-guided feature optimization: performing convolution, batch normalization, and ReLU activation operations on the infrared image and the visible light image, respectively, to generate an enhanced infrared feature map and an enhanced visible light feature map; using a convolution operation to reduce the number of channels of the enhanced infrared feature map to 1, and using a Sigmoid function to limit the output value range to between 0 and 1, to generate an infrared feature weight mask; using a convolution operation to reduce the number of channels of the enhanced visible light feature map to 1, and using a Sigmoid function to limit the output value range to between 0 and 1, to generate a visible light feature weight mask; performing pixel-by-pixel weighting on the enhanced infrared feature map based on the infrared feature weight mask to generate an optimized The enhanced infrared feature map is weighted pixel by pixel based on the visible light feature weight mask to generate an optimized visible light feature map; local illumination perception mechanism: the enhanced infrared feature map and the enhanced visible light feature map are input into the local illumination perception module to extract illumination perception information; the weights of the optimized infrared feature map and the optimized visible light feature map are adjusted according to the illumination perception information to generate a visible light weighted feature map and an infrared weighted feature map; the visible light weighted feature map and the infrared weighted feature map are spliced ​​in the channel dimension to generate a spliced ​​feature map; parameter-free channel attention mechanism: the spliced ​​feature map is subjected to subsequent feature extraction operations through residual connection and parameter-free channel attention module to obtain a multimodal fusion feature map; target detection: the multimodal fusion feature map is input into the target detection network for target detection to output the detection result.

[0005] According to the lightweight target detection method based on multimodal images disclosed in the present invention, preferably, the target detection network is a YOLO network.

[0006] According to the lightweight target detection method based on multimodal images disclosed in the present invention, preferably, the YOLO network specifically includes: a backbone network, a neck and a detection head; the multimodal fusion feature map is further extracted in the backbone network part, and features of multiple resolutions are aggregated in the neck of the target detection network to enhance detection accuracy, and finally the final target bounding box and category prediction are generated by the detection head.

[0007] According to the lightweight target detection method based on multimodal images disclosed in the present invention, preferably, the calculation process of the local illumination perception module specifically includes: the input features are subjected to convolution calculation, batch normalization and ReLU activation operations through a 3×3 convolution kernel, and a local illumination perception feature map is output; the number of channels of the local illumination perception feature map is compressed to 1 through convolution calculation, and the output value range is limited to between 0 and 1 through the Sigmoid function as illumination perception information.

[0008] According to the lightweight target detection method based on multimodal images disclosed in the present invention, preferably, the calculation process of the parameter-free channel attention module specifically includes: performing global average pooling operations and global average pooling operations on the spliced ​​feature map; adding the outputs of global average pooling and global maximum pooling to obtain a fusion tensor; limiting the value range of the fusion tensor to between 0 and 1 through the Sigmoid function to generate channel-level attention weights; expanding the attention weights to the same shape as the spliced ​​feature map, and multiplying them element-by-element to adjust the feature response of each channel of the spliced ​​feature map.

[0009] The second aspect of the present invention discloses a lightweight target detection system based on multimodal images, comprising: a memory for storing program instructions; a processor for calling the program instructions stored in the memory to implement a lightweight target detection method based on multimodal images as any of the above technical solutions.

[0010] The beneficial effects of the present invention include at least: the present invention is a lightweight target detection algorithm based on the dynamic adjustment and fusion of multiple weights of multimodal images, which combines weight-guided feature optimization, local illumination perception mechanism and parameter-free channel attention mechanism to form a multiple weight adjustment module (MWAF), which performs complementary fusion of the features of infrared and visible light images at the input end, and effectively overcomes the limitations of a single modality in complex environments without generating a large amount of computation, thereby significantly improving the accuracy and stability of target detection tasks. Specifically: The feature optimization method based on weight guidance can generate a weight matrix through feature masks according to the importance of image features of different modalities, and perform weighted processing, which can make full use of the complementarity between the modalities and avoid the loss of key features caused by simple feature superposition. Secondly, selective weighted processing can alleviate the problem of information redundancy, highlighting only the features that contribute to the detection task, thereby avoiding the interference of invalid or duplicate information and improving the computational efficiency and detection accuracy of the model. The Local Illumination Perception Module (LIPM) captures local illumination information, perceives local illumination changes in the input image, and dynamically adjusts the fusion weights of multimodal features based on this information. This effectively reduces feature distortion caused by large illumination variations within the image, ensuring accurate extraction of target location information and detailed features even under complex lighting conditions, significantly improving the model's performance in multimodal tasks. The Parameter-Free Channel Attention (PFCA) module utilizes two parameter-free operations, global average pooling (GAP) and global maximum pooling (GMP), to recalibrate features across different channels, enhancing the expressiveness of important features and suppressing interference from irrelevant or redundant information, thereby improving the model's ability to extract deep semantic information from images and details of small objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of the network structure of a lightweight object detection method based on multimodal images according to an embodiment of the present invention is shown.

[0012] Figure 2 A schematic diagram of the network structure of a multiple weight adjustment module according to an embodiment of the present invention is shown.

[0013] Figure 3 A schematic diagram of the network structure of a local illumination perception module according to an embodiment of the present invention is shown.

[0014] Figure 4 A schematic diagram of the network structure of a parameter-free channel attention module according to an embodiment of the present invention is shown.

[0015] Figure 5 A schematic block diagram of a lightweight object detection system based on multimodal images according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0016] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0017] According to one embodiment of the present invention, in view of the fact that the performance of current target detection schemes will be significantly degraded in low-light environments and the shortcomings of early fusion methods such as insufficient complementarity utilization and information redundancy, a lightweight target detection method based on multimodal images is proposed, comprising the following steps:

[0018] Step 1: Build a lightweight object detection network based on dynamic adjustment and fusion of multiple weights of multimodal images;

[0019] Step 2: Input the paired infrared and visible light images into the network as training sets, perform network training, and adjust network parameters;

[0020] Step 3: Verify the network model obtained in step 2 and perform a final evaluation using the test set to ensure the robustness and generalization ability of the model in different scenarios;

[0021] Step 4: Input the infrared and visible light image pairs into the model verified in step 3 for real-time target detection.

[0022] like Figure 1 and Figure 2As shown, the lightweight object detection network based on dynamic adjustment and fusion of multiple weights of multimodal images proposed in this embodiment includes: a multiple weight adjustment module (MWAF), a backbone network (Backbone) of the object detection network, a neck (Neck), and a detection head (Head). By adding the multiple weight adjustment module before the backbone network of the detection network, the present invention can effectively fuse the features of infrared and visible light images. The fused feature map obtained after processing by the multiple weight adjustment module is fed into the backbone network of the object detection network for further feature extraction. In the neck portion of the object detection network, features at multiple resolutions are aggregated to enhance detection accuracy. Finally, the detection head portion generates the final object bounding box and category prediction, thereby achieving higher-precision object detection.

[0023] like Figure 2 As shown in the figure, the Multiple Weight Adjustment Module (MWAF) specifically includes a weight-guided feature optimization method, a local illumination perception module, and a parameter-free attention module. In the MWAF, the input image is first processed by the weight-guided feature optimization method: the input 6-channel tensor is first divided into two parts, the first 3 channels are visible light images (RGB images), and the last 3 channels are infrared images (IR images). Then, two convolution blocks are used to convolve, batch normalize, and activate the RGB and IR features respectively to generate enhanced features. This process can be expressed as:

[0024]

[0025]

[0026] Among them, f R_en is the enhanced RGB feature, f I_en is the enhanced IR feature, X R is the original RGB feature, X I is the original IR feature.

[0027] Then, a 1×1 convolution operation is used to reduce the number of channels of the feature map to 1, and the Sigmoid function is used to limit the value range of its output to between 0 and 1 to generate a weight mask. The process of generating a feature map mask is as follows:

[0028]

[0029]

[0030] Among them, mask R , mask I are the weight masks of RGB image and IR image respectively, and σ represents the Sigmoid function.

[0031] Finally, the generated RGB mask and IR mask are pixel-by-pixel weighted on the original feature map to generate the optimized RGB feature map and IR feature map. The process is as follows:

[0032]

[0033]

[0034] Among them, f R_m , f I_m They are the optimized RGB features and IR features respectively. It is an element-wise multiplication operation.

[0035] The above-mentioned weight-guided feature optimization method can weight the key information in the RGB feature map and the IR feature map, ensuring that important areas are highlighted and effectively filtering out unimportant information.

[0036] like Figure 3 As shown in Figure 1, the calculation process of the local illumination perception module includes: the input features are first extracted through a 3×3 convolution kernel. The convolution kernel operation is expressed as:

[0037]

[0038]

[0039] Among them, Y R and Y I Represents the lighting information of RGB and IR images respectively.

[0040] Then, in order to further reduce the channel dimension and extract local illumination perception information, a 1×1 convolution operation is introduced to compress the number of channels to 1. The specific operation is as follows:

[0041]

[0042]

[0043] Among them, W R and W I Represents the illumination weights for the RGB and IR images, respectively. Subtracting 1 from the local illumination value concentrates illumination information, allowing for an intuitive assessment of whether the illumination intensity in the current region is stronger or weaker than the baseline. The sigmoid function compresses the resulting value to between 0 and 1, facilitating subsequent adjustments to the weighting factors.

[0044] After obtaining the local illumination characteristics (W R and W I), and then use this feature to adjust the image features of different modalities. In this implementation, the local illumination feature is used to adjust the weights between the IR image and the RGB image.

[0045] The weighted result of RGB features is:

[0046]

[0047] The weighted result of infrared features is:

[0048]

[0049] This weighting mechanism is based on local illumination features, which enhances the expression of infrared features in low-light areas while suppressing the influence of RGB features in overly bright areas, ensuring the maintenance of illumination consistency during multimodal image fusion.

[0050] Subsequently, the fused feature map passes through the residual connection and parameter-free channel attention module to further improve the representation ability of important features.

[0051] like Figure 4 As shown, in the parameter-free channel attention module, GAP (global average pooling) and GMP (global maximum pooling) operations are first performed on the input feature map: the average value of all pixel values ​​on each channel is calculated avg and the maximum value X max ;

[0052] Add the outputs of global average pooling and global maximum pooling to obtain a tensor F that combines the two feature descriptions c :

[0053] F c =X avg +X max

[0054] The fused feature description Fc is used to generate the channel-level attention weight W through the Sigmoid activation function c :

[0055]

[0056] Among them, σ represents the Sigmoid function, which maps the input value to between (0,1).

[0057] Finally, the attention weight W c Expand to the same shape as the input feature map and multiply element-wise to adjust the feature response of each channel to obtain the output feature map F e ;

[0058] Through the above-mentioned parameter-free attention method, without introducing additional parameters, it is possible to adaptively adjust the feature response between channels, making the model more generalizable in different scenarios, improving the overall performance while maintaining computational efficiency, and successfully enhancing the model's attention to key features.

[0059] like Figure 5 As shown, according to another embodiment of the present invention, a lightweight target detection system 500 based on multimodal images is also disclosed, including: a memory 501 for storing program instructions; a processor 502 for calling the program instructions stored in the memory to implement the lightweight target detection method based on multimodal images as described in the above embodiment.

[0060] In summary, the present invention extracts features from both infrared and visible light input images. It then utilizes weight-guided feature optimization and a local illumination perception module to ensure effective feature fusion under varying lighting conditions. The fused feature maps are then processed through residual connections and a parameter-free channel attention module to further enhance the representation of important features. The fused features, processed by the MWAF module, are then fed into a YOLO network (e.g., YOLOv8) for object detection. Through an innovative dynamic adjustment mechanism for multiple weights (weight guidance, illumination-aware weights, and attention weights), the present invention successfully achieves stable and accurate object detection in complex environments (such as low light and occlusion). Compared to traditional single-modality and static weight fusion methods, the present invention significantly improves detection robustness and reliability. With a simple and efficient network architecture, the present invention achieves excellent detection accuracy and subjective perception. Furthermore, the entire method is designed with the lightweight nature of the network in mind, ensuring performance improvements without introducing additional parameters or computational complexity, maintaining low resource usage. This makes the present invention widely applicable in embedded and real-time application scenarios.

[0061] All or part of the steps in the various methods of the above embodiments can be completed by controlling related hardware through a program. The program can be stored in a readable storage medium, and the storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electronically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.

[0062] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A lightweight target detection method based on multimodal images, characterized in that: include: Receiving images to be detected: receiving paired infrared images and visible light images; Weight-guided feature optimization: performing convolution, batch normalization, and ReLU activation operations on the infrared image and the visible light image, respectively, to generate an enhanced infrared feature map and an enhanced visible light feature map; Using a convolution operation to reduce the number of channels of the enhanced infrared feature map to 1, and using a Sigmoid function to limit the output value range to between 0 and 1, so as to generate an infrared feature weight mask; Using a convolution operation to reduce the number of channels of the enhanced visible light feature map to 1, and using a Sigmoid function to limit the output value range to between 0 and 1, so as to generate a visible light feature weight mask; Performing pixel-by-pixel weighting on the enhanced infrared feature map based on the infrared feature weight mask to generate an optimized infrared feature map; performing pixel-by-pixel weighting on the enhanced visible light feature map based on the visible light feature weight mask to generate an optimized visible light feature map; Local illumination perception mechanism: inputting the enhanced infrared feature map and the enhanced visible light feature map into a local illumination perception module to extract illumination perception information; Adjusting the weights of the optimized infrared feature map and the optimized visible light feature map according to the illumination perception information to generate a visible light weighted feature map and an infrared weighted feature map; splicing the visible light weighted feature map and the infrared weighted feature map in the channel dimension to generate a spliced ​​feature map; Parameter-free channel attention mechanism: The spliced ​​feature map is subjected to subsequent feature extraction operations through residual connections and parameter-free channel attention modules to obtain a multimodal fusion feature map; The calculation process of the parameter-free channel attention module specifically includes: Performing a global average pooling operation and a global average pooling operation on the spliced ​​feature map; Add the outputs of global average pooling and global maximum pooling to get the fused tensor; The fused tensor is limited to a value range between 0 and 1 by a Sigmoid function to generate channel-level attention weights. Expand the attention weight to the same shape as the concatenated feature map and multiply it element-wise to adjust the feature response of each channel of the concatenated feature map; Target detection: The multimodal fusion feature map is input into the target detection network for target detection to output the detection result. The target detection network is a YOLO network.

2. The lightweight target detection method based on multimodal images according to claim 1, characterized in that: The YOLO network specifically includes: a backbone network, a neck, and a detection head; the multimodal fusion feature map is further extracted in the backbone network part, and features of multiple resolutions are aggregated in the neck of the target detection network to enhance detection accuracy. Finally, the final target bounding box and category prediction are generated by the detection head.

3. The lightweight target detection method based on multimodal images according to claim 1, characterized in that: The calculation process of the local illumination perception module specifically includes: The input features are convolved with a 3×3 convolution kernel, batch normalized, and activated with ReLU to output a local illumination perception feature map. The number of channels of the local illumination perception feature map is compressed to 1 through convolution calculation, and the output value range is limited to between 0 and 1 through the Sigmoid function as illumination perception information.

4. A lightweight target detection system based on multimodal images, characterized in that: include: a memory for storing program instructions; A processor, configured to call the program instructions stored in the memory to implement the lightweight target detection method based on multimodal images as described in any one of claims 1 to 3.

Images