A semi-coupled multi-modal feature enhancement small target detection method and system

Abstract

The application discloses a kind of semi-coupling multimodal feature enhancement small target detection method and system, belong to multimodal image processing technical field.The application is first to visible light image, infrared image is implemented pre-processing, is decoupled by parameter independent double branch network extraction feature, two branches do not carry out feature interaction in middle shallow layer, retain small target weak feature signal;Subsequently, a multi-index joint evaluation model is constructed to calculate the uncertainty coefficient of two modalities, and the imaging reliability of the modalities is quantified;According to the uncertainty coefficient, the fusion weight is adaptively calculated, and the two features are dynamically weighted and fused at the high-level end of the network, to suppress the contribution of high-noise modalities;Finally, based on the enhanced multimodal features, small target detection is completed.The application solves the problem that traditional multimodal fusion methods easily cause small target feature annihilation and fixed weight cannot adapt to complex scenes, effectively improves the small target detection accuracy under low illumination, occlusion and complex background conditions, and can be used in low-altitude security, remote sensing detection, all-weather infrared monitoring and other scenes.

Description

Technical Field

[0001] This invention belongs to the field of multimodal image processing and target detection technology, specifically involving a semi-coupled multimodal feature enhancement method and system for small target detection using visible light and infrared synergy. It can be applied to scenarios such as low-altitude security monitoring, remote sensing image interpretation, all-weather field surveillance, and infrared reconnaissance. Background Technology

[0002] Visible light imaging can clearly reproduce scene textures, contours, and surface details, while infrared thermal imaging relies on target thermal radiation and is not constrained by lighting conditions. The two types of information are highly complementary. Feature fusion of visible light and infrared images to improve target detection capabilities has become a common technical solution in the field of all-weather visual surveillance.

[0003] Existing multimodal fusion schemes are classified into three categories based on the feature fusion location: early fusion, mid-term fusion, and late-term fusion. Early and mid-term fusions combine visible light and infrared features at shallow or mid-level layers of the network. For small targets within an image, their feature responses are inherently weak, and the strong features formed by large areas of background in the scene can easily suppress the effective information of small targets, causing the loss of small target feature signals and ultimately leading to missed detections and false detections.

[0004] While traditional late-stage fusion methods enable independent feature extraction from two images, they mostly employ fixed weights for feature fusion. Under complex conditions such as backlighting, low illumination, smoke obstruction, and environmental heat interference, the imaging quality of a single modality significantly deteriorates. Fixed weights cannot distinguish between reliable features and noise information; noise carried by the degraded modality contaminates the fusion result, causing significant fluctuations in detection model accuracy and poor scene adaptability. Furthermore, existing solutions lack quantitative evaluation methods for modal imaging quality, making it difficult to adaptively adjust the fusion strategy according to the real-time scene, and thus failing to meet the engineering requirements for stable detection of small targets in complex environments. Summary of the Invention

[0005] Purpose of the invention To address the problems of existing visible-infrared multimodal fusion technologies, such as loss of weak features of small targets, inability to adaptively adjust fusion weights, and insufficient noise suppression in harsh environments, this invention provides a semi-coupled multimodal feature enhancement method and system for small target detection. By first decoupling and extracting features, and then dynamically fusing them at the end, combined with a modal uncertainty assessment mechanism, the method fully preserves the weak features of small targets, adaptively suppresses low-reliability modal noise, and improves the detection accuracy and robustness of small targets in complex environments.

[0006] Technical solution To achieve the above objectives, the present invention adopts the following technical solution: A semi-coupled multimodal feature enhancement method for small target detection mainly includes five stages: image preprocessing, decoupled independent feature extraction, modal uncertainty estimation, dynamic weighted feature fusion, and small target detection. This invention adopts a semi-coupled architecture, with the core idea being decoupling first and then fusing. Shallow and mid-level features remain independent throughout the process, with feature fusion only performed at the high-level ends of the network, thus protecting the weak features of small targets at the network structure level. Simultaneously, modal uncertainty quantification is introduced, dynamically adjusting the fusion weights based on the modal confidence of each frame to reduce the interference of noisy modalities on the fusion result.

[0007] The present invention also provides a corresponding detection system, which consists of an image preprocessing module, a decoupled feature extraction module, an uncertainty estimation module, a dynamic weighted fusion module, and a target detection module connected in sequence. Based on the above method, it realizes enhanced detection of small targets in visible light and infrared images.

[0008] Beneficial effects First, the present invention adopts a decoupled dual-branch structure, in which features at each level are extracted independently from visible light and infrared images. There is no cross-modal feature mixing during the forward propagation of the network, which avoids strong background features from covering weak signals of small targets and effectively reduces the probability of missing small targets.

[0009] Second, a multi-index joint modal uncertainty assessment model is introduced, which can quantify the credibility of the features of the visible light mode and infrared mode in each frame of the image, break through the limitations of traditional fixed weight fusion, dynamically suppress the contribution of high noise mode, and improve the anti-interference ability of fused features.

[0010] Third, the feature fusion operation is performed only at the high level of the network, taking into account the inherent advantages of single-modal features and the complementary characteristics of multimodal information. It completes feature fusion while retaining visible light detail information and infrared thermal information, effectively improving the accuracy and recall of small target detection.

[0011] Fourth, through comparative testing, the FLIR-ADAS public dataset and a self-built visible light-infrared low-altitude small target paired dataset were selected for verification. Compared with traditional early fusion and fixed-weight late fusion schemes, the mAP of small target detection was improved by 8.2%, 9.7%, and 7.5% respectively in low-light, smoke-covered, and complex background scenes, and the improvement in ultra-small target detection reached up to 12.3%. The overall architecture has low additional computing overhead, with a computing power increase of less than 3%, and can be adapted to lightweight deployment platforms such as embedded terminals and front-end monitoring equipment. Detailed Implementation

[0012] Step 1: Image preprocessing.

[0013] Visible light and infrared images of the same scene are acquired using a binocular synchronous acquisition device; the two images are uniformly scaled to a resolution of 640×480, and pixel registration is achieved using bilinear interpolation; grayscale adjustment is completed using gamma correction, with the correction coefficient selected in the range of 0.8 to 1.2; basic noise reduction is implemented using a 5×5 Gaussian filter to eliminate high-frequency noise from the acquisition device and ensure strict alignment of the spatial coordinates of the two images.

[0014] Step 2: Decoupled independent feature extraction.

[0015] Two symmetrical MobileNetV2 networks with independent parameter sharing were constructed, and preprocessed visible light and infrared images were input respectively. The visible light branch extracted shallow edge texture, mid-level detail features, and high-level semantic features layer by layer; the infrared branch extracted thermal contours, temperature distribution features, and high-level heat source semantic features layer by layer. There were no feature interactions or feature merging operations during the operation of the two branches, thus completely preserving the weak feature responses corresponding to small targets.

[0016] Step 3: Modal uncertainty estimation.

[0017] For the high-level feature maps output by the two branches, the following metrics are calculated: the local pixel variance is calculated using a 3×3 sliding window to characterize feature stability; the mean pixel residual between the original image and the preprocessed image is calculated to characterize the noise level; and a threshold of 0.5, the global mean of the feature map, is set, and the percentage of pixels above the threshold is counted as the effective feature coverage.

[0018] After normalizing the three indicators using min-max, they are entered into the formula: Calculate the uncertainty coefficients for the visible light mode and the infrared mode, respectively.

[0019] Step 4: Dynamic weighted feature fusion.

[0020] According to the weighting calculation formula: , Solve for the two-way modal fusion weights, satisfying The high-level visible light features and high-level infrared features are multiplied by their corresponding weights and then summed to obtain the noise-reduced and enhanced multimodal fusion feature map.

[0021] Step 5: Small target detection.

[0022] The fused features are fed into the YOLOv8 detection head to complete candidate box selection, target classification, bounding box regression optimization, and output the location, category, and confidence information of small targets in the image.

[0023] The detection system in this embodiment includes an image preprocessing module, a decoupled feature extraction module, an uncertainty estimation module, a dynamic weighted fusion module, and a target detection module. The target detection module can use detection networks such as YOLOv5, YOLOv8, and SSD, and can be flexibly replaced according to the hardware platform. The entire system processes no more than 30ms per frame, meeting the requirements for real-time monitoring.

[0024] It should be understood that the above description is a preferred embodiment of the present invention and is not intended to limit the present invention. Any simple modifications or equivalent substitutions made within the scope of the inventive concept should be included within the protection scope of the present invention. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall implementation architecture of the present invention; Figure 2 This is a flowchart illustrating the overall process of the small target detection method of the present invention. Figure 3 This is a schematic diagram illustrating the modal uncertainty estimation and dynamic weight calculation of the present invention; Figure 4 This is a schematic diagram of the dual-branch decoupling feature extraction network structure of the present invention; Figure 5 This is a schematic diagram comparing the detection performance of different algorithms in this invention.

Claims

1. A semi-coupled multimodal feature-enhanced small target detection method, characterized in that, Includes the following steps: S1. Image preprocessing: Acquire visible light and infrared images of the same spatial scene simultaneously, perform size normalization, grayscale correction and basic noise filtering on the two images, complete image spatial registration, and ensure that the pixel positions of the visible light image and the infrared image correspond one-to-one. S2. Decoupled Independent Feature Extraction: A dual-branch feature extraction network is constructed to independently extract features from the preprocessed visible light image and infrared image. The visible light branch extracts texture, edge, and surface detail features, while the infrared branch extracts temperature radiation and thermal contour features. The two branches do not interact across modal features during the entire feature extraction process and independently retain the weak feature information corresponding to small targets in the image. S3. Modal Uncertainty Estimation: For the high-level visible light feature map and high-level infrared feature map output by the dual-branch network, a joint evaluation model is constructed from three dimensions: feature stability, noise interference level, and effective feature coverage. The visible light modal uncertainty coefficient is calculated for each model. With infrared mode uncertainty coefficient The uncertainty coefficient ranges from 0 to 1 and is used to characterize the reliability of the current frame features of the corresponding modality. S4. Dynamic weighted feature fusion: Calculate the mode fusion weight based on the uncertainty coefficients of the two modes, assign more weight to modes with low uncertainty coefficients and higher reliability, and reduce the weight contribution of modes that are severely affected by noise and have high uncertainty coefficients; use the obtained weights to perform weighted fusion of the two high-level features to generate enhanced multimodal features; S5. Small Target Detection: The enhanced multimodal features are input into the detection network head to generate target candidate boxes, classify targets and regress bounding boxes, and output the detection results of small targets in the image.

2. The semi-coupled multimodal feature enhancement method for small target detection according to claim 1, characterized in that: In step S2, the dual-branch feature extraction network has a symmetrical structure and independent network parameters. There are no cross-branch feature interaction operations such as feature splicing or pixel-by-pixel calculation between the two branches, and the visible light features and infrared features are extracted in a completely decoupled manner.

3. The semi-coupled multimodal feature enhancement method for small target detection according to claim 1, characterized in that, The modal uncertainty coefficient in step S3 is calculated as follows: Calculate the local pixel variance of the feature map respectively Image noise residuals before and after preprocessing effective feature pixel ratio ; Maximum and minimum value normalization was applied to the three indicators; According to the formula α β γ Calculate the modal uncertainty coefficient; where α+β+γ=1, and α, β, and γ are all preset coefficients greater than 0; The larger the value, The lower the reliability of modal characteristics.

4. The semi-coupled multimodal feature enhancement method for small target detection according to claim 1, characterized in that, The fusion weight calculation method in step S4 is as follows: Visible mode weights: Infrared mode weights: In the formula, .

5. The semi-coupled multimodal feature enhancement method for small target detection according to claim 1, characterized in that: Step S4 only performs fusion operations on high-level features at the end of the network, while shallow and mid-level features remain decoupled to avoid early fusion causing weak features of small targets to be covered by background information.

6. A semi-coupled multimodal feature-enhanced small target detection system, characterized in that, The detection method according to any one of claims 1 to 5 includes: The image preprocessing module is used to receive synchronously acquired visible light and infrared images, and to perform size adjustment, grayscale correction, image registration and preliminary noise reduction. The decoupled feature extraction module is connected to the image preprocessing module and includes independent visible light feature extraction branches and infrared feature extraction branches, which are used to separate and extract image features of two modalities. The uncertainty estimation module, whose input is connected to the decoupled feature extraction module, is used to calculate the uncertainty coefficients of the visible light mode and the infrared mode. The dynamic weighted fusion module is connected to the decoupled feature extraction module and the uncertainty estimation module respectively. It adaptively generates fusion weights based on the uncertainty coefficient and completes high-level feature fusion. The target detection module is connected to the dynamic weighted fusion module at its input end, and completes the localization and classification of small targets based on the fusion enhancement features.

7. A semi-coupled multimodal feature-enhanced small target detection system according to claim 6, characterized in that: Within the decoupled feature extraction module, the two branches are only connected to the dynamic weighted fusion module at the network output end, and there is no cross-branch information exchange channel between different levels in the network.

8. The semi-coupled multimodal feature enhancement method for small target detection according to claim 3, characterized in that: α=0.4, β=0.3, γ=0.3; a 3×3 sliding window is used to calculate the local pixel variance; 0.5 times the global mean of the feature map is used as the threshold to filter effective feature pixels.

9. The semi-coupled multimodal feature enhancement method for small target detection according to claim 1, characterized in that: The dual-branch feature extraction network uses a lightweight backbone network such as MobileNetV2, ShuffleNet, or ResNet18. Both branches fully retain the multi-level feature extraction structure, and all cross-modal interaction modules are removed.

10. The semi-coupled multimodal feature enhancement method for small target detection according to claim 1, characterized in that: In the image preprocessing stage, the images were uniformly scaled to a resolution of 640×480, and bilinear interpolation was used to complete the registration; the gamma correction coefficient was set to 0.8 to 1.2; and a 5×5 Gaussian convolution kernel was used for initial noise reduction.

Images