Small target identification method and system for multi-modal fusion image in complex environment

Abstract

The invention discloses a small target recognition method and system for a multi-modal fusion image in a complex environment, and belongs to the technical field of computer vision and image recognition, and the method comprises the steps: obtaining a visible light image, an infrared image and environment sensor data; image registration is carried out on visible light and infrared images, and a multi-scale image feature pyramid is constructed. And respectively extracting visible light and infrared image features to obtain visible light and infrared imaging feature data. And performing multi-modal data fusion on the visible light and infrared imaging feature data based on a cross-modal attention mechanism, and adaptively adjusting a fusion weight based on environmental sensor data to generate fusion features. And performing space-time enhancement processing on the fusion feature to obtain an enhanced fusion feature. And performing target tracking detection on the small target, and outputting position and category information of the small target. According to the method, the small target recognition capability in a severe environment is remarkably improved, and high precision and robustness can still be kept in a foggy, low-visibility and dark scene.

Description

Technical Field

[0001] This application belongs to the field of computer vision and image recognition technology, and specifically relates to a method and system for small target recognition in multimodal fusion images under complex environments. Background Technology

[0002] Small target recognition is a significant challenge in computer vision. These small targets typically have a small pixel ratio, are prone to losing detail, are susceptible to environmental noise in complex backgrounds, and exhibit blurred edges and low contrast at long distances, making them difficult for traditional recognition methods to effectively capture and distinguish. Currently, there are three main technical solutions: First, single-modal small target detection methods: These methods are typically represented by deep learning models such as YOLO and Faster R-CNN, directly detecting and classifying targets on the input image. Under sufficient lighting and with clear targets, these methods achieve good detection results. However, in environments such as haze, low visibility, and low light, the detection rate of these methods for small targets decreases significantly due to a substantial drop in image contrast and clarity, making them unsuitable for practical applications. Second, infrared imaging-based small target detection methods: These methods utilize thermal imaging cameras to acquire data and combine it with convolutional neural networks for recognition. Their advantage lies in maintaining a certain level of imaging and detection capability in nighttime or low-light scenarios. However, in dense fog or when the temperature difference between the target and the background is not significant, small targets often do not stand out well in infrared images. Furthermore, the low resolution of infrared images can easily lead to target loss or inaccurate identification. Thirdly, multimodal image fusion methods. Existing research attempts to fuse infrared and visible light images in a simple way, such as through pixel-level overlay or traditional filtering methods to improve the overall image clarity. These methods improve target visibility to some extent, but because the fusion method is too coarse and lacks targeted optimization for small target features, it is prone to problems such as small target features being masked or information redundancy, resulting in limited improvement in recognition accuracy. In summary, existing technical solutions either rely on a single modality, which is prone to failure in complex environments, or use simple fusion, which cannot guarantee effective enhancement of small target features. Especially in complex scenarios such as fog, low visibility, and low light, existing methods are difficult to meet application requirements in terms of recognition accuracy and robustness. Summary of the Invention

[0003] To address the aforementioned issues, this invention provides a method and system for small target recognition using multimodal fusion images in complex environments. This overcomes the limitations of current small target recognition methods in complex scenarios such as fog, low visibility, low light, and nighttime conditions, as well as the high dependence on external lighting and environmental conditions, which leads to generally low accuracy in the detection and recognition of small targets.

[0004] A method for small target recognition in multimodal fusion images under complex environments includes: Acquire visible light images, infrared images, and environmental sensor data, including ambient illuminance and visibility. Image registration is performed on visible light images and infrared images, and a multi-scale image feature pyramid is constructed based on the registered images; Based on the multi-scale image feature pyramid, visible light image features and infrared image features are extracted respectively to obtain visible light and infrared imaging feature data; Multimodal data fusion of visible light and infrared imaging feature data is performed based on cross-modal attention mechanism, and the fusion weights are adaptively adjusted based on environmental sensor data to generate fused features; Spatiotemporal enhancement processing is applied to the fusion features to obtain enhanced fusion features; The system performs target tracking and detection on small targets based on enhanced fusion features, and outputs the location and category information of the small targets.

[0005] According to a specific embodiment of the present invention, image registration is performed on visible light images and infrared images, and a multi-scale image feature pyramid is constructed based on the registered images, including: The visible light and infrared cameras are time-synchronized and calibrated, and their intrinsic, extrinsic, and distortion parameters are obtained. Based on the acquired intrinsic, extrinsic and distortion parameters, a combined geometric-radiometric correction strategy is used to geometrically align visible light and infrared images and to perform subpixel-level correction on residual parallax. Based on aligned visible light and infrared images, a deep learning network is used to generate a multi-scale image feature pyramid.

[0006] According to a specific embodiment of the present invention, multimodal data fusion of visible light and infrared imaging feature data based on a cross-modal attention mechanism, and adaptive adjustment of fusion weights based on environmental sensor data to generate fused features further includes: Based on cross-modal attention mechanism, using visible light image features Construct a query vector Q using infrared image features Construct the key vector K and value vector V respectively, calculate the first attention weight Attn1, and calculate the infrared feature-enhanced visible light feature based on the first attention weight Attn1. ,in,

[0007]

[0008]

[0009]

[0010]

[0011] Where s is the pyramid scale. , , , respectively, are the linear projection matrices of the query vector Q, key vector K, and value vector V, and d is the dimension of the internal vectors of the first attention; Infrared image features Construct a query vector Q′ using visible light image features Construct the key vector K′ and value vector V′ respectively, calculate the second attention weight Attn2, and calculate the infrared feature enhanced by visible light feature based on the second attention weight Attn2. ,in,

[0012]

[0013]

[0014]

[0015]

[0016] in, , , These are the query vector Q' and the key vector, respectively. Sum value vector The linear projection matrix, where d′ is the dimension of the internal vector of the second attention; The gating weight α is calculated based on environmental sensor data, and the enhanced visible light features are then adjusted accordingly. infrared features Weighted fusion is performed to obtain fusion features. ,in, α∈[0,1]

[0017] in, For illuminance, For visibility, Image contrast index For fog probability estimation, σ represents Sigmoid activation, and w is the weight vector. This is a bias term.

[0018] According to a specific embodiment of the present invention, the gating weight α is adaptively adjusted based on environmental sensor data.

[0019] According to a specific embodiment of the present invention, spatiotemporal enhancement processing is performed on the fusion features to obtain enhanced fusion features, which further includes: Small target motion trajectories are extracted using continuous frame difference and optical flow estimation to extract fused features; Static background suppression is performed on the motion trajectory of small targets based on infrared thermal features, resulting in enhanced fusion features.

[0020] According to a specific embodiment of the present invention, the extraction of fused features from the small target motion trajectory using continuous frame difference and optical flow estimation further includes: An optical flow estimation network is used to obtain the current video frame t and its neighboring frames t. Displacement field between 1 ; Using consecutive frame difference to separate adjacent frames t 1 fusion features The model is then transformed in reverse to the current frame coordinate system, and the fusion features with the current video frame t are calculated. difference residuals The calculation formula is:

[0021] In the formula, This is a sampling function based on bilinear interpolation.

[0022] According to a specific embodiment of the present invention, the enhanced fusion feature obtained by static background suppression of the motion trajectory of a small target based on infrared thermal features further includes: Thermal saliency map obtained from infrared image Constructing motion attention The calculation formula is as follows:

[0023] motion attention Fusion features with the current video frame t Adding them together yields enhanced fusion features. The calculation formula is as follows:

[0024] In the formula, η is the scaling factor for motion attention.

[0025] According to a specific embodiment of the present invention, target tracking and detection of small targets based on enhanced fusion features, and outputting the position and category information of the small targets, further includes: Construct a small target enhancement network to enhance and optimize the features of small targets; The optimized small targets are tracked and detected, and their location and category information are output.

[0026] According to a specific embodiment of the present invention, constructing a small target enhancement network to enhance and optimize the features of small targets further includes: Generate candidate regions at a high-resolution scale of fused features; Differentiable affine sampling is performed on the features within the candidate region to construct a small target enhancement network; Small target enhancement networks are used to enhance the contrast and perform super-resolution reconstruction on small targets, generating high-resolution features.

[0027] According to a specific embodiment of the present invention, the method further includes training and inference of the small target enhancement network, specifically including: During the training phase, registered visible light and infrared image pairs are used as sample inputs, and synthetic fog, low-light conditions, and thermal noise are added to perform thresholded random augmentation on the small target enhancement network. Its total loss function is... Represented as:

[0028] in,

[0029]

[0030] in, , , , These are the weighting coefficients for each loss term. For scale-aware classification loss, The IoU / CIoU localization loss is a scale-weighted method. For time-series consistency loss, For super-resolution network loss terms, For gated monotonicity regularization, Let i be the area of ​​target i. This is the lower bound of the target area in the dataset. β is the focus factor, and β is the weighting index. For the first The predicted probability of a sample, where N is the number of samples. For area weighting, The fusion features of the current video frame t, For adjacent frames t The fusion characteristics of 1 Here, U is the sampling function based on bilinear interpolation, and U is the displacement field matrix. During the inference phase, a full-image coarse inspection and fusion is performed based on the input visible light and infrared image samples. Then, the candidate regions are refined by super-resolution and subjected to secondary discrimination, outputting stable motion trajectories and confidence scores for small targets.

[0031] According to a specific embodiment of the present invention, it further includes: The scene recognition mode is adaptively switched based on environmental sensor data.

[0032] According to a specific embodiment of the present invention, adaptively switching the scene recognition mode based on environmental sensor data further includes: The current scene state S is estimated jointly based on environmental sensor data and image statistics, where S∈{normal during the day, foggy day, night, extreme}. Based on the current scenario state S, a state machine is constructed using a hysteresis threshold mechanism, and the gating weights are adjusted. Apply a monotonically consistent regularization constraint, where the monotonically consistent regularization constraint... Represented as:

[0033] In the formula, Visibility.

[0034] A small target recognition system based on multimodal fusion images in complex environments includes: The data acquisition module is used to acquire visible light images, infrared images, and environmental sensor data, including ambient illuminance and visibility. The image registration module is used to register visible light images and infrared images, and to construct a multi-scale image feature pyramid based on the registered images; The feature extraction module is used to extract visible light image features and infrared image features based on a multi-scale image feature pyramid, respectively, to obtain visible light and infrared imaging feature data. The feature fusion module is used to perform multimodal data fusion of visible light and infrared imaging feature data based on cross-modal attention mechanism, and adaptively adjust the fusion weights based on environmental sensor data to generate fused features; The spatiotemporal enhancement module is used to perform spatiotemporal enhancement processing on the fused features to obtain enhanced fused features; The target detection module is used to perform target tracking and detection on small targets based on enhanced fusion features, and outputs the location and category information of the small targets.

[0035] According to a specific embodiment of the present invention, the image registration module further includes: The calibration and parameter acquisition module is used to perform time synchronization and calibration of visible light and infrared cameras, and to acquire the intrinsic, extrinsic and distortion parameters of the visible light and infrared cameras. The correction module is used to perform geometric alignment of visible light and infrared images based on the acquired intrinsic, extrinsic and distortion parameters, and to perform subpixel-level correction of residual parallax. The feature pyramid generation module is used to generate multi-scale image feature pyramids based on aligned visible light and infrared images using a deep learning network.

[0036] According to a specific embodiment of the present invention, the feature fusion module further includes: The visible light feature calculation module is used to construct a query vector with visible light image features and construct a key vector and a value vector with infrared image features based on the cross-modal attention mechanism, calculate the first attention weight, and calculate the infrared feature-enhanced visible light features based on the first attention weight. The infrared feature calculation module is used to construct a query vector with infrared image features, construct a key vector and a value vector with visible light image features respectively, calculate the second attention weight, and calculate the infrared features enhanced by visible light features based on the second attention weight. The fusion feature calculation module is used to calculate the gating weights based on environmental sensor data, and to perform weighted fusion of enhanced visible light features and infrared features according to the gating weights to obtain fused features.

[0037] According to a specific embodiment of the present invention, the spatiotemporal enhancement module further includes: The trajectory feature extraction module is used to extract the fused features of small target motion trajectories using continuous frame difference and optical flow estimation. The fusion feature enhancement module is used to suppress the static background of the motion trajectory of small targets based on infrared thermal features, thereby obtaining enhanced fusion features.

[0038] According to a specific embodiment of the present invention, the trajectory feature extraction module further includes: The displacement field calculation module is used to obtain the displacement field between the current video frame and adjacent frames using an optical flow estimation network. The differential residual calculation module is used to reverse-transform the fusion features of adjacent frames to the coordinate system of the current frame using continuous frame differential, and to calculate the differential residual of the fusion features with the current video frame.

[0039] According to a specific embodiment of the present invention, the fusion feature enhancement module further includes: The motion attention construction module is used to construct motion attention based on the thermal saliency map obtained from infrared images; The fusion computing module is used to add motion attention to the fusion features of the current video frame to obtain enhanced fusion features.

[0040] According to a specific embodiment of the present invention, the target detection module further includes: The small target enhancement and optimization module is used to construct a small target enhancement network to enhance and optimize the features of small targets. The tracking, detection, and output module is used to track and detect the optimized small targets and output the position and category information of the small targets.

[0041] According to a specific embodiment of the present invention, the small target enhancement and optimization module further includes: The candidate region generation module is used to generate candidate regions at a high-resolution scale of fused features; The small target enhancement network construction module is used to perform differentiable affine sampling on the features in the candidate region to construct the small target enhancement network. The high-resolution feature generation module is used to perform contrast enhancement and super-resolution reconstruction on small targets based on the small target enhancement network, generating high-resolution features.

[0042] According to a specific embodiment of the present invention, it further includes: The adaptive scene switching module is used to adaptively switch scene recognition modes based on environmental sensor data.

[0043] According to a specific embodiment of the present invention, the adaptive scene switching module further includes: The scene state estimation module is used to jointly estimate the current scene state based on environmental sensor data and image statistics. The state machine construction module is used to construct a state machine based on the current scenario state using a hysteresis threshold mechanism, and to apply monotonically consistent regular constraints to the gating weights.

[0044] Compared with the prior art, this application has the following advantages: 1. This invention introduces a small target enhancement network, which enhances the feature representation of low-resolution small targets through contrast enhancement and super-resolution reconstruction methods. Simultaneously, an improved small target-specific loss function is introduced into the detection model, making the model pay more attention to small targets during training. Even with small target sizes, the system can significantly improve detection accuracy and recall.

[0045] 2. This invention proposes a deep fusion method for visible light and infrared images based on an attention mechanism, achieving deep fusion of visible light and infrared images at the feature level. During the fusion process, the attention mechanism and Transformer structure are combined to adaptively adjust modal weights under different environmental conditions (such as fog, nighttime, and normal lighting), highlighting small target features, suppressing redundant information, and achieving robust recognition across different scenes. The system can fully utilize modal complementarity to maintain stable recognition performance under different environmental conditions.

[0046] 3. This invention proposes a spatiotemporally enhanced small target detection method, which extracts the dynamic trajectory features of small targets through continuous frame difference and optical flow estimation, and combines infrared thermal features to suppress background interference, thereby enhancing the temporal consistency and motion saliency of small targets. This method can improve detection stability and robustness in complex backgrounds and multi-interference environments.

[0047] 4. This invention proposes a feature enhancement and optimization training mechanism for small targets. It designs a small target enhancement network, uses regional super-resolution and contrast enhancement to enhance the features of small targets, and combines a special scale-weighted loss function to guide the detector to focus more on small targets during training, thereby improving detection accuracy and recall.

[0048] 5. This invention incorporates an adaptive scene switching mechanism. The system integrates illumination and visibility sensors, dynamically switching recognition modes based on environmental conditions: visible light is prioritized in normal daylight conditions, infrared imaging is enhanced in foggy weather, thermal imaging is relied upon at night, and a fusion strategy is employed under extreme conditions. The system can achieve mode switching and fusion optimization in diverse environments, thereby ensuring the real-time performance and stability of recognition. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart of a method for small target recognition in multimodal fusion images under complex environments, provided by an embodiment of the present invention.

[0051] Figure 2 This is a flowchart of a method for image registration and constructing an image feature pyramid according to an embodiment of the present invention.

[0052] Figure 3 This is a flowchart of a method for multimodal data fusion of visible light and infrared imaging feature data according to an embodiment of the present invention.

[0053] Figure 4 This is a flowchart of a method for spatiotemporal enhancement processing of fused features according to an embodiment of the present invention.

[0054] Figure 5 This is a flowchart of a method for extracting the motion trajectory of a small target based on a fusion feature according to an embodiment of the present invention.

[0055] Figure 6This is a flowchart of a method for static background suppression of small target motion trajectory based on infrared thermal features according to an embodiment of the present invention.

[0056] Figure 7 This is a flowchart of a method for target tracking and detection of small targets according to an embodiment of the present invention.

[0057] Figure 8 This is a flowchart of a method for constructing a small target enhancement network to enhance and optimize the features of small targets according to an embodiment of the present invention.

[0058] Figure 9 This is a flowchart of a method for adaptively switching scene recognition modes based on environmental sensor data according to an embodiment of the present invention.

[0059] Figure 10 This is a schematic diagram of the small target recognition process in a multimodal fusion image according to an embodiment of the present invention.

[0060] Figure 11 This is a schematic diagram of an image fusion framework provided according to an embodiment of the present invention.

[0061] Figure 12 This is a structural diagram of a small target recognition system for multimodal fusion images in complex environments, provided by an embodiment of the present invention.

[0062] Figure 13 This is a structural diagram of an image registration module provided according to an embodiment of the present invention.

[0063] Figure 14 This is a structural diagram of a feature fusion module provided according to an embodiment of the present invention.

[0064] Figure 15 This is a structural diagram of a spatiotemporal enhancement module according to an embodiment of the present invention.

[0065] Figure 16 This is a structural diagram of a trajectory feature extraction module provided according to an embodiment of the present invention.

[0066] Figure 17 This is a structural diagram of a fusion feature enhancement module provided according to an embodiment of the present invention.

[0067] Figure 18 This is a structural diagram of a target detection module according to an embodiment of the present invention.

[0068] Figure 19 This is a structural diagram of a small target enhancement and optimization module provided according to an embodiment of the present invention.

[0069] Figure 20 This is a structural diagram of an adaptive scene switching module according to an embodiment of the present invention.

[0070] Figure label: 01-Data Acquisition Module; 02-Image Registration Module; 03-Feature Extraction Module; 04-Feature Fusion Module; 05-Spatiotemporal Enhancement Module; 06-Target Detection Module; 07-Adaptive Scene Switching Module; 021-Calibration and parameter acquisition module; 022-Correction module; 023-Feature pyramid generation module; 041-Visible light feature calculation module; 042-Infrared feature calculation module; 043-Fusion feature calculation module; 051 - Trajectory Feature Extraction Module; 052 - Fusion Feature Enhancement Module; 0511 - Displacement field calculation module; 0512 - Differential residual calculation module; 0521 - Motion Attention Construction Module; 0522 - Fusion Computation Module; 061-Small target enhancement and optimization module; 062-Tracking detection and output module; 0611 - Candidate Region Generation Module; 0612 - Small Target Enhancement Network Construction Module; 0613 - High-Resolution Feature Generation Module; 071-Scene State Estimation Module; 072-State Machine Construction Module. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0072] Example 1 Additional aspects and advantages of embodiments of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of embodiments of the invention. Figures 1-11 This invention provides a method for small target recognition in multimodal fusion images under complex environments, including: S1: Acquire visible light images, infrared images, and environmental sensor data, including ambient illuminance and visibility.

[0073] S2: Perform image registration on visible light and infrared images, and construct a multi-scale image feature pyramid based on the registered images.

[0074] S3: Based on the multi-scale image feature pyramid, visible light image features and infrared image features are extracted respectively to obtain visible light and infrared imaging feature data.

[0075] S4: Multimodal data fusion of visible light and infrared imaging feature data is performed based on cross-modal attention mechanism, and the fusion weights are adaptively adjusted based on environmental sensor data to generate fused features.

[0076] S5: Perform spatiotemporal enhancement processing on the fusion features to obtain enhanced fusion features.

[0077] S6: Perform target tracking and detection on small targets based on enhanced fusion features, and output the location and category information of the small targets.

[0078] Specifically, in step S1, which involves acquiring visible light images, infrared images, and environmental sensor data, the present invention acquires visible light images and infrared images simultaneously through a binocular imaging unit composed of a visible light camera and an infrared camera, and acquires environmental sensor data through an environmental sensor, which includes an illumination and visibility sensor, and acquires ambient illumination and visibility through the illumination and visibility sensor respectively.

[0079] Specifically, step S2 involves image registration of the visible light image and the infrared image, and constructing a multi-scale image feature pyramid based on the registered image, including: S21: Perform time synchronization and calibration of the visible light and infrared cameras, and obtain the intrinsic, extrinsic and distortion parameters of the visible light and infrared cameras.

[0080] In a specific embodiment of this invention, to ensure the temporal and spatial consistency of visible light and infrared image data, it is necessary to perform time-series synchronization and high-precision calibration of the visible light and infrared cameras. During the time-series synchronization phase, to achieve frame-level time alignment and avoid motion blur or registration errors caused by differences in acquisition time, the system adopts a hardware synchronization scheme. The visible light and infrared cameras are rigidly mounted via fixed bases and driven by a unified hardware trigger signal to achieve time-series synchronization. This ensures that the two cameras can expose and capture scene images at the same time, thereby obtaining strictly matched visible light-infrared image pairs, fundamentally eliminating the mismatch caused by operating system scheduling delays in software synchronization. During the calibration phase, the purpose is to accurately obtain the internal parameters (intrinsic parameters), lens distortion parameters, and the relative position and attitude relationship (extrinsic parameters) between the two cameras for each camera. This invention uses a high-contrast checkerboard calibration board or a heat source calibration board. By acquiring multiple calibration board images at different distances and angles, the intrinsic parameters and distortion parameters of the two cameras are obtained respectively. After obtaining the intrinsic parameters of the two cameras, dual-target calibration is used to obtain the extrinsic parameters. By synchronizing and calibrating visible light and infrared cameras in time, the spatiotemporal consistency of visible light and infrared images is ensured, providing a reliable data foundation for subsequent feature alignment, deep fusion, and small target recognition.

[0081] S22: Based on the acquired intrinsic, extrinsic and distortion parameters, a combined geometric-radiometric correction strategy is used to geometrically align the visible light and infrared images and to perform sub-pixel level correction on residual parallax.

[0082] In a specific embodiment of the present invention, considering the significant differences between infrared and visible light in optical imaging (such as perspective differences and band differences), the present invention adopts a joint geometric-radiometric correction strategy. First, using the intrinsic parameters, distortion parameters, and extrinsic parameters obtained in step S21, the visible light image and the infrared image are geometrically aligned using a classic pinhole camera model. This aims to eliminate most of the macroscopic geometric deviations and project the two images onto a common imaging plane. To further eliminate sub-pixel-level residual parallax caused by nonlinear imaging differences, temperature drift, or mechanical stress, a lightweight learning-based alignment module is introduced into the feature layer. This module learns the minute local deformation fields between the visible light feature map and the infrared feature map through a differentiable spatial transformation network or convolutional layer, and performs sub-pixel-level nonlinear correction on the feature map. This adaptively compensates for subtle deviations that the physical model cannot fully correct, achieving sub-pixel-level correction of residual parallax.

[0083] S23: Based on aligned visible light and infrared images, a deep learning network is used to generate a multi-scale image feature pyramid.

[0084] In one specific embodiment of the present invention, the aligned visible light image and infrared image are respectively input into a deep learning network (e.g., a convolutional neural network with a feature pyramid network), and the output is a multi-scale visible light feature pyramid and an infrared feature pyramid that are accurately registered at both the geometric and feature levels, laying the foundation for subsequent multimodal fusion. Preferably, the scale is selected as 1 / 4, 1 / 8, and 1 / 16 of the original image downsampling layer. The 1 / 16 scale features have a larger receptive field, which is beneficial for detecting small targets at a distance, while the 1 / 4 scale features retain rich detail information, which can ensure the distinguishability and accurate localization of small targets. Thus, a balance is achieved between large-area perception and accurate localization of small targets, optimizing the allocation efficiency of computing resources.

[0085] Specifically, in step S3, visible light image features and infrared image features are extracted based on a multi-scale image feature pyramid to obtain visible light and infrared imaging feature data. After accurate image registration, this invention employs a feature pyramid network architecture to perform multi-scale feature extraction on both visible light and infrared images to obtain visible light image features. and infrared image features (s represents the pyramid scale), and channel alignment is achieved through 1×1 convolution, aiming to construct a feature hierarchy that can simultaneously represent rich detailed information and high-level semantic information.

[0086] Specifically, step S4, based on a cross-modal attention mechanism, performs multimodal data fusion of visible light and infrared imaging feature data, and adaptively adjusts the fusion weights based on environmental sensor data to generate fused features, further including: S41: Based on cross-modal attention mechanism, using visible light image features Construct a query vector Q using infrared image features Construct the key vector K and value vector V respectively, calculate the first attention weight Attn1, and calculate the infrared feature-enhanced visible light feature based on the first attention weight Attn1. ,in, (1) (2) (3) (4) (5) Where s is the pyramid scale. , , are linear projection matrices of the query vector Q, key vector K, and value vector V, respectively, used to map the features to the Q, K, and V spaces, respectively, and d is the dimension of the internal vector of the first attention.

[0087] In a specific embodiment of the present invention, to fully utilize the stable representation capability of infrared images for target contours under adverse environments (such as foggy weather or low light) and to enhance the detail information of visible light images, a cross-modal attention mechanism is designed. This mechanism uses visible light features as guidance and infrared features to enhance the visible light image. Specifically, it obtains channel-aligned visible light image features... Features of infrared images Then, it is first projected onto the query, key, and value spaces respectively through linear transformation, using visible light image features. As the "questioner" in information retrieval. Through the weight matrix. right Perform a linear transformation to construct the query vector Q. Use infrared image features... As the "respondent" providing information, weight matrices are used respectively. and right A linear transformation is performed to construct key vector K and value vector V, aiming to map features from different modalities to a common, comparable feature subspace. Then, the correlation between query vector Q and key vector K is calculated based on an attention mechanism to obtain the first attention weight Attn1. After obtaining the first attention weight Attn1, it is used as the coefficient for a weighted summation, based on infrared image features. Complementary information is extracted from the value vector V and compared with the original visible light image features. The components are fused to generate enhanced visible light features. "Attn1 V can be interpreted as: retrieving the most relevant and complementary information from infrared features based on the "question" posed by visible light features. For example, in a visible light image, the query vector of a target edge region blurred by fog will receive high attention weights from the key vector of the corresponding contour region in the infrared image, thereby extracting clear contour information from the infrared value vector to enhance that region. This is achieved by comparing the query vector with the original visible light image features. The purpose of fusion is to preserve the rich texture details of the original visible light features, ensuring that the enhancement process is a complementary optimization rather than a complete replacement.

[0088] S42: Based on infrared image features Construct a query vector Q′ using visible light image features Construct the key vector K′ and value vector V′ respectively, calculate the second attention weight Attn2, and calculate the infrared feature enhanced by visible light feature based on the second attention weight Attn2. ,in, (6) (7) (8) (9) (10) in, , , These are the query vector Q' and the key vector, respectively. Sum value vector The linear projection matrix, where d′ is the dimension of the second attention internal vector.

[0089] In a specific embodiment of the present invention, in order to achieve bidirectional complementary enhancement of visible light and infrared, after constructing a first attention weight with visible light features as the query, the present invention symmetrically designs a second attention weight with infrared features as the query. This aims to utilize the rich texture and detail information of visible light images to compensate for the deficiencies in resolution and detail representation of infrared images, thereby generating infrared features with enhanced detail. Specifically, using infrared image features... As the "questioner" in information retrieval, it uses a learnable weight matrix. Perform a linear transformation to construct the query vector Q′, using visible light image features. As information providers, they respectively use learnable weight matrices. and A linear transformation is performed to construct key vector K′ and value vector V′, aiming to find supplementary details in visible light features based on their own feature representation needs. Then, the correlation between query vector Q′ and key vector K′ is calculated based on an attention mechanism to obtain a second attention weight Attn2. After obtaining the second attention weight Attn2, it is used as the coefficient of the weighted summation from visible light image features. Complementary information is extracted from the value vector V′ and compared with the original infrared image features. The infrared signature is fused to generate an enhanced infrared signature. "Attn2 V′" can be interpreted as: based on the "requirements" of infrared features, extracting the most relevant texture and detail information from the value vector of visible light features to enhance the materiality and surface pattern details of targets in infrared images. This is achieved by comparing the infrared features with those of the original infrared image. The purpose of fusion is to preserve the core outline and salient information of infrared features that are sensitive to thermal radiation, while introducing visible light details.

[0090] Combining steps S41 and S42, the infrared-enhanced visible light features are calculated by introducing a first attention weight Attn1. This study leverages the stability of infrared light to enhance the robustness of visible light in harsh environments. It calculates enhanced infrared features based on visible light characteristics by introducing a second attention weight, Attn2. By leveraging the details of visible light to enhance the characterization quality of infrared images, and combining the two, a two-way complementary depth and feature fusion of visible and infrared light is achieved. The features are then enhanced through bidirectional attention. and Each incorporates complementary advantages rather than simply adding up information.

[0091] S43: Calculate the gating weight α based on environmental sensor data, and apply the gating weight α to the enhanced visible light features. infrared features Weighted fusion is performed to obtain fusion features. The gating weight α is adaptively adjusted based on environmental sensor data.

[0092] (11) (12) in, For illuminance, For visibility, Image contrast index For fog probability estimation, σ represents Sigmoid activation, and w is the weight vector. This is a bias term.

[0093] In a specific embodiment of the present invention, after obtaining the enhanced visible light features... and infrared features Subsequently, this invention introduces an environment-adaptive gating weight α, which is jointly determined by sensor and image statistics. The aim is to dynamically fuse bimodal information based on real-time environmental conditions to achieve adaptive adjustment according to environmental changes. Preferably, the gating threshold is smoothly adjusted with visibility and illuminance within the range α ∈ [0.35, 0.75]. Specifically, the illuminance... ,visibility Image contrast index Fog probability estimation As a key factor in constructing the gating weight α, it includes not only environmental sensor data from the hardware but also soft statistics of the image itself to ensure the comprehensiveness and robustness of the decision. Then, the constructed gating weight α is applied to enhanced visible light features. infrared features Weighted fusion is performed to obtain the final fusion features. By constructing a gating weight α, adaptive weight allocation can be achieved under different visibility and illumination conditions, maintaining stable feature separability and contrast in hazy and nighttime scenes. For example, when fog or low light reduces visibility... When the gating weight decreases, Monotonic increases will cause the system to automatically increase the weight of the infrared channel; under normal lighting conditions, the gating weight will be adjusted accordingly. Reduced size while preserving visible light texture details.

[0094] In a specific embodiment of the present invention, in addition to using the attention mechanism to achieve feature-level fusion, multimodal fusion can also achieve similar fusion and control effects by concatenating and weighting convolutional features, modeling modal relationships based on graph convolution, or dynamically determining the weights of each modality through modal confidence prediction.

[0095] Specifically, step S5 performs spatiotemporal enhancement processing on the fused features to obtain enhanced fused features, which further include: S51: Small target motion trajectory is extracted using continuous frame difference and optical flow estimation to extract fused features.

[0096] S52: Static background suppression is performed on the motion trajectory of small targets based on infrared thermal features to obtain enhanced fusion features.

[0097] In one specific embodiment of the present invention, the temporal continuity of a video sequence is utilized, and continuous frame difference and optical flow estimation are employed to extract the motion trajectory features of small targets from the dynamic changes of consecutive video frames. This compensates for the deficiency that targets in single-frame images are easily overlooked due to their small size and weak features. After extracting the motion trajectory of the small target, infrared thermal features are introduced as prior knowledge to further suppress static background interference, thereby purifying and enhancing the motion trajectory of the small target.

[0098] Furthermore, step S51, which uses continuous frame difference and optical flow estimation to extract fused features of the small target motion trajectory, also includes: S511: Use an optical flow estimation network to obtain the current video frame t and adjacent frames t. Displacement field between 1 .

[0099] S512: Uses continuous frame difference to separate adjacent frames t 1 fusion features The model is then transformed in reverse to the current frame coordinate system, and the fusion features with the current video frame t are calculated. difference residuals The calculation formula is: (13) In the formula, This is a sampling function based on bilinear interpolation.

[0100] In a specific embodiment of the present invention, given the fusion features of the current video frame t... and the previous video frame t 1 fusion features First, a lightweight optical flow estimation network is used to calculate the displacement field between two frames. This displacement field precisely describes the displacement of each feature pixel from t The motion vectors from frame 1 to frame t are then used to calculate the displacement field. fusion features of the previous frame The features are then transformed in reverse to the coordinate system of the current frame t to obtain the aligned features. Then calculate the difference residuals according to formula (13). The calculated difference residuals It effectively captures changes in the scene caused by motion. For static backgrounds, their features are basically consistent with the features of the current frame after optical flow alignment, and the difference value approaches zero; while for small moving targets, even if their displacement is small, they will produce a significant differential response and thus be highlighted.

[0101] Furthermore, step S52, which performs static background suppression on the small target's trajectory based on infrared thermal features to obtain enhanced fusion features, also includes: S521: Acquiring thermal saliency maps based on infrared images Constructing motion attention The calculation formula is as follows: (14) S522: Focus on motion Fusion features with the current video frame t Adding them together yields enhanced fusion features. The calculation formula is as follows: (15) In the formula, η is the scaling factor for motion attention.

[0102] In a specific embodiment of the present invention, after extracting the differential residual containing motion information, infrared thermal features are introduced as prior knowledge to further suppress static background interference and purify and enhance the motion trajectory of small targets. First, a thermal saliency map is generated based on the infrared image of the current frame. (Obtained by normalizing the temperature difference in the image), and then the difference residuals are... The "potentiality of motion" and the thermosaliency map are represented By combining the represented "target probability", a motion attention mechanism can be constructed. This is used to enhance the thermal target area and suppress background thermal noise. Thermal saliency map. It can highlight all objects in a scene that emit heat (including moving targets and static heat sources), ultimately generating motion attention. As an adaptive weight, it is superimposed on the fused features of the original current video frame t. The spatiotemporally enhanced fusion features were obtained above. By combining "motion trajectory extraction" with "thermal feature verification", the system can track the faint motion trajectory of small targets, reduce inter-frame missed detections and flickering. At the same time, by utilizing the physical properties of infrared thermal features, it can effectively distinguish real moving targets from background disturbances, and improve the temporal consistency and detection stability of small targets under complex interference and camera shake conditions. In a specific embodiment of the present invention, in addition to using optical flow and inter-frame difference, spatiotemporal enhancement can also use recurrent neural networks or long short-term memory networks to perform temporal modeling of the sequence, or directly extract the spatiotemporal features of the video through three-dimensional convolution or temporal convolutional networks. At the same time, background modeling and subtraction methods can be used to suppress interference.

[0103] Specifically, step S6, which performs target tracking and detection on small targets based on enhanced fusion features and outputs the location and category information of the small targets, further includes: S61: Construct a small target enhancement network to enhance and optimize the features of small targets.

[0104] S62: Track and detect the optimized small targets and output the position and category information of the small targets.

[0105] In a specific embodiment of the present invention, a small target enhancement network is constructed to enhance the features of small targets, thereby solving the problem of weak features and easy missed detection of small targets. After feature enhancement, tracking technology is used to complete the final identification and localization, ensuring the accuracy and stability of the output results.

[0106] Furthermore, step S61, which involves constructing a small target enhancement network to enhance and optimize the features of small targets, also includes: S611: Generate candidate regions at a high-resolution scale of fused features.

[0107] S612: Perform differentiable affine sampling on the features within the candidate region to construct a small target enhancement network.

[0108] S613: Based on the small target enhancement network, small targets are contrast-enhanced and super-resolution reconstructed to generate high-resolution features.

[0109] In one specific embodiment of the present invention, candidate regions are first generated at a high-resolution scale of fused features by an anchorless candidate head. These candidate regions are primarily optimized for small- to medium-scale targets. For each candidate region, corresponding feature blocks are precisely cropped using differentiable affine sampling, and these feature blocks are input into a small target enhancement network constructed within the candidate region. This enhancement network contains multi-scale residual dense blocks to perform a ×2 operation. Super-resolution reconstruction with a magnification of ×4 is mainly based on ×2, but can be increased to ×4 in extremely small target scenes. By performing super-resolution reconstruction of low-resolution feature blocks by 2x or 4x, the detailed information lost due to downsampling can be recovered.

[0110] Simultaneously, an adaptive histogram layer is inserted to achieve learnable contrast stretching. By introducing a learnable contrast stretching layer, the contrast within the feature block is adaptively enhanced, making the distinction between small targets and the background more obvious. Finally, the reconstructed refined features are written back to the main detection branch, achieving a closed loop of "enlarge first, then discriminate". By writing back the enhanced high-resolution, high-contrast features to the main detection network branch, subsequent detectors can perform more accurate class judgments and bounding box fine-tuning based on these optimized refined features. The small target enhancement network can improve the discernible details and contrast of small targets without increasing the overall input resolution, significantly improving the recall and precision of small targets, while controlling computational overhead.

[0111] Furthermore, step S612, which involves differentiable affine sampling of features within the candidate region to construct the small target enhancement network, also includes: Training and inference of the small target augmentation network specifically includes: During the training phase, registered visible light and infrared image pairs are used as sample inputs, and synthetic fog, low-light conditions, and thermal noise are added to perform thresholded random augmentation on the small target enhancement network. Its total loss function is... Represented as: (16) in, (17) (18) (19) in, , , , These are the weighting coefficients for each loss term. For scale-aware classification loss, The IoU / CIoU localization loss is a scale-weighted method. For time-series consistency loss, For super-resolution network loss terms, For gated monotonicity regularization, Let i be the area of ​​target i. This is the lower bound of the target area in the dataset. β is the focus factor, and β is the weighting index. For the first The predicted probability of a sample, where N is the number of samples. For area weighting, The fusion features of the current video frame t, For adjacent frames t The fusion characteristics of 1 U is a sampling function based on bilinear interpolation, and U is the displacement field matrix.

[0112] During the inference phase, a full-image coarse inspection and fusion is performed based on the input visible light and infrared image samples. Then, the candidate regions are refined by super-resolution and subjected to secondary discrimination, outputting stable motion trajectories and confidence scores for small targets.

[0113] In a specific embodiment of the present invention, during the training phase, registered visible light and infrared paired samples are used as input. To simulate various harsh conditions in the real world and prevent the model from overfitting to ideal laboratory data, an enhanced domain randomization augmentation strategy is introduced during training, supplemented by threshold randomization augmentations such as synthetic fog, low illumination, and thermal noise to ensure distribution coverage. This allows the small target augmentation network to be trained from low-quality input to extract detailed features beneficial for small target recognition and to adapt to various complex real-world environments. To ensure that training focuses on small targets, a scale-aware classification loss is introduced into the total loss function. The localization loss employs an IoU / CIoU coupling with scale weights to reduce the dominance of small bounding box bias on the overall loss. Used to constrain the smoothness and consistency of features between adjacent frames, suppressing jitter. A self-supervised degradation reconstruction strategy is employed when high-resolution ground truth is lacking. During the inference phase, for the input visible light and infrared images, a multimodal fusion and spatiotemporal enhancement process is first run to obtain a fused feature map. Then, rapid preliminary target detection (coarse detection) is performed at a higher feature level to efficiently locate candidate regions that may contain targets. Subsequently, high-value candidate regions are selected from the coarse detection results. Only these selected candidate regions are activated with a small target enhancement network to perform super-resolution reconstruction and contrast enhancement on the features within the candidate regions, restoring their detailed information. The refined high-resolution features are then fed back to a lightweight classifier and regressor for secondary discrimination. Finally, the multi-target tracking head outputs a stable trajectory and small target confidence score. During inference, the system can adaptively adjust modal weights based on sensor signals and image statistics to achieve robust adaptation to foggy and low-light conditions.

[0114] In one specific embodiment of the present invention, when the system is deployed at the edge, it prioritizes coarse detection at a high-level scale, triggering the small target enhancement network and secondary discrimination only for high-confidence small targets. This concentrates computation on a small number of candidate regions. The number of heads and channel width of the cross-modal attention are adaptively pruned according to the platform's computing power. The optical flow network employs subthreshold early stopping and pyramid cascading to reduce the cost of large displacements. Thus, while ensuring recognition performance, real-time performance is met, achieving end-to-end performance of 25–40 FPS on common edge GPUs / ASICs.

[0115] In a specific embodiment of the present invention, in addition to relying on super-resolution and contrast enhancement, small target enhancement can also use generative adversarial networks to compensate for small target details, utilize multi-branch detectors to specifically optimize small target branches, or enhance the detection performance of small targets through knowledge distillation and scale-weighted loss functions.

[0116] Specifically, the methods also include: S7: Adaptively switches scene recognition modes based on environmental sensor data.

[0117] Furthermore, step S7, which adaptively switches the scene recognition mode based on environmental sensor data, also includes: S71: Estimate the current scene state S based on environmental sensor data and image statistics, where S∈{normal daytime, foggy daytime, nighttime, extreme}; S72: Based on the current scene state S, construct a state machine using a hysteresis threshold mechanism and adjust the gating weights. Apply a monotonically consistent regularization constraint, where the monotonically consistent regularization constraint... Represented as:

[0118] In the formula, Visibility.

[0119] In one specific embodiment of the present invention, the system jointly estimates the scene state from sensor and image statistics. The state machine uses a hysteresis threshold to avoid frequent jitter and applies gating weights. A monotonic consistency regularization is applied to constrain the physical consistency that "infrared weights do not decrease even when visibility decreases." This enables interpretable and stable policy switching, resulting in robust and flicker-free outputs during cross-scene inference.

[0120] Example 2 This invention also provides a small target recognition system for multimodal fusion images in complex environments, such as... Figures 12-20 As shown, it includes: The data acquisition module 01 is used to acquire visible light images, infrared images, and environmental sensor data, including ambient illuminance and visibility.

[0121] Image registration module 02 is used to register visible light images and infrared images, and to construct a multi-scale image feature pyramid based on the registered images.

[0122] Feature extraction module 03 is used to extract visible light image features and infrared image features based on a multi-scale image feature pyramid, respectively, to obtain visible light and infrared imaging feature data.

[0123] Feature fusion module 04 is used to perform multimodal data fusion of visible light and infrared imaging feature data based on cross-modal attention mechanism, and adaptively adjust the fusion weights based on environmental sensor data to generate fused features.

[0124] The spatiotemporal enhancement module 05 is used to perform spatiotemporal enhancement processing on the fused features to obtain enhanced fused features.

[0125] The target detection module 06 is used to perform target tracking and detection on small targets based on enhanced fusion features, and output the position and category information of the small targets.

[0126] The adaptive scene switching module 07 is used to adaptively switch the scene recognition mode based on environmental sensor data.

[0127] Specifically, the present invention acquires visible light images, infrared images, and environmental sensor data simultaneously through the data acquisition module 01, wherein the environmental sensor data includes ambient illuminance and visibility.

[0128] Specifically, the image registration module 02 also includes: The calibration and parameter acquisition module 021 is used to perform time synchronization and calibration of visible light and infrared cameras, and to acquire the intrinsic, extrinsic and distortion parameters of the visible light and infrared cameras.

[0129] The correction module 022 is used to perform geometric alignment of visible light images and infrared images based on the acquired intrinsic parameters, extrinsic parameters and distortion parameters, and to perform sub-pixel level correction of residual parallax.

[0130] Feature pyramid generation module 023 is used to generate multi-scale image feature pyramids based on aligned visible light and infrared images using a deep learning network.

[0131] In a specific embodiment of the present invention, the visible light and infrared cameras are first synchronized and calibrated using a calibration and parameter acquisition module 021. A unified hardware trigger signal drives the synchronization, ensuring temporal and spatial consistency between the visible light and infrared image data, thereby obtaining a strictly matched visible light-infrared image pair. Simultaneously, the internal parameters (intrinsic parameters), lens distortion parameters, and relative position and attitude relationship (extrinsic parameters) of each camera are accurately acquired. Then, the visible light and infrared images are geometrically aligned using a correction module 022 to eliminate most macroscopic geometric deviations and project the two images onto a common imaging plane. To further eliminate subpixel-level residual parallax caused by nonlinear imaging differences, temperature drift, or mechanical stress, a lightweight learning-based alignment module is introduced. Through a differentiable spatial transformation network or convolutional layer, it learns the minute local deformation fields between the visible light and infrared feature maps and performs subpixel-level nonlinear correction on the feature maps. This adaptively compensates for subtle deviations that the physical model cannot fully correct, achieving subpixel-level correction of residual parallax. Finally, the aligned visible light image and infrared image are input into a deep learning network (e.g., a convolutional neural network with a feature pyramid network) via the feature pyramid generation module 023. The output is a multi-scale visible light feature pyramid and an infrared feature pyramid that are accurately registered at both the geometric and feature levels, laying the foundation for subsequent multimodal fusion. Preferably, the scale is selected as 1 / 4, 1 / 8, and 1 / 16 of the original image downsampling layer.

[0132] Specifically, feature extraction module 03 performs multi-scale feature extraction on visible light and infrared images respectively to obtain visible light image features. and infrared image features (s represents the pyramid scale), and channel alignment is achieved through 1×1 convolution, aiming to construct a feature hierarchy that can simultaneously represent rich detailed information and high-level semantic information.

[0133] Specifically, feature fusion module 04 also includes: The visible light feature calculation module 041 is used to construct a query vector with visible light image features and construct a key vector and a value vector with infrared image features based on the cross-modal attention mechanism, calculate the first attention weight, and calculate the infrared feature-enhanced visible light features based on the first attention weight.

[0134] The infrared feature calculation module 042 is used to construct a query vector with infrared image features, construct a key vector and a value vector with visible light image features respectively, calculate the second attention weight, and calculate the infrared features enhanced by visible light features based on the second attention weight.

[0135] The fusion feature calculation module 043 is used to calculate the gating weights based on environmental sensor data, and to perform weighted fusion of the enhanced visible light features and infrared features according to the gating weights to obtain the fused features.

[0136] In a specific embodiment of the present invention, a first attention weight is first calculated by the visible light feature calculation module 041, and the visible light feature enhanced by infrared features is calculated based on the first attention weight. Then, a second attention weight is calculated by the infrared feature calculation module 042, and the infrared feature enhanced by visible light features is calculated based on the second attention weight. Finally, a gating weight is calculated by the fusion feature calculation module 043, and the enhanced visible light feature and infrared feature are weighted and fused according to the gating weight to obtain the fused feature. By introducing the first attention weight to calculate the visible light feature enhanced by infrared features, the stability of infrared is used to enhance the robustness of visible light in harsh environments. By introducing the second attention weight to calculate the infrared feature enhanced by visible light features, the detail of visible light is used to improve the representation quality of infrared images. Combining the two achieves deep bidirectional complementarity and feature fusion of visible light and infrared. The visible light feature and infrared feature after bidirectional attention enhancement incorporate complementary advantageous information, rather than simple information superposition. By introducing an environment-adaptive gating weight α, the dual-modal information is dynamically fused according to real-time environmental conditions to achieve adaptive adjustment according to environmental changes.

[0137] Specifically, the spatiotemporal enhancement module 05 also includes: The trajectory feature extraction module 051 is used to extract the motion trajectory of small targets with fused features by employing continuous frame difference and optical flow estimation. The fusion feature enhancement module 052 is used to suppress the static background of the motion trajectory of small targets based on infrared thermal features, so as to obtain enhanced fusion features.

[0138] In a specific embodiment of the present invention, the trajectory feature extraction module 051 utilizes the temporal continuity of the video sequence and employs continuous frame difference and optical flow estimation to extract the motion trajectory features of small targets from the dynamic changes of continuous video frames, thereby compensating for the deficiency that targets are easily ignored in single-frame images due to their small size and weak features. The fusion feature enhancement module 052 introduces infrared thermal features as prior knowledge to further suppress static background interference, purify and enhance the motion trajectory of small targets.

[0139] Furthermore, the trajectory feature extraction module 051 also includes: The displacement field calculation module 0511 is used to obtain the displacement field between the current video frame and adjacent frames using an optical flow estimation network.

[0140] The differential residual calculation module 0512 is used to reverse the fusion features of adjacent frames to the coordinate system of the current frame by using continuous frame differential, and to calculate the differential residual of the fusion features with the current video frame.

[0141] In a specific embodiment of the present invention, the displacement field between the current video frame and the previous video frame is calculated by the displacement field calculation module 0511. This displacement field accurately describes the displacement field of each feature pixel from t... The motion vectors from frame 1 to frame t are then used in reverse transformation of the fused features of the previous frame to the coordinate system of the current frame using the differential residual calculation module 0512. This yields aligned features, and the differential residual between these features and the fused features of the current video frame is calculated. The calculated differential residual effectively captures changes in the scene caused by motion. For static backgrounds, their features are essentially consistent with the features of the current frame after optical flow alignment, and the difference value approaches zero. However, for small moving targets, even slight displacement will produce a significant differential response, thus highlighting them.

[0142] Furthermore, the fusion feature enhancement module 052 also includes: Motion attention construction module 0521 is used to construct motion attention based on thermal saliency maps obtained from infrared images; The fusion computing module 0522 is used to add the motion attention to the fusion features of the current video frame to obtain enhanced fusion features.

[0143] In a specific embodiment of the present invention, a thermal saliency map is first generated by the motion attention construction module 0521. (Obtained by normalizing the temperature difference in the image), and then the difference residuals are... The "potentiality of motion" and the thermosaliency map are represented By combining the represented "target probability", a motion attention mechanism can be constructed. This is used to enhance the thermal target area and suppress background thermal noise. Thermal saliency map. It can highlight all objects with thermal radiation in the scene (including moving targets and static heat sources), and then use the fusion computing module 0522 to generate motion attention. As an adaptive weight, it is superimposed on the original fusion features of the current video frame to obtain the spatiotemporally enhanced fusion features. By combining "motion trajectory extraction" with "thermal feature verification", the system can track the weak motion trajectory of small targets, reduce inter-frame missed detection and flickering. At the same time, by utilizing the physical characteristics of infrared thermal features, it can effectively distinguish real moving targets from background disturbances, and improve the temporal consistency and detection stability of small targets under complex interference and camera shake conditions. Specifically, the target detection module 06 also includes: The small target enhancement and optimization module 061 is used to construct a small target enhancement network to enhance and optimize the features of small targets.

[0144] The tracking, detection and output module 062 is used to track and detect the optimized small targets and output the position and category information of the small targets.

[0145] In a specific embodiment of the present invention, a small target enhancement network is constructed by the small target enhancement and optimization module 061 to enhance the features of small targets, thereby solving the problem of weak features and easy missed detection of small targets. After feature enhancement, the tracking detection and output module 062 is used to complete the final identification and positioning, ensuring the accuracy and stability of the output results.

[0146] Furthermore, the small goal enhancement and optimization module 061 also includes: Candidate region generation module 0611 is used to generate candidate regions at a high-resolution scale of fused features.

[0147] The small target enhancement network construction module 0612 is used to perform differentiable affine sampling on the features in the candidate region to construct the small target enhancement network.

[0148] The high-resolution feature generation module 0613 is used to perform contrast enhancement and super-resolution reconstruction on small targets based on the small target enhancement network to generate high-resolution features.

[0149] In a specific embodiment of the present invention, a candidate region generation module 0611 is first used to generate candidate regions at a high-resolution scale of the fused features. These candidate regions are mainly optimized for small and medium-scale targets. Then, a small target enhancement network construction module 0612 is used to precisely crop the corresponding feature blocks from each candidate region using differentiable affine sampling, and these feature blocks are input into a small target enhancement network constructed within the candidate regions. This enhancement network contains multi-scale residual dense blocks to perform ×2. After performing a 4x super-resolution reconstruction, a high-resolution feature generation module 0613 is used to reconstruct the low-resolution feature blocks at a 2x or 4x super-resolution, which can recover the details lost due to downsampling. Simultaneously, an adaptive histogram layer is inserted to achieve learnable contrast stretching. By introducing a learnable contrast stretching layer, the contrast within the feature blocks is adaptively enhanced, making the distinction between small targets and the background more obvious.

[0150] Specifically, the adaptive scene switching module 07 also includes: Scene state estimation module 071 is used to jointly estimate the current scene state based on environmental sensor data and image statistics; The state machine construction module 072 is used to construct a state machine based on the current scene state using a hysteresis threshold mechanism, and to apply monotonically consistent regular constraints to the gating weights.

[0151] In a specific embodiment of the present invention, the scene state estimation module 071 first estimates the scene state jointly from sensor and image statistics, and then the state machine construction module 072 constructs the state machine and applies monotonically consistent regularization constraints to the gating weights to constrain the physical consistency of "reduced visibility, unchanged infrared weights". This enables interpretable and stable policy switching, and the output is robust and flicker-free during cross-scene inference.

[0152] In summary, the method and system for small target recognition in multimodal fusion images under complex environments described in this invention have the following advantages: 1. This invention introduces a small target enhancement network, which enhances the feature representation of low-resolution small targets through contrast enhancement and super-resolution reconstruction methods. Simultaneously, an improved small target-specific loss function is introduced into the detection model, making the model pay more attention to small targets during training. Even with small target sizes, the system can significantly improve detection accuracy and recall.

[0153] 2. This invention proposes a deep fusion method for visible light and infrared images based on an attention mechanism, achieving deep fusion of visible light and infrared images at the feature level. During the fusion process, the attention mechanism and Transformer structure are combined to adaptively adjust modal weights under different environmental conditions (such as fog, nighttime, and normal lighting), highlighting small target features, suppressing redundant information, and achieving robust recognition across different scenes. The system can fully utilize modal complementarity to maintain stable recognition performance under different environmental conditions.

[0154] 3. This invention proposes a spatiotemporally enhanced small target detection method, which extracts the dynamic trajectory features of small targets through continuous frame difference and optical flow estimation, and combines infrared thermal features to suppress background interference, thereby enhancing the temporal consistency and motion saliency of small targets. This method can improve detection stability and robustness in complex backgrounds and multi-interference environments.

[0155] 4. This invention proposes a feature enhancement and optimization training mechanism for small targets. It designs a small target enhancement network, uses regional super-resolution and contrast enhancement to enhance the features of small targets, and combines a special scale-weighted loss function to guide the detector to focus more on small targets during training, thereby improving detection accuracy and recall.

[0156] 5. This invention incorporates an adaptive scene switching mechanism. The system integrates illumination and visibility sensors, dynamically switching recognition modes based on environmental conditions: visible light is prioritized in normal daylight conditions, infrared imaging is enhanced in foggy weather, thermal imaging is relied upon at night, and a fusion strategy is employed under extreme conditions. The system can achieve mode switching and fusion optimization in diverse environments, thereby ensuring the real-time performance and stability of recognition.

[0157] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for small target recognition from multimodal fusion images in complex environments, characterized in that, include: Acquire visible light images, infrared images, and environmental sensor data, including ambient illuminance and visibility; The visible light image and the infrared image are registered, and a multi-scale image feature pyramid is constructed based on the registered image. Based on the multi-scale image feature pyramid, visible light image features and infrared image features are extracted respectively to obtain visible light and infrared imaging feature data; Multimodal data fusion of visible light and infrared imaging feature data is performed based on a cross-modal attention mechanism, and the fusion weights are adaptively adjusted based on the environmental sensor data to generate fused features; The fusion features are subjected to spatiotemporal enhancement processing to obtain enhanced fusion features; The system performs target tracking and detection on small targets based on enhanced fusion features, and outputs the location and category information of the small targets.

2. The method for small target recognition in multimodal fusion images under complex environments according to claim 1, characterized in that, The step of performing image registration on the visible light image and the infrared image, and constructing a multi-scale image feature pyramid based on the registered image, includes: The visible light and infrared cameras are time-synchronized and calibrated, and their intrinsic, extrinsic, and distortion parameters are obtained. Based on the acquired intrinsic, extrinsic and distortion parameters, a combined geometric-radiometric correction strategy is used to geometrically align visible light and infrared images and to perform subpixel-level correction on residual parallax. Based on aligned visible light and infrared images, a deep learning network is used to generate a multi-scale image feature pyramid.

3. The method for small target recognition in multimodal fusion images under complex environments according to claim 1, characterized in that, The process of fusing visible light and infrared imaging feature data based on a cross-modal attention mechanism, and adaptively adjusting the fusion weights based on the environmental sensor data to generate fused features, further includes: Based on cross-modal attention mechanism, using visible light image features Construct a query vector Q using infrared image features Construct the key vector K and value vector V respectively, calculate the first attention weight Attn1, and calculate the infrared feature-enhanced visible light feature based on the first attention weight Attn1. ,in, Where s is the pyramid scale. , , , respectively, are the linear projection matrices of the query vector Q, key vector K, and value vector V, and d is the dimension of the internal vectors of the first attention; Infrared image features Construct a query vector Q′ using visible light image features Construct the key vector K′ and value vector V′ respectively, calculate the second attention weight Attn2, and calculate the infrared feature enhanced by visible light feature based on the second attention weight Attn2. ,in, in, , , These are the query vector Q' and the key vector, respectively. Sum value vector The linear projection matrix, where d′ is the dimension of the internal vector of the second attention; The gating weight α is calculated based on environmental sensor data, and the enhanced visible light features are then adjusted according to the gating weight α. infrared features Weighted fusion is performed to obtain fusion features. ,in, ，α∈[0,1] in, For illuminance, For visibility, Image contrast index For fog probability estimation, σ represents Sigmoid activation, and w is the weight vector. This is a bias term.

4. The method for small target recognition in multimodal fusion images under complex environments according to claim 3, characterized in that, The gating weight α is adaptively adjusted based on environmental sensor data.

5. The method for small target recognition in multimodal fusion images under complex environments according to claim 1, characterized in that, The process of performing spatiotemporal enhancement processing on the fused features to obtain enhanced fused features further includes: Small target motion trajectories are extracted using continuous frame difference and optical flow estimation to extract fused features; Static background suppression is performed on the motion trajectory of small targets based on infrared thermal features, resulting in enhanced fusion features.

6. The method for small target recognition in multimodal fusion images under complex environments according to claim 5, characterized in that, The small target motion trajectory extracted using continuous frame difference and optical flow estimation further includes: An optical flow estimation network is used to obtain the current video frame t and its neighboring frames t. Displacement field between 1 ; Using consecutive frame difference to separate adjacent frames t 1 fusion features The model is then transformed in reverse to the current frame coordinate system, and the fusion features with the current video frame t are calculated. difference residuals The calculation formula is: In the formula, This is a sampling function based on bilinear interpolation.

7. The method for small target recognition in multimodal fusion images under complex environments according to claim 5, characterized in that, The enhanced fusion features obtained by static background suppression of small target motion trajectories based on infrared thermal features further include: Thermal saliency map obtained from infrared image Constructing motion attention The calculation formula is as follows: motion attention Fusion features with the current video frame t Adding them together yields enhanced fusion features. The calculation formula is as follows: In the formula, η is the scaling factor for motion attention.

8. The method for small target recognition in multimodal fusion images under complex environments according to claim 1, characterized in that, The method of performing target tracking and detection on small targets based on enhanced fusion features, and outputting the location and category information of the small targets, further includes: Construct a small target enhancement network to enhance and optimize the features of small targets; The optimized small targets are tracked and detected, and their location and category information are output.

9. The method for small target recognition in multimodal fusion images under complex environments according to claim 8, characterized in that, The construction of the small target enhancement network to enhance and optimize the features of small targets further includes: Generate candidate regions at a high-resolution scale of fused features; Differentiable affine sampling is performed on the features within the candidate region to construct a small target enhancement network; Small target enhancement networks are used to enhance the contrast and perform super-resolution reconstruction on small targets, generating high-resolution features.

10. The method for small target recognition in multimodal fusion images under complex environments according to claim 9, characterized in that, This also includes training and inference on small target augmentation networks, specifically including: During the training phase, registered visible light and infrared image pairs are used as sample inputs, and synthetic fog, low-light conditions, and thermal noise are added to perform thresholded random augmentation on the small target enhancement network. Its total loss function is... Represented as: in, in, , , , These are the weighting coefficients for each loss term. For scale-aware classification loss, The IoU / CIoU localization loss is a scale-weighted method. For time-series consistency loss, For super-resolution network loss terms, For gated monotonicity regularization, Let i be the area of ​​target i. This is the lower bound of the target area in the dataset. β is the focus factor, and β is the weighting index. For the first The predicted probability of a sample, where N is the number of samples. For area weighting, The fusion features of the current video frame t, For adjacent frames t The fusion characteristics of 1 Here, U is the sampling function based on bilinear interpolation, and U is the displacement field matrix. During the inference phase, a full-image coarse inspection and fusion is performed based on the input visible light and infrared image samples. Then, the candidate regions are refined by super-resolution and subjected to secondary discrimination, outputting stable motion trajectories and confidence scores for small targets.

11. The method for small target recognition in multimodal fusion images under complex environments according to claim 1, characterized in that, Also includes: The scene recognition mode is adaptively switched based on the environmental sensor data.

12. The method for small target recognition in multimodal fusion images under complex environments according to claim 11, characterized in that, The adaptive switching of scene recognition mode based on the environmental sensor data further includes: Based on the environmental sensor data and image statistics, the current scene state S is jointly estimated, where S∈{normal during the day, foggy day, night, extreme}; Based on the current scenario state S, a state machine is constructed using a hysteresis threshold mechanism, and the gating weights are adjusted. Apply a monotonically consistent regularization constraint, where the monotonically consistent regularization constraint... Represented as: In the formula, Visibility.

13. A small target recognition system based on multimodal fusion images in complex environments, characterized in that, include: The data acquisition module is used to acquire visible light images, infrared images, and environmental sensor data, including ambient illuminance and visibility. The image registration module is used to perform image registration between the visible light image and the infrared image, and to construct a multi-scale image feature pyramid based on the registered image; The feature extraction module is used to extract visible light image features and infrared image features based on a multi-scale image feature pyramid, respectively, to obtain visible light and infrared imaging feature data. The feature fusion module is used to perform multimodal data fusion of visible light and infrared imaging feature data based on a cross-modal attention mechanism, and adaptively adjust the fusion weights based on the environmental sensor data to generate fused features; The spatiotemporal enhancement module is used to perform spatiotemporal enhancement processing on the fused features to obtain enhanced fused features; The target detection module is used to perform target tracking and detection on small targets based on enhanced fusion features, and outputs the location and category information of the small targets.

14. The small target recognition system for multimodal fusion images in complex environments according to claim 13, characterized in that, The image registration module also includes: The calibration and parameter acquisition module is used to perform time synchronization and calibration of visible light and infrared cameras, and to acquire the intrinsic, extrinsic and distortion parameters of the visible light and infrared cameras. The correction module is used to perform geometric alignment of visible light and infrared images based on the acquired intrinsic, extrinsic and distortion parameters, and to perform subpixel-level correction of residual parallax. The feature pyramid generation module is used to generate multi-scale image feature pyramids based on aligned visible light and infrared images using a deep learning network.

15. The small target recognition system for multimodal fusion images in complex environments according to claim 13, characterized in that, The feature fusion module also includes: The visible light feature calculation module is used to construct a query vector with visible light image features and construct a key vector and a value vector with infrared image features based on the cross-modal attention mechanism, calculate the first attention weight, and calculate the infrared feature-enhanced visible light features based on the first attention weight. The infrared feature calculation module is used to construct a query vector with infrared image features, construct a key vector and a value vector with visible light image features respectively, calculate the second attention weight, and calculate the infrared features enhanced by visible light features based on the second attention weight. The fusion feature calculation module is used to calculate the gating weights based on environmental sensor data, and to perform weighted fusion of the enhanced visible light features and infrared features according to the gating weights to obtain the fused features.

16. The small target recognition system for multimodal fusion images in complex environments according to claim 13, characterized in that, The spatiotemporal enhancement module also includes: The trajectory feature extraction module is used to extract the fused features of small target motion trajectories using continuous frame difference and optical flow estimation. The fusion feature enhancement module is used to suppress the static background of the motion trajectory of small targets based on infrared thermal features, thereby obtaining enhanced fusion features.

17. The small target recognition system for multimodal fusion images in complex environments according to claim 16, characterized in that, The trajectory feature extraction module also includes: The displacement field calculation module is used to obtain the displacement field between the current video frame and adjacent frames using an optical flow estimation network. The differential residual calculation module is used to reverse-transform the fusion features of adjacent frames to the coordinate system of the current frame using continuous frame differential, and to calculate the differential residual of the fusion features with the current video frame.

18. The small target recognition system for multimodal fusion images in complex environments according to claim 16, characterized in that, The fusion feature enhancement module also includes: The motion attention construction module is used to construct motion attention based on the thermal saliency map obtained from infrared images; The fusion computing module is used to add motion attention to the fusion features of the current video frame to obtain enhanced fusion features.

19. The small target recognition system for multimodal fusion images in complex environments according to claim 16, characterized in that, The target detection module also includes: The small target enhancement and optimization module is used to construct a small target enhancement network to enhance and optimize the features of small targets. The tracking, detection, and output module is used to track and detect the optimized small targets and output the position and category information of the small targets.

20. The small target recognition system for multimodal fusion images in complex environments according to claim 19, characterized in that, The small target enhancement and optimization module also includes: The candidate region generation module is used to generate candidate regions at a high-resolution scale of fused features; The small target enhancement network construction module is used to perform differentiable affine sampling on the features in the candidate region to construct the small target enhancement network. The high-resolution feature generation module is used to perform contrast enhancement and super-resolution reconstruction on small targets based on the small target enhancement network, generating high-resolution features.

21. The small target recognition system for multimodal fusion images in complex environments according to claim 13, characterized in that, Also includes: An adaptive scene switching module is used to adaptively switch the scene recognition mode based on the environmental sensor data.

22. The small target recognition system for multimodal fusion images in complex environments according to claim 21, characterized in that, The adaptive scene switching module also includes: The scene state estimation module is used to jointly estimate the current scene state based on the environmental sensor data and image statistics; The state machine construction module is used to construct a state machine based on the current scenario state using a hysteresis threshold mechanism, and to apply monotonically consistent regular constraints to the gating weights.

Images