Intelligent camera dual-light image fusion method and dual-light integrated intelligent camera

Abstract

The invention relates to the technical field of intelligent vision and image fusion, and discloses an intelligent camera dual-light image fusion method and a dual-light integrated intelligent camera, and the method comprises the steps: collecting an original infrared image and an original visible light image; preprocessing the original infrared image and the original visible light image to obtain a double-path preprocessed image; adopting an improved optical flow algorithm to calculate an optical flow field for the two-way pre-processed image; generating an infrared band motion sensing feature set and a visible light band motion sensing feature set; generating a preliminary fusion image based on the infrared band motion sensing feature set and the visible light band motion sensing feature set; for the preliminary fusion image, obtaining an evaluation result including definition, motion consistency and detail richness evaluation indexes; dynamically adjusting a cross-modal attention weight and a gating value according to an evaluation result; and obtaining an optimized final fusion feature based on the dynamically adjusted cross-modal attention weight and the gating value. According to the invention, the fusion image has no dislocation when the target moves rapidly.

Description

Technical Field

[0001] This invention relates to the field of intelligent vision and image fusion technology, specifically to a dual-light image fusion method for intelligent cameras and a dual-light integrated intelligent camera. Background Technology

[0002] In the current mainstream smart camera market, single-modal cameras still dominate, meaning they are either equipped with only visible light sensors or only infrared sensors. There is a significant technological gap and application limitation in truly integrated camera products that achieve deep fusion of "infrared + visible light" dual modes, mainly in the following aspects:

[0003] The scene adaptability limitations of single-modal cameras:

[0004] Visible light cameras rely on ambient light. In low-light conditions such as nighttime, fog, rain, and smoke, or in complex weather conditions, the image signal-to-noise ratio drops sharply, and target details are severely lost. Infrared cameras, while able to capture target thermal radiation information and maintain target outline visibility in harsh environments, lack crucial details such as texture and color, failing to meet the needs of fine recognition (such as license plate recognition and facial recognition). When used alone, neither type of camera can achieve reliable visual perception in all weather conditions and all scenarios.

[0005] The limitations of existing dual-optical devices in "pseudo-fusion":

[0006] Most of the few devices on the market that claim to have "dual-light" technology use a simple combination of infrared and visible light sensors.

[0007] The lack of a unified synchronization mechanism in the hardware means that the timing error of dual sensor acquisition can reach the millisecond level, resulting in obvious misalignment between the two images in dynamic scenes.

[0008] The software achieves "fusion" by simply stitching together (such as splitting the screen left and right) or by weighting and overlaying pixels, without taking into account the difference in modal characteristics between infrared and visible light (the outer edge structure is emphasized by infrared, while the texture and color are emphasized by visible light). As a result, the fused images often have problems such as blurred edges and conflicting details.

[0009] The preprocessing stage lacks dedicated hardware acceleration for dual-modality processing, and issues such as blind pixels and non-uniformity in infrared images, as well as noise and color shift in visible light images, are not completely resolved, further deteriorating the fusion quality. Summary of the Invention

[0010] This invention provides a dual-light image fusion method for intelligent cameras and a dual-light integrated intelligent camera to solve the above-mentioned problems.

[0011] This invention is achieved through the following technical solution:

[0012] A dual-light image fusion method for intelligent cameras, characterized by comprising:

[0013] Raw infrared and raw visible light images are acquired simultaneously using infrared and visible light sensors.

[0014] The original infrared image and the original visible light image are subjected to dual-channel parallel preprocessing to obtain a dual-channel preprocessed image, which includes a first infrared image and a first visible light image obtained after preprocessing.

[0015] An improved optical flow algorithm with dual-modal feature weighting constraints and dynamic confidence-guided optimization is used to calculate the optical flow field of the dual-path preprocessed images. Based on the optical flow field After motion compensation is applied to the dual-path preprocessed images, multi-scale features of the motion-compensated dual-path preprocessed images are extracted using a multi-branch convolutional network. Optical flow information is then embedded into the features of each layer of the motion-compensated dual-path preprocessed images to generate an infrared motion sensing feature set. and the set of motion sensing features in the visible light band ,in, , represents the infrared feature with motion sensing in the i-th layer and the visible light feature with motion sensing in the i-th layer, respectively, and n represents the number of scale levels;

[0016] Based on the infrared band motion sensing feature set and the visible light band motion sensing feature set, the optical flow field is fused. Calculate cross-modal attention weights and The gating value is calculated by introducing an index of the intensity of optical flow field motion. Dynamically adjust the multi-scale feature fusion weights to generate the final fused features. And based on the final fusion features, a preliminary fused image is generated. ;

[0017] The preliminary fused image yields results including sharpness (C), motion consistency (M), and detail richness. The evaluation results of the evaluation metrics; and the dynamic adjustment of the cross-modal attention weights and gating values ​​based on the evaluation results. ;

[0018] The cross-modal attention weights and gating values ​​are based on dynamic adjustments. The optimized final fusion features are obtained, and the optimized preliminary fusion image is obtained; then the optimized preliminary fusion image is further optimized through a generative adversarial network (GAN) to generate the final fusion image.

[0019] As an optimization, performing dual-path parallel preprocessing on the original infrared image and the original visible light image to obtain a dual-path preprocessed image includes:

[0020] The first infrared image is obtained by sequentially performing blind pixel correction, non-uniformity correction, and sharpness enhancement processing on the original infrared image obtained through the infrared sensor.

[0021] The raw visible light image output by the visible light sensor is sequentially processed by detector calibration, bad pixel correction, autofocus, auto exposure, auto white balance, gamma correction, noise reduction and sharpening to obtain a YUV domain image, which is the first visible light image; at the same time, the pre-processed infrared image is received, and the dual-channel pre-processed images are uniformly processed by encoding and streaming, OSD overlay and forwarding.

[0022] As an optimization, an improved optical flow algorithm with dual-modal feature weighting constraints and dynamic confidence-guided optimization is used to calculate the optical flow field of the dual-path preprocessed images. The specific process is as follows:

[0023] Extract the infrared edge structure features of the first infrared image Visible light texture color features of the first visible light image ;

[0024] Infrared mode adaptation weights are calculated based on the signal-to-noise ratio of the first infrared image and the contrast of the first visible light image. Adaptation weights for visible light modes ,in, , ,in, This represents the signal-to-noise ratio of the first infrared image. Indicates the contrast of the first visible light image;

[0025] An initial optical flow field optimization model incorporating dual-modal feature weighted constraints is constructed and solved. The formula for the initial optical flow field is expressed as: ; Indicates the initial optical flow field; Indicates the first infrared image, These are pixel coordinates; This represents the first visible light image; , These represent the first infrared image and the first visible light image in pixels, respectively. The grayscale value at that location; ρ represents the components of the optical flow vector in the x and y directions; ρ represents the robust loss function. , For regularization parameters; The weighting coefficients represent the motion terms of optical flow. The infrared edge structure features of the first infrared image; The visible light texture and color features of the first visible light image; Infrared edge features after motion; The visible light texture color characteristics after motion; , ;

[0026] According to the initial optical flow field Calculate the dynamic confidence score C(x,y) for each pixel using the following formula: ;in, For the first infrared image And the first visible light image The standard deviation is obtained after statistical analysis of the grayscale differences between them; The closer the value is to 1, the more reliable the optical flow estimation for that pixel;

[0027] For dynamic confidence For low-confidence pixels, a 3×3 neighborhood optical flow weighted average is used for optimization.

[0028] The optimized optical flow vector components are substituted into the initial optical flow field to replace the original dynamic confidence level. The optical flow vectors of low-confidence pixels are used to obtain the final optical flow field. .

[0029] As an optimization, based on the optical flow field After motion compensation is applied to the dual-path preprocessed images, multi-scale features of the motion-compensated dual-path preprocessed images are extracted using a multi-branch convolutional network. Optical flow information is then embedded into the features of each layer of the motion-compensated dual-path preprocessed images to generate a motion-aware feature set. The specific process is as follows:

[0030] For the first infrared image According to the optical flow field The motion vectors (u(x,y), v(x,y)) of each pixel (x,y) are mapped to spatial positions aligned with the visible light image using a bilinear interpolation algorithm, resulting in the motion-compensated second infrared image. The second visible light image after performing reverse motion compensation on the first visible light image ;

[0031] The second infrared image is extracted using a structurally symmetrical dual-branch convolutional network. Second visible light image Multi-layer features are used to obtain a multi-scale feature set of infrared images. Multiscale feature sets of visible light images , where n is the number of scale layers;

[0032] For the optical flow field Downsampling and channel compression are performed to generate optical flow feature maps that match the features of each layer. , This represents the optical flow feature map corresponding to the i-th layer features;

[0033] The optical flow feature map is compared with the multi-scale feature set of the infrared image. Multiscale feature sets of visible light images By adding them together, we obtain the feature sets for infrared motion sensing and the feature sets for visible light motion sensing, where... , , This is the optical flow weighting coefficient.

[0034] As an optimization, cross-modal attention weights and The specific formula is:

[0035] ;

[0036] This represents the characteristics of infrared band motion sensing at the i-th scale. This represents the optical flow feature map corresponding to the i-th layer features; d represents the feature of visible light band motion perception at the i-th scale; d is the feature dimension.

[0037] ;

[0038] This represents the characteristics of visible light band motion perception at the i-th scale. This represents the characteristics of infrared band motion sensing at the i-th scale.

[0039] The threshold value The specific formula is:

[0040] ;

[0041] in, The Sigmoid activation function maps the input to the (0,1) interval, as shown in the formula. ; The weight matrix is ​​a learnable matrix. For learnable bias terms; This represents the characteristics of infrared band motion sensing at the i-th scale. This represents the characteristics of visible light band motion perception at the i-th scale. The i-th scale represents the intensity of optical flow motion, and GlobalAvgPool is a global average pooling operation.

[0042] As an optimization, the multi-scale feature fusion weights are dynamically adjusted to generate the final fused features. And based on the final fusion features, a preliminary fused image is generated. The specific process is as follows:

[0043] Infrared band motion sensing features at the i-th scale Visible light band motion sensing characteristics Combined with cross-modal attention weights and Feature fusion is performed to obtain intermediate fused features. , , This indicates element-wise multiplication;

[0044] For the intermediate fusion features The final fused features are obtained by dynamically adjusting the weights. : , Indicates the convolution operation;

[0045] Generated at various scales Where n is the number of scale levels, a feature pyramid fusion strategy is adopted, which upsamples the fusion features at different scales and adds them element by element to obtain multi-scale fusion features. : Where Upsample is the upsampling operation;

[0046] The multi-scale fused features are processed through convolutional layers and activation functions. Mapped to image space, generating a preliminary fused image. .

[0047] As an optimization, the preliminary fused image is used to obtain parameters including sharpness (C), motion consistency (M), and detail richness. The evaluation results of the evaluation metrics; and the dynamic adjustment of the cross-modal attention weights and gating values ​​based on the evaluation results. The specific process is as follows:

[0048] The Tenengrad gradient function is used to initially fuse the images. The gradient of each pixel is calculated, and the mean of all pixel gradients is then used to obtain the sharpness C. The structural similarity (SSIM) between the preliminary fused image and the second infrared image, and between the preliminary fused image and the second visible light image, is calculated separately, and then combined with the optical flow field... The motion vectors are weighted to obtain the motion consistency M; the detail richness of the preliminary fused image is calculated using the entropy function. ;

[0049] The aforementioned sharpness C, motion consistency M, and detail richness A weighted combination is performed to obtain a comprehensive evaluation score S, and an adjustment amount is constructed based on a preset optimal score range. ;

[0050] For the cross-modal attention weights and According to the adjustment amount Adjustments will be made:

[0051] ;

[0052] ;

[0053] in, For learning rate, , The comprehensive evaluation scores S are respectively , The gradient;

[0054] According to the adjustment amount For the gate value Adjustments will be made:

[0055] ;

[0056] in, To comprehensively evaluate the score S for the gate value The gradient.

[0057] As an optimization, the preliminary fused image is further refined using a Generative Adversarial Network (GAN) to generate the final fused image. The specific process is as follows:

[0058] A Generative Adversarial Network (GAN) is constructed, comprising a generator and a discriminator, wherein the loss function of the generator is: , To combat the losses, Where D(*) is the probability of authenticity output by the discriminator. For randomly generated fake images, The candidate fused image is generated by the generator from the initial fused image; the loss function of the discriminator is... , This is an ideal fused image that is manually labeled during the training phase.

[0059] This invention also discloses a dual-light integrated smart camera for executing the aforementioned dual-light image fusion method for a smart camera, comprising a dual-light image acquisition module, a preprocessing module, an optical flow calculation module, a feature fusion module, an image optimization module, and a control module integrated on the same component:

[0060] The dual-light image acquisition module is used to simultaneously acquire infrared and visible light images of the same scene and output the original infrared and visible light images.

[0061] The preprocessing module is connected to the dual-light image acquisition module and is used to preprocess the original infrared image and the original visible light image to output dual-channel preprocessed images.

[0062] The optical flow calculation module, connected to the preprocessing module, is used to calculate the optical flow field based on the dual-path preprocessed image using an optical flow algorithm that includes dual-modal feature weighting constraints and dynamic confidence optimization. ;

[0063] The feature fusion module is connected to both the preprocessing module and the optical flow calculation module, and is used to perform calculations based on the optical flow field. Motion compensation is performed on the dual-channel preprocessed images; multi-scale features of the motion-compensated dual-channel images are extracted using a multi-branch convolutional network, and the optical flow field is then processed. Motion information is embedded into features at each layer to generate an infrared motion-aware feature set and a visible light motion-aware feature set. Based on cross-modal attention weights and dynamic gating values, the motion-aware dual-modal feature sets are fused at multiple scales to output a preliminary fused image. ;

[0064] The image optimization module is connected to the feature fusion module and is used to process the initially fused image through a generative adversarial network. Perform detailed optimizations and output the final fused image;

[0065] The control module is connected to the dual-light image acquisition module, preprocessing module, optical flow calculation module, feature fusion module, and image optimization module respectively, and is used to coordinate the working timing of the dual-light image acquisition module, preprocessing module, optical flow calculation module, feature fusion module, and image optimization module to realize real-time acquisition, processing, fusion, and output of dual-light images.

[0066] As an optimization, the dual-light image acquisition module includes an infrared sensor and a visible light sensor. The preprocessing module includes an infrared image preprocessing unit and a visible light image preprocessing unit. The infrared image preprocessing unit is located in the FPGA and is used to receive the raw infrared image transmitted by the infrared sensor, preprocess the raw infrared image, and output a first infrared image. The visible light processing unit is located in the AI ​​chip and is used to receive the raw visible light image transmitted by the visible light sensor, preprocess the raw visible light image to obtain a first visible light image, and simultaneously receive the first infrared image. The first infrared image and the first visible light image are fused, optical flow field calculated, multi-scale feature extracted / embedded, and image optimized by the optical flow calculation module, feature fusion module, image optimization module, and control module located in the AI ​​chip to obtain the final fused image.

[0067] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0068] This invention overcomes the limitations of conventional dual-optical fusion through a synergistic technical solution of optical flow optimization, dynamic fusion, closed-loop adjustment, and GAN enhancement, achieving the following effects:

[0069] Significantly improved motion consistency in dynamic scenes: Conventional techniques are prone to ghosting due to inaccurate optical flow. In this invention, S3 uses dual-modal constraints and dynamic confidence to calculate accurate optical flow, S4 embeds optical flow into features and guides fusion through bidirectional attention, and S5 adjusts parameters according to motion consistency index, so that the fused image has no misalignment when the target is moving fast (such as traffic or running pedestrians), and the motion consistency is higher than that of conventional methods.

[0070] Dual preservation of details and modal advantages in complex environments: Traditional fusion often loses some features (such as infrared thermal targets or visible light textures). In this invention, S4 uses dynamic gating to balance dual-modal features, and S6 uses GAN to enhance details on the optimized preliminary image. Even in low-light and weak-texture scenes (such as nighttime buildings and vegetation on cloudy days), the fused image shows infrared thermal targets and clearly restores details such as wall textures and leaf veins, thus improving the richness of details.

[0071] Full-scene adaptive fusion without manual parameter tuning: Conventional techniques rely on fixed parameters, and the effect drops sharply when the scene changes (such as day and night switching, sudden changes in lighting). This invention S5 dynamically adjusts attention weights and gating values ​​through multi-index evaluation, and S6's GAN is further adapted and optimized to achieve adaptive fusion from low light to strong light and static to dynamic scenes without manual intervention, adapting to the all-time and all-scene usage needs of smart cameras. Attached Figure Description

[0072] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and constitute a part of this invention, are not intended to limit the scope of the invention. In the drawings:

[0073] Figure 1 This is a flowchart of a dual-light image fusion method for an intelligent camera according to the present invention;

[0074] Figure 2 This is a schematic diagram of the hardware connection of the dual-light integrated smart camera of the present invention;

[0075] Figure 3 This is an envelope diagram of the overall structure of the dual-light integrated smart camera of the present invention. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0077] This embodiment 1 provides a dual-light image fusion method for intelligent cameras, such as... Figure 1 As shown, it includes:

[0078] S1. Simultaneously acquire raw infrared images and raw visible light images using infrared sensors and visible light sensors.

[0079] In some embodiments, a synchronization mechanism combining timestamp synchronization and hardware triggering is adopted to allocate a unified time reference for the infrared sensor and the visible light sensor. The FPGA generates a hardware trigger signal to control the dual sensors to acquire images synchronously, and the acquisition time error is controlled at the microsecond level. At the same time, a high-speed preprocessing pipeline is constructed through ping-pong caching technology, so that the FPGA and AI chip can start preprocessing tasks in parallel during the image acquisition process.

[0080] Here, the same time standard, such as a high-precision clock signal, is assigned to both infrared and visible light sensors. The FPGA (programmable chip) generates electrical signal instructions to trigger both sensors to take pictures simultaneously, ensuring that both sensors acquire images at the same instant. The error is controlled at the microsecond level, which solves the problem of image misalignment in dynamic scenes. Ping-pong caching technology is used to perform acquisition and processing in parallel, avoiding the waiting time of storing data before processing, greatly improving efficiency, and ensuring a high-speed and smooth image processing chain.

[0081] S2. Perform dual-channel parallel preprocessing on the original infrared image and the original visible light image to obtain a dual-channel preprocessed image, wherein the dual-channel preprocessed image includes a first infrared image and a first visible light image obtained after preprocessing.

[0082] In some embodiments, the original infrared image obtained by the infrared sensor is sequentially subjected to blind pixel correction, non-uniformity correction and sharpness enhancement processing to obtain a first infrared image;

[0083] The raw visible light image output by the visible light sensor is sequentially processed by detector calibration, bad pixel correction, autofocus, auto exposure, auto white balance, gamma correction, noise reduction and sharpening to obtain a YUV domain image, which is the first visible light image; at the same time, the first infrared image is received, and the dual-channel preprocessed images are uniformly processed by encoding and streaming, OSD overlay and forwarding.

[0084] The FPGA executes a hardware-accelerated algorithm to sequentially perform blind pixel correction, non-uniformity correction, and sharpness enhancement on the raw infrared sensor data, solving the problems of blind pixels, non-uniformity, and poor sharpness in the raw data. After processing, the data is transmitted to the AI ​​chip through a data transmission interface (such as PCIe, MIPI, BT1120, etc., this invention uses the MIPI interface). The RK3588 AI chip, through its on-chip ISP unit, sequentially performs detector calibration, bad pixel correction, autofocus, auto exposure, auto white balance, gamma correction, noise reduction, and sharpening on the raw image output from the visible light sensor to obtain a YUV domain image. Simultaneously, it receives the infrared pre-processed image input through the data transmission interface (such as PCIe, MIPI, BT1120, etc., this invention uses the MIPI interface), and performs unified encoding, streaming, OSD overlay, and forwarding processing on both images.

[0085] Blind pixel correction: The 3×3 neighborhood mean replacement method is used to identify and replace blind pixels whose gray values ​​deviate from the neighborhood mean ±3σ.

[0086] Non-uniformity correction: Combining single-point correction and two-point correction (-20℃, 50℃ blackbody calibration) with scene adaptive correction (dynamically adjusting gain to compensate for temperature drift).

[0087] Sharpness enhancement: Image edge details are enhanced by sharpening the convolution kernel using Laplacian sharpening. Blind pixel correction and non-uniformity correction are existing technologies and will not be elaborated upon here.

[0088] Defect correction: Defects are identified by dark current detection and bright spot detection, and replaced by 4-neighborhood linear interpolation.

[0089] Automatic exposure: Based on histogram brightness analysis, the exposure time (1ms-100ms) and gain (1-32dB) are dynamically adjusted to maintain the average image brightness at a gray value of 128±10.

[0090] Encoding and streaming: H.265 encoding format is used, and the bitrate is dynamically adjusted between 1Mbps and 20Mbps.

[0091] S3. The optical flow field is calculated using an improved optical flow algorithm with dual-modal feature weighting constraints and dynamic confidence-guided optimization on the dual-path preprocessed images. Based on the optical flow field After motion compensation is applied to the dual-path preprocessed images, multi-scale features of the motion-compensated dual-path preprocessed images are extracted using a multi-branch convolutional network. Optical flow information is then embedded into the features of each layer of the motion-compensated dual-path preprocessed images to generate a motion-aware feature set. and ,in, , Let represent the infrared features and visible light features of the i-th layer with motion sensing, respectively, and n represent the number of scale levels.

[0092] In the process of multi-scale feature extraction, the image is downsampled to different degrees to obtain features at multiple scales (levels). n is the total number of these scale levels, which is used to describe the number of scale levels contained in the bimodal feature set (infrared band motion sensing feature set and visible light band motion sensing feature set).

[0093] In some embodiments, an improved optical flow algorithm with dual-modal feature weighting constraints and dynamic confidence-guided optimization is used to calculate the optical flow field of the dual-path preprocessed image. The specific process is as follows:

[0094] S3.1 Extract the infrared edge structure features of the first infrared image. Visible light texture color features of the first visible light image .

[0095] This step involves extracting core features from the infrared preprocessed image and the visible light preprocessed image, respectively.

[0096] Infrared feature extraction: Edge features are extracted from the infrared preprocessed image using edge detection operators (such as the Canny operator). This feature reflects the edge contour information of the target in the infrared image.

[0097] Visible light feature extraction: Texture features are extracted from the preprocessed visible light image using texture extraction algorithms (such as Local Binary Pattern (LBP)). This feature reflects the texture details in a visible light image.

[0098] S3.2 Calculate the infrared mode adaptation weights based on the signal-to-noise ratio of the first infrared image and the contrast of the first visible light image. Adaptation weights for visible light modes ,in, , ,in, This represents the signal-to-noise ratio of the first infrared image. This indicates the contrast of the first visible light image.

[0099] This is obtained by calculating the ratio of the first infrared image signal power to the noise power. These two weights are obtained by calculating the degree of difference in brightness in different regions of the visible light image. They are used to measure the relative importance of infrared features and visible light features in the optical flow field calculation.

[0100] S3.3 Construct and solve an initial optical flow field optimization model that includes weighted constraints based on dual-modal features. The formula for the initial optical flow field is as follows: ; Indicates the initial optical flow field; Indicates the first infrared image, These are pixel coordinates; This represents the first visible light image; , These represent the first infrared image and the first visible light image in pixels, respectively. The grayscale value at that location; These are the components of the optical flow vector in the x and y directions; This represents the robust loss function, used to reduce the impact of outliers on optical flow field calculations; , This is a regularization parameter used to constrain the smoothness of the optical flow field; This represents the constraint weight coefficient, used to adjust the degree of influence of the bimodal constraint term; The infrared edge structure features of the first infrared image; The visible light texture and color features of the first visible light image; The infrared edge features after motion (after displacement u, v); The image texture features after motion (after displacement u, v); , ;

[0101] This step involves constructing an initial optical flow field optimization model that includes weighted constraints based on bimodal features, with pixel grayscale differences, optical flow field smoothness, and bimodal feature differences as optimization objectives.

[0102] This part is based on the fundamental optical flow constraints of pixel grayscale, and uses a robust loss function. Minimize infrared image With optical flow displacement The subsequent visible light image The grayscale differences between them are used to capture pixel-level motion relationships.

[0103] This is a regularization term used to constrain the smoothness of the optical flow field. and It is a regularization parameter to prevent excessive oscillations in the optical flow field that do not conform to the actual motion laws. , Let represent the gradients of u and v, respectively. It represents the square of the L2 norm, used to calculate the "length" of the gradient vector (by summing the squares of each component of the gradient to measure the overall size of the gradient).

[0104] It is a bimodal constraint term. and These are the mode adaptation weights for infrared and visible light, which respectively constrain infrared edge features. Visible light texture features Its post-movement characteristics , The differences are addressed by utilizing dual-modal features (infrared edges and visible light textures) to improve the accuracy of optical flow calculation in dual-light scenarios.

[0105] S3.4, Based on the initial optical flow field Calculate the dynamic confidence score C(x,y) for each pixel using the following formula: ;in, For the first infrared image And the first visible light image The standard deviation is obtained after statistical analysis of the grayscale differences between them; The closer the value is to 1, the more reliable the optical flow estimation for that pixel.

[0106] S3.5, Regarding dynamic confidence levels For low-confidence pixels, a 3×3 neighborhood optical flow weighted mean is used for optimization, as shown in the formula:

[0107] , ;in, Let (x, y) be the 3×3 neighborhood of pixel (x, y). , For the optimized optical flow vector components, This represents the confidence threshold, which is typically set to 0.3.

[0108] S3.6 Substitute the optimized optical flow vector components into the initial optical flow field to replace the original dynamic confidence level. The optical flow vectors of low-confidence pixels are used to obtain the final optical flow field. .

[0109] In some embodiments, based on the optical flow field After motion compensation is applied to the dual-path preprocessed images, multi-scale features of the motion-compensated dual-path preprocessed images are extracted using a multi-branch convolutional network. Optical flow information is then embedded into the features of each layer of the motion-compensated dual-path preprocessed images to generate a motion-aware feature set. The specific process is as follows:

[0110] S3.7, regarding the first infrared image According to the optical flow field The motion vectors (u(x,y), v(x,y)) of each pixel (x,y) are mapped to spatial positions aligned with the visible light image using a bilinear interpolation algorithm, resulting in the motion-compensated second infrared image. : Where BilinearInterp is a bilinear interpolation function used to calculate pixel values ​​for non-integer coordinates. These are the coordinates of the first infrared image before motion compensation; for the first visible light image... Perform reverse motion compensation (correcting deviations caused by sensor displacement or scene motion) to obtain a second visible light image after motion compensation. This ensures that the two-light images are accurately aligned in the spatial coordinate system.

[0111] It should be noted that the first visible light image In the process of motor compensation, it is Similarly, the first infrared image In the process of motor compensation, it is .

[0112] S3.8. Employ a structurally symmetrical dual-branch convolutional network (the infrared branch and the visible light branch share the network architecture but have independent parameters) to extract multi-scale features from the motion-compensated dual-path images:

[0113] The second infrared image is extracted using a structurally symmetrical dual-branch convolutional network. Second visible light image Multi-layer features are used to obtain a multi-scale feature set of infrared images. Multiscale feature sets of visible light images , where n is the number of scale layers.

[0114] For example, infrared branch feature extraction:

[0115] The second infrared image Input the infrared branch, and extract features through hierarchical convolution operations:

[0116] Shallow features (layers 1-2): 3×3 convolutional kernel (stride 1, number of channels 64), capturing local thermal radiation gradients in infrared images (such as grayscale differences between the target and background), output features. ;

[0117] Mid-layer features (layers 3-4): 3×3 convolutional kernel (stride 2, number of channels 128), capturing the edge contours of the target (such as vehicle contours, human contours), outputting features. ;

[0118] Deep features (layers 5-6): 5×5 convolutional kernel (stride 2, number of channels 256), capturing semantic information of the target (such as the "dynamic / static target" attribute), outputting features. ;

[0119] Generate infrared multi-scale feature sets ;

[0120] Visible light branch feature extraction:

[0121] The second visible light image Input visible light branch (with the same structure as the infrared branch), extract features:

[0122] Shallow features (layers 1-2): capture texture details (such as clothing texture, road surface texture), output ;

[0123] Layer features (layers 3-4): Capture shape features (such as car window structure, limb shape), output ;

[0124] Deep features (layers 5-6): capture category information (such as "vehicle / pedestrian" semantics), output ;

[0125] Generate a multi-scale feature set of visible light .

[0126] S3.9 Scale adaptation and feature embedding of optical flow information, including scale transformation of optical flow features and fusion of motion sensing features.

[0127] Scale transformation of optical flow characteristics: for the optical flow field (Dimensions H×W×2) are downsampled and channel compressed to generate optical flow feature maps that match the features of each layer:

[0128] For the i-th layer features (size) ),Will downsampling to Compress the number of channels using a 1×1 convolution to... To obtain adaptation features , This represents the height of the i-th layer feature map, which is the number of pixels in the feature map in the vertical direction. This represents the width of the feature map at layer i, which is the number of pixels in the feature map in the horizontal direction. This represents the number of channels in the i-th layer feature map. The number of channels reflects the number of feature types or dimensions contained in the feature map. Different channels may correspond to different feature patterns or attributes. This represents the optical flow feature map corresponding to the i-th layer feature.

[0129] Motion sensing feature fusion:

[0130] Optical flow information is embedded into dual-path features through element-wise addition, preserving feature dimensions while injecting motion attributes:

[0131] Infrared motion sensing features: ( (This is the optical flow weighting coefficient, with a value ranging from 0.1 to 0.3).

[0132] Visible light band motion sensing characteristics: .

[0133] S3.10, Generate a feature set with motion awareness:

[0134] The final output consists of two sets of features:

[0135] Infrared motion sensing feature set: ;

[0136] The set of features for motion sensing in visible light bands: .

[0137] The aforementioned features simultaneously include the inherent properties of dual-light images (infrared thermal radiation / visible light texture) and motion state information reflected by the optical flow field (target motion direction and velocity), providing dual content-motion information support for subsequent cross-modal fusion.

[0138] In this step, the features of the image after motion, such as , It is extracted from the motion-compensated image (including the second infrared image and the second visible light image). This process forms a strict logical closed loop with motion compensation, and the specific relationship is as follows:

[0139] 1. Motion compensation provides a spatial alignment basis for feature extraction:

[0140] Original dual-channel preprocessed image ( Because the target's movement may cause spatial misalignment, for example, the same vehicle may be positioned to the left in an infrared image but to the right in a visible light image.

[0141] Through optical flow field After motion compensation, the results were obtained (Infrared image after motion compensation) and Spatial alignment was achieved in the motion-compensated visible light image, ensuring that the pixel positions of the same target were consistent in the two-light image.

[0142] 2. Features after motion are directly derived from the motion-compensated image.

[0143] Infrared edge features after motion From motion-compensated infrared images Extracted from the image, it reflects the motion-corrected infrared target edge (spatially aligned with the corresponding target edge in the visible light image).

[0144] Visible light texture features after motion From motion-compensated visible light images Extracted from the image, it reflects the motion-corrected visible light target texture (spatially aligned with the corresponding target texture in the infrared image).

[0145] Characteristics before exercise ( The images extracted from the uncompensated preprocessed images, namely the first infrared image and the first visible light image, may have a mismatch in bimodal features due to spatial misalignment. For example, the region corresponding to the infrared edge and the region corresponding to the visible light texture are not the same target.

[0146] Characteristics after exercise ( The image extracted from the motion-compensated image ensures that the bimodal features are spatially aligned with the same target, providing a reliable feature matching basis for subsequent bimodal feature weighting constraints (optical flow field optimization) and cross-modal fusion.

[0147] To make it easier to understand, let me briefly explain the order of S3:

[0148] Step 1: Calculate the initial optical flow field based on the original preprocessed image.

[0149] First, use the dual-channel preprocessed images (first infrared image) without motion compensation. First visible light image The initial optical flow field is calculated using an improved optical flow algorithm that incorporates bimodal constraints and dynamic confidence optimization. ;

[0150] The optical flow field at this point is a preliminary estimate of the pixel motion relationship between the two images, used to guide the first motion compensation.

[0151] Step 2: Perform the first motion compensation based on the initial optical flow field.

[0152] Using the initial optical flow field obtained in the first step Motion compensation is performed on the dual-channel preprocessed images to obtain the image after the first motion compensation. ).

[0153] The purpose of this step is to initially align the two images in space, reducing pixel misalignment caused by motion.

[0154] The third step is optional: iterative optimization (used to improve accuracy).

[0155] If high precision is required for the optical flow field, iteration can be performed:

[0156] Based on the image after the first motion compensation (which has been preliminarily aligned), the optical flow field is recalculated to obtain the optimized optical flow field. (Because the images are already aligned, the optical flow field calculation is less affected by motion misalignment and has higher accuracy.)

[0157] Then use the optimized optical flow field A second motion compensation is performed to obtain a more accurately aligned image. ).

[0158] In the optimization model of the initial optical flow field, (u,v) is a non-zero eigenvector derived from the original features and optical flow assumptions to constrain the optical flow field.

[0159] This step is one of the innovative aspects of this invention, and it has the following specific effects:

[0160] Traditional optical flow algorithms are susceptible to large modal differences (infrared thermal radiation vs. visible light texture) and weak texture regions (such as smooth infrared surfaces and low-contrast visible light regions) in dual-light scenarios, leading to incorrect optical flow estimation.

[0161] This invention introduces a dual-modal feature weighted constraint: combining infrared edge structure features and visible light texture color features, and utilizing the complementarity of the two modes (infrared excels at capturing the edges of hot targets, while visible light excels at providing texture details), the optical flow field can be more accurately constrained in both edge and texture regions. For example, the hot edges of a vehicle in an infrared image and the texture details of a vehicle in a visible light image jointly guide optical flow calculation, reducing optical flow misjudgments caused by insufficient single-modal information.

[0162] By using dynamic confidence-guided optimization, low-confidence pixels in the optical flow field (such as those with weak textures or blurred edges) are corrected using a 3×3 neighborhood weighted correction, which solves the problem of insufficient robustness of traditional optical flow in complex scenes (such as rapid target movement or sudden changes in scene illumination). For example, when a visible light image experiences local texture blurring due to insufficient illumination, the optical flow vector of the current pixel can be corrected using more reliable optical flow information within the neighborhood, ensuring the overall continuity of the optical flow field.

[0163] Furthermore, the dynamic confidence level of this invention has the effect of accurately modeling the dual-modal grayscale difference and efficiently evaluating the reliability of optical flow.

[0164] Specifically, this is reflected in: In this invention, The first infrared image And the first visible light image The results were obtained through statistical analysis of the grayscale differences. This dual-modal joint statistics can accurately characterize the inherent grayscale differences of the two modes in the current scene. For example, in a daytime scene, the visible light texture is rich and the infrared thermal radiation is uniform. This difference will be reflected; in nighttime scenes, infrared thermal differences are more pronounced, while visible light is dimmer. It will also adapt and adjust itself.

[0165] in the formula It represents the grayscale residual between infrared pixels and visible light pixels after motion, under the influence of optical flow (u,v). Simultaneously constraining the dual-modality utilizes the spatial correspondence between infrared thermal targets and visible light texture targets, allowing the confidence level to better reflect the reliability of optical flow under dual-modality conditions.

[0166] use The form ensures that:

[0167] Smoothness: Small changes in the grayscale residual will not cause drastic fluctuations in confidence (avoiding misjudgment);

[0168] Discrimination: When the residual is much larger than At that time, the confidence level rapidly approaches 0 (accurately identifying unreliable optical flow).

[0169] Other methods, if they use linear functions (such as C=1-residual / threshold), are prone to problems such as a sharp drop in confidence due to a slightly large residual or a high confidence level despite a large residual, which are insufficient in distinguishing the reliability of optical flow.

[0170] Furthermore, the confidence level is directly based on the initial optical flow field. The calculation (optical flow (u,v) derived from the initial optical flow field) can reflect the quality of the optical flow estimation in real time. For example, when the optical flow field is accurately estimated in edge regions, the bimodal grayscale residual is small, and the confidence level is close to 1; when the optical flow field is incorrectly estimated in weakly textured regions, the residual is large, and the confidence level is close to 0. This real-time capability makes subsequent optimization of low-confidence optical flow (such as neighborhood weighted correction) more targeted.

[0171] The dynamic confidence level of this invention guides the motion compensation process: for pixels with high confidence, motion compensation is performed directly using the optical flow field; for pixels with low confidence, optical flow is optimized using neighborhood weighting before compensation. Compared to other methods that perform indiscriminate compensation (or simply filter low-confidence optical flow), this invention enables more accurate motion compensation for dual-light images, providing better spatial alignment input for subsequent feature fusion.

[0172] The feature fusion of motion-compensated two-light images relies on the spatial correspondence of bimodal features. The confidence level of this invention ensures that the optical flow involved in the fusion is reliable, thereby enabling more accurate spatial matching of infrared edge features and visible light texture features. This improves the effectiveness of cross-modal attention weight calculation and multi-scale feature fusion, ultimately resulting in better motion consistency and detail richness in the fused image.

[0173] After motion compensation of the dual-path preprocessed images based on optical flow field, multi-scale features are extracted and embedded with optical flow information to generate... (Infrared motion sensing features) and (Visible light band motion sensing features) achieves alignment of bimodal features in the motion dimension. For example, after motion compensation, the edge features of the moving target in the infrared image and the texture features of the target in the visible light image are accurately matched in spatial position, providing a feature basis for motion consistency and spatial alignment for subsequent cross-modal fusion (step S4).

[0174] The embedding of optical flow information expands bimodal features from containing only inherent image attributes (infrared thermal radiation, visible light texture) to include motion state information (target motion direction, velocity). This dual-dimensional feature of content + motion enables the subsequent cross-modal attention mechanism (step S4) to more accurately measure which regions' bimodal features are more motion-matched and more worthy of fusion, thus improving the targeting of the fusion.

[0175] The precise optical flow field and motion-aware bimodal feature set generated in this step are the core inputs for subsequent steps (cross-modal fusion, GAN detail optimization), directly affecting the quality of the final fused image.

[0176] This provides motion-aware features for the cross-modal attention weight calculation in step S4, enabling the attention weights to better reflect the feature importance of motion-consistent regions.

[0177] This provides a preliminary fused image for GAN detail optimization in step S6. This image already has good motion consistency and modal complementarity, so GAN only needs to enhance the detail texture level, which reduces the difficulty of subsequent optimization.

[0178] S4. Based on the infrared band motion sensing feature set and the visible light band motion sensing feature set, fuse the optical flow field. Calculate cross-modal attention weights and The gating value is calculated by introducing an index of the intensity of optical flow field motion. Dynamically adjust the multi-scale feature fusion weights to generate the final fused features. And based on the final fusion features, a preliminary fused image is generated. .

[0179] In some embodiments, cross-modal attention weights and The specific formula is:

[0180] ;

[0181] This represents the attention weight from infrared to visible light at the i-th scale. This represents the attention weight from visible light to infrared at the i-th scale; This represents the characteristics of infrared band motion sensing at the i-th scale. This represents the optical flow feature map corresponding to the i-th layer features; d represents the feature of visible light band motion perception at the i-th scale; d is the feature dimension.

[0182] ;

[0183] This represents the characteristics of visible light band motion perception at the i-th scale. This represents the characteristics of infrared band motion sensing at the i-th scale.

[0184] After convolution and other operations, features will form feature vectors or feature maps with a certain dimension. d is used to describe the scale of these features in terms of dimension. This parameter is used when scaling the dimension of features during operations such as calculating attention weights.

[0185] In some embodiments, the gate value The specific formula is:

[0186] ;

[0187] in, is the gate value at the i-th scale, ranging from (0,1), used to control the weight ratio when fusing features at multiple scales (the closer the value is to 1, the greater the weight of the corresponding scale feature fusion). The Sigmoid activation function maps the input to the (0,1) interval, as shown in the formula. ; It is a learnable weight matrix used to perform a linear transformation on the features after global average pooling, adjusting the influence of different feature channels; These are learnable bias terms used to compensate for the results after linear transformation, thereby increasing the model's expressive power. The feature of infrared band motion sensing at the i-th scale (multi-scale features extracted from the motion-compensated infrared image). The features representing motion perception of the visible light band at the i-th scale (multi-scale features extracted from the motion-compensated visible light image). The feature representing the intensity of optical flow motion at the i-th scale is the mean magnitude of the optical flow vector (reflecting the intensity of pixel motion at this scale; the more intense the motion, the larger the mean magnitude).

[0188] GlobalAvgPool represents the global average pooling operation, which averages the input feature map (of size H×W×C) across the spatial dimensions (H and W) to obtain a feature vector of dimension (1×1×C), which is used to compress spatial information and preserve channel-level features.

[0189] Spatial information of motion sensing features and optical flow intensity features of dual optical bands is compressed by global average pooling, and then linearly transformed ( and ) and Sigmoid activation to generate gate values The gating value can dynamically adjust the weights during multi-scale feature fusion based on the semantic information of the dual-light features and the intensity of optical flow motion, making the fusion process more adaptable to different motion scenarios (such as regions with intense motion, where more emphasis is placed on optical flow-guided feature fusion).

[0190] In some embodiments, the multi-scale feature fusion weights are dynamically adjusted to generate fused features. And generate a preliminary fused image based on the fusion features. The specific process is as follows:

[0191] S4.1 Cross-modal attention fusion of multi-scale features. This involves focusing on infrared motion sensing features at the i-th scale. Visible light band motion sensing characteristics Combined with cross-modal attention weights and Feature fusion is performed to obtain intermediate fused features. , , This represents element-wise multiplication, which uses cross-modal attention weights to weight bimodal features, enabling bidirectional fusion of infrared-guided visible light and visible light-guided infrared.

[0192] S4.2, Introduce a gating value to adjust the fusion weights. Combine the gating value... Intermediate fusion features after cross-modal fusion The final fused features are obtained by dynamically adjusting the weights. : .

[0193] This indicates a convolution operation, used to generate reference features. By controlling the fusion ratio between cross-modal fusion features and convolutional reference features, the fusion process can be better adapted to the characteristics of optical flow motion and dual-light features at the current scale.

[0194] S4.3 Multi-scale feature fusion.

[0195] Generated at various scales Where n is the number of scale levels, a feature pyramid fusion strategy is adopted, which upsamples the fusion features at different scales and adds them element by element to obtain multi-scale fusion features. : Upsample is an upsampling operation that unifies features from all scales to the finest scale, such as the first scale. .

[0196] S4.4 Generate a preliminary fused image.

[0197] Features after multi-scale fusion By using convolutional layers and activation functions (such as sigmoid), features are mapped to the image space to generate a preliminary fused image. : Convolution operations convert high-dimensional features into image dimensions (such as a 3-channel RGB image), and the Sigmoid activation function maps pixel values ​​to the [0,1] interval to obtain preliminary fused image results.

[0198] This step achieves precise complementary fusion of motion-sensing-driven dual-modal features, with the following specific results:

[0199] 1. Conventional dual-light fusion techniques (such as weighted average and simple feature stitching) are mostly based on the "static scene assumption" and directly superimpose the dual-modal features in space without considering the pixel motion misalignment problem in dynamic scenes, which makes the fused image prone to motion inconsistencies such as "ghosting" and "blurred edges".

[0200] This invention embeds the motion information (u, v components) of the optical flow field into a dual-modal feature. , Generate motion-aware images , This allows features to not only include image content but also pixel motion states; cross-modal attention weights and The calculation is directly related to optical flow features, tilting the weights towards regions with consistent motion (such as areas where the edges of moving targets align with textures in the dual-modality model). In dynamic scenes (such as moving vehicles and pedestrians), the fused features maintain spatiotemporal continuity, avoiding the problem of "misaligned fusion of infrared thermal targets and visible light textures" in conventional techniques, thus achieving preliminary image fusion. The motion consistency index M improved by more than 30%.

[0201] 2. Conventional cross-modal fusion often uses unidirectional attention (such as using only visible light features to guide infrared fusion) or fixed weight fusion, which cannot fully utilize the complementary advantages of dual modes (infrared is good at capturing thermal targets, while visible light is good at presenting detailed textures), resulting in fused images either losing the saliency of thermal targets or lacking texture details.

[0202] This invention is achieved through (Infrared-guided visible light) and The bidirectional interaction (visible light guiding infrared) allows the dual-modal features to "dominate" each other; for example, the infrared thermal target region is characterized by... Enhanced weighting, visible light texture rich areas are... Enhanced weights; gate values Combining optical flow intensity characteristics The method dynamically adjusts the cross-modal fusion weights in areas of high motion (high optical flow intensity) and retains more single-modal advantages in static areas. The fused features can simultaneously preserve the thermal target contrast of infrared images (such as the distinction between pedestrians and background at night) and the texture details of visible light images (such as clothing texture and road markings). The detail richness index H is improved by 25% compared with conventional methods, solving the fusion dilemma of sacrificing one aspect for the other.

[0203] 3. Conventional techniques often perform feature fusion at a single scale (such as the size of the input image), making it difficult to take into account both global semantics (such as the overall outline of the target) and local details (such as edge sharpness), resulting in fused images that are either globally blurred or locally distorted.

[0204] This invention obtains multi-scale features (i=1,2,...,n) through a multi-branch convolutional network, corresponding to high-resolution details to low-resolution semantics; and fuses features at each scale. Upsampling and aggregation are performed to accurately fuse features at different levels (such as large-scale infrared thermal regions and small-scale visible textures) at their corresponding scales. Preliminary fused image. It maintains clarity in both global structures (such as building outlines) and local details (such as window textures), with a clarity index C (Tenengrad mean) that is 20% higher than that of single-scale fusion methods, especially in complex scenes (such as urban building complexes).

[0205] In summary, this step, through a collaborative design of motion-aware embedding, bidirectional attention interaction, dynamic gating adjustment, and multi-scale aggregation, overcomes the limitations of conventional dual-light fusion technology in terms of dynamic consistency, modal complementarity, and scale adaptability. It provides high-quality basic features for subsequent GAN detail optimization, ultimately enabling the fused image to simultaneously meet the application requirements of intelligent cameras (such as security monitoring and autonomous driving visual perception) that require continuous motion, prominent thermal targets, and clear textures.

[0206] S5. From the preliminary fused image, obtain the following parameters: sharpness (C), motion consistency (M), and detail richness. The evaluation results of the evaluation metrics; and the dynamic adjustment of the cross-modal attention weights and gating values ​​based on the evaluation results. .

[0207] The specific process is as follows:

[0208] S5.1 Calculate the evaluation indicators:

[0209] Sharpness C calculation: The Tenengrad gradient function is used to calculate the sharpness of the initially fused images. Calculate the gradient for each pixel, and then calculate the average of all pixel gradients. The formula is: Where H and W represent the height and width of the initially fused image, , These are the gradients of the initial fused image in the x and y directions (calculated using the Sobel operator), respectively. The larger the value, the clearer the image.

[0210] Motion consistency M calculation: This involves calculating the initial fused image and the motion-compensated dual-path preprocessed image. Structural similarity (SSIM), combined with optical flow field The motion vectors are weighted, and the formula is:

[0211] ;

[0212] Weights are applied to infrared modes. Weights are adapted for visible light modes; Indicates the initial fused image Second infrared image Structural similarity between them; Indicates the initial fused image Second visible light image Structural similarity between them; The weighting coefficients represent the motion terms of optical flow. , represents the motion vector component of the optical flow field at (x,y). A larger value indicates better motion consistency.

[0213] Richness of detail Calculation: The entropy function is used to measure the uniformity of gray-level distribution and the richness of detail in the initially fused image. The formula is:

[0214] ;

[0215] This is used to initially fuse the probability of pixels with a gray value of k appearing in the image; the larger the value, the richer the image details.

[0216] S5.2 Establish a mapping relationship between evaluation results and parameter adjustments.

[0217] The aforementioned sharpness C, motion consistency M, and detail richness By weighting and combining the results, we obtain the overall evaluation score S:

[0218] ;

[0219] , , These are respectively: Clarity (C), Motion Consistency (M), and Detail Richness. The weighting is based on the comprehensive evaluation score S and the preset optimal score range (e.g., ...). To determine the deviation, an adjustment amount is constructed. , This is the actual comprehensive evaluation score. These represent the lower and upper limits of the optimal score range, respectively. ; This represents the mean of the optimal score range.

[0220] S5.3 Dynamically adjust cross-modal attention weights.

[0221] Cross-modal attention weights and According to the adjustment amount Adjustments will be made:

[0222] ;

[0223] ;

[0224] in, For learning rate, , The comprehensive evaluation scores S are respectively , The gradient (calculated through backpropagation) makes the adjusted attention weights more conducive to improving the overall evaluation score;

[0225] S5.4, also based on the adjustment amount For the gate value Adjustments will be made:

[0226] ;

[0227] in, To comprehensively evaluate the score S for the gate value The gradient of the gate value can be adjusted to more accurately control the multi-scale feature fusion weights, thereby optimizing the fusion effect.

[0228] The core effect of this step is to construct a closed-loop feedback mechanism for fusion effect and parameter optimization. Compared with the static parameters and one-step fusion mode of conventional dual-light fusion technology, it achieves adaptability and accuracy in the fusion process, which is specifically reflected in the following three aspects:

[0229] 1. The parameters such as cross-modal attention weights and gating values ​​of conventional dual-light fusion technology are mostly fixed during the training phase or preset by human experience, which cannot cope with dynamic scenarios such as sudden changes in lighting (such as day / night switching) and changes in motion state (such as target acceleration / deceleration).

[0230] This invention calculates sharpness (C), motion consistency (M), and detail richness. The preliminary fused image is comprehensively evaluated from three dimensions: image quality, motion matching, and detail preservation. Based on the gradient of the evaluation results, such as... , Cross-modal attention weights Gating value Dynamic fine-tuning is performed. In tunnel entrance / exit scenarios (sudden changes in lighting), conventional methods result in visible light overexposure / infrared blurring in the fused image due to static parameters. However, step S5 of this invention can adjust parameters in real time, improving the sharpness C of the fused image by 35%, motion consistency M by 40%, and detail richness. An increase of 25%.

[0231] 2. The core challenge of dual-modal fusion is that each mode has its own defects (e.g., infrared has low contrast under strong light, and visible light has high noise under weak light). Conventional technologies lack a mechanism for sensing and compensating for "modal defects".

[0232] This invention utilizes motion consistency M (bimodal motion matching degree) and detail richness. (Texture / edge integrity) indirectly senses whether the current modality is reliable (e.g., missing visible light details in low light). It will decrease); if a certain modal defect (such as visible light detail) is perceived. If the level is low, then increase the attention weight of the other modality (e.g., increase the level of attention). (This allows infrared features to guide the fusion process more effectively). In low-light nighttime scenes, visible light images are noisy and have blurred textures. The S5 of this invention improves the thermal target recognition of the fused image by 40% by reducing the visible light attention weight and increasing the gating value's reliance on infrared features, while suppressing visible light noise and preserving more complete details.

[0233] 3. The conventional fusion process is forward-biased (without intermediate evaluation from acquisition to fusion output). If a certain link fails (such as optical flow calculation deviation), the error will be propagated to the final result and cannot be corrected.

[0234] This invention's evaluation of the initial fused image can promptly detect problems such as optical flow guidance errors and feature fusion imbalances. Through multiple iterative adjustments (e.g., optimizing parameters every 10 frames), the cross-modal attention and gating values ​​gradually approach the optimal solution for the current scene. In multi-target complex motion scenarios (such as intertwined traffic), conventional methods result in target ghosting and blurred boundaries in the fused image due to the accumulation of optical flow errors. However, step S5 of this invention, through iterative optimization, improves the motion consistency M from 0.65 (conventional method) to 0.92, approaching the ideal motion matching degree of fusion.

[0235] The S5 of this invention upgrades dual-light fusion from passive scene adaptation to active effect optimization through multi-dimensional evaluation, gradient optimization, and iterative feedback. It solves the pain points of poor scene adaptability, difficulty in compensating for modal defects, and lack of error correction of conventional technologies. It provides accurate, robust, and adaptive fusion features for subsequent GAN detail optimization, and ultimately enables smart cameras to output high-quality fused images in all time periods and all scenes.

[0236] S6. Based on the dynamically adjusted cross-modal attention weights and gating values The optimized final fusion features are obtained, and the optimized preliminary fusion image is obtained; then the optimized preliminary fusion image is further optimized through a generative adversarial network (GAN) to generate the final fusion image.

[0237] In some embodiments, the specific implementation process of S6 is as follows:

[0238] S6.1 Generate the optimized final fusion features.

[0239] S6.1.1. Re-fuse features based on adjusted parameters. Employ dynamically adjusted cross-modal attention weights. , and gate value Re-execute multi-scale feature fusion:

[0240] ;

[0241] ;

[0242] Optimized fusion features at various scales ;

[0243] S6.1.2 Optimized fusion features at each scale Using the same feature pyramid fusion strategy as in step S3, aggregated optimized features are generated. :

[0244] ;

[0245] S6.1.3. Generating the optimized preliminary fused image: The image is mapped to the image space through convolutional layers and activation functions to obtain the optimized preliminary fused image. : .

[0246] S6.2 Construct a Generative Adversarial Network (GAN) and perform detailed optimizations.

[0247] GAN network architecture design:

[0248] Generator: Employs the U-Net architecture; input is the optimized preliminary fused image. The encoder-decoder structure captures multi-scale details (such as edge sharpening and texture enhancement), and the output is a candidate fused image with optimized details. .

[0249] Discriminator: Employs a convolutional neural network (CNN), with candidate fused images as input. Two-light image after motion compensation The output is the image authenticity probability (determining whether the candidate image is close to the ideal fused image).

[0250] GAN loss function design:

[0251] The generator loss consists of adversarial loss and content loss: .

[0252] To counteract the loss and measure the generator's ability to deceive the discriminator, a binary cross-entropy loss is employed. Where D(*) is the probability of authenticity output by the discriminator. These are randomly generated fake images.

[0253] For content loss, to constrain the content consistency between the candidate image and the two-light image, structural similarity (SSIM) loss is adopted: ; Represents candidate fused images Second infrared image Structural similarity between them; Represents candidate fused images Second visible light image The structural similarity between them.

[0254] The discriminator loss is: , This is an ideal fused image that is manually labeled during the training phase.

[0255] S6.3 Generate the final fused image.

[0256] GAN Training and Iterative Optimization. Alternating training of the generator and discriminator: the generator is optimized by minimizing... To improve the ability to generate details, the discriminator minimizes... Improve the ability to distinguish between true and false data until the network converges (the loss function tends to stabilize).

[0257] Output the final fused image. After training, the generator receives the optimized preliminary fused image. Output the final fused image with enhanced details. It also retains:

[0258] thermal target salience in infrared images;

[0259] Texture details in visible light images;

[0260] Consistency of motion guided by optical flow field.

[0261] The core effect of this step is to further enhance details and improve realism based on accurate feature fusion. Compared with conventional dual-light fusion techniques (such as direct weighting and traditional CNN fusion), it overcomes the limitations of lack of detail and insufficient realism in fused images, specifically in the following three aspects:

[0262] 1. Conventional dual-light fusion techniques (such as cross-modal attention fusion and multi-scale feature fusion) can achieve modal complementarity, but they are still insufficient in preserving fine-grained details (such as texture and edge sharpness) (e.g., edge blurring in infrared thermal imaging and texture loss under low light conditions in visible light).

[0263] The generator, employing the U-Net architecture, uses an encoder-decoder structure to process the optimized initial fused image. Mining multi-scale details (such as small object edges and subtle textures); the discriminator uses motion-compensated two-light images. Candidate images output by the constraint generator, which are real samples. It is closer to the real scene in terms of texture, edges, and color. In scenes such as building wall textures and vegetation leaf vein details, conventional fusion images may lose details due to modal defects (such as no texture in infrared and blur in weak visible light), while the final fusion image generated by S6 of the present invention can clearly restore these fine-grained details, and the detail richness is improved by 30% to 40% compared with conventional methods.

[0264] 2. When conventional GANs are used for image fusion, they are mostly trained end-to-end statically, that is, the parameters of the generator and discriminator are fixed, which cannot adapt to dynamic scene changes (such as target movement and lighting fluctuations).

[0265] First, S5 adjusts the cross-modal attention weights and gating values ​​to obtain a more accurate preliminary fused image. GAN further optimizes details based on the accurate preliminary fused image, rather than directly generating the original two-light image indiscriminately. In scenes with moving targets (such as running people), conventional GANs may generate fusion results with ghosting and detail distortion due to motion misalignment of the input image. However, S6 of this invention improves the motion consistency M of the fused image by 25% through a collaborative process of dynamic parameter optimization of preliminary fusion (solving motion misalignment) → GAN optimization of details, while making details (such as clothing wrinkles) more realistic.

[0266] 3. The core contradiction of dual-light fusion is the large difference in modalities (infrared thermal radiation vs. visible light texture) and the human eye's need for realism. Conventional techniques either overemphasize one modality (such as excessive infrared proportion leading to strong false color in the image) or result in an unnatural image after fusion, resembling a composite image.

[0267] In this invention, the adversarial loss of GAN makes the generated image closer to the real scene in terms of overall visual style (such as natural colors and texture distribution), and the content loss is constrained by structural similarity (SSIM) to ensure the consistency of content between the generated image and the two-light image (such as the location of infrared thermal targets and visible light texture regions).

[0268] The final fused image retains the salience of infrared thermal targets (such as the high brightness of a fire heat source) and the natural texture and color of visible light images (such as the realistic texture of the environment around the heat source). In subjective visual evaluation, the realism score is improved by 40% compared with conventional methods, which is more in line with the human eye's expectations for imaging real scenes.

[0269] In summary, steps S3 (optical flow field calculation and motion-aware feature generation), S4 (cross-modal feature fusion and preliminary fused image generation), S5 (evaluation and dynamic parameter adjustment), and S6 (GAN detail optimization to generate the final fused image) in this invention achieve a synergistic effect of 1+1>2 through the design of motion-aware integration, dynamic closed-loop optimization, and multimodal deep collaboration. This is specifically reflected in the following three dimensions:

[0270] I. The entire process of motion perception is integrated: from optical flow to features and then to images.

[0271] S3's motion information foundation: Through weighted constraints of bimodal features and dynamic confidence optimization, an accurate and robust optical flow field is generated, and optical flow information is embedded into bimodal features, enabling the features to have motion perception capabilities (not only including image content, but also carrying pixel motion state).

[0272] S4's motion-guided fusion: The calculation of cross-modal attention weights and gating values ​​is directly related to motion-aware features, which tilts the fusion process towards regions with consistent motion (such as regions where the edges and textures of moving targets are aligned in the dual modality, which are fused first). The initial fused image naturally possesses motion continuity.

[0273] S5 motion quality assessment: The motion consistency M index directly quantifies the motion matching degree between the fused image and the two-light preprocessed image, providing feedback on the motion dimension for parameter adjustment;

[0274] S6 Motion Detail Enhancement: When optimizing details, GAN uses motion-compensated dual-light images as real samples to ensure that detail enhancement does not destroy motion consistency.

[0275] Synergistic effect: Motion perception is integrated throughout the entire process from optical flow calculation to final image, enabling the fused image to maintain both motion continuity and rich detail in dynamic scenes (such as moving objects), solving the pain point of conventional two-light fusion which makes it difficult to balance motion and detail (conventional methods either have consistent motion but blurry details, or rich details but misaligned motion).

[0276] II. Iterative optimization of dynamic closed loop: from evaluation to adjustment and then to optimization.

[0277] S4 outputs the initial fused image: used as the evaluation object for S5;

[0278] S5 Multidimensional Assessment (C, M, H) f And adjust the parameters: based on the gradient of the evaluation results, dynamically optimize the cross-modal attention weights and gating values ​​to generate a more accurate preliminary fusion image;

[0279] S6 takes the optimized initial image as input: GAN further enhances the details on this basis, rather than directly processing the original two-light image.

[0280] Synergistic effect: A closed-loop iteration is formed, consisting of initial fusion → evaluation and adjustment → re-fusion → re-evaluation and adjustment → GAN optimization, making each round of fusion more accurate than the previous one. For example, the parameters adjusted in S5 make the fusion in S4 more suitable for the current scene, while S6 optimizes details on the more accurate initial image. The final fused image has 40% to 60% higher clarity, motion consistency, and detail richness than step-by-step fusion without a closed loop.

[0281] III. Deep Collaboration of Multimodal Models: From Features to Weights to Generation.

[0282] S3's bimodal feature extraction and optical flow embedding: provides motion-aware bimodal features for S4, enabling cross-modal fusion with accurate feature raw materials;

[0283] S4's bidirectional cross-modal attention and dynamic gating: allows bimodal features to dominate each other (infrared guiding visible light, visible light guiding infrared), achieving complementary advantages;

[0284] S5's multi-dimensional evaluation: quantifies the effect of dual-modal fusion from three dimensions: sharpness, motion consistency, and detail richness, providing a basis for multi-modal balance in parameter adjustment;

[0285] S6's GAN multimodal constraint: Through content loss (SSIM), it ensures that the generated image retains both the saliency of infrared thermal targets and the texture details of visible light.

[0286] Synergistic effect: The dual-modal approach achieves deep synergy across all layers, from the feature layer to the fusion layer, then to the evaluation layer, and finally to the generation layer, rather than being dominated by a single modality or simply superimposed. For example, in low-light scenes with moving targets, the infrared thermal target guides the fusion weights, while the visible light texture details are enhanced by GAN, resulting in an image that clearly shows both the target (infrared advantage) and the details (visible light advantage).

[0287] These four steps, through the design of motion perception throughout the entire process, dynamic closed-loop iterative optimization, and multimodal full-level collaboration, deeply bind the accuracy of optical flow calculation, the complementarity of cross-modal fusion, the adaptability of parameter adjustment, and the detail of GAN generation. Ultimately, the fused image far surpasses the effect of individual optimization or simple superposition of each step in the three key indicators of motion consistency, detail richness, and visual realism, truly achieving a technological breakthrough of 1+1>2.

[0288] Example 2 discloses a dual-light integrated smart camera for executing the dual-light image fusion method of a smart camera in Example 1, including a dual-light image acquisition module, a preprocessing module, an optical flow calculation module, a feature fusion module, an image optimization module, and a control module integrated on the same component;

[0289] The dual-light image acquisition module is used to simultaneously acquire infrared and visible light images of the same scene and output the original infrared and visible light images.

[0290] The preprocessing module is connected to the dual-light image acquisition module and is used to preprocess the original infrared image and the original visible light image to output dual-channel preprocessed images.

[0291] The optical flow calculation module, connected to the preprocessing module, is used to calculate the optical flow field based on the dual-path preprocessed image using an optical flow algorithm that includes dual-modal feature weighting constraints and dynamic confidence optimization. ;

[0292] The feature fusion module is connected to both the preprocessing module and the optical flow calculation module, and is used to perform calculations based on the optical flow field. Motion compensation is performed on the dual-channel preprocessed images; multi-scale features of the motion-compensated dual-channel images are extracted using a multi-branch convolutional network, and the optical flow field is then processed. Motion information is embedded into features at each layer to generate a motion-aware bimodal feature set (infrared motion-aware feature set and visible light motion-aware feature set); based on cross-modal attention weights and dynamic gating values, the motion-aware bimodal feature set is fused at multiple scales to output a preliminary fused image. ;

[0293] The image optimization module is connected to the feature fusion module and is used to process the initially fused image through a generative adversarial network. Perform detailed optimizations and output the final fused image;

[0294] The control module is connected to each of the above modules and is used to coordinate the working sequence of each module to realize the real-time acquisition, processing, fusion and output of dual-light images.

[0295] In some embodiments, the dual-light image acquisition module includes an infrared sensor and a visible light sensor, and the preprocessing module includes an infrared image preprocessing unit and a visible light image preprocessing unit. The infrared image preprocessing unit is disposed in an FPGA and is used to receive the raw infrared image transmitted by the infrared sensor and preprocess the raw infrared image to output a first infrared image. The visible light processing unit is disposed in an AI chip and is used to receive the raw visible light image transmitted by the visible light sensor and preprocess the raw visible light image to obtain a first visible light image. At the same time, it receives the first infrared image and performs fusion, optical flow field calculation, multi-scale feature extraction / embedding, and image optimization on the first infrared image and the first visible light image through an optical flow calculation module, a feature fusion module, an image optimization module, and a control module disposed in the AI ​​chip to obtain a final fused image.

[0296] like Figure 2As shown, the dual-sensor integrated smart camera in this embodiment breaks through the bottleneck of traditional separate architectures such as CCD visible light detector core + IR infrared detector core + image information processing board through an integrated architecture of dual sensors + computing power base (FPGA + AI), enabling the sensing device to achieve integration, miniaturization, low cost, and AI intelligence. While possessing image data acquisition and processing capabilities, it empowers core AI technologies such as target tracking, high-speed autonomous target recognition, and image fusion. The system adopts an integrated architecture of dual sensors + FPGA + AI. The infrared sensor resolution is 640×512@50fps, and the visible light sensor resolution is 2448×2048@50fps, supporting simultaneous input of infrared video signals and visible light video signals. The FPGA realizes the acquisition and preprocessing of infrared images, and the AI ​​chip realizes the acquisition and preprocessing of visible light images, as well as post-processing functions such as infrared and visible light image data fusion and target recognition and tracking. The processed image data can be output to the display through the CML digital image output interface or the PAL analog video image interface, or transmitted to other devices through the network port.

[0297] The main chip consists of one ZYNQ JFMQL15T485 from Shanghai Fudan Microelectronics, one RK3588 from Rockchip, one H6412C1SD from Hikvision Microelectronics, one GMAX3405 from Changguang Chenxin, and other peripheral interface chips.

[0298] It adopts an integrated architecture of dual sensors + computing base (FPGA + AI), and provides an efficient heat dissipation solution through low power design and integrated heat dissipation technology, thereby reducing power consumption and achieving hardware and software co-optimization.

[0299] IR infrared sensors are susceptible to heat sources, which can affect imaging performance. The computing power base of the dual-light integrated smart camera has stringent requirements for power density and requires low-power design. However, low-power design increases the complexity of hardware design and the difficulty of component selection, and may sacrifice some performance. It is necessary to overcome the challenges of achieving low power consumption while ensuring performance and reducing design complexity.

[0300] The dual-light integrated smart camera mainly consists of an RK3588 board, an infrared sensor board, an FPGA interface board, an FPGA board, a TV sensor board, and matching housings and covers. The infrared sensor board, FPGA interface board, FPGA board, and TV sensor board are connected by a flexible board, as shown in the module structure diagram below. Figure 3 As shown.

[0301] The infrared image preprocessing unit is implemented by FPGA, which uses hardware-accelerated algorithms to process problems such as blind pixels, non-uniformity, and poor clarity in the raw data of infrared sensors.

[0302] The visible light image preprocessing unit is completed by the AI ​​chip. The RK3588 serves as the image processing unit. It uses the on-chip ISP unit to preprocess the image output from the TV sensor, including detector calibration, bad pixel correction, autofocus / exposure / white balance, gamma correction, noise reduction, and sharpening. After processing, the image in the YUV domain is obtained. The YUV image is then encoded, streamed, OSD processed, and forwarded for subsequent processing. The infrared image input from the data transmission interface is also encoded, streamed, OSD processed, and forwarded.

[0303] The multi-source image fusion processing module group (including optical flow calculation module, feature fusion module, and image optimization module) is completed by the AI ​​chip. It fuses and complements the pre-processed infrared and visible light images to further improve image quality and provide more multi-dimensional and complete feature information for the target recognition and tracking algorithm, thereby improving the accuracy of target recognition and tracking.

[0304] Through the above-described configuration, the present invention possesses the following advantages:

[0305] High-resolution, high-frame-rate video acquisition input: Both the infrared and visible light sensors adopt high-resolution, high-frame-rate sensors, which can obtain clearer and higher-quality raw images, providing richer detailed information for subsequent multi-source image data fusion processing, thereby further improving the image analysis effect (such as target detection, matching, recognition, tracking and other analysis results), improving the camera's intelligent operation capability and working reliability, and meeting the needs of complex task scenarios such as long distance, low light, strong clutter, weak targets and fast-moving targets;

[0306] An edge intelligent computing platform based on a fully domestically produced FPGA+AI heterogeneous architecture provides intelligent cameras with large-scale image data processing and complex visual computing capabilities to meet the real-time processing demands brought about by the surge in high-resolution, high-frame-rate, and dual-light video image data, reducing image processing latency. Through intelligent switching between day and night perception modes, it achieves full-coverage scene adaptability: during the day, relying on the full-color information of visible light images, it accurately restores the target's color, texture, and other details; at night or in low-light environments, the high contrast characteristics of infrared images ensure that the target's outline and structure are clearly discernible. The complementary advantages of these two types of images endow the system with stable perception capabilities under complex lighting conditions such as strong light, backlight, and low light, providing an all-weather visual solution that combines real-time performance and robustness.

[0307] The high-resolution dual-light integrated intelligent camera computing base breaks through the technical bottleneck of the traditional separate architecture of CCD visible light detector core + IR infrared detector core + image information processing board through the integrated architecture of dual sensors + computing base (FPGA + AI), enabling the sensing device to achieve integration, miniaturization, low cost and AI intelligence.

[0308] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for dual-light image fusion in an intelligent camera, characterized in that, include: Raw infrared and raw visible light images are acquired simultaneously using infrared and visible light sensors. The original infrared image and the original visible light image are subjected to dual-channel parallel preprocessing to obtain a dual-channel preprocessed image, which includes a first infrared image and a first visible light image obtained after preprocessing. An improved optical flow algorithm with dual-modal feature weighting constraints and dynamic confidence-guided optimization is used to calculate the optical flow field of the dual-path preprocessed image; After motion compensation is performed on the dual-path preprocessed image based on the optical flow field, multi-scale features of the motion-compensated dual-path preprocessed image are extracted by a multi-branch convolutional network, and optical flow information is embedded into the features of each layer of the motion-compensated dual-path preprocessed image to generate an infrared band motion sensing feature set and a visible light band motion sensing feature set. Based on the infrared band motion sensing feature set and the visible light band motion sensing feature set, the optical flow field is fused to calculate the cross-modal attention weights; the intensity index of optical flow field motion is introduced to calculate the gating value, the multi-scale feature fusion weights are dynamically adjusted to generate the final fused features, and a preliminary fused image is generated based on the final fused features. The preliminary fused image is evaluated to obtain evaluation results including indicators of sharpness, motion consistency, and detail richness; the cross-modal attention weights and gating values ​​are dynamically adjusted based on the evaluation results. Based on the dynamically adjusted cross-modal attention weights and gating values, the optimized final fusion features are obtained, and the optimized preliminary fusion image is obtained. The optimized preliminary fused image is then further refined using a Generative Adversarial Network (GAN) to generate the final fused image.

2. The intelligent camera dual-light image fusion method according to claim 1, characterized in that, The dual-channel parallel preprocessing of the original infrared image and the original visible light image to obtain a dual-channel preprocessed image includes: The first infrared image is obtained by sequentially performing blind pixel correction, non-uniformity correction, and sharpness enhancement processing on the original infrared image obtained through the infrared sensor. The raw visible light image output by the visible light sensor is sequentially processed by detector calibration, bad pixel correction, autofocus, auto exposure, auto white balance, gamma correction, noise reduction and sharpening to obtain a YUV domain image, which is the first visible light image; at the same time, the pre-processed infrared image is received, and the dual-channel pre-processed images are uniformly processed by encoding and streaming, OSD overlay and forwarding.

3. The intelligent camera dual-light image fusion method according to claim 1, characterized in that, An improved optical flow algorithm with dual-modal feature weighting constraints and dynamic confidence-guided optimization is used to calculate the optical flow field of the dual-path preprocessed images. The specific process is as follows: Extract the infrared edge structure features of the first infrared image Visible light texture color features of the first visible light image ; Infrared mode adaptation weights are calculated based on the signal-to-noise ratio of the first infrared image and the contrast of the first visible light image. Adaptation weights for visible light modes ,in, , ,in, This represents the signal-to-noise ratio of the first infrared image. Indicates the contrast of the first visible light image; An initial optical flow field optimization model incorporating dual-modal feature weighted constraints is constructed and solved. The formula for the initial optical flow field is expressed as: ; Indicates the initial optical flow field; Indicates the first infrared image, These are pixel coordinates; This represents the first visible light image; , These represent the first infrared image and the first visible light image in pixels, respectively. The grayscale value at that location; ρ represents the components of the optical flow vector in the x and y directions; ρ represents the robust loss function. , For regularization parameters; The weighting coefficients represent the motion terms of optical flow. The infrared edge structure features of the first infrared image; The visible light texture and color features of the first visible light image; Infrared edge features after motion; The visible light texture color characteristics after motion; , ; According to the initial optical flow field Calculate the dynamic confidence score C(x,y) for each pixel using the following formula: ;in, For the first infrared image And the first visible light image The standard deviation is obtained after statistical analysis of the grayscale differences between them; The closer the value is to 1, the more reliable the optical flow estimation for that pixel; For dynamic confidence For low-confidence pixels, a 3×3 neighborhood optical flow weighted average is used for optimization. The optimized optical flow vector components are substituted into the initial optical flow field to replace the original dynamic confidence level. The optical flow vectors of low-confidence pixels are used to obtain the final optical flow field. .

4. The intelligent camera dual-light image fusion method according to claim 1, characterized in that, Based on the optical flow field After motion compensation is applied to the dual-path preprocessed images, multi-scale features of the motion-compensated dual-path preprocessed images are extracted using a multi-branch convolutional network. Optical flow information is then embedded into the features of each layer of the motion-compensated dual-path preprocessed images to generate a motion-aware feature set. The specific process is as follows: For the first infrared image According to the optical flow field The motion vectors (u(x,y), v(x,y)) of each pixel (x,y) are mapped to spatial positions aligned with the visible light image using a bilinear interpolation algorithm, resulting in the motion-compensated second infrared image. The second visible light image after performing reverse motion compensation on the first visible light image ; The second infrared image is extracted using a structurally symmetrical dual-branch convolutional network. Second visible light image Multi-layer features are used to obtain a multi-scale feature set of infrared images. Multiscale feature sets of visible light images , where n is the number of scale layers; For the optical flow field Downsampling and channel compression are performed to generate optical flow feature maps that match the features of each layer. , This represents the optical flow feature map corresponding to the i-th layer features; The optical flow feature map is compared with the multi-scale feature set of the infrared image. Multiscale feature sets of visible light images By adding them together, we obtain the feature sets for infrared motion sensing and the feature sets for visible light motion sensing, where... , , This is the optical flow weighting coefficient.

5. The intelligent camera dual-light image fusion method according to claim 4, characterized in that, Cross-modal attention weights and The specific formula is: ； This represents the characteristics of infrared band motion sensing at the i-th scale. This represents the optical flow feature map corresponding to the i-th layer features; This represents the characteristics of visible light band motion perception at the i-th scale. d represents the feature dimension; ； This represents the characteristics of visible light band motion perception at the i-th scale. This represents the characteristics of infrared band motion sensing at the i-th scale. The threshold value The specific formula is: ； in, The Sigmoid activation function maps the input to the (0,1) interval, as shown in the formula. ; The weight matrix is ​​a learnable matrix. For learnable bias terms; This represents the characteristics of infrared band motion sensing at the i-th scale. This represents the characteristics of visible light band motion perception at the i-th scale. The i-th scale represents the intensity of optical flow motion, and GlobalAvgPool is a global average pooling operation.

6. The intelligent camera dual-light image fusion method according to claim 5, characterized in that, Dynamically adjust the multi-scale feature fusion weights to generate the final fused features. And based on the final fusion features, a preliminary fused image is generated. The specific process is as follows: Infrared band motion sensing features at the i-th scale Visible light band motion sensing characteristics Combined with cross-modal attention weights and Feature fusion is performed to obtain intermediate fused features. , , This indicates element-wise multiplication; For the intermediate fusion features The final fused features are obtained by dynamically adjusting the weights. : , Indicates the convolution operation; Generated at various scales Where n is the number of scale levels, a feature pyramid fusion strategy is adopted, which upsamples the fusion features at different scales and adds them element by element to obtain multi-scale fusion features. : Where Upsample is the upsampling operation; The multi-scale fused features are processed through convolutional layers and activation functions. Mapped to image space, generating a preliminary fused image. .

7. The intelligent camera dual-light image fusion method according to claim 1, characterized in that, The preliminary fused image yields results including sharpness (C), motion consistency (M), and detail richness. The evaluation results of the evaluation metrics; and the dynamic adjustment of the cross-modal attention weights and gating values ​​based on the evaluation results. The specific process is as follows: The Tenengrad gradient function is used to initially fuse the images. The gradient is calculated for each pixel, and the mean of the gradients of all pixels is then calculated to obtain the sharpness C. By calculating the structural similarity (SSIM) between the preliminary fused image and the second infrared image, and between the preliminary fused image and the second visible light image, and then combining it with the optical flow field... The motion vectors are weighted to obtain the motion consistency M; the detail richness of the preliminary fused image is calculated using the entropy function. ; The aforementioned sharpness C, motion consistency M, and detail richness A weighted combination is performed to obtain a comprehensive evaluation score S, and an adjustment amount is constructed based on a preset optimal score range. ; For the cross-modal attention weights and According to the adjustment amount Adjustments will be made: ； ； in, For learning rate, , The comprehensive evaluation scores S are respectively , The gradient; According to the adjustment amount For the gate value Adjustments will be made: ； in, To comprehensively evaluate the score S for the gate value The gradient.

8. The intelligent camera dual-light image fusion method according to claim 1, characterized in that, The optimized preliminary fused image is further refined using a Generative Adversarial Network (GAN) to generate the final fused image. The specific process is as follows: A Generative Adversarial Network (GAN) is constructed, comprising a generator and a discriminator, wherein the loss function of the generator is: , To combat the losses, Where D(*) is the probability of authenticity output by the discriminator. For randomly generated fake images, The candidate fused image is generated by the generator from the initial fused image; the loss function of the discriminator is... , This is an ideal fused image that is manually labeled during the training phase.

9. A dual-light integrated intelligent camera, used to execute the dual-light image fusion method for an intelligent camera as described in any one of claims 1-8, characterized in that, This includes a dual-light image acquisition module, a preprocessing module, an optical flow calculation module, a feature fusion module, an image optimization module, and a control module, all integrated on the same component. The dual-light image acquisition module is used to simultaneously acquire infrared and visible light images of the same scene and output the original infrared and visible light images. The preprocessing module is connected to the dual-light image acquisition module and is used to preprocess the original infrared image and the original visible light image to output dual-channel preprocessed images. The optical flow calculation module, connected to the preprocessing module, is used to calculate the optical flow field based on the dual-path preprocessed image using an optical flow algorithm that includes dual-modal feature weighting constraints and dynamic confidence optimization. ; The feature fusion module is connected to both the preprocessing module and the optical flow calculation module, and is used to perform calculations based on the optical flow field. Motion compensation is performed on the dual-channel preprocessed images; multi-scale features of the motion-compensated dual-channel images are extracted using a multi-branch convolutional network, and the optical flow field is then... Motion information is embedded into features of each layer to generate an infrared motion sensing feature set and a visible light motion sensing feature set. Based on cross-modal attention weights and dynamic gating values, the motion-aware bimodal feature set is fused at multiple scales to output a preliminary fused image. ; The image optimization module is connected to the feature fusion module and is used to process the initially fused image through a generative adversarial network. Perform detailed optimizations and output the final fused image; The control module is connected to the dual-light image acquisition module, preprocessing module, optical flow calculation module, feature fusion module, and image optimization module respectively, and is used to coordinate the working timing of the dual-light image acquisition module, preprocessing module, optical flow calculation module, feature fusion module, and image optimization module to realize real-time acquisition, processing, fusion, and output of dual-light images.

10. A dual-light integrated intelligent camera according to claim 9, characterized in that, The dual-light image acquisition module includes an infrared sensor and a visible light sensor. The preprocessing module includes an infrared image preprocessing unit and a visible light image preprocessing unit. The infrared image preprocessing unit is located in the FPGA and is used to receive the raw infrared image transmitted by the infrared sensor, preprocess the raw infrared image, and output a first infrared image. The visible light processing unit is located in the AI ​​chip and is used to receive the raw visible light image transmitted by the visible light sensor, preprocess the raw visible light image to obtain a first visible light image, and simultaneously receive the first infrared image. The first infrared image and the first visible light image are fused, optical flow field calculated, multi-scale feature extracted / embedded, and image optimized by the optical flow calculation module, feature fusion module, image optimization module, and control module located in the AI ​​chip to obtain the final fused image.

Images