Intelligent camera dual-light image fusion method and dual-light integrated intelligent camera

Abstract

The application relates to the field of intelligent vision and image fusion technology, and discloses a dual-light image fusion method of an intelligent camera and a dual-light integrated intelligent camera, which comprises the following steps: collecting original infrared images and original visible light images; preprocessing the original infrared images and the original visible light images to obtain dual-path preprocessed images; calculating an optical flow field by adopting an improved optical flow algorithm on the dual-path preprocessed images; generating an infrared band motion perception feature set and a visible light band motion perception feature set; generating a preliminary fusion image based on the infrared band motion perception feature set and the visible light band motion perception feature set; obtaining an evaluation result including evaluation indexes of clarity, motion consistency and detail richness by taking the preliminary fusion image as an input; dynamically adjusting a cross-modal attention weight and a gate value according to the evaluation result; and obtaining an optimized final fusion feature based on the dynamically adjusted cross-modal attention weight and the gate value. The application enables the fusion image to be free of dislocation when a target moves rapidly.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent vision and image fusion, and particularly relates to a dual-light image fusion method of an intelligent camera and a dual-light integrated intelligent camera. BACKGROUND

[0002] In the current mainstream intelligent camera market, single-mode cameras still dominate the market, that is, either only visible light sensors are equipped or only infrared sensors are equipped, and there is a significant technical gap and application limitation for a truly integrated camera product that realizes dual-mode deep fusion of "infrared + visible light", which mainly reflects in the following aspects:

[0003] Scene adaptability defects of single-mode cameras:

[0004] A visible light camera relies on ambient light, and in low light or complex weather conditions such as night, fog, rain, and smoke, the image signal-to-noise ratio decreases sharply, and target details are lost seriously. An infrared camera can capture target thermal radiation information and keep the target outline visible in harsh environments, but it lacks key details such as texture and color, and cannot meet the fine identification requirements (such as license plate recognition and face recognition). When the two types of cameras are used alone, it is difficult to achieve reliable visual perception in all-weather and all-scenarios.

[0005] Limitations of "pseudo-fusion" of existing dual-light devices:

[0006] A few devices on the market that claim to be "dual-light" mostly use a simple splicing scheme of "infrared + visible light" sensors:

[0007] There is no unified synchronization mechanism on the hardware, and the time error of the dual-sensor acquisition can reach milliseconds, resulting in obvious misalignment of the two images in dynamic scenes;

[0008] On the software side, "fusion" is achieved only by simple splicing (such as left and right split screens) or pixel-level weighted superposition, without considering the modal characteristic differences between infrared and visible light (infrared emphasizes edge structure, and visible light emphasizes texture and color), and the fused image often has problems such as edge blurring and detail conflict;

[0009] Special hardware acceleration is not designed for dual-mode in the preprocessing link, and problems such as blind pixels, non-uniformity of infrared images, and noise and color deviation of visible light images are not completely solved, further deteriorating the fusion quality. SUMMARY

[0010] The present application provides a dual-light image fusion method of an intelligent camera and a dual-light integrated intelligent camera to solve the above problems.

[0011] The present application is implemented by the following technical solutions:

[0012] A dual-light image fusion method of an intelligent camera, characterized by comprising:

[0013] synchronously collecting a raw infrared image and a raw visible light image through an infrared sensor and a visible light sensor;

[0014] obtaining a dual-channel preprocessed image through dual-channel parallel preprocessing of the raw infrared image and the raw visible light image, wherein the dual-channel preprocessed image comprises a first infrared image and a first visible light image obtained after preprocessing;

[0015] calculating an optical flow field through an improved optical flow algorithm with a dual-modal feature weighting constraint and dynamic confidence guidance optimization on the dual-channel preprocessed image ; based on the optical flow field extracting multi-scale features of the dual-channel preprocessed image after motion compensation through a multi-branch convolutional network, and embedding optical flow information into each layer feature of the dual-channel preprocessed image after motion compensation, to generate an infrared motion-aware feature set and a visible light motion-aware feature set , wherein, , respectively represent the i-th layer infrared motion-aware feature and the i-th layer visible light motion-aware feature, and n represents the number of scale levels;

[0016] based on the infrared motion-aware feature set and the visible light motion-aware feature set, fusing the optical flow field to calculate a cross-modal attention weight and ; introducing an optical flow field motion intensity index to calculate a gating value , dynamically adjusting a multi-scale feature fusion weight, generating a final fusion feature , and generating a preliminary fusion image based on the final fusion feature ;

[0017] obtaining evaluation results including clarity C, motion consistency M and detail richness evaluation indicators for the preliminary fusion image; and dynamically adjusting the cross-modal attention weight and the gating value according to the evaluation results;

[0018] based on the dynamically adjusted cross-modal attention weight and the gating value , obtaining an optimized final fusion feature, and obtaining an optimized preliminary fusion image; and performing detail optimization on the optimized preliminary fusion image through a generative adversarial network (GAN) to generate a final fusion image.

[0019] As an optimization, the dual-channel parallel preprocessing of the raw infrared image and the raw visible light image comprises:

[0020] The original infrared image obtained by the infrared sensor is sequentially subjected to blind element correction, non-uniformity correction and definition enhancement processing to obtain a first infrared image;

[0021] The original visible light image output by the visible light sensor is sequentially subjected to detector calibration, bad pixel correction, auto focus, auto exposure, auto white balance, gamma correction, denoising and sharpening processing to obtain a YUV domain image, which is a first visible light image; meanwhile, the preprocessed infrared image is received, and the two-way preprocessed images are uniformly subjected to encoding push stream, OSD superposition and forwarding processing.

[0022] As an optimization, an improved optical flow algorithm with double-modal feature weighting constraint and dynamic confidence guidance optimization is used to calculate the optical flow field of the two-way preprocessed images The specific process is as follows:

[0023] The infrared edge structure features of the first infrared image are extracted and the visible light texture color features of the first visible light image ;

[0024] The infrared modal adaptation weight and the visible light modal adaptation weight are calculated based on the signal-to-noise ratio of the first infrared image and the contrast of the first visible light image , wherein, , wherein, SNR represents the signal-to-noise ratio of the first infrared image, and CR represents the contrast of the first visible light image;

[0025] An initial optical flow field optimization model containing double-modal feature weighting constraint is constructed and solved, and the formula of the initial optical flow field is represented as: ; represents the initial optical flow field; represents the first infrared image, is the pixel coordinate; represents the first visible light image; , respectively represent the gray value of the first infrared image and the first visible light image at the pixel ; is the component of the optical flow vector in the x and y directions; and ρ represents the robust loss function; , is the regularization parameter; represents the weight coefficient of the optical flow motion term; is the infrared edge structure feature of the first infrared image; is the visible light texture color feature of the first visible light image; an infrared edge feature after motion; a visible light texture color feature after motion; , ;

[0026] According to the initial optical flow field , a dynamic confidence C(x, y) of each pixel point is calculated, and the formula is: ; wherein, is a standard deviation obtained by statistical analysis on the gray difference between the first infrared image and the first visible light image ; , the value closer to 1 indicates that the optical flow estimation of the pixel point is more reliable;

[0027] For low-confidence pixels of the dynamic confidence , a 3x3 neighborhood optical flow weighted mean optimization is adopted;

[0028] The optimized optical flow vector component is substituted into the initial optical flow field to replace the optical flow vector of the low-confidence pixel of the dynamic confidence , and a final optical flow field is obtained.

[0029] As an optimization, after motion compensation of the dual-channel preprocessed images based on the optical flow field , multi-scale features of the motion-compensated dual-channel preprocessed images are extracted through a multi-branch convolutional network, and the optical flow information is embedded in each layer feature of the motion-compensated dual-channel preprocessed images, and the specific process of generating a feature set with motion perception is as follows:

[0030] According to the motion vector (u(x, y), v(x, y)) of each pixel (x, y) in the optical flow field , the pixel is mapped to a spatial position aligned with the visible light image through a bilinear interpolation algorithm, and a second infrared image after motion compensation is obtained , and a second visible light image after reverse motion compensation of the first visible light image is obtained;

[0031] A structure-symmetric dual-branch convolutional network is used to extract multi-layer features of the second infrared image and the second visible light image , and obtain a multi-scale feature set of the infrared image and a multi-scale feature set of the visible light image , wherein n is the number of scale layers;

[0032] According to the optical flow field Downsampling and channel compression are performed to generate optical flow feature maps that match the features of each layer. , This represents the optical flow feature map corresponding to the i-th layer features;

[0033] The optical flow feature map is compared with the multi-scale feature set of the infrared image. Multiscale feature sets of visible light images By adding them together, we obtain the feature sets for infrared motion sensing and the feature sets for visible light motion sensing, where... , , This is the optical flow weighting coefficient.

[0034] As an optimization, cross-modal attention weights and The specific formula is:

[0035] ;

[0036] This represents the characteristics of infrared band motion sensing at the i-th scale. This represents the optical flow feature map corresponding to the i-th layer features; d represents the feature of visible light band motion perception at the i-th scale; d is the feature dimension.

[0037] ;

[0038] This represents the characteristics of visible light band motion perception at the i-th scale. This represents the characteristics of infrared band motion sensing at the i-th scale.

[0039] The threshold value The specific formula is:

[0040] ;

[0041] in, The Sigmoid activation function maps the input to the (0,1) interval, as shown in the formula. ; The weight matrix is ​​a learnable matrix. For learnable bias terms; This represents the characteristics of infrared band motion sensing at the i-th scale. This represents the characteristics of visible light band motion perception at the i-th scale. The i-th scale represents the intensity of optical flow motion, and GlobalAvgPool is a global average pooling operation.

[0042] As optimization, the multi-scale feature fusion weight is dynamically adjusted to generate the final fusion feature , and a preliminary fusion image is generated based on the final fusion feature The specific process is as follows:

[0043] For the infrared band motion perception feature under the i-th scale , the visible light band motion perception feature , combined with the cross-modal attention weight and feature fusion to obtain intermediate fusion features , , denotes element-wise multiplication;

[0044] The intermediate fusion features are dynamically weighted to obtain the final fusion features : , denotes convolution operation;

[0045] For the generated under each scale, n is the number of scale levels, and a feature pyramid fusion strategy is adopted to upsample and add the fusion features of different scales element by element to obtain multi-scale fusion features : where Upsample is the upsample operation;

[0046] The multi-scale fusion features are mapped to the image space through the convolution layer and the activation function to generate the preliminary fusion image .

[0047] As optimization, for the preliminary fusion image, the evaluation results of evaluation indexes including sharpness C, motion consistency M and detail richness are obtained; and the cross-modal attention weight and the gating value are dynamically adjusted according to the evaluation results The specific process is as follows:

[0048] The gradient of each pixel of the preliminary fusion image is calculated using the Tenengrad gradient function, and the average of all pixel gradients is obtained to obtain the sharpness C; the structural similarity SSIM of the preliminary fusion image and the second infrared image and the preliminary fusion image and the second visible light image is calculated respectively, and then the motion vector of the optical flow field is combined to obtain the motion consistency M; the entropy function is used to calculate the detail richness

[0049] The sharpness C, motion consistency M and detail richness The weighted combination is performed to obtain a comprehensive evaluation score S, and an adjustment amount is constructed based on a preset optimal score range ;

[0050] The cross-modal attention weight And According to the adjustment amount Adjustment:

[0051] ;

[0052] ;

[0053] Wherein, is a learning rate, , The gradient of the comprehensive evaluation score S to , ;

[0054] According to the adjustment amount Adjust the gate value :

[0055] ;

[0056] Wherein, The gradient of the comprehensive evaluation score S to the gate value .

[0057] As optimization, the preliminary fused image after optimization is optimized in detail through a generative adversarial network GAN, and the specific process of generating the final fused image is:

[0058] The generative adversarial network GAN is constructed, and the generative adversarial network GAN includes a generator and a discriminator, wherein the loss function of the generator is , The adversarial loss is , wherein D(*) is the authenticity probability output by the discriminator, is a randomly generated false image, is a candidate fused image output by the generator from the preliminary fused image; the loss function of the discriminator is , is an ideal fused image artificially labeled in the training stage.

[0059] The application also discloses a dual-light integrated intelligent camera for executing the intelligent camera dual-light image fusion method, which comprises a dual-light image acquisition module, a preprocessing module, an optical flow calculation module, a feature fusion module, an image optimization module and a control module integrated on the same component.

[0060] The dual-optical image acquisition module is configured to synchronously acquire an infrared image and a visible light image of the same scene, and output an original infrared image and an original visible light image.

[0061] The preprocessing module is connected with the dual-optical image acquisition module, and is configured to pre-process the original infrared image and the original visible light image, and output dual-channel pre-processed images.

[0062] The optical flow calculation module is connected with the preprocessing module, and is configured to calculate an optical flow field based on the dual-channel pre-processed images by using an optical flow algorithm containing dual-modal feature weighting constraint and dynamic confidence optimization. ;

[0063] The feature fusion module is connected with the preprocessing module and the optical flow calculation module, respectively, and is configured to perform motion compensation on the dual-channel pre-processed images based on the optical flow field , extract multi-scale features of the motion-compensated dual-channel images by using a multi-branch convolutional network, embed motion information of the optical flow field into features of each layer, generate an infrared motion-aware feature set and a visible light motion-aware feature set, and perform multi-scale fusion on the dual-modal motion-aware feature sets based on cross-modal attention weights and dynamic gate values, to output a preliminary fusion image. ;

[0064] The image optimization module is connected with the feature fusion module, and is configured to perform detail optimization on the preliminary fusion image by using a generative adversarial network, to output a final fusion image.

[0065] The control module is connected with the dual-optical image acquisition module, the preprocessing module, the optical flow calculation module, the feature fusion module and the image optimization module, respectively, and is configured to coordinate working time sequences of the dual-optical image acquisition module, the preprocessing module, the optical flow calculation module, the feature fusion module and the image optimization module, to realize real-time acquisition, processing, fusion and output of dual-optical images.

[0066] ​As optimization, the dual-optical image acquisition module comprises an infrared sensor and a visible light sensor, the preprocessing module comprises an infrared image preprocessing unit and a visible light image preprocessing unit, the infrared image preprocessing unit is arranged in an FPGA, is used for receiving an original infrared image transmitted by the infrared sensor and outputting a first infrared image after preprocessing the original infrared image, and the visible light preprocessing unit is arranged in an AI chip, is used for receiving an original visible light image transmitted by the visible light sensor and obtaining a first visible light image after preprocessing the original visible light image, and simultaneously, receiving the first infrared image, and through a light flow calculation module, a feature fusion module, an image optimization module and a control module arranged in the AI chip, the first infrared image and the first visible light image are fused, a light flow field is calculated, multi-scale features are extracted / embedded and image optimization is performed to obtain a final fusion image.

[0067] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0068] The present application breaks through the limitation of conventional dual-optical fusion through the cooperative technical scheme of light flow optimization-dynamic fusion-closed loop adjustment-GAN enhancement, and has the following effects:

[0069] Significant improvement in motion consistency in dynamic scenes: conventional technology is prone to fusion ghosting due to inaccurate light flow, the present application uses dual-mode constraint + dynamic confidence algorithm to accurately calculate light flow in S3, embeds light flow in features in S4 and fuses through bidirectional attention guidance, and adjusts parameters according to the motion consistency index in S5, so that the fusion image has no displacement when the target moves quickly (such as traffic flow and running pedestrians), and the motion consistency is higher than that of conventional methods.

[0070] Dual reservation of details and modal advantages in complex environments: traditional fusion often loses one to save the other (such as losing infrared thermal targets or visible light textures), the present application uses dynamic gating to balance dual-modal features in S4, and enhances details on the preliminary image after optimization through GAN in S6, so that even in low-light and weak-texture scenes (such as night buildings and overcast vegetation), the fusion image clearly shows infrared thermal targets and restores wall textures, leaf veins and other details, and the detail richness is improved.

[0071] Self-adaptive fusion in all scenes without manual parameter adjustment: conventional technology relies on fixed parameters, and the effect decreases sharply when the scene changes (such as day-night switching and sudden changes in light), the present application dynamically adjusts the attention weight and the gating value through multi-index evaluation in S5, and the GAN in S6 further optimizes the adaptation, realizing self-adaptive fusion from low light to strong light, static to dynamic scenes without manual intervention, and adapting to the full-time and full-scene use requirements of intelligent cameras. BRIEF DESCRIPTION OF DRAWINGS

[0072] The drawings described herein are intended to provide further understanding of the embodiments of the present application, form a part of the present application, and do not constitute a limitation on the embodiments of the present application. In the drawings:

[0073] Figure 1 A flow chart of a dual-light image fusion method of an intelligent camera of the present application;

[0074] Figure 2 A hardware connection schematic diagram of a dual-light integrated intelligent camera of the present application;

[0075] Figure 3 An envelope diagram of the overall structure of a dual-light integrated intelligent camera of the present application. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below in combination with embodiments and drawings, the illustrative embodiments of the present application and the description thereof are only used to explain the present application, and do not constitute a limitation on the present application.

[0077] Embodiment 1 of the present application provides a dual-light image fusion method of an intelligent camera, as shown in the following formula (I), which comprises the following steps: Figure 1

[0078] S1, synchronously collecting original infrared images and original visible light images by infrared sensors and visible light sensors.

[0079] In some embodiments, a synchronization mechanism combining time stamp synchronization and hardware triggering is adopted, a uniform time reference is assigned to the infrared sensors and the visible light sensors, a hardware triggering signal is generated by an FPGA to control the dual-sensor synchronous image acquisition, and the acquisition time error is controlled within microseconds; at the same time, a high-speed preprocessing pipeline is constructed by using the ping-pong buffer technology, so that the FPGA and the AI chip can start the preprocessing task in parallel during the image acquisition process.

[0080] Here, the same time standard, such as a high-precision clock signal, is assigned to the infrared and visible light sensors, an electrical signal instruction is generated by an FPGA (programmable chip) to trigger the two sensors to take pictures at the same time, so as to ensure that the two sensors collect images at the same instant, with an error controlled within microseconds, thus solving the image misalignment problem in a dynamic scene, and the acquisition and processing are carried out in parallel by using the ping-pong buffer technology, which avoids the waiting time caused by full storage before processing, greatly improves the efficiency, and ensures a high-speed and smooth image processing link.

[0081] S2, performing dual-path parallel preprocessing on the original infrared images and the original visible light images to obtain dual-path preprocessed images, wherein the dual-path preprocessed images comprise a first infrared image and a first visible light image obtained after preprocessing.

[0082] ​In some embodiments, the original infrared image obtained by the infrared sensor is sequentially subjected to blind element correction, non-uniformity correction and definition enhancement processing to obtain a first infrared image.

[0083] The original visible light image output by the visible light sensor is sequentially subjected to detector calibration, bad point correction, automatic focusing, automatic exposure, automatic white balance, gamma correction, denoising and sharpening processing to obtain a YUV domain image, which is the first visible light image; at the same time, the first infrared image is received, and the two-way preprocessed images are uniformly subjected to encoding push stream, OSD superposition and forwarding processing.

[0084] The FPGA executes a hardware acceleration algorithm to sequentially perform blind element correction, non-uniformity correction and definition enhancement processing on the original data of the infrared sensor, thereby solving the problems of blind elements, non-uniformity and poor definition of the original data, and the processed data is transmitted to an AI chip through a data transmission interface (such as PCIe, MIPI, BT1120, etc., and the MIPI interface is used in the present application); the RK3588 AI chip performs detector calibration, bad point correction, automatic focusing, automatic exposure, automatic white balance, gamma correction, denoising and sharpening processing on the original image output by the visible light sensor to obtain a YUV domain image; at the same time, the infrared preprocessed image input through the data transmission interface (such as PCIe, MIPI, BT1120, etc., and the MIPI interface is used in the present application) is received, and the two-way images are uniformly subjected to encoding push stream, OSD superposition and forwarding processing.

[0085] Blind element correction: a 3x3 neighborhood mean replacement method is used to identify and replace blind elements with a gray value deviating from the neighborhood mean by ±3σ;

[0086] Non-uniformity correction: single-point correction and two-point correction (-20℃, 50℃ blackbody calibration) are combined with scene adaptive correction (dynamic adjustment of gain compensation for temperature drift).

[0087] Definition enhancement: image edge details are enhanced through Laplacian sharpening convolution kernel. Blind element correction and non-uniformity correction are prior art and will not be described here.

[0088] Bad point correction: bad points are identified through dark current detection and bright point detection, and are replaced by 4-neighborhood linear interpolation;

[0089] Automatic exposure: based on histogram brightness analysis, the exposure time (1ms-100ms) and gain (1-32dB) are dynamically adjusted to maintain the average brightness of the image at 128±10 gray values;

[0090] Encoding push stream: H.265 encoding format is used, and the code rate is dynamically adjusted between 1Mbps and 20Mbps.

[0091] S3, using an improved optical flow algorithm with dual-modal feature weighting constraint and dynamic confidence guidance optimization to calculate the optical flow field of the dual-channel preprocessed images ; based on the optical flow field After motion compensation on the dual-channel preprocessed images, multi-scale features of the motion-compensated dual-channel preprocessed images are extracted respectively by a multi-branch convolutional network, and optical flow information is embedded into each layer of features of the motion-compensated dual-channel preprocessed images to generate a feature set with motion perception and wherein, , respectively represent the i-th layer of infrared features with motion perception and the i-th layer of visible light features with motion perception, and n represents the number of scale levels.

[0092] In the multi-scale feature extraction process, the image will be down-sampled to different degrees and other operations to obtain features of multiple scales (levels), and n is the total number of these scale levels, which is used to describe the number of scale levels contained in the dual-modal feature set (infrared feature set with motion perception, visible light feature set with motion perception).

[0093] In some embodiments, the improved optical flow algorithm with dual-modal feature weighting constraint and dynamic confidence guidance optimization is used to calculate the optical flow field of the dual-channel preprocessed images The specific process is as follows:

[0094] S3.1, extract the infrared edge structure features of the first infrared image and the visible light texture color features of the first visible light image .

[0095] This step is to extract core features from the infrared preprocessed image and the visible light preprocessed image respectively:

[0096] Infrared feature extraction: using an edge detection operator (such as Canny operator) to extract edge features from the infrared preprocessed image , which reflects the edge contour information of the target in the infrared image.

[0097] Visible light feature extraction: using a texture extraction algorithm (such as local binary pattern LBP) to extract texture features from the visible light preprocessed image , which reflects the texture detail information in the visible light image.

[0098] S3.2, calculate the infrared modal adaptive weight and the visible light modal adaptive weight wherein, , wherein, a signal-to-noise ratio of the first infrared image, a contrast of the first visible light image.

[0099] obtained by calculating the ratio of the first infrared image signal power to noise power, obtained by calculating the difference degree of luminance in different regions of the visible light image, the two weights are used to measure the importance of infrared features and visible light features in the calculation of the optical flow field.

[0100] S3.3, construct an initial optical flow field optimization model containing bimodal feature weighting constraints and solve it, the formula of the initial optical flow field is represented as: represents the initial optical flow field; represents the first infrared image, is the pixel coordinate; represents the first visible light image; , respectively represent the gray value of the first infrared image and the first visible light image at the pixel ; is the component of the optical flow vector in the x and y directions; represents a robust loss function, which is used to reduce the influence of outliers on the calculation of the optical flow field; , is a regularization parameter, which is used to constrain the smoothness of the optical flow field; represents a constraint term weight coefficient, which is used to adjust the influence degree of the bimodal constraint term; is the infrared edge structure feature of the first infrared image; is the visible light texture color feature of the first visible light image; is the infrared edge feature after motion (after displacement u, v); is the image texture feature after motion (after displacement u, v); , ;

[0101] This step is to construct an initial optical flow field optimization model containing bimodal feature weighting constraints, taking pixel gray difference, optical flow field smoothness and bimodal feature difference as optimization objectives.

[0102] This part is the basic constraint term of the optical flow based on pixel gray, through the robust loss function , minimize the gray difference between the infrared image and the visible light image after optical flow displacement , capture the pixel-level motion relationship.

[0103] ​ This is a regularization term used to constrain the smoothness of the optical flow field. and It is a regularization parameter to prevent excessive oscillations in the optical flow field that do not conform to the actual motion laws. , Let u and v represent the gradients, respectively. It represents the square of the L2 norm, used to calculate the "length" of the gradient vector (by summing the squares of each component of the gradient to measure the overall size of the gradient).

[0104] It is a bimodal constraint term. and These are the mode adaptation weights for infrared and visible light, which respectively constrain infrared edge features. Visible light texture features Its post-movement characteristics , The differences are addressed by utilizing dual-modal features (infrared edges and visible light textures) to improve the accuracy of optical flow calculation in dual-light scenarios.

[0105] S3.4, Based on the initial optical flow field Calculate the dynamic confidence score C(x,y) for each pixel using the following formula: ;in, For the first infrared image And the first visible light image The standard deviation is obtained after statistical analysis of the grayscale differences between them; The closer the value is to 1, the more reliable the optical flow estimation for that pixel.

[0106] S3.5, Regarding dynamic confidence levels For low-confidence pixels, a 3×3 neighborhood optical flow weighted mean is used for optimization, as shown in the formula:

[0107] , ;in, Let (x, y) be the 3×3 neighborhood of pixel (x, y). , For the optimized optical flow vector components, This represents the confidence threshold, which is typically set to 0.3.

[0108] S3.6 Substitute the optimized optical flow vector components into the initial optical flow field to replace the original dynamic confidence level. The optical flow vectors of low-confidence pixels are used to obtain the final optical flow field. .

[0109] In some embodiments, based on the optical flow field After motion compensation of the dual-path preprocessed images, multi-scale features of the motion-compensated dual-path preprocessed images are extracted through a multi-branch convolutional network, and optical flow information is embedded into each layer feature of the motion-compensated dual-path preprocessed images to generate a specific process of a feature set with motion perception:

[0110] S3.7, for the first infrared image , according to the motion vector (u(x,y), v(x,y)) of each pixel (x,y) in the optical flow field , the pixel is mapped to the spatial position aligned with the visible light image through a bilinear interpolation algorithm to obtain a motion-compensated second infrared image : where BilinearInterp is a bilinear interpolation function for processing non-integer coordinate pixel value calculation, respectively, are the coordinates of the first infrared image before motion compensation; for the first visible light image , reverse motion compensation (correcting the deviation caused by sensor displacement or scene motion) is performed to obtain a motion-compensated second visible light image , ensuring that the dual light images are accurately aligned in the spatial coordinate system.

[0111] It should be noted that the first visible light image , in the motion compensation process is . Similarly, the first infrared image , in the motion compensation process is .

[0112] S3.8, a structure-symmetrical dual-branch convolutional network (infrared branch and visible light branch share network architecture but independent parameters) is used to extract multi-scale features from the motion-compensated dual-path images:

[0113] A structure-symmetrical dual-branch convolutional network is used to extract multi-layer features of the second infrared image and the second visible light image , to obtain a multi-scale feature set of the infrared image and a multi-scale feature set of the visible light image , where n is the number of scale layers.

[0114] For example, infrared branch feature extraction:

[0115] The second infrared image is input into the infrared branch, and features are extracted through hierarchical convolution operation:

[0116] Shallow features (1-2 layers): 3x3 convolution kernel (stride 1, channel number 64), capturing local thermal radiation gradient (e.g. gray difference between target and background) of infrared image, output feature ;

[0117] Middle features (3-4 layers): 3x3 convolution kernel (stride 2, channel number 128), capturing target edge contour (e.g. vehicle contour, human contour), output feature ;

[0118] Deep features (5-6 layers): 5x5 convolution kernel (stride 2, channel number 256), capturing target semantic information (e.g. "dynamic / static target" attribute), output feature ;

[0119] Generate infrared multi-scale feature set ;

[0120] Visible light branch feature extraction:

[0121] Input the second visible light image to the visible light branch (consistent with the infrared branch structure), extract features:

[0122] Shallow features (1-2 layers): capture texture details (e.g. clothing texture, road texture), output ;

[0123] Layer features (3-4 layers): capture shape features (e.g. vehicle window structure, limb shape), output ;

[0124] Deep features (5-6 layers): capture class information (e.g. "vehicle / pedestrian" semantics), output ;

[0125] Generate visible light multi-scale feature set .

[0126] S3.9, scale adaptation and feature embedding of optical flow information, including scale conversion of optical flow features and motion perception feature fusion.

[0127] Scale conversion of optical flow features: downsample and channel compress the optical flow field (size HxWx2) to generate optical flow feature maps matching each layer of features:

[0128] For the i-th layer of features (size ), downsample to , compress the channel number to through 1x1 convolution to obtain adaptive features , Height of the i-th layer feature map, i.e., the number of pixels in the vertical direction, Width of the i-th layer feature map, i.e., the number of pixels in the horizontal direction, Channel of the i-th layer feature map, which reflects the number of feature categories or dimensions contained in the feature map, and different channels may correspond to different feature patterns or attributes, Optical flow feature map corresponding to the i-th layer feature.

[0129] Motion-aware feature fusion:

[0130] Embedding optical flow information into dual-channel features through element-level addition, which retains the feature dimension while injecting motion attributes:

[0131] Infrared band motion-aware feature: is the optical flow weight coefficient, with a value range of 0.1-0.3;

[0132] Visible band motion-aware feature:

[0133] S3.10, generating a set of motion-aware features:

[0134] Finally output two sets of features:

[0135] Infrared band motion-aware feature set:

[0136] Visible band motion-aware feature set:

[0137] The above features contain both the inherent attributes of dual-light images (infrared thermal radiation / visible light texture) and the motion state information reflected by the optical flow field (target motion direction, speed), providing dual information support of content-motion for subsequent cross-modal fusion.

[0138] In this step, the image features after motion, such as are extracted from the motion-compensated images (including the second infrared image and the second visible light image), and this process forms a strict logical closed loop with motion compensation, with the following specific associations:

[0139] 1. Motion compensation provides a spatial alignment basis for feature extraction:

[0140] The original dual-channel preprocessed images ( ) may have spatial misalignment due to target motion, for example, the same vehicle is located to the left in the infrared image and to the right in the visible light image. ​​​​​

[0141] by optical flow field After motion compensation, the obtained (infrared image after motion compensation) and visible light image after motion compensation) achieve spatial alignment, and the pixel positions of the same target in the dual light images are consistent.

[0142] 2. The features after motion are directly derived from the images after motion compensation

[0143] Infrared edge features after motion : extracted from the infrared image after motion compensation , reflecting the infrared target edge after motion correction (spatially aligned with the corresponding target edge in the visible light image);

[0144] Visible light texture features after motion : extracted from the visible light image after motion compensation , reflecting the visible light target texture after motion correction (spatially aligned with the corresponding target texture in the infrared image).

[0145] Features before motion are extracted from the uncompensated preprocessed images, i.e. the first infrared image and the first visible light image, which may cause mismatch between dual modal features due to spatial misalignment, for example, the infrared edge corresponding area and the visible light texture corresponding area are not the same target.

[0146] Features after motion are extracted from the images after motion compensation, ensuring that the dual modal features are spatially aligned to the same target, providing a reliable feature matching basis for subsequent dual modal feature weighting constraints (optical flow field optimization) and cross-modal fusion.

[0147] For ease of understanding, let's briefly describe the order of S3:

[0148] Step 1: Calculate the initial optical flow field based on the original preprocessed images.

[0149] First, use the dual-channel preprocessed images (first infrared image , first visible light image ) that have not been motion compensated, calculate the initial optical flow field by using an improved optical flow algorithm that includes dual modal constraints and dynamic confidence optimization;

[0150] At this point, the optical flow field is a preliminary estimate of the pixel motion relationship between the dual-channel images, used to guide the first motion compensation.

[0151] Step 2: Perform the first motion compensation based on the initial optical flow field.

[0152] Using the initial optical flow field obtained in the first step Motion compensation is performed on the dual-path preprocessed image to obtain a first motion-compensated image ).

[0153] The purpose of this step is to preliminarily align the dual-path images in space and reduce pixel misplacement caused by motion.

[0154] Third step (optional): Iterative optimization (adopted when precision is improved).

[0155] If the accuracy of the optical flow field is high, iteration can be performed:

[0156] Based on the first motion-compensated image (which has been preliminarily aligned), the optical flow field is recalculated to obtain an optimized optical flow field (the image has been aligned, so the calculation of the optical flow field is less disturbed by motion misplacement and has higher precision);

[0157] The optimized optical flow field is used to perform a second motion compensation to obtain more accurately aligned images ).

[0158] In the optimization model of the initial optical flow field, (u, v) is a non-zero feature vector derived from the original features + optical flow hypothesis and used to constrain the optical flow field.

[0159] This step is one of the innovations of the present application and has the following effects:

[0160] Traditional optical flow algorithms are easily affected by large modal differences (infrared thermal radiation vs. visible light texture) and weak texture areas (such as infrared smooth surfaces and visible light low-contrast areas) in dual optical scenes, leading to incorrect optical flow estimation.

[0161] The present application introduces a dual-modal feature weighting constraint: combining infrared edge structure features and visible light texture color features, and using the complementarity of dual modalities (infrared excels at capturing thermal target edges, and visible light excels at providing texture details), the optical flow field can be more accurately constrained in edge and texture areas. For example, the thermal edge of a vehicle in an infrared image and the texture details of the vehicle in a visible light image jointly guide the optical flow calculation, which can reduce the misjudgment of the optical flow caused by insufficient single-modal information.

[0162] Through dynamic confidence-guided optimization, 3x3 neighborhood weighted correction is adopted for low-confidence pixels in the optical flow field (such as weak texture and fuzzy edges), solving the problem of insufficient robustness of traditional optical flow in complex scenes (such as fast target motion and scene light abrupt change). For example, when the local texture of the visible light image is blurred due to insufficient light, the optical flow vector of the current pixel can be corrected by more reliable optical flow information in the neighborhood to ensure the continuity of the overall optical flow field.

[0163] And, the dynamic confidence of the present application has the effects of precise modeling of the dual-mode gray scale difference and efficient evaluation of the reliability of optical flow.

[0164] Specifically embodied in the present application, is the gray scale difference between the first infrared image and the first visible light image statistically analyzed. This dual-mode joint statistics can accurately depict the inherent gray scale difference of the dual-mode in the current scene. For example, in a daytime scene, the visible light texture is rich, the infrared thermal radiation is uniform, which will reflect this difference; in a night scene, the infrared thermal difference is obvious, and the visible light is dark, which will also be adaptively adjusted.

[0165] In the formula is the gray scale residual error of the infrared pixel and the visible light pixel after motion under the action of the optical flow (u, v). At the same time, the dual-mode is constrained, and the spatial correspondence between the infrared thermal target and the visible light texture target is used, so that the confidence can better reflect the reliability of the optical flow under the dual-mode.

[0166] In the form of , it is ensured that:

[0167] Smoothness: Small changes in gray scale residual error will not cause the confidence to fluctuate dramatically (avoiding false positives);

[0168] Discrimination: When the residual error is much larger than , the confidence quickly approaches 0 (accurately identifying unreliable optical flow).

[0169] If other methods use a linear function (such as C=1-residual error / threshold), it is easy to appear that a slight residual error leads to a sharp decrease in confidence or a large residual error but a still high confidence, and the discrimination ability of the optical flow reliability is insufficient.

[0170] And, the confidence is directly calculated based on the initial optical flow field (the optical flow (u, v) comes from the initial optical flow field), which can reflect the quality of the optical flow estimation in real time. For example, when the optical flow field is accurately estimated in the edge area, the dual-mode gray scale residual error is small, and the confidence is close to 1; when the optical flow field is incorrectly estimated in the weak texture area, the residual error is large, and the confidence is close to 0. This real-time nature makes the optimization of subsequent low-confidence optical flow (such as neighborhood weighted correction) more targeted.

[0171] The dynamic confidence of the application can guide the motion compensation process: for pixels with high confidence, directly use the optical flow field for motion compensation; for pixels with low confidence, use the neighborhood weighted optimization optical flow and then compensate. Compared with other methods of indiscriminate compensation (or simple filtering of low-confidence optical flow), the application can make the motion compensation of the dual-light image more accurate, providing better spatial alignment for the input of subsequent feature fusion.

[0172] After motion compensation, the feature fusion of the dual-light image relies on the spatial correspondence of the dual-modal features. The confidence of the application ensures that the optical flow involved in the fusion is reliable, which in turn allows the infrared edge features and visible light texture features to be more accurately matched in space, improving the effect of cross-modal attention weight calculation and multi-scale feature fusion, and ultimately making the motion consistency and detail richness of the fused image better.

[0173] Based on the optical flow field, the dual-channel preprocessed images are motion compensated, and then multi-scale features are extracted and embedded with optical flow information to generate (infrared band motion-aware features) and (visible light band motion-aware features), realizing the alignment of dual-modal features in the motion dimension. For example, after motion compensation, the edge features of the moving target in the infrared image are accurately matched with the texture features of the target in the visible light image in spatial position, providing a feature basis for subsequent cross-modal fusion (step S4) with motion consistency and spatial alignment.

[0174] The embedding of optical flow information allows dual-modal features to expand from containing only image inherent properties (infrared thermal radiation, visible light texture) to containing motion state information (target motion direction, speed). This dual-dimensional feature of content + motion allows the subsequent cross-modal attention mechanism (step S4) to more accurately measure which regions of dual-modal features are more matched in motion and more worthy of fusion, improving the relevance of the fusion.

[0175] The accurate optical flow field and motion-aware dual-modal feature set generated in this step are the core input for subsequent steps (cross-modal fusion, GAN detail optimization), directly affecting the quality of the final fused image:

[0176] The motion-aware features provide motion-aware features for the cross-modal attention weight calculation of step S4, allowing the attention weight to better reflect the importance of features in motion-consistent regions;

[0177] The preliminary fusion image provided for the GAN detail optimization of step S6 already has good motion consistency and modal complementarity, so the GAN only needs to enhance the detail texture level, reducing the difficulty of subsequent optimization.

[0178] S4, based on the infrared band motion-aware feature set and the visible light band motion-aware feature set, fusing the optical flow field Calculate cross-modal attention weights and The gating value is calculated by introducing an index of the intensity of optical flow field motion. Dynamically adjust the multi-scale feature fusion weights to generate the final fused features. And based on the final fusion features, a preliminary fused image is generated. .

[0179] In some embodiments, cross-modal attention weights and The specific formula is:

[0180] ;

[0181] This represents the attention weight from infrared to visible light at the i-th scale. This represents the attention weight from visible light to infrared at the i-th scale; This represents the characteristics of infrared band motion sensing at the i-th scale. This represents the optical flow feature map corresponding to the i-th layer features; d represents the feature of visible light band motion perception at the i-th scale; d is the feature dimension.

[0182] ;

[0183] This represents the characteristics of visible light band motion perception at the i-th scale. This represents the characteristics of infrared band motion sensing at the i-th scale.

[0184] After convolution and other operations, features will form feature vectors or feature maps with a certain dimension. d is used to describe the scale of these features in terms of dimension. This parameter is used when scaling the dimension of features during operations such as calculating attention weights.

[0185] In some embodiments, the gate value The specific formula is:

[0186] ;

[0187] in, is the gate value at the i-th scale, ranging from (0,1), used to control the weight ratio when fusing features at multiple scales (the closer the value is to 1, the greater the weight of the corresponding scale feature fusion). The Sigmoid activation function maps the input to the (0,1) interval, as shown in the formula. ; It is a learnable weight matrix used to perform a linear transformation on the features after global average pooling, adjusting the influence of different feature channels; is a learnable bias term to compensate the result of linear transformation and increase the expressive ability of the model. represents the i-th scale infrared band motion-aware feature (multi-scale feature extracted from the motion-compensated infrared image). represents the i-th scale visible band motion-aware feature (multi-scale feature extracted from the motion-compensated visible image). represents the i-th scale optical flow motion intensity feature, specifically the mean length of optical flow vector (reflecting the intensity of pixel motion at this scale, the more intense the motion, the greater the mean length).

[0188] GlobalAvgPool represents the global average pooling operation, which averages the input feature map (size HxWxC) in the spatial dimension (H and W) to obtain a feature vector of dimension (1x1xC), which is used to compress spatial information and retain channel-level features.

[0189] The spatial information of the dual-band motion-aware feature and the optical flow intensity feature is compressed by global average pooling, and then linearly transformed ( and ) and Sigmoid activated to generate the gating value The gating value can dynamically adjust the weight of multi-scale feature fusion according to the semantic information of dual-band features + the intensity of optical flow motion, so that the fusion process is more adaptive to different motion scenes (such as motion-intensive areas, more emphasis on optical flow-guided feature fusion).

[0190] In some embodiments, the multi-scale feature fusion weight is dynamically adjusted to generate the fused feature , and the preliminary fusion image is generated based on the fused feature.

[0191] S4.1, multi-scale feature cross-modal attention fusion. For the i-th scale infrared band motion-aware feature , visible band motion-aware feature , combined with cross-modal attention weight and to obtain intermediate fusion feature , , represents element-wise multiplication, which weights the dual-modal features by cross-modal attention weight, realizing bidirectional fusion of infrared-guided visible light and visible light-guided infrared.

[0192] S4.2, introduce gating value to adjust fusion weight. Combined with the gating value , the intermediate fusion feature after cross-modal fusion The dynamic weight adjustment is performed to obtain a final fusion feature : .

[0193] Convolution operation is represented, and the convolution operation is used for generating a reference feature, The fusion proportion of the cross-modal fusion feature and the convolution reference feature is controlled, so that the fusion process is more adaptive to the characteristics of the optical flow motion and the dual optical features at the current scale.

[0194] S4.3, multi-scale feature fusion.

[0195] The generated fusion features at different scales are upsampled and element-wise added to obtain multi-scale fusion features , n is the number of scale levels, and a feature pyramid fusion strategy is used to upsample and element-wise add the fusion features at different scales to obtain multi-scale fusion features : , wherein Upsample is an upsample operation, and the features at different scales are unified to the size of the finest scale, such as the first scale, and the size .

[0196] S4.4, generating a preliminary fusion image.

[0197] The multi-scale fused features are mapped to the image space through a convolution layer and an activation function (such as Sigmoid) to generate a preliminary fusion image : The convolution operation converts high-dimensional features into image dimensions (such as 3-channel RGB images), and the Sigmoid activation function maps the pixel values to the interval [0, 1] to obtain a preliminary fusion image result.

[0198] This step realizes the precise complementary fusion of motion-aware driven dual modal features, and the specific effects are as follows:

[0199] 1. Conventional dual optical fusion technology (such as weighted average, simple feature splicing) is mostly based on the "static scene assumption", directly superimposes the dual modal features in space, and does not consider the pixel motion misplacement problem in dynamic scenes, resulting in motion inconsistency phenomena such as "ghosting" and "edge blur" in the fusion image.

[0200] The motion information (u, v components) of the optical flow field is embedded into the dual modal features , , the motion-aware , is generated, so that the features not only contain image content, but also carry the pixel motion state; the cross-modal attention weight and The calculation is directly related to optical flow features, tilting the weights towards regions with consistent motion (such as areas where the edges of moving targets align with textures in the dual-modality model). In dynamic scenes (such as moving vehicles and pedestrians), the fused features maintain spatiotemporal continuity, avoiding the problem of "misaligned fusion of infrared thermal targets and visible light textures" in conventional techniques, thus achieving preliminary image fusion. The motion consistency index M improved by more than 30%.

[0201] 2. Conventional cross-modal fusion often uses unidirectional attention (such as using only visible light features to guide infrared fusion) or fixed weight fusion, which cannot fully utilize the complementary advantages of dual modes (infrared is good at capturing thermal targets, while visible light is good at presenting detailed textures), resulting in fused images either losing the saliency of thermal targets or lacking texture details.

[0202] This invention is achieved through (Infrared-guided visible light) and The bidirectional interaction (visible light guiding infrared) allows the dual-modal features to "dominate" each other; for example, the infrared thermal target region is characterized by... Enhanced weighting, visible light texture rich areas are... Enhanced weights; gate values Combining optical flow intensity characteristics The method dynamically adjusts the cross-modal fusion weights in areas of high motion (high optical flow intensity) and retains more single-modal advantages in static areas. The fused features can simultaneously preserve the thermal target contrast of infrared images (such as the distinction between pedestrians and background at night) and the texture details of visible light images (such as clothing texture and road markings). The detail richness index H is improved by 25% compared with conventional methods, solving the fusion dilemma of sacrificing one aspect for the other.

[0203] 3. Conventional techniques often perform feature fusion at a single scale (such as the size of the input image), making it difficult to take into account both global semantics (such as the overall outline of the target) and local details (such as edge sharpness), resulting in fused images that are either globally blurred or locally distorted.

[0204] This invention obtains multi-scale features (i=1,2,...,n) through a multi-branch convolutional network, corresponding to high-resolution details to low-resolution semantics; and fuses features at each scale. Upsampling and aggregation are performed to accurately fuse features at different levels (such as large-scale infrared thermal regions and small-scale visible textures) at their corresponding scales. Preliminary fused image. It maintains clarity in both global structures (such as building outlines) and local details (such as window textures), with a clarity index C (Tenengrad mean) that is 20% higher than that of single-scale fusion methods, especially in complex scenes (such as urban building complexes).

[0205] In summary, this step, through a collaborative design of motion-aware embedding, bidirectional attention interaction, dynamic gating adjustment, and multi-scale aggregation, overcomes the limitations of conventional dual-light fusion technology in terms of dynamic consistency, modal complementarity, and scale adaptability. It provides high-quality basic features for subsequent GAN detail optimization, ultimately enabling the fused image to simultaneously meet the application requirements of intelligent cameras (such as security monitoring and autonomous driving visual perception) that require continuous motion, prominent thermal targets, and clear textures.

[0206] S5. From the preliminary fused image, obtain the following parameters: sharpness (C), motion consistency (M), and detail richness. The evaluation results of the evaluation metrics; and the dynamic adjustment of the cross-modal attention weights and gating values ​​based on the evaluation results. .

[0207] The specific process is as follows:

[0208] S5.1 Calculate the evaluation indicators:

[0209] Sharpness C calculation: The Tenengrad gradient function is used to calculate the sharpness of the initially fused images. Calculate the gradient for each pixel, and then calculate the average of all pixel gradients. The formula is: Where H and W represent the height and width of the initially fused image, , These are the gradients of the initial fused image in the x and y directions (calculated using the Sobel operator), respectively. The larger the value, the clearer the image.

[0210] Motion consistency M calculation: This involves calculating the initial fused image and the motion-compensated dual-path preprocessed image. Structural similarity (SSIM), combined with optical flow field The motion vectors are weighted, and the formula is:

[0211] ;

[0212] Weights are applied to infrared modes. Weights are adapted for visible light modes; Indicates the initial fused image Second infrared image Structural similarity between them; Indicates the initial fused image Second visible light image Structural similarity between them; The weighting coefficients represent the motion terms of optical flow. , represents the motion vector component of the optical flow field at (x,y). A larger value indicates better motion consistency.

[0213] Richness of detail Calculation: The entropy function is used to measure the uniformity of gray-level distribution and the richness of detail in the initially fused image. The formula is:

[0214] ;

[0215] This is used to initially fuse the probability of pixels with a gray value of k appearing in the image; the larger the value, the richer the image details.

[0216] S5.2 Establish a mapping relationship between evaluation results and parameter adjustments.

[0217] The aforementioned sharpness C, motion consistency M, and detail richness By performing weighted combination, we obtain the comprehensive evaluation score S:

[0218] ;

[0219] , , These are respectively: Clarity (C), Motion Consistency (M), and Detail Richness. The weighting is based on the comprehensive evaluation score S and the preset optimal score range (e.g., ...). To determine the deviation, an adjustment amount is constructed. , This is the actual comprehensive evaluation score. These represent the lower and upper limits of the optimal score range, respectively. ; This represents the mean of the optimal score range.

[0220] S5.3 Dynamically adjust cross-modal attention weights.

[0221] Cross-modal attention weights and According to the adjustment amount Adjustments will be made:

[0222] ;

[0223] ;

[0224] in, For learning rate, , The comprehensive evaluation scores S are respectively , The gradient (calculated through backpropagation) makes the adjusted attention weights more conducive to improving the overall evaluation score;

[0225] S5.4, also based on the adjustment amount For the gate value Adjustments will be made:

[0226] ;

[0227] in, To comprehensively evaluate the score S for the gate value The gradient of the gate value can be adjusted to more accurately control the multi-scale feature fusion weights, thereby optimizing the fusion effect.

[0228] The core effect of this step is to construct a closed-loop feedback mechanism for fusion effect and parameter optimization. Compared with the static parameters and one-step fusion mode of conventional dual-light fusion technology, it achieves adaptability and accuracy in the fusion process, which is specifically reflected in the following three aspects:

[0229] 1. The parameters such as cross-modal attention weights and gating values ​​of conventional dual-light fusion technology are mostly fixed during the training phase or preset by human experience, which cannot cope with dynamic scenarios such as sudden changes in lighting (such as day / night switching) and changes in motion state (such as target acceleration / deceleration).

[0230] This invention calculates sharpness (C), motion consistency (M), and detail richness. The preliminary fused image is comprehensively evaluated from three dimensions: image quality, motion matching, and detail preservation. Based on the gradient of the evaluation results, such as... , Cross-modal attention weights Gating value Dynamic fine-tuning is performed. In tunnel entrance / exit scenarios (sudden changes in lighting), conventional methods result in visible light overexposure / infrared blurring in the fused image due to static parameters. However, step S5 of this invention can adjust parameters in real time, improving the sharpness C of the fused image by 35%, motion consistency M by 40%, and detail richness. An increase of 25%.

[0231] 2. The core challenge of dual-modal fusion is that each mode has its own defects (e.g., infrared has low contrast under strong light, and visible light has high noise under weak light). Conventional technologies lack a mechanism for sensing and compensating for "modal defects".

[0232] This invention utilizes motion consistency M (bimodal motion matching degree) and detail richness. (Texture / edge integrity) indirectly senses whether the current modality is reliable (e.g., missing visible light details in low light). It will decrease); if a certain modal defect (such as visible light detail) is perceived. If the level is low, then increase the attention weight of the other modality (e.g., increase the level of attention). Let the infrared feature guide more fusion). In the night weak light scene, the visible light image noise is large, the texture is fuzzy, the S5 of the application reduces the visible light attention weight, enhances the dependence of the gating value on the infrared feature, makes the recognition degree of the fusion image of the hot target improve by 40%, and at the same time, the visible light noise is suppressed, and the detail reservation is more complete.

[0233] 3. The fusion process of the conventional technology is forward without feedback (no intermediate evaluation from acquisition to fusion output), once a link is wrong (such as flow calculation deviation), the error will be transmitted to the final result and cannot be corrected.

[0234] The evaluation of the preliminary fusion image of the application can timely find the problems of flow guide error and feature fusion imbalance; through multiple iteration adjustment (such as optimizing the parameters once every 10 frames), the cross-modal attention and the gating value gradually approach the optimal solution of the current scene. In the scene of multiple targets and complex motion (such as vehicle flow interweaving), the conventional method causes target ghosting and boundary blur of the fusion image due to the accumulation of flow error, and the step S5 of the application improves the motion consistency M from 0.65 (conventional method) to 0.92 through iterative optimization, close to the motion matching degree of ideal fusion.

[0235] The S5 of the application realizes the upgrade of the dual-light fusion from passive adaptation scene to active optimization effect through multi-dimensional evaluation + gradient optimization + iterative feedback, solves the pain points of poor scene adaptability, difficult compensation of modal defects and no correction of errors of the conventional technology, provides precise, robust and adaptive fusion features for subsequent GAN detail optimization, and finally enables the intelligent camera to output high-quality fusion images in all time periods and all scenes.

[0236] S6, the cross-modal attention weight and the gating value based on dynamic adjustment Obtain the optimized final fusion feature and the optimized preliminary fusion image; and then perform detail optimization on the optimized preliminary fusion image through a generative adversarial network (GAN) to generate a final fusion image.

[0237] In some embodiments, the specific implementation process of S6 is as follows:

[0238] S6.1, generating an optimized final fusion feature.

[0239] S6.1.1, re-fusing the feature based on the adjusted parameters. The cross-modal attention weight , and the gating value are dynamically adjusted, and multi-scale feature fusion is performed again:

[0240] ;

[0241] ;

[0242] Optimized multi-scale fusion features are obtained ;

[0243] S6.1.2, the optimized multi-scale fusion features are refined The same feature pyramid fusion strategy as step S3 is adopted to generate aggregated optimized features :

[0244] ;

[0245] S6.1.3, the optimized preliminary fusion image is generated The optimized preliminary fusion image is obtained by mapping to the image space through the convolution layer and the activation function : .

[0246] S6.2, a generative adversarial network (GAN) is constructed for detail optimization.

[0247] GAN network structure design:

[0248] Generator: adopts U-Net architecture, input is the optimized preliminary fusion image , captures multi-scale details (such as edge sharpening, texture enhancement) through encoder-decoder structure, and outputs the candidate fusion image after detail optimization .

[0249] Discriminator: adopts convolutional neural network (CNN), input is the candidate fusion image and the motion compensated dual light image , output is the image authenticity probability (judges whether the candidate image is close to the ideal fusion image).

[0250] GAN loss function design:

[0251] Generator loss consists of adversarial loss and content loss: .

[0252] For adversarial loss, measure the ability of the generator to deceive the discriminator, adopt binary cross entropy loss, where D(*) is the authenticity probability output by the discriminator, is a randomly generated fake image.

[0253] For content loss, constrain the content consistency of the candidate image and the dual light image, adopt structural similarity (SSIM) loss: ; denotes the candidate fusion image and the second infrared image structural similarity between the first and second visible light images; representing the structural similarity between the candidate fused image and the second visible light image.

[0254] the discriminator loss is: , is the ideal fused image artificially labeled in the training phase.

[0255] S6.3, generating the final fused image.

[0256] GAN training and iterative optimization. Alternately train the generator and the discriminator: the generator minimizes to improve the detail generation capability, and the discriminator minimizes to improve the authenticity recognition capability, until the network converges (the loss function tends to be stable).

[0257] output the final fused image. After training, the generator receives the optimized preliminary fused image , and outputs the final fused image with enhanced details , which retains:

[0258] thermal target saliency of the infrared image;

[0259] texture details of the visible light image;

[0260] motion consistency guided by the optical flow field.

[0261] The core effect of this step is to further realize detail enhancement and realism enhancement on the basis of precise feature fusion. Compared with conventional dual-light fusion techniques (such as direct weighting and traditional CNN fusion), it breaks through the limitations of missing details and insufficient realism in the fused image, which is specifically reflected in the following three aspects:

[0262] 1. Conventional dual-light fusion techniques (such as cross-modal attention fusion and multi-scale feature fusion) can achieve modal complementation, but still have deficiencies in preserving fine-grained details (such as texture and edge sharpness) (such as edge blurring of infrared thermal imaging and texture loss under low light in visible light).

[0263] The generator using the U-Net architecture can extract multi-scale details (such as small target edges and subtle textures) from the optimized preliminary fused image ; the discriminator takes the motion-compensated dual-light image as the real sample, and constrains the candidate image output by the generator ​Closer to the real scene in texture, edge, color. In the building wall texture, vegetation vein details and other scenes, the conventional fusion image may lose details due to modal defects (such as infrared without texture, visible light weak light blur), and the final fusion image generated by S6 of the application can clearly restore these fine-grained details, and the detail richness is improved by 30%-40% compared with the conventional method.

[0264] 2. When the conventional GAN is used for image fusion, it is mostly end-to-end static training, that is, the parameters of the generator and the discriminator are fixed, and it cannot adapt to dynamic scene changes (such as target motion and light fluctuation).

[0265] First, the cross-modal attention weight and the gate value are adjusted by S5 to obtain a more accurate preliminary fusion image; the GAN further optimizes the details on the basis of the accurate preliminary fusion image, rather than directly generating without difference on the original dual-light images. In the moving target (such as a running person) scene, the conventional GAN may generate a ghosting + detail distorted fusion result due to the motion misplacement of the input image; and the S6 of the application cooperates the process of dynamically parameterizing the preliminary fusion (solves the motion misplacement) → GAN optimizes the details, so that the motion consistency M of the fusion image is improved by 25%, and the details (such as clothing wrinkles) are more realistic.

[0266] 3. The core contradiction of dual-light fusion is the large modal difference (infrared thermal radiation vs. visible light texture) and the demand of human eyes for realism, and the conventional technology either excessively deviates to a certain modal (such as too high proportion of infrared leading to strong pseudo-color feeling of the image), or the fused image is unnatural and like a synthetic image.

[0267] The adversarial loss of the GAN in the application makes the generated image closer to the real scene in overall visual style (such as natural color and texture distribution), and the content loss is constrained by structural similarity (SSIM) to ensure the content consistency (such as infrared thermal target position and visible light texture area) of the generated image and the dual-light image.

[0268] The final fusion image not only retains the saliency of the infrared thermal target (such as high-light display of the fire heat source), but also has the natural texture and color of the visible light image (such as the real texture of the environment around the heat source), and in the subjective visual evaluation, the realism score is improved by 40% compared with the conventional method, which is more in line with the expectation of human eyes for real scene imaging.

[0269] In summary, the steps S3 (optical flow field calculation and motion perception feature generation), S4 (cross-modal feature fusion and preliminary fusion image generation), S5 (evaluation and dynamic parameter adjustment), and S6 (GAN detail optimization to generate the final fusion image) in the application realize the synergistic effect of 1+1>2 through the design of motion perception throughout, dynamic closed-loop optimization and multi-modal deep collaboration, which is specifically reflected in the following three dimensions:

[0270] I. Motion-aware whole-process penetration: from optical flow to features to images.

[0271] S3's motion information foundation: through bimodal feature weighting constraint and dynamic confidence optimization, generate accurate and robust optical flow field, and embed optical flow information into bimodal features, making features have motion awareness (not only containing image content, but also carrying pixel motion state);

[0272] S4's motion-guided fusion: the calculation of cross-modal attention weight and gate value is directly related to the motion-aware feature, making the fusion process tilt towards motion-consistent areas (such as the edges and textures of moving targets that align in bimodal fusion), and the preliminary fusion image naturally has motion continuity;

[0273] S5's motion quality evaluation: the motion consistency M index directly quantifies the motion matching degree between the fusion image and the bimodal preprocessed image, providing motion dimension feedback for parameter adjustment;

[0274] S6's motion detail enhancement: GAN optimizes details with motion-compensated bimodal images as real samples, ensuring that detail enhancement does not destroy motion consistency.

[0275] Synergistic effect: motion awareness is throughout the whole process from optical flow calculation to the final image, making the fusion image maintain motion continuity and detail richness in dynamic scenes (such as moving objects), solving the pain point of conventional bimodal fusion that cannot balance motion and detail (conventional methods either have motion consistency but blurred details, or have rich details but motion misalignment).

[0276] II. Dynamic closed-loop iterative optimization: from evaluation to adjustment to optimization.

[0277] S4 outputs the preliminary fusion image: as the evaluation object of S5;

[0278] S5 multi-dimensional evaluation (C, M, H f ) and parameter adjustment: based on the gradient of the evaluation results, dynamically optimize the cross-modal attention weight and gate value to generate a more accurate preliminary fusion image;

[0279] S6 takes the optimized preliminary image as input: GAN further enhances details on this basis, rather than directly processing the original bimodal image.

[0280] Synergistic effect: form a closed-loop iteration of preliminary fusion → evaluation adjustment → re-fusion → re-evaluation adjustment → GAN optimization, making each round of fusion more accurate than the previous one. For example, the parameters adjusted by S5 make the fusion of S4 more suitable for the current scene, and S6 optimizes details on a more accurate preliminary image, making the final fusion image's clarity, motion consistency, and detail richness improve by 40%~60% compared to step-by-step fusion without a closed loop.

[0281] Three, multi-modal deep synergy: from features to weights to generation.

[0282] S3's dual-modal feature extraction and optical flow embedding: provides S4 with dual-modal features with motion awareness, making the cross-modal fusion have precise feature ingredients;

[0283] S4's bidirectional cross-modal attention and dynamic gating: lets the dual-modal features dominate each other (infrared guides visible light, visible light guides infrared), achieving complementary advantages;

[0284] S5's multi-dimensional evaluation: quantifies the effect of dual-modal fusion from three dimensions of clarity, motion consistency, and detail richness, providing a multi-modal balance basis for parameter adjustment;

[0285] S6's GAN multi-modal constraint: through content loss (SSIM), ensures that the generated image retains both the thermal target saliency of infrared and the texture details of visible light.

[0286] Synergistic effect: dual-modal from feature layer to fusion layer to evaluation layer to generation layer, achieving deep synergy at all levels, rather than single-modal dominance or simple superposition. For example, in a low-light + moving target scene, the thermal target of infrared guides the fusion weight, and the texture details of visible light are enhanced through GAN, and the final image can both see the target clearly (infrared advantage) and distinguish the details clearly (visible light advantage).

[0287] The four steps are designed through motion-aware throughout the whole process, dynamic closed-loop iterative optimization, and multi-modal full-level synergy, which deeply bind the accuracy of optical flow calculation, the complementarity of cross-modal fusion, the adaptability of parameter adjustment, and the details of GAN generation. Ultimately, the fusion image in motion consistency, detail richness, and visual realism, three key indicators, all far exceed the effect of single optimization or simple superposition of each step, truly realizing the technical breakthrough of 1+1>2.

[0288] Embodiment 2 discloses a dual-light integrated intelligent camera for performing the intelligent camera dual-light image fusion method of embodiment 1, including a dual-light image acquisition module, a preprocessing module, an optical flow calculation module, a feature fusion module, an image optimization module, and a control module integrated on the same component;

[0289] The dual-light image acquisition module is configured to synchronously acquire infrared images and visible light images of the same scene, and output raw infrared images and raw visible light images;

[0290] The preprocessing module is connected with the dual-light image acquisition module and configured to preprocess the raw infrared images and raw visible light images, and output dual-channel preprocessed images;

[0291] The optical flow calculation module is connected with the preprocessing module and is configured to calculate an optical flow field based on the dual-path preprocessed images by using an optical flow algorithm containing dual-modal feature weighting constraint and dynamic confidence optimization. ;

[0292] The feature fusion module is connected with the preprocessing module and the optical flow calculation module respectively, and is configured to perform motion compensation on the dual-path preprocessed images, extract multi-scale features of the dual-path images after motion compensation by using a multi-branch convolutional network, embed motion information of the optical flow field into each layer of features, and generate a dual-modal feature set with motion perception (an infrared dual-modal feature set with motion perception and a visible light dual-modal feature set with motion perception). perform multi-scale fusion on the dual-modal feature set with motion perception based on cross-modal attention weight and dynamic gating value, and output a preliminary fusion image. ;

[0293] The image optimization module is connected with the feature fusion module, and is configured to perform detail optimization on the preliminary fusion image by using a generative adversarial network, and output a final fusion image.

[0294] The control module is connected with each of the above modules respectively, and is configured to coordinate the working time sequence of each module, and realize real-time acquisition, processing, fusion and output of dual-light images.

[0295] In some embodiments, the dual-light image acquisition module includes an infrared sensor and a visible light sensor, the preprocessing module includes an infrared image preprocessing unit and a visible light image preprocessing unit, the infrared image preprocessing unit is arranged in an FPGA, is configured to receive an original infrared image transmitted by the infrared sensor, and output a first infrared image after preprocessing the original infrared image, and the visible light preprocessing unit is arranged in an AI chip, is configured to receive an original visible light image transmitted by the visible light sensor, and output a first visible light image after preprocessing the original visible light image, and simultaneously, receive the first infrared image, and perform fusion, optical flow field calculation, multi-scale feature extraction / embedding and image optimization on the first infrared image and the first visible light image by using the optical flow calculation module, the feature fusion module, the image optimization module and the control module arranged in the AI chip, to obtain a final fusion image.

[0296] As Figure 2 ​​As shown, the dual-light integrated intelligent camera in this embodiment breaks through the technical bottleneck of traditional separated architecture such as CCD visible light detector core + IR infrared detector core + image information processing board, and realizes the integration, miniaturization, low cost and AI intelligence of the sensing device through the integrated architecture of dual-sensor + computing base (FPGA + AI). While having the image data acquisition and processing capability, the system enables core AI technologies such as target tracking, high-speed autonomous target identification, image fusion, etc. The system adopts the integrated architecture of dual-sensor + FPGA + AI, the infrared sensor resolution is 640x512@50fps, the visible light sensor resolution is 2448x2048@50fps, and it supports simultaneous input of infrared video signal and visible light video signal. The FPGA realizes the collection and preprocessing of infrared images, the AI chip realizes the collection and preprocessing of visible light images, as well as the post-processing functions such as infrared and visible light image data fusion, target identification tracking, etc. The processed image data can be output to the display through the CML digital image output interface or the PAL analog video image interface, or transmitted to other devices through the network port.

[0297] The main chips are composed of one piece of ZYNQ JFMQL15T485 of Shanghai Fudan Microelectronics, RK3588 of Ruiyi Micro, H6412C1SD of Hikvision, GMAX3405 of Changguochen, and other peripheral interface chips.

[0298] The dual-sensor + computing base (FPGA + AI) integrated architecture is adopted, and through low-power design and integrated heat dissipation technology, a high-efficiency heat dissipation solution is provided to reduce power consumption and realize software and hardware collaborative optimization.

[0299] IR infrared sensors are easily affected by heat sources, thereby affecting the imaging effect. The computing base of the dual-light integrated intelligent camera has strict requirements on power density, and low-power design is required. However, low-power design increases the complexity of hardware design and the difficulty of component selection, and may sacrifice part of the performance, so it is necessary to overcome the realization of low power consumption while ensuring performance and reducing design complexity.

[0300] The dual-light integrated intelligent camera is mainly composed of an RK3588 board, an infrared sensor board, an FPGA interface board, an FPGA board, a television sensor board, and a supporting shell and cover plate. The infrared sensor board, the FPGA interface board, the FPGA board, the television sensor board, and the four boards are connected through flexible boards to form a module structure diagram as shown in Figure 3 .

[0301] The infrared image preprocessing unit is completed by the FPGA, which processes the problems of blind elements, non-uniformity, poor definition, etc. of the original data of the infrared sensor through hardware acceleration algorithm.

[0302] The visible light image preprocessing unit is completed by an AI chip, the RK3588 is used as an image processing unit, an ISP (image signal processing) unit on the chip is used to preprocess the image output by the visible light sensor, including detector calibration, bad point correction, automatic focusing / exposure / white balance, gamma correction, denoising, sharpening and the like, and the processed image is in the YUV domain, and the YUV image is encoded and pushed, OSD, forwarded for subsequent processing; the infrared image input from the data transmission interface is encoded and pushed, OSD, forwarded and the like.

[0303] The multi-source image fusion processing module group (including an optical flow calculation module, a feature fusion module and an image optimization module) is completed by an AI chip, the preprocessed infrared image and visible light image are informationally fused and complemented to further improve the image quality and provide more dimensional and more complete feature information for a target recognition and tracking algorithm, and the target recognition and tracking accuracy is improved.

[0304] Through the above setting, the application has the following advantages:

[0305] High-resolution high-frame-rate video acquisition input: both the infrared sensor and the visible light sensor adopt a high-resolution high-frame-rate sensor, clearer and higher-quality original images can be obtained, more abundant detail information is provided for subsequent multi-source image data fusion processing, and thus the image analysis effect (for example, analysis results such as target detection, matching, recognition and tracking) is further improved, the intelligent operation ability and working reliability of the camera are improved, and the demand of complex task scenarios such as long-distance, low-light, strong clutter and small target and fast maneuvering target is met;

[0306] Edge intelligent computing base based on a national FPGA+AI heterogeneous architecture: the intelligent camera is provided with large-scale image data processing and complex visual computing capability to meet the real-time processing demand brought by high-resolution, high-frame-rate and double-light video image data, and reduce image processing delay. Through day and night perception mode intelligent switching, scene adaptability full coverage is realized: during the day, the full-color information of the visible light image is relied on to accurately restore the details such as target color and texture; at night or in a weak light environment, the high-contrast characteristic of the infrared image ensures that the target outline and structure are clear and distinguishable. The advantages of the two types of images are complementary, the system is endowed with stable perception ability under complex light conditions such as strong light, backlight and low light, and a real-time and robust all-weather visual solution is provided.

[0307] High-resolution double-light integrated intelligent camera computing base, through the integrated architecture of double-sensor+computing base (FPGA+AI), the traditional separated architecture technical bottleneck of CCD visible light detector core+IR infrared detector core+image information processing board is broken, and the perception device is realized to be integrated, miniaturized, low-cost and AI intelligent.

[0308] The above detailed description of the specific embodiments of the present application is provided for the purpose of further explaining the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of intelligent camera dual light image fusion, characterized in that, The method comprises the following steps: Synchronously collecting original infrared images and original visible light images through infrared sensors and visible light sensors; Performing double-path parallel preprocessing on the original infrared images and the original visible light images to obtain double-path preprocessed images, wherein the double-path preprocessed images comprise a first infrared image and a first visible light image obtained after preprocessing; Using an improved optical flow algorithm with double-modal feature weighting constraint and dynamic confidence guidance optimization to calculate an optical flow field based on the double-path preprocessed images; After motion compensation of the double-path preprocessed images based on the optical flow field, extracting multi-scale features of the motion-compensated double-path preprocessed images through a multi-branch convolutional network, and embedding optical flow information into each layer of features of the motion-compensated double-path preprocessed images to generate an infrared motion-aware feature set and a visible light motion-aware feature set; Based on the infrared motion-aware feature set and the visible light motion-aware feature set, fusing the optical flow field to calculate cross-modal attention weights; introducing an optical flow field motion intensity index to calculate a gating value, dynamically adjusting a multi-scale feature fusion weight, generating a final fusion feature, and generating a preliminary fusion image based on the final fusion feature; Obtaining evaluation results including clarity, motion consistency and detail richness evaluation indexes based on the preliminary fusion image; dynamically adjusting the cross-modal attention weights and the gating value based on the evaluation results; Based on the dynamically adjusted cross-modal attention weights and the gating value, obtaining an optimized final fusion feature, and obtaining an optimized preliminary fusion image; Further optimizing the optimized preliminary fusion image through a generative adversarial network (GAN) to generate a final fusion image.

2. The method of claim 1, wherein, The double-path parallel preprocessing of the original infrared images and the original visible light images comprises the following steps: Performing blind pixel correction, non-uniformity correction and clarity enhancement processing on the original infrared image obtained through an infrared sensor in sequence to obtain a first infrared image; Performing detector calibration, bad pixel correction, automatic focusing, automatic exposure, automatic white balance, gamma correction, denoising and sharpening processing on the original visible light image output by the visible light sensor in sequence to obtain a YUV domain image, which is a first visible light image; simultaneously receiving the preprocessed infrared image, and uniformly performing encoding push stream, OSD superposition and forwarding processing on the double-path preprocessed images.

3. The method of claim 1, wherein, An improved optical flow algorithm with bimodal feature weighting constraint and dynamic confidence guidance optimization is used to calculate the optical flow field of the dual-path preprocessed image The specific process is as follows: extracting infrared edge structure features of the first infrared image visible light texture color features of the first visible light image ; An infrared modality adaptation weight is computed based on a signal-to-noise ratio of the first infrared image and a contrast of the first visible light image and a visible light modality adaptation weight wherein , wherein denotes a signal-to-noise ratio of the first infrared image, denotes a contrast of the first visible light image; An initial optical flow field optimization model including a dual-modality feature weighting constraint is constructed and solved, and a formula representation of the initial optical flow field is as follows: ; represents an initial optical flow field; represents a first infrared image, is a pixel coordinate; represents a first visible light image; , respectively represent gray scale values of the first infrared image and the first visible light image at a pixel ; is a component of an optical flow vector in an x and y direction; and ρ represents a robust loss function; , is a regularization parameter; represents a weight coefficient of an optical flow motion term; is an infrared edge structural feature of the first infrared image; is a visible light texture color feature of the first visible light image; is a moved infrared edge feature; is a moved visible light texture color feature; , ; According to the initial optical flow field , a dynamic confidence C(x, y) of each pixel point is calculated, and a formula is as follows: ; wherein, is a standard deviation obtained by performing statistical analysis on a gray difference between the first infrared image and the first visible light image ; , the value closer to 1 indicates that the optical flow estimation of the pixel point is more reliable. For dynamic confidence Low confidence pixels are optimized by 3x3 neighborhood optical flow weighted mean. substituting the original optical flow vectors of the low-confidence pixels in the initial optical flow field with the optimized optical flow vector components to obtain the final optical flow field .

4. The method of claim 1, wherein, based on the optical flow field After motion compensation is performed on the dual-path preprocessed images, multi-scale features of the motion-compensated dual-path preprocessed images are respectively extracted through a multi-branch convolutional network, and optical flow information is embedded into features of each layer of the motion-compensated dual-path preprocessed images, to generate a specific process of a feature set with motion perception. to the first infrared image , according to the motion vector (u(x,y),v(x,y)) of each pixel (x,y) in the optical flow field , the pixel is mapped to the spatial position aligned with the visible light image by a bilinear interpolation algorithm, to obtain a motion compensated second infrared image , the first visible light image is reversely motion compensated to obtain a second visible light image ; A structure-symmetrical double-branch convolutional network is used to extract multi-layer features of the second infrared image and the second visible light image to obtain a multi-scale feature set of the infrared image and a multi-scale feature set of the visible light image , wherein n is the number of scale layers. down-sampling and channel compression to generate light flow feature maps matched with the features of each layer down-sampling and channel compression to generate light flow feature maps matched with the features of each layer denotes the light flow feature map corresponding to the i-th layer feature​ The optical flow feature maps are added respectively with a multi-scale feature set of the infrared image and a multi-scale feature set of the visible light image to obtain an infrared motion-aware feature set and a visible light motion-aware feature set, wherein, is a flow weight coefficient.​​​​ 5. The method of claim 4, wherein, Cross-modal attention weights and The specific formula is: ； a feature representing infrared band motion perception at the i-th scale; a feature representing the optical flow feature map corresponding to the i-th layer feature; a feature representing visible light band motion perception at the i-th scale; d is the feature dimension; ； a feature representing visible band motion perception at the i-th scale; a feature representing infrared band motion perception at the i-th scale; The gating value The specific formula is: ； where, is a Sigmoid activation function that maps the input to the interval (0, 1) with the formula ; is a learnable weight matrix, is a learnable bias term; represents the feature of infrared band motion perception at the i-th scale; represents the feature of visible light band motion perception at the i-th scale; represents the feature of the motion intensity of optical flow at the i-th scale, and GlobalAvgPool is a global average pooling operation.

6. The method of claim 5, wherein, The dynamic adjustment multi-scale feature fusion weight is generated to generate a final fusion feature and generate a preliminary fusion image based on the final fusion feature The specific process is that: infrared band motion-aware feature at the i-th scale , visible light band motion-aware feature , combined with cross-modal attention weights and feature fusion to obtain intermediate fusion features , , denotes element-wise multiplication; For the intermediate fusion features The final fused features are obtained by dynamically adjusting the weights. : , Indicates the convolution operation; The generated features at each scale are fused to obtain multi-scale fusion features , n is the number of scale levels, and a feature pyramid fusion strategy is adopted to up-sample and element-wise add the fusion features at different scales to obtain multi-scale fusion features : , wherein Upsample is an up-sampling operation The multi-scale fusion features are mapped to an image space by a convolutional layer and an activation function, to generate a preliminary fusion image. The multi-scale fusion features are mapped to an image space by a convolutional layer and an activation function, to generate a preliminary fusion image. .

7. The method of claim 4, wherein the method further comprises: For the preliminary fusion image, obtain an evaluation result of an evaluation index including definition C, motion consistency M and detail richness The evaluation result of the evaluation index; dynamically adjust the cross-modal attention weight and the gating value according to the evaluation result The specific process is as follows: A Tenengrad gradient function is used to calculate the gradient of each pixel of the preliminary fused image and the mean value of all pixel gradients is obtained to get the definition C. By calculating the structural similarity SSIM of the preliminary fusion image and the second infrared image and the preliminary fusion image and the second visible light image respectively, and combining the motion vector of the optical flow field to obtain motion consistency M; and using an entropy function to calculate the detail richness of the preliminary fusion image ; The clarity C, the motion consistency M, the detail richness The weighted combination is performed to obtain a comprehensive evaluation score S, and an adjustment amount is constructed based on a preset optimal score range ; to the cross-modal attention weights and according to the adjustment amount adjustment: ； ； wherein, is the learning rate, , are the gradients of the composite evaluation score S with respect to , , respectively. According to the adjustment amount adjusting the gating value : ； wherein, is the gradient of the gating value for the comprehensive evaluation score S.

8. The method of claim 1, wherein, The specific process of further optimizing the optimized preliminary fusion image through a generative adversarial network (GAN) to generate a final fusion image comprises the following steps: A generative adversarial network (GAN) is constructed, the GAN including a generator and a discriminator, wherein a loss function of the generator is , is an adversarial loss, wherein D(*) is a probability of authenticity of a discriminator output, is a randomly generated false image, is a candidate fusion image output by the generator through a preliminary fusion image; and a loss function of the discriminator is , is an ideal fusion image artificially labeled in a training stage.

9. A dual-optical integrated smart camera for performing the dual-optical image fusion method of any one of claims 1-8, wherein, The dual-light image acquisition module, the preprocessing module, the optical flow calculation module, the feature fusion module, the image optimization module and the control module are integrated on the same component: The dual-light image acquisition module is configured to synchronously collect infrared images and visible light images of the same scene, and output original infrared images and original visible light images; The preprocessing module is connected with the dual-light image acquisition module, and is configured to preprocess the original infrared images and the original visible light images, and output double-path preprocessed images; The optical flow calculation module is connected with the preprocessing module, and is configured to calculate an optical flow field based on the dual-path preprocessing image by using an optical flow algorithm containing dual-mode feature weighting constraint and dynamic confidence optimization ; The feature fusion module is connected with the preprocessing module and the optical flow calculation module respectively, and is configured to fuse the motion information of the optical flow field and the features of the preprocessed image based on the optical flow field Motion compensation is performed on the dual-path preprocessed image; multi-scale features of the motion-compensated dual-path image are extracted through a multi-branch convolutional network, and the motion information of the optical flow field is embedded into each layer of features to generate an infrared band motion-aware feature set and a visible light band motion-aware feature set Motion compensation is performed on the dual-path preprocessed image; multi-scale features of the motion-compensated dual-path image are extracted through a multi-branch convolutional network, and the motion information of the optical flow field is embedded into each layer of features to generate an infrared band motion-aware feature set and a visible light band motion-aware feature set Based on the cross-modal attention weight and the dynamic gating value, the dual-modal feature set with motion perception is multi-scale fused to output a preliminary fusion image ; The image optimization module is connected with the feature fusion module, and is configured to generate the preliminary fusion image through a generative adversarial network perform detail optimization, and output a final fusion image. The control module is connected with the dual-light image acquisition module, the preprocessing module, the optical flow calculation module, the feature fusion module and the image optimization module respectively, and is used for coordinating the working time sequence of the dual-light image acquisition module, the preprocessing module, the optical flow calculation module, the feature fusion module and the image optimization module, so as to realize real-time acquisition, processing, fusion and output of dual-light images.

10. The dual-optical integrated smart camera of claim 9, wherein, The dual-light image acquisition module comprises an infrared sensor and a visible light sensor, the preprocessing module comprises an infrared image preprocessing unit and a visible light image preprocessing unit, the infrared image preprocessing unit is arranged in an FPGA, is used for receiving an original infrared image transmitted by the infrared sensor, and outputs a first infrared image after preprocessing the original infrared image, and the visible light processing unit is arranged in an AI chip, is used for receiving an original visible light image transmitted by the visible light sensor, and obtains a first visible light image after preprocessing the original visible light image, and simultaneously, receives the first infrared image, and through the optical flow calculation module, the feature fusion module, the image optimization module and the control module arranged in the AI chip, the first infrared image and the first visible light image are fused, optical flow field is calculated, multi-scale features are extracted / embedded and image optimization is performed, so as to obtain a final fusion image.

Images