Dual-lens image fusion data processing system based on infrared camera and low-light television

Abstract

The application discloses a dual-lens image fusion data processing system based on infrared camera and low-light television, and relates to the technical field of data processing.The system comprises the following modules: a dual-lens data acquisition module, a multi-modal processing and task identification module, a dynamic resource scheduling module, a multi-target optimization fusion processing module and a closed-loop evaluation optimization module; the dual-lens data acquisition module acquires and stores infrared thermal imaging and low-light visible light data of the same scene; the multi-modal processing and task identification module identifies core tasks and assigns priority identification; the dynamic resource scheduling module combines GPU computing power real-time occupancy rate and FPGA DSP unit load, and allocates computing resources according to priority rules; the multi-target optimization fusion processing module evaluates quality, solves the optimal fusion path, completes fusion in the FPGA and outputs images; and the closed-loop evaluation optimization module evaluates in two dimensions, feeds back adjustment parameters, records the optimal path to the parameter library, forms a closed-loop optimization, and improves fusion quality and real-time performance.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a dual-lens image fusion data processing system based on infrared camera and low-light television. Background Technology

[0002] In low-light scenarios such as nighttime security, low-altitude detection, and autonomous driving, a single imaging device is insufficient to meet the requirements for clear imaging. Infrared cameras can capture target thermal radiation information, but they lack detailed textures; low-light television can preserve scene details, but it is easily affected by ambient light. Therefore, dual-lens image fusion technology based on infrared cameras and low-light television has become a research hotspot. By combining the advantages of both types of images, it can significantly improve target recognition capabilities in low-light scenarios.

[0003] Although current mainstream dual-lens image fusion systems can achieve basic data acquisition and fusion, two key technical bottlenecks still exist in complex dynamic scenarios: First, the task scheduling mechanism is rigid and difficult to adapt to the needs of multi-task parallelism. For example, existing systems mostly adopt a fixed resource allocation mode. The infrared and low-light data processing channels often suffer from high-priority task response delays due to the equal distribution of computing power. Moreover, when sub-tasks such as target detection and image enhancement are executed serially, low-priority tasks are prone to occupying computing power, causing the target recognition window to be missed. Second, quality and efficiency are difficult to optimize in a coordinated manner. For example, most systems only pursue fusion quality and ignore hardware computing power constraints, resulting in processing latency exceeding the application threshold. Or, in order to compress latency and simplify the algorithm, information entropy loss and target signal-to-noise ratio decrease are caused, which cannot meet the needs of high-precision recognition. Furthermore, there is a lack of dynamic evaluation and adjustment mechanisms, making it difficult to balance the relationship between the two based on real-time hardware status and scenario requirements. Summary of the Invention

[0004] The purpose of this invention is to provide a dual-lens image fusion data processing system based on infrared camera and low-light television to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] The dual-lens image fusion data processing system based on infrared camera and low-light television includes: a dual-lens data acquisition module, a multi-modal processing and task recognition module, a dynamic resource scheduling module, a multi-objective optimization fusion processing module, and a closed-loop evaluation and optimization module.

[0007] The dual-lens data acquisition module acquires infrared thermal imaging data and low-light visible light data of the same scene, and stores the acquired infrared thermal imaging data and low-light visible light data.

[0008] The multimodal processing and task identification module preprocesses the stored infrared thermal imaging data and low-light visible light data respectively, extracts target motion features from the preprocessed infrared thermal imaging data, analyzes scene complexity based on the preprocessed low-light visible light data, identifies the current core task type based on the target motion features and scene complexity, and assigns a corresponding priority label.

[0009] The dynamic resource scheduling module receives priority identifiers and, in conjunction with the real-time GPU computing power utilization rate and the FPGA DSP unit load, allocates computing resources according to priority rules.

[0010] The multi-objective optimization fusion processing module acquires preprocessed infrared thermal imaging data and low-light visible light data, performs real-time quality assessment, reads the DSP unit utilization rate of the FPGA as an efficiency constraint, solves the optimal fusion path, selects the corresponding filter kernel and fusion weight strategy, completes the fusion calculation of infrared target features and low-light details in the FPGA, and outputs the fused image.

[0011] The closed-loop evaluation and optimization module evaluates the fused image from two dimensions: quality and efficiency. If the quality index does not meet the preset qualified threshold, it is fed back to the multi-objective optimization fusion processing module to adjust the filter kernel type and fusion weight. If the processing delay exceeds the preset timeliness threshold, it is fed back to the dynamic resource scheduling module to optimize the resource allocation ratio. The optimal fusion path parameters are recorded in the parameter library to form a closed-loop optimization mechanism.

[0012] Furthermore, the dual-lens data acquisition module includes an infrared camera lens, a low-light television lens, and an FPGA-based dual-channel pipeline buffer unit;

[0013] Infrared camera lens and low-light television lens are used to simultaneously acquire infrared thermal imaging data and low-light visible light data of the same scene. The dual-channel pipeline buffer unit has two independent high-speed cache channels, which are used to temporarily store infrared thermal imaging data and low-light visible light data in parallel.

[0014] Furthermore, the multimodal processing and task recognition module includes a multimodal data preprocessing unit and a task priority determination unit;

[0015] The multimodal data preprocessing unit sequentially performs non-uniformity correction and thermal noise suppression preprocessing on the infrared thermal imaging data, and sequentially performs gain adaptive adjustment and blind pixel repair preprocessing on the low-light visible light data. Specifically, the multimodal data preprocessing unit uses a two-point correction method for non-uniformity correction of the infrared thermal imaging data, and the corresponding calculation formula is: I ir_corr (x,y)=G(x,y)·I ir (x,y)+O(x,y), where I ir(x,y) represents the original infrared pixel value, G(x,y) and O(x,y) represent the correction gain and offset, respectively, and I ir_corr (x,y) represents the corrected infrared pixel value; the infrared thermal imaging data after non-uniformity correction is filtered by a 3×3 neighborhood mean filter to suppress thermal noise, and the corresponding calculation formula is: I ir_denoise (x,y)=(1 / 9)∑ i∈[-1,1] ∑ j∈[-1,1] I ir_corr (x+i,y+j);

[0016] The multimodal data preprocessing unit employs adaptive gain adjustment based on grayscale mean for low-light visible light data. The corresponding calculation formula is: G low =μ target / μ input , where μ target The target grayscale mean, μ input To input the mean grayscale value of the low-light image, the adjusted low-light visible light data is: I low_gain (x,y)=G low ·I low (x,y), where I low (x,y) represents the grayscale value of a pixel in the original low-light visible light image; 4-neighborhood interpolation is used to repair the detected blind pixels, and the corresponding calculation formula is: I low_fix (x,y)=[I low_gain (x-1,y)+I low_gain (x+1,y)+I low_gain (x,y-1)+I low_gain (x,y+1)] / 4.

[0017] The task priority determination unit identifies the current core task type and assigns a corresponding priority label by extracting target motion features from infrared thermal imaging data and analyzing scene complexity in low-light visible light data; the specific details are as follows:

[0018] Target motion features are extracted from preprocessed infrared thermal imaging data. These features include motion velocity v and trajectory change rate α. The motion velocity v is calculated using the target coordinates of consecutive frames, specifically: v = [(x...] t -x t-1 ) 2 +(y t -y t-1 ) 2 ] (1 / 2) / Δt, where (x t ,y t ) and (x t-1 ,y t-1The coordinates of the target center at time t and time t-1 are respectively, and Δt is the frame interval; the formula for calculating the trajectory change rate α is: α = |v| t -v t-1 | / v t-1 +ε, where v t Let v be the velocity at time t. t-1 Let ε be the velocity at time t-1, and let ε be the minimum value to avoid the denominator being 0.

[0019] Scene complexity features are analyzed based on preprocessed low-light and visible light data. These features include the proportion of dynamic objects (ρ) and texture entropy (H), with the formula for calculating the proportion of dynamic objects (ρ) as: ρ = N moving / N total , where N moving N represents the number of pixels in the dynamic target. total Let H be the total number of pixels in the low-light visible light image; the formula for calculating texture entropy H is: H = -∑ k∈[0,255] P(k)log2P(k), where P(k) is the percentage of pixels with gray value k in the low-light visible light image;

[0020] The extracted target motion features and the analyzed scene complexity features are normalized to form a target feature vector D, where D = [v', α', ρ', H']. The corresponding priority score S is calculated based on the target feature vector D, where S = w1·v' + w2·α' + w3·ρ' + w4·H', and w1, w2, w3, and w4 are preset weights and satisfy 1 + w2 + w3 + w4 = 1. A high priority label is assigned when S ≥ T1, a normal priority label is assigned when T2 < S < T1, and a low priority label is assigned when S ≤ T2, where T1 and T2 are preset thresholds and T1 > T2.

[0021] Furthermore, the dynamic resource scheduling module includes a task scheduler pre-configured with a task resource allocation priority list, wherein the task resource allocation priority in the priority list is ranked from high to low as high priority tasks, regular priority tasks, and low priority tasks. The task scheduler is used to receive priority identifiers and, in conjunction with the real-time GPU computing power utilization rate and the FPGA DSP unit load, execute computing resource allocation according to the priority list. The specific analysis process is as follows:

[0022] The real-time utilization rate of GPU computing power is represented by η_GPU, and η_GPU = (current actual GPU computing power / total rated GPU computing power) × 100%; the DSP unit load rate of FPGA is represented by η_DSP, and η_DSP = (current number of occupied DSP units / total rated number of DSP units of FPGA) × 100%; both the real-time utilization rate of GPU computing power and the DSP unit load rate of FPGA are obtained through real-time sampling; the preset high load threshold of GPU is η_GPU_H and the low load threshold is η_GPU_L, the high load threshold of FPGA DSP is η_DSP_H and the low load threshold is η_DSP_L, and the minimum guaranteed ratio and maximum allocation ratio of GPU / DSP resources for each priority task are also specified. The thresholds and ratios can be dynamically adjusted according to the system hardware performance.

[0023] Priority resource allocation is performed as follows: resources are allocated to high-priority tasks first, and the resource allocation ratio is dynamically adjusted according to the threshold range of η_GPU and η_DSP to ensure that it is not lower than the minimum guaranteed ratio; after the high-priority tasks are allocated, resources are allocated to regular priority tasks based on the remaining resources, not exceeding the maximum allocation ratio, and if the remaining resources are insufficient, they are put into the waiting queue; finally, the remaining resources are allocated to low-priority tasks, and if the ratio is lower than the minimum guaranteed ratio, their calculation is suspended and rescheduled after the resources are released.

[0024] Periodically resample η_GPU and η_DSP. If the load change exceeds the preset threshold, re-execute the above priority resource allocation process and update the resource ratio of each task.

[0025] Furthermore, the multi-objective optimization fusion processing module includes a fusion quality assessment unit, a hardware efficiency monitoring unit, an optimization path solving unit, and an image fusion execution unit;

[0026] The fusion quality assessment unit performs real-time quality assessment based on preprocessed infrared thermal imaging data and low-light visible light data through a constructed quality index system; the hardware efficiency monitoring unit reads the DSP unit utilization rate of the FPGA in real time and uses it as an efficiency constraint; the optimization path solving unit combines the quality assessment results and efficiency constraints to solve the optimal fusion path through an optimization algorithm, and dynamically selects the matching filter kernel and fusion weight strategy; the image fusion execution unit performs the fusion calculation of infrared target features and low-light details in the FPGA according to the solved optimal fusion path and the selected fusion weight strategy, and generates and outputs the fused image.

[0027] Furthermore, the specific analysis process for integrating the quality assessment unit is as follows:

[0028] A multi-dimensional quality index system is constructed, which includes infrared target signal-to-noise ratio (SNR_ir), low-light image structural similarity (SSIM_low), and fused image information entropy (H_fuse). The infrared target signal-to-noise ratio (SNR_ir), low-light image structural similarity (SSIM_low), and fused image information entropy (H_fuse) are normalized, and the corresponding comprehensive quality score Q is calculated, where Q = a × SNR_ir' + b × SSIM_low' + c × H_fuse', where a, b, and c are weighting factors, and a + b + c = 1; SNR_ir', SSIM_low', and H_fuse' are the normalized infrared target signal-to-noise ratio (SNR_ir), low-light image structural similarity (SSIM_low), and fused image information entropy (H_fuse).

[0029] Furthermore, the specific analysis process of the path optimization unit is as follows:

[0030] A variable vector X is constructed using the filter kernel parameter K and the fusion weights w_ir and w_low. The filter kernel parameter K includes the kernel size and kernel type. The kernel size can be 3×3 or 5×5, and the kernel type can be a Gaussian kernel (standard deviation σ∈[0.5,2.0]) or a Laplace kernel (the sum of the coefficient matrix elements is 0). Both w_ir and w_low are ∈[0.3,0.7], and w_ir + w_low = 1. An objective function f(X) is constructed with the goal of maximizing the overall fusion quality score Q, and f(X) = Q(X). Hardware efficiency monitoring is then incorporated. The constraints of the unit are as follows: the DSP unit load rate of the FPGA η_DSP_util(X) ≤ η_DSP_lim, where η_DSP_lim is determined according to the FPGA model; the fusion processing delay τ_fuse(X) ≤ τ_lim, where τ_lim is determined according to the real-time requirements of the scenario; an improved particle swarm optimization algorithm is used to optimize and solve for the optimal fusion path, where the number of particles in the improved particle swarm optimization algorithm is 20-50, the number of iterations is 50-100, and the inertia weight ω∈[0.4,0.9].

[0031] Furthermore, the closed-loop evaluation and optimization module includes a two-dimensional evaluation unit, a feedback adjustment unit, and a parameter storage and iteration unit;

[0032] The dual-dimensional evaluation unit evaluates the fused image in terms of both quality and efficiency, checking whether the quality indicators meet the preset qualified threshold and whether the processing delay exceeds the preset timeliness threshold. The feedback adjustment unit feeds back to the multi-objective optimization fusion processing module to adjust the filter kernel type and fusion weight, and feeds back to the dynamic resource scheduling module to optimize the resource allocation ratio. The parameter storage and iteration unit records the optimal fusion path parameters to the parameter library, and through continuous accumulation and calling, forms a self-iterative closed-loop optimization mechanism for the system.

[0033] Furthermore, the specific analysis process of the two-dimensional evaluation unit is as follows:

[0034] The quality of the fused image is evaluated by calculating the target recognition score R_tar, which is calculated as: R_tar = SNR_fuse / (SNR_ir + SNR_low), where SNR_fuse is the target signal-to-noise ratio of the fused image; the detail retention score R_det is calculated as: R_det = SSIM_fuse × H_fuse'; and the overall quality index Q_eval is calculated as: Q_eval = β1 × R_tar + β2 × R_det, where β1 and β2 are weights, and their specific values ​​are set based on the target priority principle. When the overall quality index Q_eval is greater than or equal to the preset qualified threshold Qth, the quality is deemed qualified.

[0035] Efficiency is evaluated on the fused image by obtaining the end-to-end processing delay τ_total, which is equal to the sum of data acquisition delay, preprocessing delay, fusion processing delay, and image output delay. The resource utilization fluctuation coefficient Z is calculated, and Z = [max(η_DSP(t)) - min(η_DSP(t))] / μ(η_DSP), where η_DSP(t) is the real-time value of the DSP unit load rate of the FPGA in a certain period, and μ(η_DSP) is the average value of the corresponding period. When τ_total is less than or equal to the preset delay τ_th and the resource utilization fluctuation coefficient Z is less than or equal to the preset fluctuation coefficient Z0, the efficiency is deemed qualified.

[0036] Compared with existing technologies, the beneficial effects of this invention are as follows: By combining infrared thermal imaging and low-light visible light data, this invention can comprehensively capture scene information, especially improving the accuracy of target recognition in low-light or complex environments; the system can dynamically allocate computing resources according to task priority and real-time hardware resource status, ensuring that high-priority tasks receive sufficient computing support and avoiding the problem of uneven resource allocation; through multi-objective optimization and path solving, it can select the optimal filtering kernel and fusion weight strategy under the constraints of hardware resources and processing latency, ensuring a balance between image fusion quality and processing efficiency. Through real-time dual-dimensional evaluation of quality and efficiency, it can automatically adjust system parameters, optimize the fusion processing path, and continuously accumulate the optimal configuration to form a self-optimizing closed loop, adapting to different scenarios and task requirements. By comprehensively analyzing target motion characteristics and scene complexity, the system can intelligently determine the priority of the current task, thereby prioritizing the completion of key tasks under limited resources and improving system responsiveness. Attached Figure Description

[0037] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0038] Figure 1 This is a schematic diagram of the modules of the dual-lens image fusion data processing system based on infrared camera and low-light television of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Please see Figure 1 The present invention provides the following technical solution:

[0041] The dual-lens image fusion data processing system based on infrared camera and low-light television includes: a dual-lens data acquisition module, a multi-modal processing and task recognition module, a dynamic resource scheduling module, a multi-objective optimization fusion processing module, and a closed-loop evaluation and optimization module.

[0042] The dual-lens data acquisition module acquires infrared thermal imaging data and low-light visible light data of the same scene, and stores the acquired infrared thermal imaging data and low-light visible light data.

[0043] The multimodal processing and task identification module preprocesses the stored infrared thermal imaging data and low-light visible light data respectively, extracts target motion features from the preprocessed infrared thermal imaging data, analyzes scene complexity based on the preprocessed low-light visible light data, identifies the current core task type based on the target motion features and scene complexity, and assigns a corresponding priority label.

[0044] The dynamic resource scheduling module receives priority identifiers and, in conjunction with the real-time GPU computing power utilization rate and the FPGA DSP unit load, allocates computing resources according to priority rules.

[0045] The multi-objective optimization fusion processing module acquires preprocessed infrared thermal imaging data and low-light visible light data, performs real-time quality assessment, reads the DSP unit utilization rate of the FPGA as an efficiency constraint, solves the optimal fusion path, selects the corresponding filter kernel and fusion weight strategy, completes the fusion calculation of infrared target features and low-light details in the FPGA, and outputs the fused image.

[0046] The closed-loop evaluation and optimization module evaluates the fused image from two dimensions: quality and efficiency. If the quality index does not meet the preset qualified threshold, it is fed back to the multi-objective optimization fusion processing module to adjust the filter kernel type and fusion weight. If the processing delay exceeds the preset timeliness threshold, it is fed back to the dynamic resource scheduling module to optimize the resource allocation ratio. The optimal fusion path parameters are recorded in the parameter library to form a closed-loop optimization mechanism.

[0047] The dual-lens data acquisition module includes an infrared camera lens, a low-light television lens, and an FPGA-based dual-channel pipeline buffer unit;

[0048] Infrared camera lens and low-light television lens are used to simultaneously acquire infrared thermal imaging data and low-light visible light data of the same scene. The dual-channel pipeline buffer unit has two independent high-speed cache channels, which are used to temporarily store infrared thermal imaging data and low-light visible light data in parallel.

[0049] The multimodal processing and task recognition module includes a multimodal data preprocessing unit and a task priority determination unit;

[0050] The multimodal data preprocessing unit sequentially performs non-uniformity correction and thermal noise suppression preprocessing on the infrared thermal imaging data, and sequentially performs gain adaptive adjustment and blind pixel repair preprocessing on the low-light visible light data. Specifically, the multimodal data preprocessing unit uses a two-point correction method for non-uniformity correction of the infrared thermal imaging data, and the corresponding calculation formula is: I ir_corr (x,y)=G(x,y)·I ir (x,y)+O(x,y), where I ir (x,y) represents the original infrared pixel value, G(x,y) and O(x,y) represent the correction gain and offset, respectively, and I ir_corr (x,y) represents the corrected infrared pixel value; the infrared thermal imaging data after non-uniformity correction is filtered by a 3×3 neighborhood mean filter to suppress thermal noise, and the corresponding calculation formula is: I ir_denoise (x,y)=(1 / 9)∑ i∈[-1,1] ∑ j∈[-1,1] I ir_corr (x+i,y+j);

[0051] The multimodal data preprocessing unit employs adaptive gain adjustment based on grayscale mean for low-light visible light data. The corresponding calculation formula is: G low =μ target / μ input , where μ target The target grayscale mean, μ input To input the mean grayscale value of the low-light image, the adjusted low-light visible light data is: I low_gain (x,y)=G low ·I low(x,y), where I low (x,y) represents the grayscale value of a pixel in the original low-light visible light image; 4-neighborhood interpolation is used to repair the detected blind pixels, and the corresponding calculation formula is: I low_fix (x,y)=[I low_gain (x-1,y)+I low_gain (x+1,y)+I low_gain (x,y-1)+I low_gain (x,y+1)] / 4.

[0052] The task priority determination unit identifies the current core task type and assigns a corresponding priority label by extracting target motion features from infrared thermal imaging data and analyzing scene complexity in low-light visible light data; the specific details are as follows:

[0053] Target motion features are extracted from preprocessed infrared thermal imaging data. These features include motion velocity v and trajectory change rate α. The motion velocity v is calculated using the target coordinates of consecutive frames, specifically: v = [(x...] t -x t-1 ) 2 +(y t -y t-1 ) 2 ] (1 / 2) / Δt, where (x t ,y t ) and (x t-1 ,y t-1 The coordinates of the target center at time t and time t-1 are respectively, and Δt is the frame interval; the formula for calculating the trajectory change rate α is: α = |v| t -v t-1 | / v t-1 +ε, where v t Let v be the velocity at time t. t-1 Let ε be the velocity at time t-1, and let ε be the minimum value to avoid the denominator being 0.

[0054] Scene complexity features are analyzed based on preprocessed low-light and visible light data. These features include the proportion of dynamic objects (ρ) and texture entropy (H), with the formula for calculating the proportion of dynamic objects (ρ) as: ρ = N moving / N total , where N moving N represents the number of pixels in the dynamic target. total Let H be the total number of pixels in the low-light visible light image; the formula for calculating texture entropy H is: H = -∑ k∈[0,255] P(k)log2P(k), where P(k) is the percentage of pixels with gray value k in the low-light visible light image;

[0055] The extracted target motion features and the analyzed scene complexity features are normalized to form a target feature vector D, where D = [v', α', ρ', H']. The corresponding priority score S is calculated based on the target feature vector D, where S = w1·v' + w2·α' + w3·ρ' + w4·H', and w1, w2, w3, and w4 are preset weights and satisfy 1 + w2 + w3 + w4 = 1. A high priority label is assigned when S ≥ T1, a normal priority label is assigned when T2 < S < T1, and a low priority label is assigned when S ≤ T2, where T1 and T2 are preset thresholds and T1 > T2.

[0056] In this embodiment, it is assumed that in a nighttime outdoor surveillance scenario, infrared thermal imaging data captures a moving target, while low-light visible light data reflects details of the scene environment. The corresponding target motion characteristics are assumed to be: speed v = 6 m / s, trajectory change rate α = 1.2, and scene complexity characteristics: dynamic target proportion ρ = 0.35, texture entropy H = 5.2. The extracted target motion features and the analyzed scene complexity features are normalized to form a target feature vector D, assuming D = [v', α',ρ',H']=[0.6,0.6,0.35,0.65]; preset weights w1=0.3, w2=0.3, w3=0.2, w4=0.2, calculate priority score S, and S=0.3×0.6+0.3×0.6+0.2×0.35+0.2×0.65=0.56; assuming preset thresholds T1=0.7, T2=0.3, since T2<0.56<T1, assign the current core task a regular priority label.

[0057] The dynamic resource scheduling module includes a task scheduler pre-configured with a task resource allocation priority list. The priority list lists task resource allocation priorities from highest to lowest as high-priority tasks, regular-priority tasks, and low-priority tasks. The task scheduler receives priority identifiers and, in conjunction with the real-time GPU computing power utilization and the FPGA's DSP unit load, performs computing resource allocation according to the priority list. The specific analysis process is as follows:

[0058] The real-time utilization rate of GPU computing power is represented by η_GPU, and η_GPU = (current actual GPU computing power / total rated GPU computing power) × 100%; the DSP unit load rate of FPGA is represented by η_DSP, and η_DSP = (current number of occupied DSP units / total rated number of DSP units of FPGA) × 100%; both the real-time utilization rate of GPU computing power and the DSP unit load rate of FPGA are obtained through real-time sampling; the preset high load threshold of GPU is η_GPU_H and the low load threshold is η_GPU_L, the high load threshold of FPGA DSP is η_DSP_H and the low load threshold is η_DSP_L, and the minimum guaranteed ratio and maximum allocation ratio of GPU / DSP resources for each priority task are also specified. The thresholds and ratios can be dynamically adjusted according to the system hardware performance.

[0059] Priority resource allocation is performed as follows: resources are allocated to high-priority tasks first, and the resource allocation ratio is dynamically adjusted according to the threshold range of η_GPU and η_DSP to ensure that it is not lower than the minimum guaranteed ratio; after the high-priority tasks are allocated, resources are allocated to regular priority tasks based on the remaining resources, not exceeding the maximum allocation ratio, and if the remaining resources are insufficient, they are put into the waiting queue; finally, the remaining resources are allocated to low-priority tasks, and if the ratio is lower than the minimum guaranteed ratio, their calculation is suspended and rescheduled after the resources are released.

[0060] Periodically resample η_GPU and η_DSP. If the load change exceeds the preset threshold, re-execute the above priority resource allocation process and update the resource ratio of each task.

[0061] In this embodiment, it is assumed that the core parameter takes the following specific values:

[0062] Load thresholds: η_GPU_H=80% (high GPU load), η_GPU_L=40% (low GPU load); η_DSP_H=85% (high DSP load), η_DSP_L=35% (low DSP load);

[0063] Resource allocation parameters: For high-priority tasks, the minimum guaranteed GPU allocation (G_H_min) is 40%, and the minimum guaranteed DSP allocation (D_H_min) is 35%; for regular-priority tasks, the maximum GPU allocation (G_R_max) is 30%, and the maximum DSP allocation (D_R_max) is 25%; for low-priority tasks, the minimum guaranteed GPU / DSP allocation is 10%.

[0064] Sampling and adjustment period: η_GPU / η_DSP samples once every 10ms, and the dynamic adjustment trigger threshold is ≥10% of the load change, with an adjustment period of 20ms.

[0065] Example of a priority resource allocation scenario:

[0066] Scenario 1: Resource allocation for high-priority tasks (such as high-speed target tracking);

[0067] When η_GPU=30% (≤η_GPU_L=40%), η_DSP=30% (≤η_DSP_L=35%):

[0068] GPU allocation ratio G_H = G_H_min + (η_GPU_L - η_GPU) × 0.5 = 40% + (40% - 30%) × 0.5 = 45%; DSP allocation ratio D_H = D_H_min + (η_DSP_L - η_DSP) × 0.4 = 35% + (35% - 30%) × 0.4 = 37%;

[0069] When η_GPU=90% (> η_GPU_H=80%), η_DSP=90% (> η_DSP_H=85%):

[0070] G_H = G_H_min - (η_GPU - η_GPU_H) × 0.2 = 40% - (90% - 80%) × 0.2 = 38% (minimum guarantee of not less than 30%); D_H = D_H_min - (η_DSP - η_DSP_H) × 0.15 = 35% - (90% - 85%) × 0.15 = 34.25% (minimum guarantee of not less than 25%).

[0071] Scenario 2: Resource allocation for routine / low-priority tasks;

[0072] After high-priority tasks are assigned, if the remaining available GPU percentage ΔG = 25% (< G_R_max = 30%): the GPU allocation percentage for regular-priority tasks is G_R = 25%;

[0073] If the remaining available DSP percentage ΔD = 4% (< 5%): Regular priority tasks will not be allocated DSP resources and will enter the waiting queue;

[0074] After the normal priority allocation, if the GPU has a final remaining ΔG_final = 8% (<10% minimum guarantee for low priority tasks): low priority tasks will pause GPU computation and be rescheduled after resources are released.

[0075] Example of dynamic adjustment trigger: If η_GPU = 40% at time t1 and η_GPU = 55% at time t2 (20ms interval), and the load change Δη_GPU = 15% ≥ 10%, the task scheduler re-executes resource allocation, adjusting the GPU allocation ratio of high-priority tasks from 45% (at time t1) to 40% (at time t2, because η_GPU is in the range of η_GPU_L-η_GPU_H), and synchronously updating the resource ratio of regular / low-priority tasks.

[0076] The multi-objective optimization fusion processing module includes a fusion quality assessment unit, a hardware efficiency monitoring unit, an optimization path solving unit, and an image fusion execution unit;

[0077] The fusion quality assessment unit performs real-time quality assessment based on preprocessed infrared thermal imaging data and low-light visible light data through a constructed quality index system; the hardware efficiency monitoring unit reads the DSP unit utilization rate of the FPGA in real time and uses it as an efficiency constraint; the optimization path solving unit combines the quality assessment results and efficiency constraints to solve the optimal fusion path through an optimization algorithm, and dynamically selects the matching filter kernel and fusion weight strategy; the image fusion execution unit performs the fusion calculation of infrared target features and low-light details in the FPGA according to the solved optimal fusion path and the selected fusion weight strategy, and generates and outputs the fused image.

[0078] The specific analysis process for the integrated quality assessment unit is as follows:

[0079] A multi-dimensional quality index system is constructed, which includes infrared target signal-to-noise ratio (SNR_ir), low-light image structural similarity (SSIM_low), and fused image information entropy (H_fuse). The infrared target signal-to-noise ratio (SNR_ir), low-light image structural similarity (SSIM_low), and fused image information entropy (H_fuse) are normalized, and the corresponding comprehensive quality score Q is calculated, where Q = a × SNR_ir' + b × SSIM_low' + c × H_fuse', where a, b, and c are weighting factors, and a + b + c = 1; SNR_ir', SSIM_low', and H_fuse' are the normalized infrared target signal-to-noise ratio (SNR_ir), low-light image structural similarity (SSIM_low), and fused image information entropy (H_fuse).

[0080] In this embodiment, for the infrared target signal-to-noise ratio (SNR_ir), the target region and background region are divided from the preprocessed infrared thermal imaging data. The target region is the connected component extracted by the contour detection algorithm, and the background region is the region within a preset pixel range outside the target region. The mean gray value μ_tar of the target region, the mean gray value μ_bg of the background region, and the standard deviation σ_bg of the background region are calculated respectively. The formula for calculating the infrared target signal-to-noise ratio (SNR_ir) is: SNR_ir = 20 × log 10 (|μ_tar-μ_bg| / σ_bg); For the low-light image structure similarity SSIM_low, the preprocessed low-light visible light data is used as the reference image I_ref, and the low-light data after 3×3 Gaussian noise reduction is used as the image to be evaluated I_eval; 8×8 pixel blocks are selected as the calculation unit, and the calculation is performed block by block throughout the entire image. The formula is:

[0081] ;

[0082] Where μ_ref and μ_eval are the mean gray values ​​of corresponding pixel blocks in the two images, respectively, and σ_ref 2 σ_eval 2 σ_ref-eval represents the grayscale variance of corresponding pixel blocks in the two images, and σ_ref-eval represents the covariance of corresponding pixel blocks in the two images; C1=(K1×L) 2 C2 = (K2 × L) 2 K1=0.01 and K2=0.03 are preset constants, L=255 is the gray level of the low-light image, and SSIM_low takes a value range of [0,1]. The closer the value is to 1, the more complete the low-light data structure is preserved.

[0083] For the entropy H_fuse of the fused image, count the number N(k) of pixels with each gray value k in the fused image, where k∈[0,255], and calculate the pixel percentage P(k)=N(k) / N_total (where N_total is the total number of pixels in the fused image). The formula for calculating H_fuse is: H_fuse=-∑ k∈[0,255] [P(k)×log2P(k)], where H_fuse is in bits, and the larger the value, the richer the scene details contained in the fused image.

[0084] The specific analysis process of the path optimization unit is as follows:

[0085] A variable vector X is constructed using the filter kernel parameter K and the fusion weights w_ir and w_low. The filter kernel parameter K includes the kernel size and kernel type. The kernel size can be 3×3 or 5×5, and the kernel type can be a Gaussian kernel (standard deviation σ∈[0.5,2.0]) or a Laplace kernel (the sum of the coefficient matrix elements is 0). Both w_ir and w_low are ∈[0.3,0.7], and w_ir + w_low = 1. An objective function f(X) is constructed with the goal of maximizing the overall fusion quality score Q, and f(X) = Q(X). Hardware efficiency monitoring is then incorporated. The constraints of the unit are as follows: the DSP unit load rate of the FPGA η_DSP_util(X) ≤ η_DSP_lim, where η_DSP_lim is determined according to the FPGA model; the fusion processing delay τ_fuse(X) ≤ τ_lim, where τ_lim is determined according to the real-time requirements of the scenario; an improved particle swarm optimization algorithm is used to optimize and solve for the optimal fusion path, where the number of particles in the improved particle swarm optimization algorithm is 20-50, the number of iterations is 50-100, and the inertia weight ω∈[0.4,0.9].

[0086] The closed-loop evaluation and optimization module includes a two-dimensional evaluation unit, a feedback adjustment unit, and a parameter storage and iteration unit;

[0087] The dual-dimensional evaluation unit evaluates the fused image in terms of both quality and efficiency, checking whether the quality indicators meet the preset qualified threshold and whether the processing delay exceeds the preset timeliness threshold. The feedback adjustment unit feeds back to the multi-objective optimization fusion processing module to adjust the filter kernel type and fusion weight, and feeds back to the dynamic resource scheduling module to optimize the resource allocation ratio. The parameter storage and iteration unit records the optimal fusion path parameters to the parameter library, and through continuous accumulation and calling, forms a self-iterative closed-loop optimization mechanism for the system.

[0088] The specific analysis process of the two-dimensional evaluation unit is as follows:

[0089] The quality of the fused image is evaluated by calculating the target recognition score R_tar, which is calculated as: R_tar = SNR_fuse / (SNR_ir + SNR_low), where SNR_fuse is the target signal-to-noise ratio of the fused image; the detail retention score R_det is calculated as: R_det = SSIM_fuse × H_fuse'; and the overall quality index Q_eval is calculated as: Q_eval = β1 × R_tar + β2 × R_det, where β1 and β2 are weights, and their specific values ​​are set based on the target priority principle. When the overall quality index Q_eval is greater than or equal to the preset qualified threshold Qth, the quality is deemed qualified.

[0090] Efficiency is evaluated on the fused image by obtaining the end-to-end processing delay τ_total, which is equal to the sum of data acquisition delay, preprocessing delay, fusion processing delay, and image output delay. The resource utilization fluctuation coefficient Z is calculated, and Z = [max(η_DSP(t)) - min(η_DSP(t))] / μ(η_DSP), where η_DSP(t) is the real-time value of the DSP unit load rate of the FPGA in a certain period, and μ(η_DSP) is the average value of the corresponding period. When τ_total is less than or equal to the preset delay τ_th and the resource utilization fluctuation coefficient Z is less than or equal to the preset fluctuation coefficient Z0, the efficiency is deemed qualified.

[0091] In this embodiment, the specific analysis process of the feedback adjustment unit is as follows:

[0092] When the dual-dimensional evaluation unit determines that the quality is unqualified, it sends an adjustment instruction to the multi-target optimization fusion processing module: if the fusion target recognition R_tar is insufficient, increase the infrared fusion weight w_ir (adjustment step size is 0.05-0.1, and does not exceed 0.7); if the detail preservation integrity R_det is insufficient, switch to a larger size filter kernel or adjust the kernel parameters, and increase the low-light fusion weight w_low (adjustment step size is 0.05-0.1, and does not exceed 0.7), until the quality is qualified or the upper limit of 3 adjustments is reached.

[0093] When the dual-dimensional evaluation unit determines that the efficiency is unqualified, it sends an adjustment instruction to the dynamic resource scheduling module: if the end-to-end latency exceeds the limit, it increases the FPGA resource ratio of high-priority tasks by 5%-10% and reduces the GPU resource ratio of low-priority tasks by 5%-10%, and switches to a smaller-sized filter core if necessary; if the resource fluctuation exceeds the limit, it enables the load smoothing mechanism (the real-time fluctuation limit is set to 15%), and dynamically allocates idle resources to tasks with high load until the efficiency is qualified or the upper limit of two adjustments is reached.

[0094] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0095] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dual-lens image fusion data processing system based on infrared camera and low-light television, characterized in that: The system includes: a dual-lens data acquisition module, a multimodal processing and task recognition module, a dynamic resource scheduling module, a multi-objective optimization and fusion processing module, and a closed-loop evaluation and optimization module; The dual-lens data acquisition module acquires infrared thermal imaging data and low-light visible light data of the same scene, and stores the acquired infrared thermal imaging data and low-light visible light data. The multimodal processing and task identification module preprocesses the stored infrared thermal imaging data and low-light visible light data respectively, extracts target motion features from the preprocessed infrared thermal imaging data, analyzes scene complexity based on the preprocessed low-light visible light data, identifies the current core task type based on the target motion features and scene complexity, and assigns a corresponding priority identifier. The target motion features include motion velocity v and trajectory change rate α, and the motion velocity v is calculated using the target coordinates of consecutive frames, specifically: v=[(x t -x t-1 ) 2 +(y t -y t-1 ) 2 ] (1 / 2) / Δt, where (x t ,y t ) and (x t-1 ,y t-1 The coordinates of the target center at time t and time t-1 are respectively, and Δt is the frame interval; the formula for calculating the trajectory change rate α is: α = |v| t -v t-1 | / v t-1 +ε, where v t Let v be the velocity at time t. t-1 Let ε be the velocity at time t-1, and ε be the minimum value. The scene complexity includes the proportion of dynamic targets ρ and the texture entropy H, and the formula for calculating the proportion of dynamic targets ρ is: ρ = N moving / N total , where N moving N represents the number of pixels in the dynamic target. total Let H be the total number of pixels in the low-light visible light image; the formula for calculating texture entropy H is: H = -∑ k∈[0,255] P(k)log2P(k), where P(k) is the percentage of pixels with gray value k in the low-light visible light image; The extracted target motion features and the analyzed scene complexity features are normalized to form a target feature vector D, where D = [v', α', ρ', H']. The corresponding priority score S is calculated based on the target feature vector D, where S = w1·v' + w2·α' + w3·ρ' + w4·H', and w1, w2, w3, and w4 are preset weights and satisfy 1 + w2 + w3 + w4 = 1. A high priority label is assigned when S ≥ T1, a normal priority label is assigned when T2 < S < T1, and a low priority label is assigned when S ≤ T2, where T1 and T2 are preset thresholds and T1 > T2. The dynamic resource scheduling module receives priority identifiers and, combined with the real-time GPU computing power utilization rate and the FPGA DSP unit load, allocates computing resources according to priority rules. Specifically, the real-time GPU computing power utilization rate η_GPU = (current actual GPU computing power / GPU rated total computing power) × 100%, and the FPGA DSP unit load rate η_DSP = (current number of occupied DSP units / FPGA rated total number of DSP units) × 100%. The dynamic resource scheduling module prioritizes resource allocation for high-priority tasks, dynamically adjusting the resource allocation ratio based on the threshold range of η_GPU and η_DSP to ensure it does not fall below the minimum guaranteed ratio. After high-priority tasks are allocated, the remaining resources are used to allocate resources to regular-priority tasks, not exceeding the maximum allocation ratio. Finally, the remaining resources are allocated to low-priority tasks; if the ratio falls below the minimum guaranteed ratio, the task's computation is paused, and rescheduled after resource release. The multi-objective optimization fusion processing module acquires preprocessed infrared thermal imaging data and low-light visible light data, and performs real-time quality assessment. A variable vector X is constructed using the filter kernel parameter K and fusion weights w_ir and w_low, where w_ir and w_low are both ∈ [0.3, 0.7], and w_ir + w_low = 1. An objective function f(X) = Q(X) is constructed with the goal of maximizing the overall fusion quality score Q. With constraints of FPGA DSP unit load rate η_DSP_util(X) ≤ η_DSP_lim and fusion processing delay τ_fuse(X) ≤ τ_lim, an improved particle swarm optimization algorithm is used to solve for the optimal fusion path. The corresponding filter kernel and fusion weight strategy are selected, and the fusion calculation of infrared target features and low-light details is completed in the FPGA, outputting the fused image. The closed-loop evaluation and optimization module evaluates the fused image in two dimensions: quality and efficiency. If the quality index does not meet the preset qualified threshold, it is fed back to the multi-objective optimization fusion processing module to adjust the filter kernel type and fusion weight. If the processing delay exceeds the preset timeliness threshold, it is fed back to the dynamic resource scheduling module to optimize the resource allocation ratio. The optimal fusion path parameters are recorded in the parameter library to form a closed-loop optimization mechanism.

2. The dual-lens image fusion data processing system based on infrared camera and low-light television according to claim 1, characterized in that: The dual-lens data acquisition module includes an infrared camera lens, a low-light television lens, and an FPGA-based dual-channel pipeline buffer unit. The infrared camera lens and the low-light television lens are used to simultaneously acquire infrared thermal imaging data and low-light visible light data of the same scene. The dual-channel pipeline buffer unit is equipped with two independent high-speed buffer channels, which are used to temporarily store the infrared thermal imaging data and low-light visible light data in parallel.

3. The dual-lens image fusion data processing system based on infrared camera and low-light television according to claim 1, characterized in that: The multimodal processing and task identification module includes a multimodal data preprocessing unit and a task priority determination unit; The multimodal data preprocessing unit sequentially performs non-uniformity correction and thermal noise suppression preprocessing on the infrared thermal imaging data, and sequentially performs gain adaptive adjustment and blind pixel repair preprocessing on the low-light visible light data. Specifically, the multimodal data preprocessing unit uses a two-point correction method for non-uniformity correction of the infrared thermal imaging data, and the corresponding calculation formula is: I ir_corr (x,y)=G(x,y)·I ir (x,y)+O(x,y), where I ir (x,y) represents the original infrared pixel value, G(x,y) and O(x,y) represent the correction gain and offset, respectively, and I ir_corr (x,y) represents the corrected infrared pixel value; the infrared thermal imaging data after non-uniformity correction is filtered by a 3×3 neighborhood mean filter to suppress thermal noise, and the corresponding calculation formula is: I ir_denoise (x,y)=(1 / 9)∑ i∈[-1,1] ∑ j∈[-1,1] I ir_corr (x+i,y+j); The multimodal data preprocessing unit employs adaptive gain adjustment based on grayscale mean for low-light visible light data. The corresponding calculation formula is: G low =μ target / μ input , where μ target The target grayscale mean, μ input To input the mean grayscale value of the low-light image, the adjusted low-light visible light data is: I low_gain (x,y)=G low ·I low (x,y), where I low (x,y) represents the grayscale value of a pixel in the original low-light visible light image; 4-neighborhood interpolation is used to repair the detected blind pixels, and the corresponding calculation formula is: I low_fix (x,y)=[I low_gain (x-1,y)+I low_gain (x+1,y)+I low_gain (x,y-1)+I low_gain (x,y+1)] / 4; The task priority determination unit identifies the current core task type by extracting target motion features from infrared thermal imaging data and analyzing scene complexity in low-light visible light data, and assigns a corresponding priority label.

4. The dual-lens image fusion data processing system based on infrared camera and low-light television according to claim 1, characterized in that: The dynamic resource scheduling module includes a task scheduler pre-configured with a task resource allocation priority list. The priority list lists task resource allocation priorities from high to low as high priority tasks, regular priority tasks, and low priority tasks. The task scheduler receives priority identifiers and, in conjunction with the real-time GPU computing power utilization rate and the FPGA DSP unit load rate, performs computing resource allocation according to the priority list.

5. The dual-lens image fusion data processing system based on infrared camera and low-light television according to claim 1, characterized in that: The multi-objective optimization fusion processing module includes a fusion quality assessment unit, a hardware efficiency monitoring unit, an optimization path solving unit, and an image fusion execution unit; The fusion quality assessment unit performs real-time quality assessment based on preprocessed infrared thermal imaging data and low-light visible light data by constructing a quality index system. The hardware efficiency monitoring unit reads the load rate of the DSP unit of the FPGA in real time and uses it as an efficiency constraint condition; the optimal path solving unit combines the quality evaluation result and the efficiency constraint condition, and solves the optimal fusion path through an optimization algorithm, and dynamically selects a matching filter kernel and a fusion weight strategy; the image fusion execution unit completes the fusion calculation of the infrared target features and the low-light details in the FPGA according to the solved optimal fusion path and the selected fusion weight strategy, and generates and outputs a fused image.

6. The dual-lens image fusion data processing system based on infrared camera and low-light television according to claim 5, characterized in that: The specific analysis process of the fusion quality evaluation unit is as follows: Construct a multi-dimensional quality index system, where the quality index system includes the infrared target signal-to-noise ratio SNR_ir, the low-light image structural similarity SSIM_low, and the fused image information entropy H_fuse; normalize the infrared target signal-to-noise ratio SNR_ir, the low-light image structural similarity SSIM_low, and the fused image information entropy H_fuse, and calculate the corresponding comprehensive quality score Q, and Q = a×SNR_ir’ + b×SSIM_low’ + c×H_fuse’, where a, b, and c are all weight factors, and a + b + c = 1; SNR_ir’, SSIM_low’, and H_fuse’ are the normalized infrared target signal-to-noise ratio SNR_ir, the low-light image structural similarity SSIM_low, and the fused image information entropy H_fuse.

7. The dual-lens image fusion data processing system based on infrared camera and low-light television according to claim 1, characterized in that: The closed-loop evaluation and optimization module includes a two-dimensional evaluation unit, a feedback adjustment unit, and a parameter storage and iteration unit; The two-dimensional evaluation unit performs two-dimensional evaluation of the quality and efficiency of the fused image, detects whether the quality index reaches a preset qualified threshold, and whether the processing delay exceeds a preset time limit threshold; the feedback adjustment unit feeds back to the multi-objective optimization fusion processing module to adjust the filter kernel type and fusion weight, and feeds back to the dynamic resource scheduling module to optimize the resource allocation ratio; the parameter storage and iteration unit records the optimal fusion path parameters in the parameter library, and forms a closed-loop optimization mechanism for system self-iteration through continuous accumulation and invocation.

8. The dual-lens image fusion data processing system based on infrared camera and low-light television according to claim 7, characterized in that: The specific analysis process of the two-dimensional evaluation unit is as follows: Perform a quality dimension evaluation on the fused image, calculate the fusion target recognition rate R_tar, and the corresponding calculation formula is: R_tar = SNR_fuse / (SNR_ir + SNR_low), where SNR_fuse is the target signal-to-noise ratio of the fused image; calculate the detail retention integrity R_det, and the corresponding calculation formula is: R_det = SSIM_fuse×H_fuse’; calculate the comprehensive quality index Q_eval, and the corresponding calculation formula is: Q_eval = β1×R_tar + β2×R_det, where β1 and β2 are weights; when the comprehensive quality index Q_eval is greater than or equal to the preset qualified threshold Qth, it is determined that the quality is qualified; Efficiency is evaluated on the fused image by obtaining the end-to-end processing delay τ_total, which is equal to the sum of data acquisition delay, preprocessing delay, fusion processing delay, and image output delay. The resource utilization fluctuation coefficient Z is calculated, and Z = [max(η_DSP(t)) - min(η_DSP(t))] / μ(η_DSP), where η_DSP(t) is the real-time value of the DSP unit load rate of the FPGA in a certain period, and μ(η_DSP) is the average value of the corresponding period. When τ_total is less than or equal to the preset delay τ_th and the resource utilization fluctuation coefficient Z is less than or equal to the preset fluctuation coefficient Z0, the efficiency is deemed qualified.

Images