A thermal imaging enhancement method and system based on multi-modal fusion

Abstract

This invention discloses a thermal imaging enhancement method and system based on multimodal fusion, relating to the field of image processing technology. It involves sending hardware synchronization trigger signals to a thermal imaging sensor and a visible light image sensor to acquire a thermal radiation intensity matrix and a three-channel visible light image matrix. Thermal contour point sets and texture edge point sets are extracted. Using the thermal contour point sets as a reference, a displacement vector is calculated. Two-dimensional deformation field data is generated through spatial interpolation. Pixel position correction is performed on the three-channel visible light image matrix to output a visible light image. A first thermal radiation component matrix and a first surface reflection component matrix are initialized. A discriminator is used to identify the identification results and fine-tunes the values ​​in reverse until a termination condition is met. A second thermal radiation component matrix and a second surface reflection component matrix are output. Using the second thermal radiation component matrix as a base layer, dynamic weighting coefficients are calculated. Based on the dynamic weighting coefficients, the second surface reflection component matrix is ​​injected into the base layer to generate an enhanced thermal imaging image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a thermal imaging enhancement method and system based on multimodal fusion. Background Technology

[0002] Image processing technology encompasses the acquisition, storage, processing, and analysis of image data, with core aspects including image enhancement, image segmentation, image recognition, and image restoration. Image processing technology is widely applied in medicine, security, industrial inspection, satellite remote sensing, and robot vision. The overall image processing technology system covers image acquisition, image transformation, feature extraction, data analysis, and pattern recognition, driving the advancement of various automation and intelligent technologies. Traditional multimodal fusion-based thermal imaging enhancement methods aim to improve the quality and usability of thermal images by fusing image information from different sensors or different modalities. Traditional methods typically combine thermal images with other modalities through image alignment, feature extraction, and fusion algorithms. This involves converting multi-source images to a unified spatial scale and enhancing image details and contrast through weighted fusion or algorithm optimization, improving the performance of thermal images in low-light or high-noise environments for subsequent analysis and application.

[0003] Existing technologies for multimodal image fusion suffer from two major drawbacks. First, in the image alignment stage, traditional methods only perform simple spatial scale unification, failing to accurately correct the nonlinear geometric distortions caused by differences in physical location and optical characteristics between thermal imaging and visible light sensors. This directly results in feature misalignment and structural ghosting between images even after alignment, fundamentally impairing the clarity of the final fused image. Second, in the image fusion stage, traditional methods employ pixel-level weighted averaging or simple feature overlay. This approach does not physically separate and understand the core characteristics of different modal information. Key contour information characterizing temperature gradients in thermal imaging is interfered with and blurred by the complex textures of visible light. Simultaneously, valuable high-frequency detail information in visible light images is smoothed or suppressed during the fusion process. Ultimately, the output image fails to simultaneously capture the accuracy of temperature distribution and the richness of scene details, significantly diminishing its information value. Summary of the Invention

[0004] The technical problem solved by this invention is that in the processing of multimodal images, the existing technology usually only performs spatial scale unification in the alignment process, which is difficult to accurately correct the nonlinear geometric distortion caused by the difference in the physical position and optical characteristics of the sensor. As a result, feature misalignment and structural ghosting still exist between images before fusion processing, which impairs the clarity of the final image. Furthermore, in the fusion stage, traditional methods often use pixel-level weighted averaging or simple feature superposition, which do not separate the core characteristics of different modal information from a physical perspective. This often causes the key contours representing temperature gradients in thermal imaging to be interfered with by visible light textures. Conversely, high-frequency detail information in visible light images is also smoothed or suppressed during the fusion process. The final output image is difficult to balance the accuracy of temperature distribution and the richness of scene details.

[0005] To address the aforementioned technical problems, the present invention provides the following technical solution: a thermal imaging enhancement method based on multimodal fusion, comprising the following steps:

[0006] Step S1: Send hardware synchronization trigger signals to the thermal imaging sensor and the visible light image sensor to acquire the thermal radiation intensity matrix and the three-channel visible light image matrix, respectively.

[0007] Step S2: Extract the thermal contour point set based on the thermal radiation intensity matrix, and extract the texture edge point set based on the three-channel visible light image matrix. Using the thermal contour point set as a reference, calculate the displacement vector, generate two-dimensional deformation field data through spatial interpolation, and use the two-dimensional deformation field data to correct the pixel position of the three-channel visible light image matrix to output the visible light image.

[0008] Step S3: Initialize the first thermal radiation component matrix and the first surface reflection component matrix, use the discriminator to identify the first thermal radiation component matrix and the first surface reflection component matrix, obtain the identification result and fine-tune the value in reverse until the termination condition is met, and output the second thermal radiation component matrix and the second surface reflection component matrix.

[0009] Step S4: Using the second thermal radiation component matrix as the substrate layer, calculate the dynamic weighting coefficient based on the second thermal radiation component matrix, and inject the second surface reflection component matrix into the substrate layer according to the dynamic weighting coefficient to generate an enhanced thermal imaging image.

[0010] Preferably, step S1 includes the following sub-steps:

[0011] Step S101: Simultaneously send high-level pulses to the thermal imaging sensor and the visible light image sensor in the dual-light camera module as hardware synchronization trigger signals.

[0012] In step S102, after receiving the hardware synchronization trigger signal, the thermal imaging sensor and the visible light image sensor perform exposure actions at the same time point. After receiving the hardware synchronization trigger signal, the thermal imaging sensor captures first scene data and converts the first scene data into a two-dimensional thermal radiation intensity matrix representing the temperature of each pixel. After receiving the hardware synchronization trigger signal, the visible light image sensor captures second scene data and converts the second scene data into a three-channel visible light image matrix representing the color and brightness of each pixel.

[0013] Preferably, step S2 includes the following sub-steps:

[0014] Step S201: Traverse the pixels in the two-dimensional thermal radiation intensity matrix, calculate the intensity difference between each pixel and its eight neighboring pixels, mark the pixels with intensity differences greater than a preset temperature change threshold as thermal contour points, and obtain a thermal contour point set. Convert the three-channel visible light image matrix into a grayscale image and calculate the brightness gradient between pixels. Mark the pixels with the top 5% of the brightness gradient as texture edge points, and obtain a texture edge point set.

[0015] Preferably, step S2 further includes the following sub-steps:

[0016] Step S202: Based on the set of thermal contour points, for each thermal contour point, match the texture edge point in the set of texture edge points that is spatially closest and whose gradient direction angle is less than a preset angle, and calculate the displacement vector between the thermal contour point and the nearest texture edge point. By spatially interpolating all displacement vectors, generate two-dimensional deformation field data that covers the entire image plane with smooth changes.

[0017] Preferably, step S2 further includes the following sub-steps:

[0018] Step S203: Based on the two-dimensional deformation field data, calculate the original coordinates of each pixel position in the newly generated image in the three-channel visible light image matrix, obtain the pixel values ​​of the four pixels around the original coordinates, and perform a weighted average to obtain the corrected pixel information. Based on the corrected pixel information, output a visible light image that is structurally aligned with the two-dimensional thermal radiation intensity matrix.

[0019] Preferably, step S3 includes the following sub-steps:

[0020] Step S301: Initialize the first thermal radiation component matrix and the first surface reflection component matrix. In each iteration of optimization, perform pixel-by-pixel multiplication on the first thermal radiation component matrix and the first surface reflection component matrix to simulate the predicted visible light image generated by the interaction between ambient light and surface reflection in the visible light band.

[0021] Calculate the first pixel value difference between the predicted visible light image and the visible light image, take the first thermal radiation component matrix as the predicted thermal imaging image, and calculate the second pixel value difference between the predicted thermal imaging image and the two-dimensional thermal radiation intensity matrix.

[0022] Preferably, step S3 further includes the following sub-steps:

[0023] Step S302: Input the first thermal radiation component matrix and the first surface reflection component matrix into the discriminator. The discriminator identifies the temperature distribution attribute and reflectivity distribution attribute of the input matrix based on the smoothness of the values ​​in the matrix and the proportion of high-frequency components, and obtains the identification result.

[0024] Preferably, step S3 further includes the following sub-steps:

[0025] Step S303: Based on the difference between the first pixel value, the difference between the second pixel value, and the recognition result, the values ​​in the first thermal radiation component matrix and the first surface reflection component matrix are finely adjusted in reverse. When the number of iterations reaches the preset upper limit or the reconstruction difference is less than the preset value, the loop is terminated, and the second thermal radiation component matrix and the second surface reflection component matrix are output.

[0026] Preferably, step S4 includes the following sub-steps:

[0027] Step S401: Create a blank image matrix as an enhanced thermal imaging image, and use the second thermal radiation component matrix as the base layer of the enhanced thermal imaging image;

[0028] Step S402: Read the values ​​in the second surface reflection component matrix pixel by pixel, and analyze the degree of change of the values ​​in the preset surrounding area of ​​each pixel in the second thermal radiation component matrix.

[0029] If the severity value is greater than a preset severity threshold, the dynamic weighting coefficient is increased;

[0030] If the severity value is less than or equal to a preset severity threshold, the dynamic weighting coefficient is reduced.

[0031] Step S403: Multiply the reflectivity value in the second surface reflectivity component matrix by the dynamic weighting coefficient, and add it to the temperature value of the substrate layer. After completing the calculation of all pixels, output the enhanced thermal imaging image.

[0032] A thermal imaging enhancement system based on multimodal fusion includes an acquisition module, a spatial alignment module, a decoupling module, and an adaptive module;

[0033] The acquisition module is used to send hardware synchronization trigger signals to the thermal imaging sensor and the visible light image sensor to acquire the thermal radiation intensity matrix and the three-channel visible light image matrix, respectively.

[0034] The spatial alignment module is used to extract thermal contour point sets based on the thermal radiation intensity matrix and extract texture edge point sets based on the three-channel visible light image matrix. Using the thermal contour point sets as a reference, the displacement vector is calculated, and two-dimensional deformation field data is generated through spatial interpolation. The two-dimensional deformation field data is used to correct the pixel positions of the three-channel visible light image matrix and output a visible light image.

[0035] The decoupling module is used to initialize the first thermal radiation component matrix and the first surface reflection component matrix, use a discriminator to identify the first thermal radiation component matrix and the first surface reflection component matrix, obtain the identification result and fine-tune the value in reverse until the termination condition is met, and output the second thermal radiation component matrix and the second surface reflection component matrix.

[0036] The adaptive module is used to use the second thermal radiation component matrix as the base layer, calculate dynamic weighting coefficients based on the second thermal radiation component matrix, and inject the second surface reflection component matrix into the base layer according to the dynamic weighting coefficients to generate an enhanced thermal imaging image.

[0037] The beneficial effects of this invention are as follows: Addressing the issues of inaccurate registration due to differences in sensor physical positions during multimodal image fusion, and the loss of texture details and blurring of thermal features caused by simple pixel-level fusion strategies in existing technologies, this invention improves upon these problems. Focusing on the primary task of thermal imaging enhancement, it constructs a complete information processing chain, from hardware synchronous acquisition and physical field constraint alignment to adversarial iteration-based physical attribute decoupling, and finally adaptive detail injection. Through hardware and software collaboration, the synchronization of the data source is ensured, and the stable physical characteristics of thermal imaging are used to correct the geometric deformation of visible light images. Finally, by simulating the physical imaging process to decouple features in reverse, deep information fusion is achieved. This significantly improves the registration accuracy of multimodal images when there are differences in viewing angles. The generated enhanced image can simultaneously retain the high-contrast contours of thermal targets and the rich surface textures of visible light images, improving image clarity and interpretability. In applications such as nighttime security, it can effectively reduce target misses or misjudgments caused by poor image quality, thus improving the overall reliability of the system. Attached Figure Description

[0038] Figure 1 A flowchart illustrating the steps of a multimodal fusion-based thermal imaging enhancement method according to an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of the basic process of a thermal imaging enhancement system based on multimodal fusion, provided as an embodiment of the present invention. Detailed Implementation

[0040] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0041] Example 1, referring to Figure 1 A thermal imaging enhancement method based on multimodal fusion is provided, comprising the following steps:

[0042] Step S1: Send hardware synchronization trigger signals to the thermal imaging sensor and the visible light image sensor to acquire the thermal radiation intensity matrix and the three-channel visible light image matrix, respectively.

[0043] Step S2: Extract the thermal contour point set based on the thermal radiation intensity matrix, and extract the texture edge point set based on the three-channel visible light image matrix. Using the thermal contour point set as a reference, calculate the displacement vector, generate two-dimensional deformation field data through spatial interpolation, and use the two-dimensional deformation field data to correct the pixel position of the three-channel visible light image matrix to output the visible light image.

[0044] Step S3: Initialize the first thermal radiation component matrix and the first surface reflection component matrix, use a discriminator to identify the first thermal radiation component matrix and the first surface reflection component matrix, obtain the identification result and fine-tune the value in reverse until the termination condition is met, and output the second thermal radiation component matrix and the second surface reflection component matrix.

[0045] Step S4: Using the second thermal radiation component matrix as the substrate layer, calculate the dynamic weighting coefficient based on the first thermal radiation component matrix, and inject the second surface reflection component matrix into the substrate layer according to the dynamic weighting coefficient to generate an enhanced thermal imaging image.

[0046] This invention addresses the problems of inaccurate registration due to differences in the physical positions of sensors during multimodal image fusion, and the loss of texture details and blurring of thermal features caused by simple pixel-level fusion strategies in existing technologies. It proposes an optimized solution, the core of which involves constructing a decoupled calculation process that separates the temperature radiation information in thermal imaging data from the ambient light reflection information in visible light images. A deformation field is generated using stable temperature boundary information provided by thermal imaging to accurately correct and align the pixel positions in the visible light image. This is primarily applicable to dual-light camera systems and their back-end processing units in nighttime security monitoring or industrial non-destructive testing scenarios. The core data processed are the radiation intensity data representing the temperature distribution of an object's surface captured by the thermal imaging sensor, and the image brightness and color data representing the surface's reflection characteristics of ambient light captured by the visible light sensor. Through this series of processing steps, a high-definition enhanced image data that integrates temperature saliency and texture details is finally output, achieving thermal imaging enhancement.

[0047] In a specific embodiment, step S1 includes the following sub-steps:

[0048] Step S101: Simultaneously send high-level pulses to the thermal imaging sensor and the visible light image sensor in the dual-light camera module as hardware synchronization trigger signals.

[0049] In step S102, after receiving the hardware synchronization trigger signal, the thermal imaging sensor and the visible light image sensor perform exposure actions at the same time point. After receiving the hardware synchronization trigger signal, the thermal imaging sensor captures the first scene data and converts the first scene data into a two-dimensional thermal radiation intensity matrix representing the temperature of each pixel. After receiving the hardware synchronization trigger signal, the visible light image sensor captures the second scene data and converts the second scene data into a three-channel visible light image matrix representing the color and brightness of each pixel.

[0050] Due to physical differences in shutter type, circuit link length, and exposure start logic between thermal imaging sensors (uncooled infrared focal plane arrays) and visible light image sensors (CMOS / CCD), a response delay pre-calibration is first performed. Under controlled lighting conditions, the time difference between the rising edge of the high-level pulse sent by the edge computing processing unit and the exposure start feedback signal output by each sensor is monitored using an oscilloscope. The response delay of the thermal imaging sensor is recorded as follows: The response delay of a visible light image sensor is During step S101, the edge computing processing unit does not simply send two pulses simultaneously. Instead, it executes an asymmetric triggering strategy based on the calibration results, performs phase alignment calculations, and calculates the response difference between the response delay of the thermal imaging sensor and the response delay of the visible light image sensor. .like First, a trigger signal is sent to the thermal imaging sensor, and a high-precision hardware timer is started, with a delay of... After a certain time, a trigger signal is sent to the visible light image sensor. The synchronization accuracy requirement is to control the center point deviation of the exposure start of the thermal imaging sensor and the visible light image sensor within 1ms, preferably 500ns, through the hardware interrupt mechanism of the FPGA or microcontroller, to ensure that the acquired first scene data and the second scene data have physical consistency on the time axis.

[0051] To address circuit delay variations caused by ambient temperature fluctuations, a closed-loop real-time adjustment based on exposure feedback is implemented. This involves real-time reading of the response delay from the thermal imaging sensor and the timestamp of the exposure end signal returned by the visible light image sensor. If the center time deviation between the two signals exceeds a preset time threshold (2ms), the system automatically fine-tunes the hardware timer's delay parameters before acquiring the next image frame. This allows for the dynamic elimination of synchronization drift caused by environmental factors.

[0052] After receiving the data stream in step S103, the system will check the hardware timestamp attached to each frame of data. If the difference between the timestamps of the two images is greater than the preset tolerance range, the frame will be determined to be unaligned and discarded. Alternatively, a small amount of time-domain motion compensation will be performed through the optical flow algorithm to ensure that the two-dimensional thermal radiation intensity matrix and the three-channel visible light image matrix entering step S2 are strictly aligned in the time domain.

[0053] In a specific embodiment, step S2 includes the following sub-steps:

[0054] Step S201: Traverse the pixels in the two-dimensional thermal radiation intensity matrix, calculate the intensity difference between each pixel and its eight neighboring pixels, mark the pixels with intensity differences greater than a preset temperature change threshold as thermal contour points, and obtain a thermal contour point set. Convert the three-channel visible light image matrix into a grayscale image and calculate the brightness gradient between pixels. Mark the pixels with the top 5% of brightness gradients as texture edge points, and obtain a texture edge point set.

[0055] In this embodiment, a preset temperature change threshold is used. It is not a fixed constant, but is dynamically calculated based on the global distribution of the current thermal radiation intensity matrix. First, the gradient mean of the entire thermal radiation intensity matrix is ​​calculated. with standard deviation The mathematical expression for the preset temperature change threshold is:

[0056] ;

[0057] in, To preset the temperature change threshold, As an adjustment coefficient, its value typically ranges from [value range missing] in industrial testing scenarios. In a high-contrast indoor environment, The preset temperature change threshold is relatively large, and it automatically increases to filter background noise, especially in low-contrast outdoor environments. The temperature is relatively small, and the preset temperature change threshold is lowered to capture subtle temperature boundaries.

[0058] Step S202: Based on the set of thermal contour points, for each thermal contour point, match the texture edge point in the set of texture edge points that is spatially closest and whose gradient direction angle is less than a preset angle, and calculate the displacement vector between the thermal contour point and the nearest texture edge point. By spatially interpolating all displacement vectors, generate two-dimensional deformation field data that covers the entire image plane with smooth changes.

[0059] Step S203: Based on the two-dimensional deformation field data, calculate the original coordinates of each pixel position in the newly generated image in the three-channel visible light image matrix, obtain the pixel values ​​of the four pixels around the original coordinates, and perform a weighted average to obtain the corrected pixel information. Based on the corrected pixel information, output a visible light image that is structurally aligned with the two-dimensional thermal radiation intensity matrix.

[0060] In this embodiment, spatial interpolation of the displacement vector generates two-dimensional deformation field data covering the entire image. To accurately characterize the complex nonlinear distortions between sensors, a thin-plate spline interpolation algorithm combined with bilinear interpolation is used for pixel resampling, specifically including:

[0061] Due to the differences in physical position (parallax) and optical lens distortion between thermal imaging sensors and visible light sensors, the coordinate mapping relationship in the mathematical modeling of nonlinear deformation fields is nonhomogeneous. The two-dimensional deformation field is decomposed into horizontal displacement components. and vertical displacement components Fitting is performed using a thin-plate spline function, with the successfully matched thermal profile points and texture edge point pairs from step S202 serving as control points. The corresponding displacement vector serves as the interpolation constraint.

[0062] A regularization parameter is introduced to reduce the interference of local noise on the smoothness of the deformation field, ensuring that the deformation field maintains a smooth transition even in featureless regions.

[0063] After obtaining a continuous two-dimensional deformation field, step S203 corrects the visible light image using inverse mapping logic. Since the calculated original coordinates are usually non-integer, bilinear interpolation is performed to obtain high-precision pixel values ​​for one pixel in the corrected image. Pixel points are calculated through deformation fields. Floating-point coordinates in the original three-channel visible light image matrix Take coordinates The pixel values ​​of the four surrounding integer pixels (top left, top right, bottom left, and bottom right) are weighted and averaged. The weights are determined by the ratio of the distance between the floating-point coordinates and the four integer coordinates, thus ensuring that the corrected image has smooth edges without obvious jaggedness or structural breaks.

[0064] Compared with traditional global affine transformation or simple polynomial fitting, thin plate spline interpolation has significant advantages. It can accurately handle local nonlinear distortions caused by small deviations in lens assembly or different physical distances. While maintaining the physical contour of thermal imaging as a benchmark, it minimizes the stretching distortion of visible light image texture and achieves deep structural alignment between heterogeneous images.

[0065] This invention utilizes stable temperature profile information in thermal imaging to generate a spatial deformation field, and performs pixel-by-pixel nonlinear geometric correction on visible light images. This effectively overcomes the feature misalignment and structural blurring problems caused by sensor physical differences in traditional alignment methods, and achieves deep structural alignment between heterogeneous images.

[0066] In a specific embodiment, step S3 includes the following sub-steps:

[0067] Step S301: Initialize the first thermal radiation component matrix and the first surface reflection component matrix. In each iteration of optimization, perform pixel-by-pixel multiplication on the first thermal radiation component matrix and the first surface reflection component matrix to simulate the predicted visible light image generated by the interaction between ambient light and surface reflection in the visible light band.

[0068] The first thermal radiation component matrix represents the basic thermal distribution of an object, while the first surface reflection component matrix represents the surface texture and reflection characteristics of the object.

[0069] Calculate the first pixel value difference between the predicted visible light image and the visible light image, take the first thermal radiation component matrix as the predicted thermal imaging image, and calculate the second pixel value difference between the predicted thermal imaging image and the two-dimensional thermal radiation intensity matrix.

[0070] In step S302, the first thermal radiation component matrix and the first surface reflection component matrix are input into the discriminator. The discriminator identifies the temperature distribution attribute and reflectivity distribution attribute of the input matrix based on the smoothness of the values ​​in the matrix and the proportion of high-frequency components, and obtains the identification result.

[0071] Step S303: Based on the difference between the first pixel value, the difference between the second pixel value, and the recognition result, the values ​​in the first thermal radiation component matrix and the first surface reflection component matrix are finely adjusted in reverse. When the number of iterations reaches the preset upper limit or the reconstruction difference is less than the preset value, the loop is terminated, and the second thermal radiation component matrix and the second surface reflection component matrix are output.

[0072] In this embodiment, the discriminator aims to distinguish whether the input matrix belongs to the physical fundamental temperature distribution (mainly low frequency) or the surface light reflectance (mainly high frequency texture) by analyzing the statistical characteristics of the input matrix. A discriminator architecture based on a convolutional neural network is employed, combining frequency domain features for extraction. The discriminator architecture contains four convolutional layers, each using... Convolution kernels are used, and the Leaky ReLU activation function is applied.

[0073] The discriminator calculates the proportion of high-frequency components in the matrix by extracting the variance of the image gradient in the first layer and performing global average pooling in subsequent layers.

[0074] The discriminator is pre-trained to identify matrices with high total variation as reflectivity distributions and matrices with high local smoothness (i.e., small numerical gradients) as temperature distributions.

[0075] The difference in the first pixel value represents the difference in visible light reconstruction, and the difference in the second pixel value represents the difference in thermal imaging reconstruction.

[0076] To achieve the reverse fine-tuning in step S303, a multi-objective total loss function was constructed to quantify reconstruction differences, physical attribute constraints, and adversarial game dynamics. The mathematical expression of the total loss function is as follows:

[0077] ;

[0078] in, This is the total loss function value. The visible light reconstruction loss is used to calculate the difference between the predicted visible light image and the actual visible light image. Norms ensure the fidelity of reconstructed images. This is the thermal imaging reconstruction loss, used to calculate the mean square error between the first thermal radiation component matrix and the thermal radiation intensity matrix, ensuring the accuracy of the basic thermal information. As an adversarial loss, it is used to calculate the probability loss of the reflectance matrix being misclassified as a temperature distribution based on the discriminator's output, driving the reflectance matrix to include more high-frequency texture details that the discriminator cannot distinguish. To constrain the smoothness loss, a total variational regularization term is applied to the first thermal radiation component matrix, forcing the first thermal radiation component matrix to remove noise that does not belong to the temperature gradient, so that the features of the first thermal radiation component matrix tend to be smooth.

[0079] Based on the gradient generated by the total loss function, the Adam optimizer is used to inversely adjust the pixel values ​​in the first thermal radiation component matrix and the first surface reflection component matrix. The goal of the adjustment is to minimize the total loss function value. When the iterations enable the discriminator to distinguish the physical properties of the first thermal radiation component matrix and the first surface reflection component matrix with high confidence, and the total reconstruction error is less than the reconstruction error threshold, the decision information has been deeply decoupled.

[0080] The high-confidence discrimination condition is that the discriminator's predicted probability values ​​for the first surface reflection component matrix belonging to the reflectivity distribution attribute and the predicted probability values ​​for the first thermal radiation component matrix belonging to the thermal radiation distribution attribute are both greater than a preset confidence threshold. In this embodiment, the discriminator output layer uses the Sigmoid activation function, and the output range is... Let the discriminator output value be... Let represent the probability that the input matrix belongs to the surface reflection component (high-frequency texture). This represents the probability of belonging to the thermal radiation component (low-frequency smoothing). A confidence threshold is set. The value is 0.85. The total reconstruction error is the weighted sum of the differences between the first and second pixel values, used to measure how similar the simulated image is to the original acquired image. This is applied to pixel value normalization. For images within the range, the reconstruction error threshold is set to 0.01. If the reconstruction error threshold is too large, the iteration stops too early, the decoupling is incomplete, and the generated enhanced thermal imaging image will have severe ghosting. If the reconstruction error threshold is too small, the algorithm will have difficulty converging, the computation time will be too long, and the model will overfit to noise.

[0081] This invention decomposes image data into two independent components—thermal radiation and light reflectivity—that characterize physical properties by constructing an adversarial computing process. This fundamentally avoids the intermodal information interference and detail loss caused by traditional weighted fusion. Finally, in the fusion generation stage, high-frequency reflectivity information is adaptively injected into the image based on thermal radiation information as texture details, enhancing the detail representation at temperature change boundaries. This significantly improves the recognizability of target objects and the realism of the scene environment, ensuring the high fidelity and rich information content of the enhanced image.

[0082] In a specific embodiment, step S4 includes the following sub-steps:

[0083] Step S401: Create a blank image matrix as an enhanced thermal imaging image, and use the second thermal radiation component matrix as the base layer of the enhanced thermal imaging image.

[0084] Step S402: Read the values ​​in the second surface reflection component matrix pixel by pixel, and analyze the degree of change of the values ​​in the preset surrounding area of ​​each pixel in the second thermal radiation component matrix.

[0085] If the intensity value exceeds the preset threshold, the dynamic weight coefficient is increased.

[0086] If the severity value is less than or equal to the preset severity threshold, the dynamic weight coefficient is reduced.

[0087] In this embodiment, a preset degree threshold is used. Used to distinguish between smooth regions and regions of drastic change, performing [action] on the second thermal radiation component matrix. The variance calculation within the neighborhood uses a preset threshold that is the 70th to 85th percentile of the variance distribution of the entire image. This preset threshold ensures that only when the local temperature gradient is greater than a preset proportion of the background undulation is it identified as an edge region that needs detail enhancement.

[0088] To ensure the accuracy of texture injection, dynamic weighting coefficients are used. The degree of drastic change in local numerical values Using a piecewise linear mapping function, the linear mapping formula is:

[0089] ;

[0090] in, For dynamic weighting coefficients, The base injection weight is typically set to 0.2 to ensure that the base texture is preserved in smooth areas. To maximize the enhancement weight, it is typically set to 1.0, used to significantly enhance visible light details at temperature boundaries. The preset intensity saturation value is used to prevent image distortion caused by excessive local noise. To preset the degree threshold, This represents the degree of drastic change in local numerical values.

[0091] Step S403: Multiply the reflectivity value in the second surface reflectivity component matrix by the dynamic weighting coefficient, and add it to the temperature value of the substrate layer. After completing the calculation of all pixels, output the enhanced thermal imaging image.

[0092] In this embodiment, the mathematical expression for the pixel value of the enhanced thermal imaging image is:

[0093] ;

[0094] in, To enhance the pixel values ​​of thermal imaging images, The pixel values ​​of the second thermal radiation component matrix determine the basic thermal distribution. The high-frequency components are extracted from the second surface reflection component matrix after high-pass filtering. To achieve adaptive injection of details for the calculated pixel-level dynamic weight coefficients.

[0095] While there are various alternative techniques in the various subtasks of image processing, it is difficult to form a highly targeted and logically coherent overall solution to achieve the same inventive purpose as this invention.

[0096] The core advantage of this invention lies in its completeness and the synergistic effect of each stage. For example, in the image alignment stage, traditional registration methods based on feature points (such as SIFT and ORB) can be tried. However, when dealing with heterogeneous images with vastly different physical properties, such as thermal imaging and visible light, these methods struggle to find a sufficient number of reliable corresponding points. Their registration accuracy and robustness are far inferior to the scheme of this invention, which uses stable physical contours (temperature boundaries) to generate a deformation field. In the information fusion stage, more classic multi-scale fusion techniques such as wavelet transform or Laplacian pyramids can be employed. However, these techniques are essentially mathematical signal processing; they mix pixel frequency information rather than separating the underlying physical causes (thermal radiation and light reflectivity) through adversarial decoupling, as in this invention. Therefore, they cannot fundamentally avoid intermodal interference. Ultimately, the adaptive detail injection strategy of this invention is based on the successful decoupling of physical properties, which is unmatched by simple weighted averaging or fixed fusion rules. Therefore, any replacement of a single link will disrupt the complete logical chain of this solution from physical alignment and physical decoupling to the integration of physical meaning, and will not achieve the same enhancement effect that balances high fidelity and rich information content.

[0097] Example 2, refer to Figure 2 This paper presents a thermal imaging enhancement system based on multimodal fusion, which includes an acquisition module, a spatial alignment module, a decoupling module, and an adaptive module.

[0098] The acquisition module is used to send hardware synchronization trigger signals to the thermal imaging sensor and the visible light image sensor to acquire the thermal radiation intensity matrix and the three-channel visible light image matrix, respectively.

[0099] The spatial alignment module is used to extract thermal contour point sets based on the thermal radiation intensity matrix and extract texture edge point sets based on the three-channel visible light image matrix. Using the thermal contour point sets as a reference, the displacement vector is calculated, and two-dimensional deformation field data is generated through spatial interpolation. The two-dimensional deformation field data is used to correct the pixel positions of the three-channel visible light image matrix and output the visible light image.

[0100] The decoupling module is used to initialize the first thermal radiation component matrix and the first surface reflection component matrix, use a discriminator to identify the first thermal radiation component matrix and the first surface reflection component matrix, obtain the identification result and fine-tune the value in reverse until the termination condition is met, and output the second thermal radiation component matrix and the second surface reflection component matrix.

[0101] The adaptive module is used to use the second thermal radiation component matrix as the substrate layer, calculate dynamic weighting coefficients based on the second thermal radiation component matrix, and inject the second surface reflection component matrix into the substrate layer according to the dynamic weighting coefficients to generate an enhanced thermal imaging image.

[0102] This invention constructs a complete and innovative information processing chain, from physical alignment to physical attribute decoupling and adaptive fusion. First, a nonlinear geometric correction method based on the physical contours of thermal imaging is used. This method utilizes the stable and well-defined temperature boundaries in thermal imaging as a physical benchmark to generate a smooth two-dimensional deformation field, which is then used to perform pixel-by-pixel precise correction of visible light images. This method overcomes the unreliability of traditional feature matching between heterogeneous images, achieving a deep level of structural alignment. Second, a radiation and reflectivity information decoupling model based on adversarial training is used. By creating an iteratively optimized adversarial process, image data is decomposed from a pixel-mixed state into two independent physical attribute matrices: a thermal radiation component representing the object's basic temperature distribution and a light reflectivity component representing surface texture. This fundamentally avoids the mutual interference and loss of different modal information in traditional fusion methods. Third, an adaptive detail injection fusion strategy based on temperature gradients is employed. When generating the final image, the decoupled high-frequency reflectivity information (texture details) is used as enhancement data and adaptively injected into the image based on thermal radiation information. The intensity of the injection is positively correlated with the gradient of the thermal radiation substrate, that is, the details are enhanced in the boundary areas where the temperature changes drastically, thereby significantly improving the recognition of the target and the realism of the scene.

[0103] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the protection scope of the present invention.

Claims

1. A thermal imaging enhancement method based on multimodal fusion, characterized in that, Includes the following steps: Step S1: Send hardware synchronization trigger signals to the thermal imaging sensor and the visible light image sensor to acquire the thermal radiation intensity matrix and the three-channel visible light image matrix, respectively. Step S2: Extract the thermal contour point set based on the thermal radiation intensity matrix, and extract the texture edge point set based on the three-channel visible light image matrix. Using the thermal contour point set as a reference, calculate the displacement vector, generate two-dimensional deformation field data through spatial interpolation, and use the two-dimensional deformation field data to correct the pixel position of the three-channel visible light image matrix to output the visible light image. Step S3: Initialize the first thermal radiation component matrix and the first surface reflection component matrix, use the discriminator to identify the first thermal radiation component matrix and the first surface reflection component matrix, obtain the identification result and fine-tune the value in reverse until the termination condition is met, and output the second thermal radiation component matrix and the second surface reflection component matrix. Step S4: Using the second thermal radiation component matrix as the substrate layer, calculate the dynamic weighting coefficient based on the second thermal radiation component matrix, and inject the second surface reflection component matrix into the substrate layer according to the dynamic weighting coefficient to generate an enhanced thermal imaging image.

2. The thermal imaging enhancement method based on multimodal fusion as described in claim 1, characterized in that, Step S1 includes the following sub-steps: Step S101: Simultaneously send high-level pulses to the thermal imaging sensor and the visible light image sensor in the dual-light camera module as hardware synchronization trigger signals. In step S102, after receiving the hardware synchronization trigger signal, the thermal imaging sensor and the visible light image sensor perform exposure actions at the same time point. After receiving the hardware synchronization trigger signal, the thermal imaging sensor captures first scene data and converts the first scene data into a two-dimensional thermal radiation intensity matrix representing the temperature of each pixel. After receiving the hardware synchronization trigger signal, the visible light image sensor captures second scene data and converts the second scene data into a three-channel visible light image matrix representing the color and brightness of each pixel.

3. The thermal imaging enhancement method based on multimodal fusion as described in claim 2, characterized in that, Step S2 includes the following sub-steps: Step S201: Traverse the pixels in the two-dimensional thermal radiation intensity matrix, calculate the intensity difference between each pixel and its eight neighboring pixels, mark the pixels with intensity differences greater than a preset temperature change threshold as thermal contour points, and obtain a thermal contour point set. Convert the three-channel visible light image matrix into a grayscale image and calculate the brightness gradient between pixels. Mark the pixels with the top 5% of the brightness gradient as texture edge points, and obtain a texture edge point set.

4. The thermal imaging enhancement method based on multimodal fusion as described in claim 3, characterized in that, Step S2 further includes the following sub-steps: Step S202: Based on the set of thermal contour points, for each thermal contour point, match the texture edge point in the set of texture edge points that is spatially closest and whose gradient direction angle is less than a preset angle, and calculate the displacement vector between the thermal contour point and the nearest texture edge point. By spatially interpolating all displacement vectors, generate two-dimensional deformation field data that covers the entire image plane with smooth changes.

5. The thermal imaging enhancement method based on multimodal fusion as described in claim 4, characterized in that, Step S2 further includes the following sub-steps: Step S203: Based on the two-dimensional deformation field data, calculate the original coordinates of each pixel position in the newly generated image in the three-channel visible light image matrix, obtain the pixel values ​​of the four pixels around the original coordinates, and perform a weighted average to obtain the corrected pixel information. Based on the corrected pixel information, output a visible light image that is structurally aligned with the two-dimensional thermal radiation intensity matrix.

6. The thermal imaging enhancement method based on multimodal fusion as described in claim 5, characterized in that, Step S3 includes the following sub-steps: Step S301: Initialize the first thermal radiation component matrix and the first surface reflection component matrix. In each iteration of optimization, perform pixel-by-pixel multiplication on the first thermal radiation component matrix and the first surface reflection component matrix to simulate the predicted visible light image generated by the interaction between ambient light and surface reflection in the visible light band. Calculate the first pixel value difference between the predicted visible light image and the visible light image, take the first thermal radiation component matrix as the predicted thermal imaging image, and calculate the second pixel value difference between the predicted thermal imaging image and the two-dimensional thermal radiation intensity matrix.

7. The thermal imaging enhancement method based on multimodal fusion as described in claim 6, characterized in that, Step S3 further includes the following sub-steps: Step S302: Input the first thermal radiation component matrix and the first surface reflection component matrix into the discriminator. The discriminator identifies the temperature distribution attribute and reflectivity distribution attribute of the input matrix based on the smoothness of the values ​​in the matrix and the proportion of high-frequency components, and obtains the identification result.

8. The thermal imaging enhancement method based on multimodal fusion as described in claim 7, characterized in that, Step S3 further includes the following sub-steps: Step S303: Based on the difference between the first pixel value, the difference between the second pixel value, and the recognition result, the values ​​in the first thermal radiation component matrix and the first surface reflection component matrix are finely adjusted in reverse. When the number of iterations reaches the preset upper limit or the reconstruction difference is less than the preset value, the loop is terminated, and the second thermal radiation component matrix and the second surface reflection component matrix are output.

9. The thermal imaging enhancement method based on multimodal fusion as described in claim 8, characterized in that, Step S4 includes the following sub-steps: Step S401: Create a blank image matrix as an enhanced thermal imaging image, and use the second thermal radiation component matrix as the base layer of the enhanced thermal imaging image; Step S402: Read the values ​​in the second surface reflection component matrix pixel by pixel, and analyze the degree of change of the values ​​in the preset surrounding area of ​​each pixel in the second thermal radiation component matrix. If the severity value is greater than a preset severity threshold, the dynamic weighting coefficient is increased; If the severity value is less than or equal to a preset severity threshold, the dynamic weighting coefficient is reduced. Step S403: Multiply the reflectivity value in the second surface reflectivity component matrix by the dynamic weighting coefficient, and add it to the temperature value of the substrate layer. After completing the calculation of all pixels, output the enhanced thermal imaging image.

10. A thermal imaging enhancement system based on multimodal fusion, applied in a thermal imaging enhancement method based on multimodal fusion as described in any one of claims 1-9, characterized in that, It includes an acquisition module, a spatial alignment module, a decoupling module, and an adaptive module; The acquisition module is used to send hardware synchronization trigger signals to the thermal imaging sensor and the visible light image sensor to acquire the thermal radiation intensity matrix and the three-channel visible light image matrix, respectively. The spatial alignment module is used to extract thermal contour point sets based on the thermal radiation intensity matrix and extract texture edge point sets based on the three-channel visible light image matrix. Using the thermal contour point sets as a reference, the displacement vector is calculated, and two-dimensional deformation field data is generated through spatial interpolation. The two-dimensional deformation field data is used to correct the pixel positions of the three-channel visible light image matrix and output a visible light image. The decoupling module is used to initialize the first thermal radiation component matrix and the first surface reflection component matrix, use a discriminator to identify the first thermal radiation component matrix and the first surface reflection component matrix, obtain the identification result and fine-tune the value in reverse until the termination condition is met, and output the second thermal radiation component matrix and the second surface reflection component matrix. The adaptive module is used to use the second thermal radiation component matrix as the base layer, calculate dynamic weighting coefficients based on the second thermal radiation component matrix, and inject the second surface reflection component matrix into the base layer according to the dynamic weighting coefficients to generate an enhanced thermal imaging image.

Images