Multi-modal thermal imager intelligent monitoring method based on image processing

Abstract

The invention discloses a multi-modal thermal imager intelligent monitoring method based on image processing. The method comprises the following steps: S1, performing dual-band synchronous imaging; s2, carrying out cross-modal image registration; s3, multispectral image fusion: performing smoothing processing on the infrared and visible light images through anisotropic diffusion to obtain a main structure layer image, calculating to obtain an infrared and visible light detail layer image, and linearly combining the detail layer image through PCA principal component weight, and finally, combining the detail layer image and the main structure layer image according to the self-defined weight to obtain a final fusion image. And S4, gas leakage detection is carried out, so that gas leakage detection and early warning can be realized. According to the invention, a gas enhancement algorithm is adopted to realize real-time detection of leaked gas, interference of most background noise can be eliminated, and the real-time performance and reliability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent detection, and particularly to an intelligent monitoring method for a multimodal thermal imager based on image processing. Background Art

[0002] In the current intelligent management of industrial equipment, early detection of gas leakage usually uses sensors for perception, such as smoke detectors and temperature sensors. Among them, the smoke detector relies on the change in the concentration of smoke particles in the air to judge the occurrence of a fire for early warning. At the same time, in the management and maintenance of fire-fighting equipment, regular manual inspections are required, and sensor data is monitored to monitor the operation status of the equipment to ensure the normal operation of the fire-fighting equipment.

[0003] However, in the initial stage of gas leakage, smoke will not quickly spread to the position of the detector. Especially in large spaces and well-ventilated environments, the detector cannot respond in time, and relying on sensors for early detection of gas leakage will miss the best response time. In terms of daily maintenance, the efficiency of manual inspection is relatively low, and it takes a lot of time. At the same time, when a fire disaster occurs, it is difficult to inspect the equipment in the disaster scene and the equipment with harsh installation environments such as in-pipe sensors, which takes a long time and cannot detect problems in time. Therefore, there is an urgent need for an intelligent management system of a multimodal thermal imager for image processing to solve such problems.

[0004] Infrared imaging technology is a technology that generates images by capturing and converting infrared radiation emitted by objects. All objects with a temperature higher than absolute zero radiate infrared rays, which are captured by an infrared detector and converted into electrical signals, thereby generating a visual infrared image, and infrared gas leakage that cannot be captured by a visible light image can be photographed. This technology can be used under day and night conditions and is less affected by the environment, but usually has low resolution and poor texture.

[0005] Visible light images usually have high spatial resolution, a considerable amount of details and light-dark contrast; they are suitable for human visual perception and contain rich texture and detail information. Therefore, infrared and visible light images have complementary characteristics.

[0006] Existing traditional gas target detection algorithms for infrared imaging include optical flow method, frame difference method, and background modeling method. However, due to the complex and diverse forms of gas targets and industrial scenes, and traditional algorithms are difficult to distinguish the types of moving targets, it is difficult to meet the requirements of gas leakage detection. With the continuous development of deep learning, researchers have begun to add deep learning to gas leakage detection to improve the reliability and accuracy of detection. However, deep learning algorithms require higher computing power support and upfront training costs, and it is difficult to achieve fast and real-time gas leakage detection and segmentation tasks. Summary of the Invention

[0007] The object of the present invention is to provide an intelligent monitoring method for a multimodal thermal imager based on image processing, aiming at the problems in the prior art.

[0008] To achieve this, the above object of the present invention is realized by the following technical solutions:

[0009] An intelligent monitoring method for a multimodal thermal imager based on image processing includes:

[0010] S1, dual-band synchronous imaging: Using time-synchronized dual-band infrared and visible light detectors to respectively obtain three-channel visible light data and single-channel infrared data;

[0011] S2, cross-modal image registration: Using feature point matching to register the infrared data and visible light data obtained in step S1 to achieve registration of infrared and visible light images under different fields of view. Through cropping and projective transformation of the visible light data and registration with the infrared image, registration of infrared data and visible light data under different fields of view is achieved;

[0012] S3, multispectral image fusion: Smoothing the infrared and visible light images through anisotropic diffusion to obtain the main structure layer image, and calculating the detail layer images of the infrared and visible light. Linearly combining the detail layer images through PCA principal component weights, and finally obtaining the final fused image according to the custom weights by combining the detail layer images and the main structure layer image;

[0013] S4, gas leakage detection: Processing the fused image obtained in step S3 with a gas enhancement algorithm, calculating the absolute difference between adjacent frames, , representing the absolute difference between adjacent frames, performing threshold segmentation on , differentiating the gas region from the noise through an adaptive threshold, extracting the gas leakage region, and marking the connected regions in the binary image judged as the gas leakage region to achieve the detection and warning of gas leakage.

[0014] While adopting the above technical solutions, the present invention can also adopt or combine the following technical solutions:

[0015] As a preferred technical solution of the present invention: The input data in step S2 includes infrared data and visible light data , where represents the infrared image, containing the temperature information of each pixel point, represents the three-channel visible light data of RGB, containing the light intensity of each pixel point, represents the height and width of the graphic pixel number;

[0016] To ensure the alignment of multimodal data in the same coordinate system, for and Perform image registration, , where, is the aligned data, represents the registration transformation matrix, which is determined by projective transformation after manual calibration.

[0017] As a preferred technical solution of the present invention: Step S3 includes the following steps:

[0018] S31, perform anisotropic diffusion on and for smoothing to obtain and , retaining the main structure;

[0019] Calculate the detail part: and . and respectively represent the infrared image after registration and smoothing and the visible light image after smoothing;

[0020] S32, perform covariance analysis on the pixel values of and to extract the principal component directions and , linearly combine the detail images with the principal component weights, , represents the detail image after being combined with the principal component weights;

[0021] S33, combine the smoothed images and , , represents the image after weight fusion of the smoothed image, retaining the common structure, and respectively represent the weights of infrared and visible light; add and fuse the detail part and the structure part , , is the final fusion result of infrared and visible light.

[0022] As a preferred technical solution of the present invention: Step S4 specifically includes the following steps: the registered visible light image , the collected infrared image , use the continuous video frames and as the input of the gas enhancement algorithm, t represents the timestamp, and use Gaussian filtering to smooth the image, , , where x and y represent the width and height of the input image respectively. Calculate the absolute difference between adjacent frames of visible light and infrared images. , represents the absolute difference between adjacent frames. For perform threshold segmentation. By using an adaptive threshold, distinguish the moving object regions and backgrounds of infrared images and visible light images respectively. , where T(x, y) is used as the adaptive threshold and can be dynamically adjusted based on local statistics. Obtain the binary mask of the visible light image , which contains the motion changes of background noise, and the binary mask of the infrared image , which contains the motion changes of background noise and the leakage situation of infrared gas. Therefore, a binary mask representing the leakage area of infrared gas can be obtained , = , excluding the influence of background motion noise (such as people walking, vehicles passing by, etc.), mark the connected regions in the binary image and determine it as the gas leakage area.

[0023] As a preferred technical solution of the present invention: It further includes step S5, dynamic pseudo-color visualization of the gas leakage area: Establish a dynamic color mapping table, generate a gradient color according to the gas intensity or motion characteristics for pseudo-color display, visually observe the spatial distribution and dynamic changes of the gas, and locate the gas leakage area to achieve visualization.

[0024] As a preferred technical solution of the present invention: The way to perform pseudo-color analysis on the leakage gas area of infrared and fused images in step S5 is: The binary mask after frame difference method processing represents the gas area, represents the background. Establish a dynamic color mapping table, generate a gradient color according to the gas intensity or motion characteristics, and define the interpolation from the starting color to the ending color : , where s is the normalized gas feature; linearly interpolate to generate a gradient color, , mix it into the input image, and output , where R(s) refers to the intensity value of the red component calculated at the normalized ratio s, G(s) refers to the intensity value of the green component calculated at the normalized ratio s, and B(s) refers to the intensity value of the blue component calculated at the normalized ratio s; Through pseudo-color display, the spatial distribution and dynamic changes of the gas can be visually observed. And locate the gas leakage area.

[0025] Compared with the prior art, the intelligent monitoring method of the multi-modal thermal imager based on image processing of the present invention has the following beneficial effects: By capturing the abnormal gas temperature characteristics through the infrared spectrum, combining the visible light texture information, and using the PCA weighted fusion algorithm to generate a composite image, the limitations of a single modality are solved, and the detection accuracy is improved; By adopting the three-frame difference method combined with spatio-temporal joint filtering and motion compensation, instantaneous interference is eliminated, and the anti-interference ability is improved, realizing dynamic gas enhancement priority; By establishing an HSV dynamic color table for pseudo-color mapping and mask overlay, the contour accuracy of the leakage area is improved, which is better than the direct display of the thermal map, realizing visualization enhancement.

[0026] The intelligent monitoring method of the multi-modal thermal imager based on image processing of the present invention proposes a gas enhancement fusion algorithm. Through the analysis and fusion of visible light and infrared images, it can not only realize the real-time detection and segmentation of gas targets, but also eliminate the interference of other moving targets. It can be applied in various scenarios. Brief Description of the Drawings

[0027] Figure 1 It is a flow block diagram of the intelligent monitoring method of the multi-modal thermal imager based on image processing of the present invention.

[0028] Figure 2 It is an effect diagram of the registration of infrared data and visible light data in the intelligent monitoring method of the multi-modal thermal imager based on image processing of the present invention;

[0029] Figure 3 It is an effect diagram of the fusion of infrared and visible light in the intelligent monitoring method of the multi-modal thermal imager based on image processing of the present invention. Detailed Embodiments

[0030] The present invention will be further described in detail with reference to the accompanying drawings and specific embodiments.

[0031] As Figures 1 - 3 shown, the intelligent monitoring method of the multi-modal thermal imager based on image processing of the present invention is used to overcome the problems in the prior art that the sensor perception effect is lagging, and it is impossible to give an early warning of the gas leakage scene in time; the efficiency of manual inspection is relatively low, and it is difficult to inspect the equipment in the disaster scene and the equipment with harsh installation environment, which takes a long time and it is impossible to find problems in time.

[0032] The present invention relates to the technical field of industrial equipment management, and in particular to an intelligent monitoring method for a multimodal thermal imager based on image processing. The present invention adopts an early detection method for industrial gas leakage based on the fusion of thermal imaging and visible light, and realizes more accurate and earlier detection of industrial gas leakage by fusing various perception data; combines thermal imaging and visible light images, and uses a gas enhancement algorithm for real-time analysis, and the fusion provides more comprehensive environmental information, including both temperature anomalies and signs of gas leakage under visible light, such as open flames and smoke; at the same time, analyzes the changes in the visible light and infrared spectra, and detects early signs of leakage, including local temperature anomalies and smoke generation, before the gas spreads widely, so as to achieve earlier warning and improve the reliability of the system.

[0033] The intelligent monitoring method for a multimodal thermal imager based on image processing of the present invention includes: a multimodal image fusion module that combines infrared data and visible light image data, and performs real-time analysis on the fused multimodal data through a gas enhancement mode to capture temperature anomalies and signs of gas leakage under visible light; an early gas leakage warning module that uses spectral data extraction and analysis to extract visible light and infrared spectral information from the fused image data, analyzes the spectral change rate at each pixel point, and captures early signs of gas leakage, including local temperature anomalies and the generation of nascent smoke; a gas detection false colorization module that generates a false color image by establishing a mapping table between infrared gray values and color bands; a leakage location identification module that is used for gas leakage location and locates the position of gas leakage by using infrared and visible light fusion data.

[0034] The intelligent monitoring method for a multimodal thermal imager based on image processing of the present invention includes the following steps:

[0035] Dual-band synchronous imaging: Adopt time-synchronized dual-band infrared and visible light detectors to respectively obtain three-channel visible light data and single-channel infrared data;

[0036] Thermal imaging visible light image fusion: Use the feature point matching method to realize the registration of infrared data and visible light image data under different fields of view, and combine the infrared data and visible light image data

[0037] Gas leakage warning: Based on the analysis of the changes in the visible light and infrared spectra of the fused data, use a gas enhancement algorithm to perform real-time analysis on the fused multimodal data, and detect early signs of gas leakage, including local temperature anomalies and nascent smoke;

[0038] False colorization of the leaked gas area of the infrared and fused images: By establishing a mapping relationship between each gray value and the RGB triple; to enhance the visual effect or highlight the gas leakage information, use the infrared and fused image information extracted in step S1 and step S2 to perform false colorization performance of the gas area;

[0039] Furthermore, the method for thermal imaging visible light image fusion analysis is as follows: The input data includes infrared data and visible light data , where represents the thermal imaging image, containing the temperature information of each pixel point, represents the visible light image of the RGB three channels, containing the light intensity of each pixel point, represents the height and width of the graphic pixel number;

[0040] 1) To ensure the alignment of multi-modal data in the same coordinate system, perform image registration on and , where , among which is the aligned data, represents the registration transformation matrix, which is determined by projective transformation through manual calibration;

[0041] 2) Perform smoothing processing on and through anisotropic diffusion to obtain and (retaining the main structure). Calculation details: and . and respectively represent the infrared image after registration and smoothing processing and the visible light image after smoothing processing. Perform covariance analysis on the pixel values of and to extract the principal component directions and , and linearly combine the detail images with the principal component weights, , represents the detail image combined with the principal component weights; by combining the smoothed images and , , represents the weight fusion image after smoothing processing (retaining the common structure), and respectively represent the weights of infrared and visible light; add and fuse the detail part and the structure part , , is the final fusion result of infrared and visible light.

[0042] Furthermore, the method for gas leakage warning analysis is as follows: The registered visible light image , the collected infrared image , the continuous video frames and As the input of the gas enhancement algorithm, t represents the timestamp, and the image is smoothed using Gaussian filtering. , , where x and y represent the width and height of the input image respectively, and the absolute difference between adjacent frames of visible light and infrared images is calculated. , represents the absolute difference between adjacent frames. Perform threshold segmentation on , and distinguish the moving object regions and backgrounds of infrared images and visible light images respectively through adaptive thresholds. , where T(x, y) is used as the adaptive threshold and can be dynamically adjusted based on local statistics (such as mean and standard deviation). The binary mask of the visible light image (including the motion changes of background noise) and the binary mask of the infrared image (including the motion changes of background noise and the leakage situation of infrared gas) are obtained respectively. Therefore, a binary mask , = can be obtained, excluding the influence of background motion noise (such as people walking, vehicles passing by, etc.). Mark the connected regions in the binary image and determine them as gas leakage regions.

[0043] The pseudo-color analysis method for the gas leakage regions in infrared and fused images is as follows: The binary mask after being processed by the frame difference method represents the gas region, represents the background. A dynamic color mapping table is established, and a gradient color is generated according to the gas intensity or motion characteristics. Define the interpolation from the starting color to the ending color : , where s is the normalized gas feature. Linear interpolation generates a gradient color, , which is mixed into the input image, and the output is obtained. R(s) refers to the intensity value of the red component calculated at the normalized ratio s, G(s) refers to the intensity value of the green component calculated at the normalized ratio s, and B(s) refers to the intensity value of the blue component calculated at the normalized ratio s. Through pseudo-color display, the spatial distribution and dynamic changes of the gas can be visually observed. And the gas leakage regions can be located.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention adopts a gas leakage detection method based on the fusion of thermal imaging and visible light. By fusing a variety of perception data, it realizes more accurate and earlier detection of gas leakage; combines thermal imaging and visible light images, and uses a gas enhancement algorithm for real-time analysis, and the fusion provides more comprehensive environmental information, including both temperature anomalies and texture information under visible light.

[0046] The present invention uses a gas enhancement algorithm to realize real-time detection of leaked gas, and can exclude the interference of most background noises (such as people walking, cars passing by, etc.), improving the real-time performance and reliability of the system. Moreover, the algorithm used has low requirements for the computing power of hardware devices and can be deployed and run on most systems.

[0047] The present invention adopts the establishment of a gas leakage area mask and the establishment of a dynamic color mapping table to realize the false coloring of the gas leakage area, improving the readability and aesthetics of the system.

[0048] The fusion of visible light and infrared images is a key technology in the field of multimodal image processing. By organically combining the delicate texture details and distinct color features contained in the visible light image with the unique thermal radiation information captured by the infrared image, a composite image that retains the characteristics of the original two images and has new advantages is generated. The present invention significantly improves the detail expression ability of the image and greatly enhances people's cognitive and interpretation abilities for complex scenes. By using a series of advanced image fusion algorithms, including but not limited to fusion strategies based on wavelet transform, fusion methods at the frequency domain level, and fusion technologies relying on deep learning models, etc., it can efficiently integrate multi-source data from visible light sensors and infrared sensors, providing strong support for more accurate and stable target recognition tasks and comprehensive environmental perception applications.

[0049] Embodiment 1

[0050] The intelligent monitoring method of a multimodal thermal imager based on image processing in the present invention adopts an early detection method for industrial gas leakage based on thermal imaging and visible light. By fusing thermal imaging and visible light data, it realizes more accurate and earlier detection of industrial gas leakage. Specifically, it includes the following steps: 1) Simultaneously image infrared and visible light, and use time-synchronized infrared and visible light detectors to respectively obtain radiation images in the gas absorption band and non-absorption band; 2) Registration of infrared and visible light images: The field of view of the infrared detector is , and the image size is ; the field of view of the visible light detector is , and the image size is , the transformation matrix is determined by feature point matching, and then infrared and visible light image registration is achieved through projective transformation; 3) Infrared and visible light image fusion: The main structure layer and detail layer of the image are obtained through anisotropic diffusion processing, and then the detail layer is linearly combined by PCA principal component weights. Finally, the detail layer and the main structure layer are added together to obtain the infrared and visible light image fusion result; 4) Gas leakage detection: Based on the analysis of the fused data, the changes in visible light and infrared spectra are analyzed. The gas enhancement algorithm is used to perform real-time analysis on the fused multi-modal data to detect early signs of gas leakage, including local temperature anomalies and smoke; 5) False color display of gas leakage area: By establishing a mapping table between infrared gray values and color bands, a false color image is generated.

[0051] The above specific implementation manners are used to explain the present invention. They are only the preferred embodiments of the present invention and do not limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and scope of the claims of the present invention fall within the protection scope of the present invention.

Claims

1. An intelligent monitoring method for a multimodal thermal imager based on image processing, characterized in that, Including: S1, dual-band synchronous imaging: Using a time-synchronized dual-band infrared and visible light detector to respectively obtain three-channel visible light data and single-channel infrared data; S2, cross-modal image registration: Implementing the registration of infrared and visible light images under different fields of view by using the feature point matching method for the infrared data and visible light data obtained in step S1. Through the cropping and projective transformation of the visible light data and the registration with the infrared image, the registration of infrared data and visible light data under different fields of view is achieved; S3, multi-spectral image fusion: Smoothing the infrared and visible light images through anisotropic diffusion to obtain the main structure layer image, calculating the detail layer images of infrared and visible light, linearly combining the detail layer images through PCA principal component weights, and finally obtaining the final fused image according to the custom weights to combine the detail layer image and the main structure layer image; S4, Gas Leak Detection: Process the fused image obtained in step S3 with a gas enhancement algorithm, calculate the absolute difference between adjacent frames, denote the absolute difference between adjacent frames, and perform threshold segmentation on to distinguish the gas region from the noise through an adaptive threshold, extract the gas leakage region, and label the connected regions in the binary image which is determined as the gas leakage region to achieve the detection and early warning of gas leakage.

2. The intelligent monitoring method of the multimodal thermal imager based on image processing according to claim 1, wherein The input data in step S2 includes infrared data and visible light data , where represents an infrared image, containing the temperature information of each pixel point, represents the visible light data of the RGB three channels, containing the light intensity of each pixel point, represents the height and width of the number of graphic pixels; To ensure the alignment of multimodal data in the same coordinate system, and are subjected to image registration. , where is the aligned data, represents the registration transformation matrix, which is determined by the projective transformation after manual calibration.

3. The intelligent monitoring method of the multimodal thermal imager based on image processing according to claim 2, characterized in that, Step S3 includes the following steps: S31, through anisotropic diffusion for and smoothing processing, to obtain and , retaining the main structure; Calculation details: and , and represent the infrared image after registration and smoothing and the visible light image after smoothing, respectively; S32, perform covariance analysis on the pixel values of and to extract the principal component direction and , linearly combine the detail images with the principal component weights, , denote the detail image after being combined with the principal component weights; S33, by the smoothed image and are combined, , indicating the weight-fused image after smoothing, retaining the common structure, and respectively represent the weights of infrared and visible light; the detailed part and the structural part are added and fused, , which is the final result after the fusion of infrared and visible light.

4. The intelligent monitoring method of a multimodal thermal imager based on image processing according to claim 1, wherein: Step S4 Specifically, it includes the following steps: the registered visible light image collected , the infrared image collected , the continuous video frames and are used as the input of the gas enhancement algorithm. t represents the timestamp. The image is smoothed using Gaussian filtering, , , where x and y represent the width and height of the input image respectively. Calculate the absolute difference between adjacent frames of the visible light and infrared images, , represents the absolute difference between adjacent frames. Perform threshold segmentation on , and distinguish the moving object regions and backgrounds of the infrared image and visible light image respectively through adaptive thresholds, , where T(x, y) is used as the adaptive threshold and can be dynamically adjusted based on local statistics. Obtain the binary mask of the visible light image, which contains the motion change of the background noise, and the binary mask of the infrared image, which contains the motion change of the background noise and the leakage situation of the infrared gas. Therefore, a binary mask representing the leakage region of the infrared gas can be obtained, = , excluding the influence of background motion noise, and label the connected regions in the binary image judged as the gas leakage region.

5. The intelligent monitoring method of the multimodal thermal imager based on image processing according to claim 1, characterized in that, It further includes step S5, dynamic pseudo-color visualization of the gas leakage area: Establishing a dynamic color mapping table, generating a gradient color through pseudo-color display according to the gas intensity or motion characteristics, visually observing the spatial distribution and dynamic changes of the gas, and locating the gas leakage area to achieve visualization.

6. The intelligent monitoring method of the multimodal thermal imager based on image processing according to claim 1, wherein, The method for pseudo-color analysis of the infrared and fused image leakage gas region in step S5 is as follows: the binary mask after frame difference method processing represents the gas region, represents the background, establish a dynamic color mapping table, generate a gradient color according to the gas intensity or motion characteristics, and define the interpolation from the starting color to the ending color : , where s is the normalized gas feature; Generate a gradient color through linear interpolation, mix it into the input image, and output , where R(s) refers to the intensity value of the red component calculated at the normalized ratio s, G(s) refers to the intensity value of the green component calculated at the normalized ratio s, and B(s) refers to the intensity value of the blue component calculated at the normalized ratio s; through pseudocolor display, the spatial distribution and dynamic changes of the gas can be visually observed, and the gas leakage area can be located.

Images