Fire source identification and location method based on the fusion of solid-state laser radar and thermal imaging vision

Abstract

The present invention discloses a fire source identification and location method that integrates solid-state laser radar and thermal imaging vision. The method fuses laser point cloud images with thermal imaging images. Through a fire source identification and location algorithm based on the registration and fusion of thermal imaging vision and laser radar point clouds, the depth information of the laser radar point cloud is combined with the color information of the thermal imaging vision, thereby overcoming the shortcoming of single thermal imaging lacking fire source location information. The method of the present invention first processes the thermal imaging image using image thresholding to address the issue of low thermal image resolution. The processed thermal imaging image is then subjected to a posture transformation to colorize the point cloud, addressing the shortcoming of the point cloud lacking color information. Ultimately, the points of the two images are fused to obtain a fused image with depth and color information, thereby effectively improving the accuracy of fire source identification and location based on the fused image.

Description

Technical Field

[0001] The present invention relates to a flame identification and positioning method, specifically a fire source identification and positioning method that integrates solid-state laser radar and thermal imaging vision, and belongs to the technical field of fire early warning and positioning. Background Art

[0002] Fire accidents are among the most frequent, dangerous, and devastating of all disasters. They pose serious challenges, including widespread harm and widespread destruction, causing significant losses to social production, people's lives, and property. In recent years, the gradual expansion of petrochemical industries, ultra-high voltage power transmission and transformation systems, and large-scale warehousing and logistics parks has significantly increased the number of fire hazards and various disaster-causing factors, making these areas key areas for fire prevention.

[0003] Traditional fire detection technologies typically detect fire sources based on temperature changes, combustion gas composition, and aerosols during a fire. However, these characteristic signals gradually weaken during spatial propagation and are significantly affected by wind speed and direction surrounding the fire. Consequently, traditional fire source detection methods have low accuracy. With the advancement of digital image processing technology, image and video-based fire detection technologies have gradually emerged and are being applied. However, traditional fire image feature extraction methods have limited ability to distinguish different fire scenes and types, resulting in low recognition accuracy.

[0004] Using deep learning frameworks for flame detection and recognition, and using multi-sensor fusion for flame image processing and recognition, have become two of the most commonly used approaches. One study proposed a flame edge detector based on convolutional neural networks (CNNs), which effectively generates and extracts edge maps. However, fires often occur in complex environments, and a single fire source is rarely present, which can lead to significant recognition errors. Others have combined CNNs and their improved models with video streams to extract fire source features. However, these are limited by the network's bulky structure and demanding hardware configuration, making them impractical for computation on standard equipment and placing significant constraints on the algorithm. Another approach uses deep learning neural networks combined with video streams, employing multi-task learning strategies to jointly identify smoke and estimate optical flow, simultaneously capturing both intra-frame appearance features and inter-frame motion features. Deep learning can adaptively extract fire source features and is less susceptible to environmental influences. However, most deep learning approaches using video streams for feature extraction fail to obtain depth information about the fire source, making it ineffective for guiding firefighting robots to extinguish fires.

[0005] Therefore, using thermal images alone cannot represent depth information, and LiDAR images cannot obtain temperature information. Therefore, an image fusion strategy is needed to fuse visible light and LiDAR images. Multi-sensor fusion, which leverages the data features of various sensors for fire source identification and detection, is another mainstream approach. Existing multi-sensor fusion methods for flame identification mostly use visible light vision fusion. However, if infrared thermal imaging vision is fused with laser images, the differences between laser and infrared images also limit the information fusion between the images. Due to the number of laser lines, the accuracy of laser images is somewhat different from that of infrared images. Laser images have clear objects and sharp edges, while thermal images have blurred images and low differentiation. This makes it difficult to directly superimpose and fuse laser and thermal images. Currently, there is no better fusion method. Therefore, how to fuse laser and thermal images to improve the accuracy of flame identification and location is one of the research directions in this industry. Summary of the Invention

[0006] In response to the problems existing in the above-mentioned prior art, the present invention provides a fire source identification and positioning method that integrates solid-state laser radar and thermal imaging vision, which can better integrate laser point cloud images and thermal imaging images, thereby improving the accuracy of flame identification and positioning.

[0007] To achieve the above objectives, the present invention adopts a technical solution: a method for fire source identification and positioning that integrates solid-state laser radar and thermal imaging vision, which specifically comprises the following steps:

[0008] A. Install solid-state laser radar and thermal imaging cameras at the locations to be monitored, and connect them to a computer to complete the deployment of the flame recognition and positioning system.

[0009] B. Use a computer to calibrate the thermal imaging camera's internal parameter coefficients using Zhang Zhengyou's calibration method;

[0010] C. Use a computer to unify the clock sources of the thermal imaging camera and solid-state lidar, and then align the sampling timestamps of the two to complete the time synchronization of the thermal imaging camera and solid-state lidar;

[0011] D. When the fire source identification and location begins, the solid-state laser radar and thermal imaging camera begin to collect radar point cloud images and thermal imaging images respectively, and transmit the collected images to the computer;

[0012] E. Use computer to solve and obtain the external parameters for the joint calibration of thermal imaging camera and solid-state lidar;

[0013] F. The computer first selects a frame of thermal imaging image acquired in step D, and then performs binary thresholding processing on the frame of image;

[0014] G. The computer selects a radar point cloud image frame that is time-synchronized with the thermal imaging frame based on the thermal imaging frame processed in step F. Then, based on the extrinsic parameters determined in step E, the computer transforms the coordinates of each point in the radar point cloud image so that the points correspond one-to-one with the thermal imaging frame, thereby fusing each point of the two images and ultimately obtaining a fused image with depth and color information.

[0015] H. Determine the scope and location of the fire source based on the fused image obtained in step G.

[0016] Furthermore, the calibration of the camera intrinsic coefficients in step B is specifically as follows:

[0017] Assume that the pixel coordinates of each point in the image taken by the thermal imaging camera are [u, v] T , let Oxyz be the camera coordinate system. After pinhole projection in the pinhole model, the coordinates of the space point P[X′, Y′, Z′] in the real world on the imaging plane are P camera [X,Y,Z], the camera focal length is f, then we have

[0018]

[0019] Set a pixel plane ouv on the physical imaging plane and get the pixel coordinates of P′ on the pixel plane: [u, v] T ; The original o is located in the upper left corner of the image, the u axis is parallel to the x axis to the right, and the v axis is parallel to the y axis downward; the pixel coordinates are scaled by α times on the u axis and β times on the v axis. At the same time, the origin is translated [c x , c y ] T Then, the coordinates of P′ and the pixel coordinates [u, v] T The relationship is

[0020]

[0021] And let αf=f x , βf=f y , then

[0022]

[0023] Rewriting the formula into matrix form, using homogeneous coordinates, we can get

[0024]

[0025] In formula (4), K is the internal parameter of the camera, and the internal parameter coefficient of the thermal imaging camera is calculated.

[0026] Furthermore, the specific process of jointly calibrating the external parameters in step E is as follows:

[0027] Assume that each pixel of the thermal image is [u, v] T , each pixel of the radar point cloud image is P[X′,Y′,Z′], solve the external parameters between the thermal imaging camera and the solid-state lidar

[0028]

[0029] According to formula (5), the external parameter The value of

[0030] Furthermore, the specific process of performing binary thresholding processing in step F is as follows:

[0031] Assume that the temperature of each pixel in a thermal imaging image is T (x,y) , the unit is ℃, and the RGB value corresponding to each pixel is C (x,y) (i, j, k), set the threshold as ε, and reassign the RGB value of each pixel of the thermal imaging image according to the following rules:

[0032]

[0033] When the temperature of a pixel point T (x,y) When the pixel value is greater than or equal to the threshold ε, the RGB value corresponding to the pixel is reassigned to (255, 0, 0). Otherwise, the RGB value of the pixel is maintained. After all pixels of the thermal imaging image are processed, the binary thresholding process of the thermal imaging image is completed.

[0034] Furthermore, the specific process of coordinate transformation of each pixel point of the radar point cloud image in step G is as follows:

[0035] According to the obtained intrinsic parameter K of the thermal imaging camera and the external parameter of the joint calibration of the thermal imaging camera and the solid-state laser radar Then each radar point cloud point P of the laser radar lidar =(x, y, z) is converted to the coordinates of the camera coordinate system. The specific conversion process is:

[0036]

[0037] Through the above transformation, the pixel points of the thermal imaging image can be compared with the point cloud point P lidar =(x, y, z), so that each point of the radar point cloud image corresponds to the frame of thermal imaging image after coordinate transformation, realizing the fusion of each point of the two images, and finally obtaining a fused image with depth information and color information.

[0038] Compared with the existing technology, in order to cope with the variability of fire situations and improve the efficiency of fire source identification and positioning, the present invention fuses laser point cloud images with thermal imaging images. Through a fire source identification and positioning algorithm based on thermal imaging vision and lidar point cloud registration and fusion, the depth information of the lidar point cloud is combined with the color information of the thermal imaging vision, thereby making up for the shortcoming that single thermal imaging has no fire source location information; in addition, the present invention first calibrates the internal parameters of the thermal imaging camera, then calibrates the camera and radar with external parameters, and then uses image thresholding to process the thermal imaging image to solve the problem of low resolution of the thermal imaging image. Then, the processed thermal imaging image is transformed into the point cloud according to the external parameter calibration, thereby solving the shortcoming that the point cloud has no color information. Finally, the fusion of each point of the two images is realized to obtain a fused image with depth information and color information, thereby effectively improving the accuracy of fire source identification and positioning according to the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention;

[0040] Figure 2 is a flowchart of a binary thresholding process according to an embodiment of the present invention;

[0041] Figure 3 is a thermal imaging image after binary thresholding processing in an embodiment of the present invention;

[0042] Figure 4 4a is a fused three-dimensional fire scene image in an embodiment of the present invention, wherein 4a is a left view and 4b is a right view;

[0043] Figure 5 5a is a comparison diagram before and after fusion according to an embodiment of the present invention, wherein 5a is an image not fused according to the present invention, and 5b is an image fused according to the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below.

[0045] like Figure 1 As shown, the specific steps of the present invention are:

[0046] A. Install solid-state laser radar and thermal imaging cameras at the locations to be monitored, and connect them to a computer to complete the deployment of the flame recognition and positioning system.

[0047] B. Use a computer to calibrate the thermal imaging camera using the Zhang Zhengyou calibration method to calibrate the camera's intrinsic parameter coefficients. The intrinsic parameters of a thermal imaging camera are the camera's intrinsic properties, reflecting the transformation relationship between spatial points and pixel points. In the process of registering solid-state laser radar with thermal imaging images, the camera's intrinsic parameters affect the registration pose accuracy. The thermal imaging camera in this embodiment is a Hikvision DS-2TD2166-7 / V1 thermal imaging network camera. The steps for performing the internal parameter calibration are as follows: First, place the calibration plate in front of the heater and heat it for 5 minutes. To achieve better results, control the distance between the calibration plate and the heater within 10 cm; then, use the thermal imaging camera to take pictures of the thermal imaging calibration plate from different angles to obtain calibration images; finally, input all the obtained calibration images into the camera calibration toolbox under Matlab, and the calibration toolbox will automatically eliminate images that do not meet the calibration requirements. The toolbox uses the Zhang Zhengyou calibration method to calibrate the camera's intrinsic parameter coefficients, specifically:

[0048] Assume that the pixel coordinates of each point in the image taken by the thermal imaging camera are [u, v] T , let Oxyz be the camera coordinate system. After pinhole projection in the pinhole model, the coordinates of the space point P[X′, Y′, Z′] in the real world on the imaging plane are P camera [X,Y,Z], the camera focal length is f, then we have

[0049]

[0050] Set a pixel plane ouv on the physical imaging plane and get the pixel coordinates of P′ on the pixel plane: [u, v] T ; The original o is located in the upper left corner of the image, the u axis is parallel to the x axis to the right, and the v axis is parallel to the y axis downward; the pixel coordinates are scaled by α times on the u axis and β times on the v axis. At the same time, the origin is translated [c x , c y ] T Then, the coordinates of P′ and the pixel coordinates [u, v] T The relationship is

[0051]

[0052] And let αf=f x , βf=f y , then

[0053]

[0054] Rewriting the formula into matrix form, using homogeneous coordinates, we can get

[0055]

[0056] In formula (4), K is the internal parameter of the camera,

[0057] Finally, the internal parameter matrix of the thermal imaging camera is obtained:

[0058]

[0059] C. Use a computer to unify the clock sources of the thermal imaging camera and solid-state LiDAR, and then align the sampling timestamps of the two. Since the sampling frequency of the thermal imaging camera is higher than that of the solid-state LiDAR, after each LiDAR sampling cycle, the nearest camera data frame is found and the two frames are aligned to complete the time synchronization of the thermal imaging camera and solid-state LiDAR.

[0060] D. When the fire source identification and location begins, the solid-state laser radar and thermal imaging camera begin to collect radar point cloud images and thermal imaging images respectively, and transmit the collected images to the computer;

[0061] E. Use computer to solve and obtain the external parameters for the joint calibration of thermal imaging camera and solid-state lidar. The specific process is as follows:

[0062] Assume that each pixel of the thermal image is [u, v] T , each pixel of the radar point cloud image is P[X′,Y′,Z′], solve the external parameters between the thermal imaging camera and the solid-state lidar

[0063]

[0064] According to formula (5), the external parameter Value:

[0065]

[0066] F. The computer first selects a frame of thermal imaging image acquired in step D, and then performs binary threshold processing on the frame of image, such as Figure 2 As shown, the specific process is:

[0067] Assume that the temperature of each pixel in a thermal imaging image is T (x,y) , the unit is ℃, and the RGB value corresponding to each pixel is C (x,y) (i, j, k), set the threshold as ε, and reassign the RGB value of each pixel of the thermal imaging image according to the following rules:

[0068]

[0069] When the temperature of a pixel point T (x,y)When the value is greater than or equal to the threshold ε, the RGB value corresponding to the pixel is reassigned to (255, 0, 0). Otherwise, the RGB value of the pixel is maintained or reassigned to (85, 85, 85). After all the pixels of the thermal imaging image are processed, Figure 3 As shown, the binary thresholding process of the thermal imaging image frame is completed.

[0070] G. The computer selects a frame of radar point cloud image that is time-synchronized with the frame of thermal imaging image after processing in step F, and then performs coordinate transformation on each point cloud point of the radar point cloud image according to the external parameters determined in step E. The specific process is as follows:

[0071] According to the obtained intrinsic parameter K of the thermal imaging camera and the external parameter of the joint calibration of the thermal imaging camera and the solid-state laser radar Then each radar point cloud point P of the laser radar lidar =(x, y, z) is converted to the coordinates of the camera coordinate system. The specific conversion process is:

[0072]

[0073] Through the above transformation, the pixel points of the thermal imaging image can be compared with the point cloud point P lidar =(x, y, z), so that each point of the radar point cloud image corresponds to the frame of thermal imaging image after coordinate transformation, realizing the fusion of each point of the two images, such as Figure 4 As shown, a fused image with depth information and color information is finally obtained; then the above steps are repeated for the remaining frames of the captured image, so that a fused image of each frame can be obtained;

[0074] H. Determine the scope and location of the fire source based on the fused image obtained in step G.

[0075] Effect verification:

[0076] In order to verify the effectiveness and accuracy of the method proposed in this invention, an unmanned vehicle platform was used to conduct an indoor fire source identification and positioning experiment. The unmanned vehicle platform is equipped with a Hikvision DS-2TD2166-7 / V1 thermal imaging network camera and a DJI Livox-avia laser radar. The thermal imaging camera has a resolution of 1280×720 thermal imaging images and a sampling frequency of 25Hz. The detection range of the laser radar is 0.5 to 150m, and it uses petal-type non-repetitive scanning, with a horizontal range of 70.4° and a vertical range of 77.2°, and a sampling frequency of 25Hz. In order to ensure the safety and accuracy of the experiment, the fire source was replaced with a heater that also generates heat. During the experiment, the position of the heat source was changed, and the heat source was located using a fusion positioning algorithm. The measured value was compared with the true value to evaluate the algorithm:

[0077] Heat source identification experiment:

[0078] The recognition effect of the image before and after image threshold processing is compared. The experimental variables are different lighting conditions and different distances, so as to analyze and compare the algorithm performance. Figure 5 As shown, the thermal imaging image after image processing is significantly different from the original image, and has a significant degree of distinction in the identification of fire sources and their range. Two sets of comparative experiments were conducted under different conditions, namely comparative experiments under different lighting environments and comparative experiments at different distances. RGB information was extracted from the thermal imaging image after image processing, and the effectiveness of the algorithm was verified by comparing the proportion of red information in the two. The proportion is the ratio of the number of color information R = 255 pixels in the image before and after fusion, and the accuracy is the probability of successfully assigning color information R = 255 to the fire source point cloud after fusion. The accuracy of image processing was verified by extracting the RGB information of the point cloud after image fusion. The specific data is shown in Table 1 below.

[0079] Table 1 Image processing data

[0080]

[0081]

[0082] The data in Table 1 shows that image processing reduces the proportion of fire sources in the image, improving fire source identification efficiency. After image fusion, the laser point cloud has a high probability of containing red points in the fire source, and the accuracy of coloring the fire source point cloud is close to 100%, demonstrating the high accuracy of the proposed fusion method for fire source identification and range determination.

[0083] Heat source positioning experiment:

[0084] For the heat source positioning experiment, in order to verify the accuracy of image fusion in locating the fire source, a total of 10 sets of data were tested under different lighting conditions and different distances. The actual coordinates of the object are (x, y, y), and the measured coordinates of the object obtained by the algorithm are (x', y', z'). The Euclidean distance is used. To measure the positioning error of the algorithm, the relative error is used To describe the accuracy of the final positioning, the results are shown in Table 2.

[0085] Table 2(a) Relative positioning error of fire source (sufficient lighting)

[0086]

[0087] Table 2(b) Relative positioning error of fire source (insufficient lighting)

[0088]

[0089]

[0090] Table 2 shows that after image processing and image fusion, the relative error remains within 5% under both sufficient and insufficient lighting conditions and when the fire source is within 15m. The relative positioning error under insufficient lighting conditions is improved compared to the relative positioning error under sufficient lighting conditions. Due to the influence of the sensor's own acquisition distance, accurate positioning data cannot be obtained when the measurement range exceeds 20m. Therefore, it can be concluded that the image fusion proposed by the present invention can accurately locate the fire source at a short distance (within 15m) and is not greatly affected by lighting.

[0091] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A fire source identification and positioning method based on the fusion of solid-state laser radar and thermal imaging vision, characterized in that: The specific steps are: A. Install solid-state laser radar and thermal imaging cameras at the locations to be monitored, and connect them to a computer to complete the deployment of the flame recognition and positioning system. B. Use a computer to calibrate the thermal imaging camera's internal parameter coefficients using Zhang Zhengyou's calibration method; C. Use a computer to unify the clock sources of the thermal imaging camera and solid-state lidar, and then align the sampling timestamps of the two to complete the time synchronization of the thermal imaging camera and solid-state lidar; D. When the fire source identification and location begins, the solid-state laser radar and thermal imaging camera begin to collect radar point cloud images and thermal imaging images respectively, and transmit the collected images to the computer; E. Use computer to solve and obtain the external parameters for the joint calibration of thermal imaging camera and solid-state lidar; F. The computer first selects a frame of thermal imaging image acquired in step D, and then performs binary thresholding processing on the frame of image; G. The computer selects a radar point cloud image frame that is time-synchronized with the thermal imaging frame based on the thermal imaging frame processed in step F. Then, based on the extrinsic parameters determined in step E, the computer transforms the coordinates of each point in the radar point cloud image so that the points correspond one-to-one with the thermal imaging frame, thereby fusing each point of the two images and ultimately obtaining a fused image with depth and color information. H. Determine the scope and location of the fire source based on the fused image obtained in step G.

2. The fire source identification and positioning method based on the fusion of solid-state laser radar and thermal imaging vision according to claim 1 is characterized in that: The calibration of the camera intrinsic coefficients in step B is specifically as follows: Assume that the pixel coordinates of each point in the image taken by the thermal imaging camera are [u, v] T , let Oxyz be the camera coordinate system, after the pinhole projection in the pinhole model, the space point P[X ′ ,Y ′ ,Z ′ ]The coordinates on the imaging plane are P camera [X,Y,Z], the camera focal length is f, then we have Set a pixel plane ouv on the physical imaging plane and get P under the pixel plane ′ Pixel coordinates: [u, v] T ; The original o is located in the upper left corner of the image, the u axis is parallel to the x axis to the right, and the v axis is parallel to the y axis downward; the pixel coordinates are scaled by α times on the u axis and β times on the v axis. At the same time, the origin is translated [c x , c y ] T Then, the coordinates of P′ and the pixel coordinates [u, v] T The relationship is And let αf=f x , βf=f y , then Rewriting the formula into matrix form, using homogeneous coordinates, we can get In formula (4), K is the internal parameter of the camera, and the internal parameter coefficient of the thermal imaging camera is calculated.

3. The fire source identification and positioning method based on the fusion of solid-state laser radar and thermal imaging vision according to claim 1 is characterized in that: The specific process of jointly calibrating the external parameters in step E is as follows: Assume that each pixel of the thermal image is [u, v] T , each pixel of the radar point cloud image is P[X′,Y′,Z′], solve the external parameters between the thermal imaging camera and the solid-state lidar According to formula (5), the external parameter The value of .

4. The fire source identification and positioning method based on the fusion of solid-state laser radar and thermal imaging vision according to claim 1 is characterized in that: The specific process of performing binary thresholding processing in step F is as follows: Assume that the temperature of each pixel in a thermal imaging image is T (x,y) , the unit is ℃, and the RGB value corresponding to each pixel is C (x,y) (i, j, k), set the threshold as ε, and reassign the RGB value of each pixel of the thermal imaging image according to the following rules: When the temperature of a pixel point T (x,y) When the pixel value is greater than or equal to the threshold ε, the RGB value corresponding to the pixel is reassigned to (255, 0, 0). Otherwise, the RGB value of the pixel is maintained. After all pixels of the thermal imaging image are processed, the binary thresholding process of the thermal imaging image is completed.

5. The fire source identification and positioning method based on the fusion of solid-state laser radar and thermal imaging vision according to claim 1 is characterized in that: The specific process of coordinate transformation of each pixel point of the radar point cloud image in step G is as follows: According to the obtained intrinsic parameter K of the thermal imaging camera and the external parameter of the joint calibration of the thermal imaging camera and the solid-state laser radar Then each radar point cloud point P of the laser radar lidar =(x, y, z) is converted to the coordinates of the camera coordinate system. The specific conversion process is: Through the above transformation, the pixel points of the thermal imaging image can be compared with the point cloud point P lidar =(x, y, z), so that each point of the radar point cloud image corresponds to the frame of thermal imaging image after coordinate transformation, realizing the fusion of each point of the two images, and finally obtaining a fused image with depth information and color information.

Images