Fire-fighting robot fire source recognition and positioning system based on binocular vision and thermal imaging

Abstract

The application discloses a fire source recognition and positioning system of a fire-fighting robot based on binocular vision and thermal imaging, a binocular depth camera acquires binocular stereo images and structured light images in a target scene, records visual information, a thermal imager acquires thermal images and temperature data under the same scene, space registration is performed on the acquired binocular stereo images, structured light images and thermal images, so that the three are aligned under the same coordinate system, the binocular stereo images, the structured light images and the thermal images are fused, and the fused images are processed, the processed fused images are input into a trained fire source recognition network model for detection, after a preliminary fire source point is detected, a temperature value is compared with a preset temperature threshold value, if the temperature value exceeds the preset temperature threshold value, the fire source is determined, binocular imaging triangulation and structured light positioning fusion algorithms are used for fire source positioning, and the application has the advantages of quick fire source positioning and quick fire extinguishing.

Description

Technical Field

[0001] This invention relates to the field of fire protection technology, specifically a fire source identification and positioning system for fire-fighting robots based on binocular vision and thermal imaging. Background Technology

[0002] Electricity and thermal energy have become indispensable resources in people's daily lives. While bringing convenience, the frequency of their use also increases the likelihood of fires. Because firefighting is a high-risk specialized industry, firefighting robots have been developed to reduce safety hazards for firefighters. However, existing firefighting robots typically rely on manual identification of the fire source and manual adjustment of the water cannon angle. This method is not only inefficient, but also prone to delays in firefighting operations due to the rapid movement or changes in the fire source in complex fire scenes.

[0003] Therefore, there is an urgent need for a fire-fighting robot that can automatically identify the location of a fire and extinguish it autonomously, thereby improving fire-fighting efficiency and accuracy. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a fire source identification and positioning system for fire-fighting robots based on binocular vision and thermal imaging, which can quickly locate fire sources and extinguish fires rapidly.

[0005] The technical solution adopted by this invention to solve its technical problem is:

[0006] A fire source identification and positioning system for firefighting robots based on binocular vision and thermal imaging, characterized by the following method for identifying and locating fire sources:

[0007] Step S1: The binocular depth camera acquires binocular stereo images and structured light images of the target scene and records visual information; the thermal imager acquires thermal images and temperature data of the same scene.

[0008] Step S2: Spatial registration is performed on the acquired binocular stereo image, structured light image and thermal image to align the three in the same coordinate system;

[0009] Step S3: Fuse the binocular stereo image, structured light image, and thermal image, and process the fused image;

[0010] Step S4: Input the processed fused image into the trained fire source recognition network model for detection. After detecting the potential fire source, extract the temperature value and compare it with the preset temperature threshold. If the temperature value exceeds the preset temperature threshold, it is determined to be a fire source and step S5 is executed. If the temperature value does not exceed the preset temperature threshold, it jumps to step S1.

[0011] Step S5: Use the algorithm of binocular imaging triangulation and structured light positioning fusion to locate the fire source;

[0012] The combination of binocular depth cameras and thermal imagers can identify and locate fire sources.

[0013] After locating the fire source in step S5 of this invention, step S6 is executed.

[0014] Step S6: Combining the three-dimensional coordinates of the binocular depth camera, the fire source, and the water cannon, calculate the target pitch angle and left and right rotation angle of the water cannon. The fire robot's vehicle controller controls the water cannon to rotate to the target position and controls the water cannon to spray water to complete the fire extinguishing operation; ensuring that it can be aimed at the fire source to achieve efficient fire extinguishing.

[0015] In step S2 of this invention, the OpenCV tool is used to calibrate the three types of images. Since the thermal imager and the binocular depth camera are installed at a certain distance, the OpenCV tool can be used for calibration to solve the registration deviation caused by different viewing angles and focal lengths.

[0016] In step S2 of this invention, a pairwise registration method is used: the binocular stereo image is spatially registered with the structured light image, and the binocular stereo image is spatially registered with the thermal image.

[0017] The spatial registration method between the binocular stereo image and the thermal image in step S2 of this invention is as follows:

[0018] Multiple calibrations were performed, and the transformation matrix was calculated using the affine transformation formula. Multiple sets of affine transformation matrices were obtained. Using the least squares method, the transformation matrix with the smallest error was calculated. The feature point coordinates in the binocular stereo image are (x... stereo ,y stereo ,z stereo The coordinates of the feature points in the thermal image are (x... thermal ,y thermal ,z thermal The transformation formula is as follows:

[0019]

[0020] In the formula, a, b, c, d, e, f, g, h, i are the parameters of the linear transformation, while t x , t y , t z These are the components of the translation vector;

[0021] By acquiring n sets of feature points, a 3n×12 design matrix P and a 3n×1 observation vector Q are constructed. The affine transformation formula can be written in matrix form, and then the least squares method is used to solve for the optimal solution.

[0022] PX = Q;

[0023] X = (P) T P) -1 P T Q;

[0024] In the formula, X is the parameter vector of the affine transformation to be solved;

[0025] After the coordinate transformation is completed, the coordinate systems of the two images are basically aligned. Then, the fields of view of the two images are compared and cropped to the same size.

[0026] By scaling the images to the same size, bilinear interpolation is used for scaling, and a single pixel (x) is taken from the thermal image. ti ,y ti The target pixel in the corresponding stereo image is (x). bsi ,y bsi The pixel value I(x) is calculated using bilinear interpolation. bsi ,y bsi Take the four nearest integer coordinates of each pixel in the original thermal image: The calculation formula is:

[0027]

[0028] In the formula, Let dx and dy be the pixel values ​​of the nearest integer coordinates, and dx and dy be the weights, where dx and dy are the values ​​of the nearest integer coordinates.

[0029] By scaling, a one-to-one correspondence is achieved between pixels on the thermal image and the stereoscopic image.

[0030] In step S3 of this invention, before fusing the binocular stereo image, structured light image, and thermal image, it is necessary to preprocess the binocular stereo image, structured light image, and thermal image respectively.

[0031] The preprocessing method involves performing illumination correction on the binocular stereo image, structured light image, and thermal image to enhance image brightness, and using histogram equalization to improve contrast.

[0032] The temperature data of the thermal image is normalized to unify the pixel data range of the thermal image, the binocular stereo image, and the structured light image in order to realize the visualization of temperature information in the thermal image and linearly reduce the temperature value to [0,255].

[0033]

[0034] In the formula, Temp_img_normal is the normalized temperature information, Temp_img is the original data value, max(Temp_img) is the maximum value in the original dataset, and min(Temp_img) is the minimum value in the original dataset.

[0035] In step S3 of this invention, Gaussian filtering for noise reduction and edge sharpening are used to process the fused image.

[0036] In step S5 of this invention, the binocular imaging triangulation and structured light localization employ a weighted fusion algorithm based on error estimation;

[0037] Z final (x,y)=w1(x,y)Z stereo (x,y)+w1(x,y)Z struct (x,y);

[0038]

[0039] In the formula, Z stereo (x,y) is the depth map of the binocular imaging system, d(x,y) is the disparity map obtained by the stereo matching algorithm in the left and right images through the binocular imaging system, f is the camera focal length, B is the baseline distance between the two cameras of the binocular depth camera, and Z is the depth map of the binocular depth camera. struct (x,y) represents the depth map of the structured light image, e1(x,y) and e2(x,y) are the error estimates of the binocular stereo image and the structured light image at point (x,y), respectively, w1(x,y) and w2(x,y) are the weights of the binocular imaging triangulation and structured light localization fusion, respectively, Z final (x,y) is the weighted fused depth map;

[0040] From the fused depth map, the depth value of the fire source region is extracted. The fire source's planar coordinates were obtained through initial fire source identification. Based on the fire source's planar coordinates, depth value, and calibration parameters of the stereo depth camera, the 3D world coordinates of the fire source point are calculated.

[0041]

[0042] Z = Z final (x,y);

[0043] In the formula, f is the camera focal length, and c x c y (X,Y,Z) represents the pixel coordinates of the image center, and (X,Y,Z) represents the precise location of the fire source in the three-dimensional world coordinate system.

[0044] In step S6 of this invention, the method for calculating the target elevation angle and left and right rotation angle of the water cannon is as follows:

[0045]

[0046] In the formula, α is the left and right rotation angle of the water cannon target, β is the elevation angle of the water cannon target, (x PS ,y PS ,z PS (x) represents the three-dimensional coordinates of the water cannon. fire ,y fire ,z fire () represents the three-dimensional coordinates of the fire source.

[0047] The fire-fighting robot of this invention has a patrol mode and a follow-up mode. In both working modes, the fire-fighting robot can actively locate the fire source. When the binocular depth camera and thermal imager detect and locate the fire source, the fire-fighting robot activates the fire extinguishing control system and autonomously controls the water cannon to aim at the fire source and spray water to extinguish the fire.

[0048] In follow mode, the water cannon's horizontal and vertical movements follow the vehicle-mounted gimbal. The water cannon is manually switched according to the fire level. When the fire level is high and there is an external water supply, the large water cannon is selected. When the fire level is low or there is no external water supply, the small water cannon is selected.

[0049] During the follow-up process, the fire robot vehicle controller determines whether the target motion angle exceeds the water cannon's critical angle. If it does not exceed the critical angle, the fire robot vehicle controller sends a motion command to move the water cannon to the target angle. If it exceeds the critical angle, the operator controls the fire robot to start the vehicle motion control system to adjust its posture and achieve the follow-up function.

[0050] The beneficial effects of this invention are: the combination of a binocular depth camera and a thermal imager can identify and locate the fire source, thereby improving fire extinguishing efficiency and accuracy. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the fire-fighting robot structure of the present invention.

[0052] Figure 2 This is a flowchart of the fire source identification and positioning process of the present invention.

[0053] Figure 3 This is a three-dimensional coordinate diagram of the binocular depth camera, fire source, and water cannon of the present invention.

[0054] Attached image label: Small water cannon-1;

[0055] Vehicle-mounted gimbal-2;

[0056] Water cannon-3;

[0057] Binocular Depth Camera-4. Detailed Implementation

[0058] The present invention will now be described in conjunction with the accompanying drawings and embodiments.

[0059] As attached Figure 1 As shown, a fire-fighting robot is equipped with a vehicle-mounted gimbal 2, a binocular depth camera 4, a fire extinguishing control system, an intelligent controller, a vehicle controller, and a vehicle motion control system. The binocular depth camera 4 is located in front of the fire-fighting robot and includes two cameras arranged on the left and right sides, with a certain baseline distance between the two cameras.

[0060] The vehicle-mounted gimbal 2 integrates a thermal imager, a wide-angle camera, a zoom camera, a high-precision servo motor, and a switch. The vehicle controller controls the high-precision servo motor, which improves the adjustment speed and accuracy of the vehicle-mounted gimbal, enabling it to rotate 360° and achieve all-round monitoring of the environment around the fire-fighting robot. The thermal imager and binocular depth camera work together to quickly identify and locate fire sources. The wide-angle camera and zoom camera help operators observe the factory environment more intuitively and make decisions as quickly as possible in the event of a fire. The switch uses the RS485 protocol to achieve dual signal output function, which can centrally output video signals from the wide-angle camera, zoom camera, and thermal imager.

[0061] The intelligent controller is equipped with a multi-core processor and deep algorithms to analyze and process sensor data. After calculation by the fire source identification and positioning model, it can output the fire source location information and the target rotation angle of the water cannon to the vehicle controller.

[0062] The vehicle controller is used to receive the algorithm output of the intelligent controller and the control commands of the operator, and at the same time control the vehicle-mounted gimbal 2, the fire extinguishing control system and the vehicle motion control system to realize the normal operation of the vehicle. In this embodiment, the vehicle controller can be a PLC controller.

[0063] The vehicle motion control system includes an attitude sensing module, a hydraulic motor control system, and a safety monitoring system. The attitude sensing module mainly senses the vehicle's tilt angle, direction, and speed through accelerometers and gyroscopes, and adjusts the vehicle's attitude through the hydraulic motor control system.

[0064] The fire extinguishing control system includes a large water cannon 3 and a small water cannon 1. When the small water cannon 1 is selected, the small water cannon 1 is assisted by its own water tank and fire pump to output water. When switching to the large water cannon 3, water is supplied by the external fire hose. At the same time, the system also needs to output the angle and status information of the two water cannons and receive the water cannon rotation and water output commands from the vehicle controller to ensure that the water cannons can effectively aim at the fire source and complete the fire extinguishing work.

[0065] As attached Figure 2-3As shown, a fire source identification and positioning system for firefighting robots based on binocular vision and thermal imaging uses the following method to identify and locate fire sources:

[0066] Step S1: The binocular depth camera 4 acquires binocular stereo images and structured light images of the target scene and records visual information; the thermal imager acquires thermal images and temperature data of the same scene.

[0067] Step S2: Spatial registration is performed on the acquired binocular stereo image, structured light image and thermal image to align the three in the same coordinate system;

[0068] A feature-point-based registration method is adopted, which uses corner and edge feature points to calculate the transformation matrix and achieves calibration of three types of images;

[0069] Since the thermal imager and the binocular depth camera 4 are installed at a certain distance, OpenCV is used to calibrate the three images, which can solve the registration deviation caused by different viewing angles and focal lengths.

[0070] The binocular stereo image and the structured light image are spatially registered using a pairwise registration method, and the binocular stereo image and the thermal image are spatially registered.

[0071] The spatial registration method for binocular stereo images and thermal images is as follows:

[0072] Multiple calibrations were performed, and the transformation matrix was calculated using the affine transformation formula. Multiple sets of affine transformation matrices were obtained. Using the least squares method, the transformation matrix with the smallest error was calculated. The feature point coordinates in the binocular stereo image are (x... stereo ,y stereo ,z stereo The coordinates of the feature points in the thermal image are (x... thermal ,y thermal ,z thermal The transformation formula is as follows:

[0073]

[0074] In the formula, a, b, c, d, e, f, g, h, i are the parameters of the linear transformation, while t x , t y , t z These are the components of the translation vector;

[0075] By acquiring n sets of feature points, a 3n×12 design matrix P and a 3n×1 observation vector Q are constructed. The affine transformation formula can be written in matrix form, and then the least squares method is used to solve for the optimal solution.

[0076] PX = Q;

[0077] X = (P) T P)-1 P T Q;

[0078] In the formula, X is the parameter vector of the affine transformation to be solved;

[0079] After the coordinate transformation is completed, the coordinate systems of the two images are basically aligned. Then, the fields of view of the two images are compared and cropped to the same size.

[0080] By scaling the images to the same size, bilinear interpolation is used for scaling, and a single pixel (x) is taken from the thermal image. ti ,y ti The target pixel in the corresponding stereo image is (x). bsi ,y bsi The pixel value I(x) is calculated using bilinear interpolation. bsi ,y bsi Take the four nearest integer coordinates of each pixel in the original thermal image: The calculation formula is:

[0081]

[0082] In the formula, Let dx and dy be the pixel values ​​of the nearest integer coordinates, and dx and dy be the weights, where dx and dy are the values ​​of the nearest integer coordinates.

[0083] By scaling, a one-to-one correspondence is achieved between pixels on the thermal image and the binocular stereo image;

[0084] Step S3: Preprocess the stereo image, structured light image and thermal image respectively, fuse the stereo image, structured light image and thermal image, and process the fused image by using Gaussian filtering for noise reduction and edge sharpening.

[0085] The preprocessing method involves performing illumination correction on the binocular stereo image, structured light image, and thermal image to enhance image brightness, and using histogram equalization to improve contrast.

[0086] The temperature data of the thermal image is normalized to unify the pixel data range of the thermal image, the binocular stereo image, and the structured light image in order to realize the visualization of temperature information in the thermal image and linearly reduce the temperature value to [0,255].

[0087]

[0088] In the formula, Temp_img_normal is the normalized temperature information, Temp_img is the original data value, max(Temp_img) is the maximum value in the original dataset, and min(Temp_img) is the minimum value in the original dataset.

[0089] Step S4: After processing the previously acquired image dataset, input it into the constructed fire source recognition network model for training. Input the processed fused image into the trained fire source recognition network model for detection. After detecting the potential fire source, extract the temperature value and compare it with the preset temperature threshold. If the temperature value exceeds the preset temperature threshold, it is determined to be a fire source and step S5 is executed. If the temperature value does not exceed the preset temperature threshold, jump to step S1.

[0090] In this step, the fire source identification network model is based on the YOLOv4 deep neural network learning framework. The neural network built on this framework is a dual-channel data-driven neural network. It uses visual graphics data and deep graphics data for network learning. Based on the preliminary fire source points identified by the neural network, combined with the judgment results of thermal imaging temperature values, the fire source is confirmed.

[0091] Step S5: Use the algorithm of binocular imaging triangulation and structured light positioning fusion to locate the fire source;

[0092] Binocular imaging triangulation and structured light localization employ a weighted fusion algorithm based on error estimation;

[0093] Z final (x,y)=w1(x,y)Z stereo (x,y)+w1(x,y)Z struct (x,y);

[0094]

[0095] In the formula, Z stereo (x,y) is the depth map of the binocular imaging, d(x,y) is the disparity map obtained through binocular stereo matching, f is the camera focal length, B is the baseline distance between the two cameras of the binocular depth camera, and Z is the depth map of the binocular depth camera. struct (x,y) represents the depth map of the structured light image, e1(x,y) and e2(x,y) are the error estimates of the binocular stereo image and the structured light image at point (x,y), respectively, w1(x,y) and w2(x,y) are the weights of the binocular imaging triangulation and structured light localization fusion, respectively, Z final (x,y) is the weighted fused depth map;

[0096] From the fused depth map, the depth value of the fire source region is extracted. Initially, the fire source's planar coordinates are obtained by identifying the fire source. Based on the fire source's planar coordinates, depth value, and calibration parameters of the stereo depth camera, the 3D world coordinates of the fire source point are calculated.

[0097]

[0098] Z = Z final (x,y);

[0099] In the formula, f is the camera focal length, and c x c y Let (X,Y,Z) be the pixel coordinates of the image center, and (X,Y,Z) be the precise location of the fire source in the three-dimensional world coordinate system.

[0100] Step S6: Combining the three-dimensional coordinates of the binocular depth camera, the fire source, and the water cannon, calculate the target pitch angle and left and right rotation angle of the water cannon. The fire robot's vehicle controller controls the water cannon to rotate to the target position and controls the water cannon to spray water to complete the fire extinguishing operation.

[0101] The calculation method for the target elevation angle and left and right rotation angle of the water cannon is as follows:

[0102]

[0103] In the formula, α is the left and right rotation angle of the water cannon target, β is the elevation angle of the water cannon target, (x PS ,y PS ,z PS (x) represents the three-dimensional coordinates of the water cannon. fire ,y fire ,z fire () represents the three-dimensional coordinates of the fire source.

[0104] The combination of binocular depth camera and thermal imager can identify and locate the fire source. Calculating the target elevation angle and left and right rotation angle of the water cannon can effectively aim at the fire source and achieve efficient fire extinguishing.

[0105] In this embodiment, the fire-fighting robot is equipped with a patrol mode and a follow-up mode. In both modes, the fire-fighting robot can actively locate the fire source. When the binocular depth camera and thermal imager detect and locate the fire source, the fire-fighting robot activates the fire extinguishing control system and autonomously controls the water cannon to aim at the fire source and spray water to extinguish the fire.

[0106] In patrol mode, when the fire source identification network model trained in step S4 of this embodiment identifies the fire source and the coordinates of the fire source point are located in step S5, the target motion angle of the water cannon is calculated according to step S6. The fire-fighting robot automatically starts the fire extinguishing control system and the vehicle motion control system, and autonomously controls the water cannon to aim at the fire source and spray water to extinguish the fire.

[0107] In follow-up mode, in this embodiment, the operator uses a remote control to control the water cannon to spray water. When the fire source recognition network model trained in step S4 identifies the fire source, the operator can simultaneously observe the image of the wide-angle camera on the vehicle-mounted gimbal. If obvious flames are seen, the operator can determine whether fire extinguishing is necessary. If so, the fire extinguishing control system is activated in time, and the water cannon is operated to spray water for fire extinguishing. Alternatively, when the fire source recognition network model trained in step S4 identifies the fire source, and the coordinates of the fire source are located in step S5, the target motion angle of the water cannon is calculated according to step S6. The fire robot then automatically activates the fire extinguishing control system and the vehicle motion control system, and autonomously controls the water cannon to aim at the fire source and spray water to extinguish the fire.

[0108] In follow mode, the horizontal and vertical movements of the water cannon follow the vehicle-mounted gimbal 2. The water cannon is manually switched according to the fire level. When the fire level is high and there is an external water supply, the large water cannon 3 is selected. When the fire level is low or there is no external water supply, the small water cannon 1 is selected.

[0109] In this embodiment, during the follow-up process, the vehicle controller determines whether the target movement angle of the water cannon exceeds the critical angle of the water cannon. If it does not exceed the critical angle, the vehicle controller sends a movement command to make the water cannon move autonomously to the target angle. If it exceeds the critical angle, a prompt message is sent to the remote control display interface, and the operator controls the fire-fighting robot to start the vehicle motion control system to adjust the vehicle posture and realize the follow-up function.

[0110] The thermal imager can acquire temperature data of the field of view and has a wide temperature recognition range, typically ranging from 0°C to 600°C. It can set a temperature threshold and detect in real time whether any area exceeds the predetermined temperature threshold point; it can cover the entire range from ambient temperature to high-temperature flame.

Claims

1. A fire source identification and positioning system for a firefighting robot based on binocular vision and thermal imaging, characterized in that, The following method is used to identify and locate the fire source: Step S1: The binocular depth camera acquires binocular stereo images and structured light images of the target scene and records visual information; the thermal imager acquires thermal images and temperature data of the same scene. Step S2: Spatial registration is performed on the acquired binocular stereo image, structured light image and thermal image to align the three in the same coordinate system; The binocular stereo image and the structured light image are spatially registered using a pairwise registration method, and the binocular stereo image and the thermal image are spatially registered. Step S3: Fuse the binocular stereo image, structured light image, and thermal image, and process the fused image; Step S4: Input the processed fused image into the trained fire source recognition network model for detection. After detecting the potential fire source, extract the temperature value and compare it with the preset temperature threshold. If the temperature value exceeds the preset temperature threshold, it is determined to be a fire source and step S5 is executed. If the temperature value does not exceed the preset temperature threshold, it jumps to step S1. In this step, the fire source identification network model is built based on the YOLOv4 deep neural network learning framework; Step S5: Use the algorithm of binocular imaging triangulation and structured light positioning fusion to locate the fire source; Binocular imaging triangulation and structured light localization employ a weighted fusion algorithm based on error estimation; ； ； ； ； In the formula, This is a depth map for binocular imaging. This is a disparity map obtained by using a stereo matching algorithm in the left and right images through a binocular imaging system. For camera focal length, The baseline distance between the two cameras of a binocular depth camera. This is the depth map of the structured light image. and Binocular stereo images and structured light images at points Error estimation at the location, and These are the weights for binocular imaging triangulation and structured light localization fusion, respectively. This is the depth map after weighted fusion; From the fused depth map, the depth value of the fire source region is extracted. The fire source's planar coordinates were obtained through initial fire source identification. Based on the fire source's planar coordinates, depth value, and calibration parameters of the stereo depth camera, the 3D world coordinates of the fire source point are calculated. ； ； ； In the formula, f is the camera focal length. , The pixel coordinates of the image center. This refers to the precise location of the fire source in the three-dimensional world coordinate system. Step S6: Combining the three-dimensional coordinates of the binocular depth camera, the fire source, and the water cannon, calculate the target pitch angle and left and right rotation angle of the water cannon. The fire robot's vehicle controller controls the water cannon to rotate to the target pose and controls the water cannon to spray water to complete the fire extinguishing operation. The calculation method for the target elevation angle and left and right rotation angle of the water cannon is as follows: ； ； In the formula, The target rotation angle of the water cannon is to the left and right. The target elevation angle for the water cannon. The coordinates of the water cannon are shown in three dimensions. The fire-fighting robot is equipped with a patrol mode and a follow-up mode. In patrol mode, when the fire source identification network model trained in step S4 identifies the fire source and the coordinates of the fire source point are located in step S5, the target motion angle of the water cannon is calculated according to step S6. The fire-fighting robot automatically starts the fire extinguishing control system and the vehicle motion control system, and autonomously controls the water cannon to aim at the fire source and spray water to extinguish the fire. In follow-up mode, when the fire source recognition network model trained in step S4 identifies a fire source, the operator can simultaneously observe the image from the wide-angle camera on the vehicle-mounted gimbal. If a clear flame is seen, the operator can determine whether fire extinguishing is necessary. If so, the fire extinguishing control system can be activated in a timely manner, and the water cannon can be operated to spray water for fire extinguishing. Alternatively, when the fire source recognition network model trained in step S4 identifies a fire source, and the coordinates of the fire source are located in step S5, the target motion angle of the water cannon is calculated according to step S6. The fire robot can then automatically activate the fire extinguishing control system and the vehicle motion control system, and autonomously control the water cannon to aim at the fire source and spray water to extinguish the fire.

2. The fire source identification and positioning system for a fire-fighting robot based on binocular vision and thermal imaging as described in claim 1, characterized in that, In step S2, OpenCV tools are used to calibrate the three types of images.

3. A fire source identification and positioning system for a firefighting robot based on binocular vision and thermal imaging as described in claim 1 or 2, characterized in that, The spatial registration method between the stereo image and the thermal image in step S2 is as follows: Multiple calibrations were performed, and the transformation matrix was calculated using the affine transformation formula. Multiple sets of affine transformation matrices were obtained. Then, using the least squares method, the transformation matrix with the smallest error was calculated. The coordinates of the feature points in the stereo image are... The coordinates of the feature points in the thermal image are The transformation formula is as follows: ； In the formula, a, b, c, d, e, f, g, h, i are the parameters of the linear transformation, while , , These are the components of the translation vector; By obtaining n sets of feature points, a Design Matrix and observation vector The affine transformation formula can be written in matrix form, and then the least squares method can be used to find the optimal solution: ； ； In the formula, Let be the parameter vector of the affine transformation to be solved; After the coordinate transformation is completed, the coordinate systems of the two images are basically aligned. Then, the fields of view of the two images are compared and cropped to the same size. By scaling the images to the same size, bilinear interpolation is used for scaling, and a single pixel from the thermal image is extracted. The target pixel in the corresponding binocular stereo image is The pixel values ​​are calculated using bilinear interpolation. Take the four nearest integer coordinates of each pixel in the original thermal image: , , , The calculation formula is: ； In the formula, , , , The pixel value is the nearest integer coordinate point. For the weights, where , ; By scaling, a one-to-one correspondence is achieved between pixels on the thermal image and the stereoscopic image.

4. A fire source identification and positioning system for a firefighting robot based on binocular vision and thermal imaging as described in claim 1 or 2, characterized in that, In step S3, before fusing the stereo image, structured light image, and thermal image, the stereo image, structured light image, and thermal image need to be preprocessed separately. The preprocessing method involves performing illumination correction on the binocular stereo image, structured light image, and thermal image to enhance image brightness, and using histogram equalization to improve contrast. To visualize temperature information in thermal images, the temperature data is normalized. This unifies the pixel data range of thermal images with binocular stereo images and structured light images, linearly reducing the temperature values ​​to a smaller value. ; 55； In the formula, The temperature information after normalization. The original data values, The maximum value in the original dataset. It is the minimum value in the original dataset.

5. A fire source identification and positioning system for a firefighting robot based on binocular vision and thermal imaging as described in claim 1 or 2, characterized in that, In step S3, Gaussian filtering for noise reduction and edge sharpening are used to process the fused image.

6. A fire source identification and positioning system for a firefighting robot based on binocular vision and thermal imaging as described in claim 1 or 2, characterized in that, In follow mode, the water cannon's horizontal and vertical movements follow the vehicle-mounted gimbal. The water cannon is manually switched according to the fire level. When the fire level is high and there is an external water supply, the large water cannon is selected. When the fire level is low or there is no external water supply, the small water cannon is selected. During the follow-up process, the fire robot vehicle controller determines whether the target motion angle exceeds the water cannon's critical angle. If it does not exceed the critical angle, the fire robot vehicle controller sends a motion command to move the water cannon to the target angle. If it exceeds the critical angle, the operator controls the fire robot to start the vehicle motion control system to adjust its posture and achieve the follow-up function.

Images