Fire-fighting robot fire source identification and positioning system based on binocular vision and thermal imaging

Abstract

The invention discloses a fire-fighting robot fire source identification and positioning system based on binocular vision and thermal imaging, which is characterized in that a binocular depth camera acquires a binocular stereo image and a structured light image in a target scene and records visual information, and a thermal imager acquires a thermal image and temperature data in the same scene; performing spatial registration on the acquired binocular stereo image, the structured light image and the thermal image to enable the three images to be aligned under the same coordinate system, fusing the binocular stereo image, the structured light image and the thermal image, processing the fused image, and inputting the processed fused image into a trained fire source identification network model for detection to obtain a fire source identification result. After a prepared fire source point is detected, a temperature value is extracted and compared with a preset temperature threshold value, if the temperature value exceeds the preset temperature threshold value, a fire source is judged, and a binocular imaging triangulation and structured light positioning fusion algorithm is utilized to position the fire source, and the method has the advantages of being capable of rapidly positioning the fire source, rapidly extinguishing fire and the like.

Description

Technical Field

[0001] The invention relates to the technical field of fire fighting, in particular to a fire source identification and positioning system of a fire fighting robot based on binocular vision and thermal imaging. Background Art

[0002] Electricity and fire energy have become an indispensable resource in people's daily lives. While it brings convenience to people, the frequency of its use also increases the probability of fire. Since firefighting is a high-risk special industry, in order to reduce the safety hazards of firefighters, firefighting robots have been developed to perform firefighting operations. Existing firefighting robots usually rely on manual judgment of the location of the fire source and manual adjustment of the water cannon angle to extinguish the fire. This method is not only inefficient, but also in complex fire scenes, the rapid movement or change of the fire source may cause delays in firefighting operations.

[0003] Therefore, there is an urgent need for a firefighting robot that can automatically identify the location of the fire source and extinguish the fire autonomously, thereby improving the efficiency and accuracy of firefighting. Summary of the invention

[0004] The purpose of the present invention is to solve the above-mentioned deficiencies in the prior art and to provide a fire source identification and positioning system for a fire-fighting robot based on binocular vision and thermal imaging, which can quickly locate the fire source and extinguish the fire quickly.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A fire source identification and positioning system for a fire-fighting robot based on binocular vision and thermal imaging, characterized in that the fire source is identified and positioned using the following method:

[0007] Step S1: The binocular depth camera obtains the binocular stereo image and structured light image in the target scene and records the visual information, and the thermal imager obtains the thermal image and temperature data of the same scene;

[0008] Step S2: spatially registering the acquired binocular stereo image, structured light image and thermal image so that the three are aligned in the same coordinate system;

[0009] Step S3: fusing the binocular stereo image, the structured light image and the thermal image, and processing the fused image;

[0010] Step S4: Input the processed fusion image into the trained fire source identification network model for detection. After the preliminary fire source point is detected, the temperature value is extracted and compared with the preset temperature threshold. If the temperature value exceeds the preset temperature threshold, it is determined to be a fire source and step S5 is executed. If the temperature value does not exceed the preset temperature threshold, jump to step S1;

[0011] Step S5: using binocular imaging triangulation and structured light positioning fusion algorithm to locate the fire source;

[0012] The combination of binocular depth cameras and thermal imagers can determine and locate the source of fire.

[0013] After the fire source is located in step S5 of the present invention, step S6 is continued;

[0014] Step S6: Combine the three-dimensional coordinates of the binocular depth camera, fire source and water cannon to calculate the target pitch angle and left and right rotation angle of the water cannon. The fire fighting robot vehicle controller controls the water cannon to rotate to the target position and controls the water cannon to spray water to complete the fire extinguishing operation; ensure that it can be aimed at the fire source to achieve efficient fire extinguishing.

[0015] In step S2 of the present invention, the OpenCV tool is used to calibrate the three images; because the thermal imager and the binocular depth camera are installed at a certain distance, the OpenCV tool is used for calibration to solve the alignment deviation caused by different viewing angles and focal lengths.

[0016] Step S2 of the present invention adopts a pairwise registration method, the binocular stereo image is spatially registered with the structured light image, and the binocular stereo image is spatially registered with the thermal image.

[0017] The spatial registration method of the binocular stereo image and the thermal image in step S2 of the present invention is:

[0018] Perform multiple calibrations, use the affine transformation formula to calculate the transformation matrix, obtain multiple sets of affine transformation matrices, and use the least squares method to calculate the transformation matrix with a smaller error. The coordinates of the feature points in the binocular stereo image are (x stereo ,y stereo ,z stereo ), the coordinates of the feature points in the thermal image are (x thermal ,y thermal ,z thermal ), the transformation formula is as follows:

[0019]

[0020] Where a, b, c, d, e, f, g, h, i are the parameters of the linear transformation, and t x , t y , t z are the components of the translation vector;

[0021] By obtaining n sets of feature points, a 3n×12 design matrix P and a 3n×1 observation vector Q are constructed. The affine transformation formula can be written in matrix form, and then the least squares method is used to solve the optimal solution:

[0022] PX = Q;

[0023] X=(P T P) -1 P T Q;

[0024] Where X is the parameter vector of the affine transformation to be solved;

[0025] After the coordinate transformation is completed, the coordinate systems of the two images are basically aligned, and then the fields of view of the two images are compared and the fields of view are cropped to the same size;

[0026] By scaling the images, the two images are scaled to the same size, and bilinear interpolation is used for scaling. A pixel point (x ti ,y ti ), the corresponding target pixel in the binocular stereo image is (x bsi ,y bsi ), use bilinear interpolation to calculate its pixel value I(x bsi ,y bsi ), take the four adjacent integer coordinates of the pixel in the original thermal image: The calculation formula is:

[0027]

[0028] In the formula, is the pixel value of the adjacent integer coordinate point, dx, dy are weights, where

[0029] Through scaling, the pixels on the thermal image and the binocular stereo image are matched one to one.

[0030] In step S3 of the present invention, before the binocular stereo image, the structured light image and the thermal image are fused, the binocular stereo image, the structured light image and the thermal image need to be preprocessed respectively;

[0031] The preprocessing method is to perform illumination correction on binocular stereo images, structured light images and thermal images, enhance image brightness, and use histogram equalization to improve contrast;

[0032] Normalize the temperature data of thermal images to realize the visualization of temperature information in thermal images, unify the pixel data range of thermal images, binocular stereo images, and structured light images, and linearly reduce the temperature values ​​to [0,255];

[0033]

[0034] Where Temp_img_normal is the normalized temperature information, Temp_img is the original data value, max(Temp_img) is the maximum value in the original data set, and min(Temp_img) is the minimum value in the original data set.

[0035] In step S3 of the present invention, the fused image is processed by using Gaussian filtering to remove noise and sharpen edges.

[0036] In step S5 of the present invention, binocular imaging triangulation and structured light positioning adopt a weighted fusion algorithm based on error estimation;

[0037] Z final (x,y)=w1(x,y)Z stereo (x,y)+w1(x,y)Z struct (x,y);

[0038]

[0039] In the formula, Z stereo (x, y) is the depth map of binocular imaging, d(x, y) is the disparity map obtained by the stereo matching algorithm in the left and right images of the binocular imaging system, f is the focal length of the camera, B is the baseline distance between the two cameras of the binocular depth camera, and Z struct (x, y) is the depth map of the structured light image, e1(x, y) and e2(x, y) are the error estimates of the binocular stereo image and the structured light image at point (x, y), w1(x, y) and w2(x, y) are the weights of the binocular imaging triangulation and structured light positioning fusion, Z final (x, y) is the depth map after weighted fusion;

[0040] From the fused depth map, the depth value of the fire source area is extracted. The plane coordinates of the fire source are obtained by identifying the fire source in the early stage. According to the plane coordinates of the fire source, the depth value and the calibration parameters of the binocular depth camera, the three-dimensional world coordinates of the fire source point are calculated:

[0041]

[0042] Z=Z final (x,y);

[0043] Where f is the focal length of the camera, c x , c y is the pixel coordinate of the center of the image, and (X, Y, Z) is the exact position of the fire source in the three-dimensional world coordinate system.

[0044] In step S6 of the present invention, the target pitch angle and left and right rotation angle of the water cannon are calculated as follows:

[0045]

[0046] Where α is the target left and right rotation angle of the water cannon, β is the target elevation angle of the water cannon, (x PS ,y PS ,z PS ) is the three-dimensional coordinate of the water cannon, (x fire ,y fire ,z fire ) is the three-dimensional coordinate of the fire source.

[0047] The fire-fighting robot of the present invention is provided with a patrol mode and a follow-up mode. In these two working modes, the fire-fighting robot can actively search for the fire source. When the binocular depth camera and the thermal imager detect and locate the fire source, the fire-fighting robot starts the fire-fighting control system, autonomously controls the water cannon to aim at the fire source, and sprays water to extinguish the fire.

[0048] In the follow-up mode, the horizontal and pitch movements of the water cannon follow the vehicle-mounted PTZ. The water cannon is manually switched according to the fire severity. When the fire severity is high and there is an external water supply, a large water cannon is selected. When the fire severity is low or no external water supply is available, a small water cannon is selected.

[0049] During the following process, the fire-fighting robot vehicle controller determines whether the target motion angle exceeds the critical angle of the water cannon. If not, the fire-fighting robot vehicle controller sends a motion command to move the water cannon to the target angle. If it exceeds the critical angle, the operator controls the fire-fighting robot to start the vehicle motion control system to adjust the posture and realize the following function.

[0050] The beneficial effects of the present invention are as follows: the combination of the binocular depth camera and the thermal imager can determine and locate the fire source, thereby improving the efficiency and accuracy of fire extinguishing. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a structural schematic diagram of the fire-fighting robot of the present invention.

[0052] Figure 2 It is a flow chart of fire source identification and positioning of the present invention.

[0053] Figure 3 It is a schematic diagram of the three-dimensional coordinates of the binocular depth camera, fire source and water cannon of the present invention.

[0054] Reference numerals: small water cannon-1;

[0055] Vehicle-mounted PTZ-2;

[0056] Large water cannon-3;

[0057] Stereo depth camera-4. DETAILED DESCRIPTION

[0058] The present invention is described below in conjunction with the accompanying drawings and embodiments.

[0059] As attached Figure 1 As shown, a firefighting robot is provided with a vehicle-mounted pan / tilt 2, a binocular depth camera 4, a fire extinguishing control system, an intelligent controller, a vehicle controller and a vehicle motion control system. The binocular depth camera 4 is located directly in front of the firefighting robot. The binocular depth camera 4 includes two cameras arranged on the left and right, and there is a certain baseline distance between the two cameras.

[0060] The vehicle-mounted PTZ 2 integrates a thermal imager, a wide-angle camera, a zoom camera, a high-precision servo motor and a switch. The vehicle controller controls the action of the high-precision servo motor. The high-precision servo motor improves the adjustment speed and accuracy of the vehicle-mounted PTZ, allowing the vehicle-mounted PTZ to rotate 360°, realizing all-round monitoring of the environment around the fire-fighting robot. The thermal imager and the binocular depth camera can cooperate to quickly identify and locate the fire source. The wide-angle camera and the zoom camera can help operators observe the factory environment more intuitively and make decisions as soon as possible when a fire occurs. The switch uses the RS485 protocol to realize the dual-signal output function, which can centrally output the video signals of the wide-angle camera, the zoom camera, and the thermal imager.

[0061] The intelligent controller is equipped with a multi-core processor and deep algorithms to analyze and process sensor data. After calculation of the fire source identification and positioning model, it can output the fire source location information and the target rotation angle of the water cannon to the vehicle controller.

[0062] The vehicle controller is used to receive the algorithm output of the intelligent controller and the control instructions of the operator, and control the vehicle-mounted pan / tilt 2, the fire extinguishing control system and the vehicle motion control system to achieve the normal operation of the vehicle. In this embodiment, the vehicle controller can be a PLC controller;

[0063] The vehicle motion control system includes a posture sensing module, a hydraulic motor control system and a safety monitoring system. The posture sensing module mainly senses the vehicle's tilt angle, direction and speed through an accelerometer and a gyroscope, and adjusts the vehicle's posture through a hydraulic motor control system.

[0064] The fire extinguishing control system includes a large water cannon 3 and a small water cannon 1. When the small water cannon 1 is selected, the water tank and fire pump provided by the small water cannon 1 assist the small water cannon 1 in discharging water. When the large water cannon 3 is switched to, water is supplied by an external fire hose. At the same time, the system also needs to output the angle and status information of the two water cannons, and receive water cannon rotation and water discharge instructions from the vehicle controller to ensure that the water cannon can effectively aim at the fire source and complete the fire extinguishing work.

[0065] As attached Figure 2-3As shown, a fire source identification and positioning system of a fire fighting robot based on binocular vision and thermal imaging uses the following method to identify and locate the fire source:

[0066] Step S1: The binocular depth camera 4 obtains a binocular stereo image and a structured light image in the target scene, records the visual information, and the thermal imager obtains a thermal image and temperature data in the same scene;

[0067] Step S2: spatially registering the acquired binocular stereo image, structured light image and thermal image so that the three are aligned in the same coordinate system;

[0068] The registration method based on feature points is adopted to calculate the transformation matrix using corner points and edge feature points to achieve the calibration of the three images;

[0069] Since the thermal imager and the binocular depth camera 4 are installed at a certain distance, the three images are calibrated using the OpenCV tool to solve the registration deviation caused by different viewing angles and focal lengths;

[0070] Using a pairwise registration method, binocular stereo images are spatially registered with structured light images, and binocular stereo images are spatially registered with thermal images.

[0071] The spatial registration method of binocular stereo image and thermal image is:

[0072] Perform multiple calibrations, use the affine transformation formula to calculate the transformation matrix, obtain multiple sets of affine transformation matrices, and use the least squares method to calculate the transformation matrix with a smaller error. The coordinates of the feature points in the binocular stereo image are (x stereo ,y stereo ,z stereo ), the coordinates of the feature points in the thermal image are (x thermal ,y thermal ,z thermal ), the transformation formula is as follows:

[0073]

[0074] Where a, b, c, d, e, f, g, h, i are the parameters of the linear transformation, and t x , t y , t z are the components of the translation vector;

[0075] By obtaining n sets of feature points, a 3n×12 design matrix P and a 3n×1 observation vector Q are constructed. The affine transformation formula can be written in matrix form, and then the least squares method is used to solve the optimal solution:

[0076] PX = Q;

[0077] X=(P T P)-1 P T Q;

[0078] Where X is the parameter vector of the affine transformation to be solved;

[0079] After the coordinate transformation is completed, the coordinate systems of the two images are basically aligned, and then the fields of view of the two images are compared and the fields of view are cropped to the same size;

[0080] By scaling the images, the two images are scaled to the same size, and bilinear interpolation is used for scaling. A pixel point (x ti ,y ti ), the corresponding target pixel in the binocular stereo image is (x bsi ,y bsi ), use bilinear interpolation to calculate its pixel value I(x bsi ,y bsi ), take the four adjacent integer coordinates of the pixel in the original thermal image: The calculation formula is:

[0081]

[0082] In the formula, is the pixel value of the adjacent integer coordinate point, dx, dy are weights, where

[0083] Through scaling, the pixels on the thermal image and the binocular stereo image are one-to-one corresponding;

[0084] Step S3: pre-processing the binocular stereo image, the structured light image and the thermal image respectively, fusing the binocular stereo image, the structured light image and the thermal image, and processing the fused image by using Gaussian filtering denoising and edge sharpening methods;

[0085] The preprocessing method is to perform illumination correction on binocular stereo images, structured light images and thermal images, enhance image brightness, and use histogram equalization to improve contrast;

[0086] Normalize the temperature data of thermal images to realize the visualization of temperature information in thermal images, unify the pixel data range of thermal images, binocular stereo images, and structured light images, and linearly reduce the temperature values ​​to [0,255];

[0087]

[0088] Where Temp_img_normal is the normalized temperature information, Temp_img is the original data value, max(Temp_img) is the maximum value in the original data set, and min(Temp_img) is the minimum value in the original data set;

[0089] Step S4: after processing the image data set acquired in the early stage, input it into the constructed fire source recognition network model for training, input the processed fusion image into the trained fire source recognition network model for detection, after detecting the prepared fire source point, extract the temperature value and compare it with the preset temperature threshold, if the temperature value exceeds the preset temperature threshold, it is determined to be a fire source, and step S5 is executed, if the temperature value does not exceed the preset temperature threshold, jump to step S1;

[0090] The fire source identification network model in this step is based on the YOLOv4 deep neural network learning framework. A dual-channel data-driven neural network is built based on this framework. Visual graphics data and deep graphics data are used for network learning. The fire source is confirmed based on the prepared fire source points identified by the neural network and the judgment results of the thermal imaging temperature value.

[0091] Step S5: using binocular imaging triangulation and structured light positioning fusion algorithm to locate the fire source;

[0092] The binocular imaging triangulation and structured light positioning adopt a weighted fusion algorithm based on error estimation;

[0093] Z final (x,y)=w1(x,y)Z stereo (x,y)+w1(x,y)Z struct (x,y);

[0094]

[0095] In the formula, Z stereo (x, y) is the depth map of binocular imaging, d(x, y) is the disparity map obtained by binocular stereo matching, f is the focal length of the camera, B is the baseline distance between the two cameras of the binocular depth camera, and Z struct (x, y) is the depth map of the structured light image, e1(x, y) and e2(x, y) are the error estimates of the binocular stereo image and the structured light image at point (x, y), w1(x, y) and w2(x, y) are the weights of the binocular imaging triangulation and structured light positioning fusion, Z final (x, y) is the depth map after weighted fusion;

[0096] From the fused depth map, the depth value of the fire source area is extracted. The plane coordinates of the fire source are obtained by identifying the fire source in the early stage. According to the plane coordinates of the fire source, the depth value and the calibration parameters of the binocular depth camera, the three-dimensional world coordinates of the fire source point are calculated:

[0097]

[0098] Z=Z final (x,y);

[0099] Where f is the focal length of the camera, c x , c y is the pixel coordinate of the center of the image, (X, Y, Z) is the exact position of the fire source in the three-dimensional world coordinate system;

[0100] Step S6: Combine the three-dimensional coordinates of the binocular depth camera, the fire source and the water cannon to calculate the target pitch angle and left and right rotation angle of the water cannon. The fire fighting robot vehicle controller controls the water cannon to rotate to the target position and controls the water cannon to spray water to complete the fire extinguishing operation;

[0101] The calculation method of the target pitch angle and left and right rotation angle of the water cannon is:

[0102]

[0103] Where α is the target left and right rotation angle of the water cannon, β is the target elevation angle of the water cannon, (x PS ,y PS ,z PS ) is the three-dimensional coordinate of the water cannon, (x fire ,y fire ,z fire ) is the three-dimensional coordinate of the fire source;

[0104] The combination of binocular depth camera and thermal imager can determine and locate the source of fire, calculate the target pitch angle and left and right rotation angle of the water cannon, and effectively aim at the fire source to achieve efficient fire extinguishing.

[0105] In this embodiment, the fire-fighting robot is provided with a patrol mode and a follow-up mode. In these two working modes, the fire-fighting robot can actively search for the fire source. When the binocular depth camera and the thermal imager detect and locate the fire source, the fire-fighting robot activates the fire-fighting control system, autonomously controls the water cannon to aim at the fire source, and sprays water to extinguish the fire.

[0106] In the patrol mode, when the fire source recognition network model trained in step S4 of this embodiment recognizes the fire source and the coordinates of the fire source point are located in step S5, the target movement angle of the water cannon is calculated according to step S6, and the fire-fighting robot automatically starts the fire-fighting control system and the vehicle motion control system, and autonomously controls the water cannon to aim at the fire source and spray water to extinguish the fire.

[0107] In the follow-up mode, in this embodiment, the operator uses a remote control to control the water cannon to discharge water. When the fire source recognition network model trained in step S4 identifies the fire source, the operator can simultaneously observe the image of the wide-angle camera on the vehicle-mounted gimbal. If an obvious flame is seen, the operator determines whether it is necessary to extinguish the fire. If necessary, the fire extinguishing control system is activated in time to operate the water cannon to spray water and extinguish the fire. Alternatively, when the fire source recognition network model trained in step S4 identifies the fire source and the coordinates of the fire source are located in step S5, the target movement angle of the water cannon is calculated according to step S6, and the fire-fighting robot automatically activates the fire extinguishing control system and the vehicle motion control system, and autonomously controls the water cannon to aim at the fire source and spray water to extinguish the fire.

[0108] In the follow-up mode, the horizontal and pitch movements of the water cannon follow the vehicle-mounted pan / tilt 2. The water cannon is manually switched according to the fire severity. When the fire severity is high and there is an external water supply, the large water cannon 3 is selected. When the fire severity is low or no external water supply is available, the small water cannon 1 is selected.

[0109] In this embodiment, during the following process, the vehicle controller determines whether the target movement angle of the water cannon exceeds the critical angle of the water cannon. If not, the vehicle controller sends a motion command to make the water cannon move autonomously to the target angle. If it exceeds the critical angle, a prompt message is sent to the remote control display interface, and the operator controls the fire-fighting robot to start the vehicle motion control system to adjust the vehicle posture and realize the following function.

[0110] The thermal imager can obtain temperature data of the field of view and has a wide temperature recognition range, usually in the temperature range of 0° to 600°. The temperature threshold is set to detect in real time whether there is an area exceeding the established temperature threshold point; it meets the full range of coverage from ambient temperature to high-temperature flames.

Claims

1. A fire source identification and positioning system for a fire-fighting robot based on binocular vision and thermal imaging, characterized in that: Use the following methods to identify and locate the fire source: Step S1: The binocular depth camera obtains the binocular stereo image and structured light image in the target scene and records the visual information, and the thermal imager obtains the thermal image and temperature data of the same scene; Step S2: spatially registering the acquired binocular stereo image, structured light image and thermal image so that the three are aligned in the same coordinate system; Step S3: fusing the binocular stereo image, the structured light image and the thermal image, and processing the fused image; Step S4: Input the processed fusion image into the trained fire source identification network model for detection. After the preliminary fire source point is detected, the temperature value is extracted and compared with the preset temperature threshold. If the temperature value exceeds the preset temperature threshold, it is determined to be a fire source and step S5 is executed. If the temperature value does not exceed the preset temperature threshold, jump to step S1; Step S5: Using the binocular imaging triangulation and structured light positioning fusion algorithm to locate the fire source.

2. According to claim 1, a fire source identification and positioning system for a fire-fighting robot based on binocular vision and thermal imaging is characterized in that: After the fire source is located in step S5, proceed to step S6; Step S6: Combine the three-dimensional coordinates of the binocular depth camera, fire source and water cannon to calculate the target pitch angle and left and right rotation angle of the water cannon. The fire fighting robot vehicle controller controls the water cannon to rotate to the target position and controls the water cannon to spray water to complete the fire extinguishing operation.

3. A fire source identification and positioning system for a fire-fighting robot based on binocular vision and thermal imaging according to claim 1 or 2, characterized in that: In step S2, the three images are calibrated using the OpenCV tool.

4. A fire source identification and positioning system for a firefighting robot based on binocular vision and thermal imaging according to claim 1 or 2, characterized in that: Step S2 adopts a pairwise registration method, and the binocular stereo image is spatially registered with the structured light image, and the binocular stereo image is spatially registered with the thermal image.

5. The fire source identification and positioning system of a fire-fighting robot based on binocular vision and thermal imaging according to claim 4 is characterized in that: The spatial registration method of the binocular stereo image and the thermal image in step S2 is: Perform multiple calibrations, use the affine transformation formula to calculate the transformation matrix, obtain multiple sets of affine transformation matrices, and use the least squares method to calculate the transformation matrix with a smaller error. The coordinates of the feature points in the binocular stereo image are (x stereo ,y stereo ,z stereo ), the coordinates of the feature points in the thermal image are (x thermal ,y thermal ,z thermal ), the transformation formula is as follows: Where a, b, c, d, e, f, g, h, i are the parameters of the linear transformation, and t x , t y , t z are the components of the translation vector; By obtaining n sets of feature points, a 3n×12 design matrix P and a 3n×1 observation vector Q are constructed. The affine transformation formula can be written in matrix form, and then the least squares method is used to solve the optimal solution: PX = Q; X=(P T P) -1 P T Q; Where X is the parameter vector of the affine transformation to be solved; After the coordinate transformation is completed, the coordinate systems of the two images are basically aligned, and then the fields of view of the two images are compared and the fields of view are cropped to the same size; By scaling the images, the two images are scaled to the same size, and bilinear interpolation is used for scaling. A pixel point (x ti ,y ti ), the corresponding target pixel in the binocular stereo image is (x bsi ,y bsi ), use bilinear interpolation to calculate its pixel value I(x bsi ,y bsi ), take the four adjacent integer coordinates of the pixel in the original thermal image: The calculation formula is: In the formula, is the pixel value of the adjacent integer coordinate point, dx, dy are weights, where Through scaling, the pixels on the thermal image and the binocular stereo image are matched one to one.

6. A fire source identification and positioning system for a firefighting robot based on binocular vision and thermal imaging according to claim 1, 2 or 5, characterized in that: In step S3, before the binocular stereo image, the structured light image and the thermal image are fused, the binocular stereo image, the structured light image and the thermal image need to be preprocessed respectively; The preprocessing method is to perform illumination correction on binocular stereo images, structured light images and thermal images, enhance image brightness, and use histogram equalization to improve contrast; Normalize the temperature data of thermal images to realize the visualization of temperature information in thermal images, unify the pixel data range of thermal images, binocular stereo images, and structured light images, and linearly reduce the temperature values ​​to [0,255]; Where Temp_img_normal is the normalized temperature information, Temp_img is the original data value, max(Temp_img) is the maximum value in the original data set, and min(Temp_img) is the minimum value in the original data set.

7. A fire source identification and positioning system for a firefighting robot based on binocular vision and thermal imaging according to claim 1, 2 or 5, characterized in that: In step S3, the fused image is processed by using Gaussian filtering to remove noise and sharpen edges.

8. A fire source identification and positioning system for a firefighting robot based on binocular vision and thermal imaging according to claim 1, 2 or 5, characterized in that: In step S5, binocular imaging triangulation and structured light positioning adopt a weighted fusion algorithm based on error estimation; Z final (x,y)=w1(x,y)Z stereo (x,y)+w1(x,y)Z struct (x,y); In the formula, Z stereo (x, y) is the depth map of binocular imaging, d(x, y) is the disparity map obtained by the stereo matching algorithm in the left and right images of the binocular imaging system, f is the focal length of the camera, B is the baseline distance between the two cameras of the binocular depth camera, and Z struct (x, y) is the depth map of the structured light image, e1(x, y) and e2(x, y) are the error estimates of the binocular stereo image and the structured light image at point (x, y), w1(x, y) and w2(x, y) are the weights of the binocular imaging triangulation and structured light positioning fusion, Z final (x, y) is the depth map after weighted fusion; From the fused depth map, the depth value of the fire source area is extracted. The plane coordinates of the fire source are obtained by identifying the fire source in the early stage. According to the plane coordinates of the fire source, the depth value and the calibration parameters of the binocular depth camera, the three-dimensional world coordinates of the fire source point are calculated: Z=Z final (x,y); Where f is the focal length of the camera, c x , c y is the pixel coordinate of the center of the image, and (X, Y, Z) is the exact position of the fire source in the three-dimensional world coordinate system.

9. The fire source identification and positioning system of a fire-fighting robot based on binocular vision and thermal imaging according to claim 2, characterized in that: In step S6, the target pitch angle and left and right rotation angle of the water cannon are calculated as follows: Where α is the target left and right rotation angle of the water cannon, β is the target elevation angle of the water cannon, (x PS ,y PS ,z PS ) is the three-dimensional coordinate of the water cannon, (x fire ,y fire ,z fire ) is the three-dimensional coordinate of the fire source.

10. A fire source identification and positioning system for a firefighting robot based on binocular vision and thermal imaging according to claim 1 or 2 or 5 or 9, characterized in that: The fire-fighting robot is provided with a patrol mode and a follow-up mode. In these two working modes, the fire-fighting robot can actively search for the fire source. When the binocular depth camera and the thermal imager detect and locate the fire source, the fire-fighting robot activates the fire-fighting control system, autonomously controls the water cannon to aim at the fire source, and sprays water to extinguish the fire; In the follow-up mode, the horizontal and pitch movements of the water cannon follow the vehicle-mounted PTZ. The water cannon is manually switched according to the fire severity. When the fire severity is high and there is an external water supply, a large water cannon is selected. When the fire severity is low or no external water supply is available, a small water cannon is selected. During the following process, the fire-fighting robot vehicle controller determines whether the target motion angle exceeds the critical angle of the water cannon. If not, the fire-fighting robot vehicle controller sends a motion command to move the water cannon to the target angle. If it exceeds the critical angle, the operator controls the fire-fighting robot to start the vehicle motion control system to adjust the posture and realize the following function.

Images