Fire extinguishing method, equipment and medium of intelligent fire-fighting robot based on binocular vision

Abstract

The invention discloses an automatic fire extinguishing method of an intelligent fire-fighting robot based on binocular vision, which belongs to the field of automatic fire extinguishing of fire-fighting robots, and comprises the following steps: step 1, flame detection: when extinguishing a fire, a binocular camera of an intelligent fire-fighting robot is detected by using a target detection model to obtain a color binocular image containing a flame, and the three-dimensional position of the flame is determined; step 2, a water cannon yaw angle is calculated: according to the three-dimensional position relationship between the intelligent fire-fighting robot and the flame, the water cannon yaw angle is calculated, so that the water cannon points to the flame; step 3, a water cannon pitch angle is adjusted: a water cannon jet trajectory is measured by using a deep binocular matching network based on matching cost, and the water cannon pitch angle is adjusted with the difference between the water cannon jet trajectory and the flame position as feedback, so that the water cannon jet continuously approaches the flame; step 4, fire extinguishing hit judgment: when the flame area in the acquired thermal imaging image continues to decrease, it is confirmed that the jet hits the flame, the water cannon angle is maintained, and the jet continuously transports the fire extinguishing medium until the flame area is zero, and the fire extinguishing task is terminated.

Description

Technical Field

[0001] The present invention relates to the field of automatic fire extinguishing of intelligent fire fighting robots, and in particular to an automatic fire extinguishing method of an intelligent fire fighting robot based on binocular vision. Background Art

[0002] At present, robots are widely used in firefighting operations. Intelligent firefighting robots can replace human firefighters to complete reconnaissance or firefighting work in high-temperature, high-pressure, toxic and harmful fire environments. However, the current intelligent firefighting robots mainly rely on human operators to remotely control firefighting, and their intelligence level is insufficient. Improving the intelligence of intelligent firefighting robots and enabling them to have the ability to autonomously extinguish fires is crucial to the development of intelligent firefighting robots.

[0003] For intelligent firefighting robots, an automatic firefighting strategy is required to complete autonomous firefighting tasks. The goal of the automatic firefighting strategy is to control the trajectory of the jet and deliver a certain amount of fire extinguishing medium to the flame through the jet, thereby extinguishing the flame. In order for the jet to hit the flame, the pitch and yaw angles of the water cannon need to be adjusted. Due to the fragmentation characteristics of the jet, it is difficult to establish an accurate physical model of the jet, and it is difficult to construct an automatic firefighting method based on the jet model. If the three-dimensional trajectory of the jet is calculated using visual measurement, since the texture of the central area of ​​the jet in the image is relatively sparse, it is difficult to determine the corresponding points of the jet in the two images using traditional features, that is, it is impossible to use traditional image features to accurately calculate the parallax of the jet area.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The purpose of the present invention is to provide an automatic fire extinguishing method for an intelligent fire-fighting robot based on binocular vision, which uses a deep binocular matching network to calculate the matching cost to calculate the trajectory of the jet, thereby avoiding measuring the complex physical properties of the jet. At the same time, the angle of the water cannon is controlled based on visual feedback to ensure the hit rate of the jet hitting the flame, thereby solving the above-mentioned technical problems existing in the prior art.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] An automatic fire extinguishing method for an intelligent fire-fighting robot based on binocular vision is used to control an intelligent fire-fighting robot equipped with a binocular camera, a laser radar and a thermal imaging camera to perform an automatic fire extinguishing task, comprising the following steps:

[0008] Step 1, flame detection: When the intelligent fire-fighting robot performs an automatic fire-fighting task in an actual fire-fighting scene, the binocular camera carried by the intelligent fire-fighting robot obtains a color binocular image containing the flame, uses the target detection model to determine the flame position in the obtained color binocular image, and determines the three-dimensional position of the flame based on the detection frame of the flame position;

[0009] Step 2, calculating the yaw angle of the water cannon: according to the three-dimensional position relationship between the current intelligent fire-fighting robot and the three-dimensional position of the flame, the yaw angle of the water cannon is calculated, and the yaw angle is adjusted to make the water cannon point to the flame position;

[0010] Step 3, adjust the pitch angle of the water cannon: use the pre-built and trained deep binocular matching network based on matching cost to measure the current jet trajectory of the water cannon, and use the gap between the current jet trajectory of the water cannon and the flame position as feedback to adjust the pitch angle of the water cannon so that the jet of the water cannon is constantly close to the flame position;

[0011] Step 4, fire extinguishing hit judgment: when it is judged that the flame area of ​​the flame position in the thermal imaging image continuously acquired by the thermal imaging camera carried by the intelligent fire-fighting robot continues to decrease, it is confirmed that the jet hits the flame at the flame position, and the current yaw angle and pitch angle of the water cannon are maintained, so that the jet of the water cannon continues to transport the fire extinguishing medium until the flame area of ​​the flame position in the newly acquired thermal imaging image is judged to be zero, then the fire extinguishing task is terminated.

[0012] A processing device, comprising:

[0013] at least one memory for storing one or more programs;

[0014] At least one processor can execute one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor can implement the method described in the present invention.

[0015] A readable storage medium stores a computer program, which can implement the method described in the present invention when the computer program is executed by a processor.

[0016] Compared with the prior art, the automatic fire extinguishing method of the intelligent fire-fighting robot based on binocular vision provided by the present invention has the following beneficial effects:

[0017] The intelligent fire-fighting robot measures the three-dimensional trajectory of the current jet of the water cannon through a pre-built and trained deep binocular matching network based on matching cost in a visual measurement manner, and adjusts the pitch angle and yaw angle of the water cannon according to the measured three-dimensional trajectory of the jet as feedback, so that the jet of the water cannon can be accurately aimed at the flame at the flame position to perform accurate automatic fire-fighting operations. Since a complex jet physical model is not required, the measurement of a large number of physical quantities is avoided, and it can be easily applied to the automatic fire-fighting task of the intelligent fire-fighting robot. The method fuses the angle quantity of the water cannon with the visual information to calculate the matching cost, overcomes the problem of sparse features in the jet image area, and improves the accuracy of the jet trajectory measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0019] Figure 1 A flow chart of an automatic fire extinguishing method for an intelligent fire fighting robot based on binocular vision provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the specific content of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0021] First, the terms that may be used in this article are explained as follows:

[0022] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y means both “X” or “Y” and “X and Y”.

[0023] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0024] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0025] Unless otherwise specified or limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this article can be understood according to specific circumstances.

[0026] When concentration, temperature, pressure, size or other parameters are expressed in the form of a numerical range, the numerical range should be understood to specifically disclose all ranges formed by the pairing of any upper limit, lower limit, and preferred value in the numerical range, regardless of whether the range is explicitly stated; for example, if a numerical range of "2 to 8" is stated, the numerical range should be interpreted as including ranges such as "2 to 7", "2 to 6", "5 to 7", "3 to 4 and 6 to 7", "3 to 5 and 7", "2 and 5 to 7", etc. Unless otherwise specified, the numerical ranges stated herein include both their end values ​​and all integers and fractions within the numerical range.

[0027] The orientation or position relationship indicated by terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. are based on the orientation or position relationship shown in the drawings and are only for the convenience and simplification of description, and do not explicitly or implicitly indicate that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation of this document.

[0028] The scheme provided by the present invention is described in detail below. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professionals in the field. If the specific conditions are not specified in the embodiments of the present invention, they are carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. If the manufacturers are not specified in the reagents or instruments used in the embodiments of the present invention, they are all conventional products that can be purchased commercially.

[0029] like Figure 1 As shown, an embodiment of the present invention provides an automatic fire extinguishing method for an intelligent fire-fighting robot based on binocular vision, which is used to control an intelligent fire-fighting robot equipped with a binocular camera and a laser radar to perform an automatic fire extinguishing task, comprising the following steps:

[0030] Step 1, flame detection: When the intelligent fire-fighting robot performs an automatic fire-fighting task in an actual fire-fighting scene, the binocular camera carried by the intelligent fire-fighting robot obtains a color binocular image containing the flame, uses the target detection model to determine the flame position in the obtained color binocular image, and determines the three-dimensional position of the flame based on the detection frame of the flame position;

[0031] Step 2, calculating the yaw angle of the water cannon: according to the three-dimensional position relationship between the current intelligent fire-fighting robot and the three-dimensional position of the flame, the yaw angle of the water cannon is calculated, and the yaw angle is adjusted to make the water cannon point to the flame position;

[0032] Step 3, adjust the pitch angle of the water cannon: use the pre-built and trained deep binocular matching network based on matching cost to measure the current jet trajectory of the water cannon, and use the gap between the current jet trajectory of the water cannon and the flame position as feedback to adjust the pitch angle of the water cannon so that the jet of the water cannon is constantly close to the flame position;

[0033] Step 4, fire extinguishing hit judgment: when it is judged that the flame area of ​​the flame position in the continuously acquired thermal imaging image continues to decrease, confirm that the jet hits the flame at the flame position, maintain the current water cannon angle, and make the water cannon jet continue to deliver the fire extinguishing medium until it is judged that the flame area of ​​the flame position in the newly acquired thermal imaging image is zero, then end this fire extinguishing task.

[0034] Preferably, in step 1 of the above method, the target detection model can adopt an existing model, such as YOLO, Faster R-CNN, etc.

[0035] Preferably, in the above method, the deep binocular matching network based on matching cost in step 3 is constructed and trained in the following manner, including:

[0036] Step A31, construct a feature extractor: The feature extractor is composed of a fully connected layer containing 256 neurons and six convolutional blocks connected in parallel. With yaw angle As the input of the feature extractor, the water cannon pitch angle With yaw angle It is processed by a fully connected layer with 256 neurons, and the output of the fully connected layer is tiled to The input one-dimensional vector is copied in the spatial dimension of the convolution feature map to obtain an output vector, and the obtained output vector is added to the input feature map in the sixth convolution block to obtain a three-dimensional feature map that combines the angle information of the water cannon with the color binocular image information containing the jet in the feature map as the output feature of the feature extractor;

[0037] Step A32, calculating the real disparity through the matching cost calculation network: using the output features of the feature extractor as input, using the matching cost calculation network to calculate the external parameters of the binocular camera through the matching relationship of the feature points by using the matching cost calculation method, and performing disparity estimation on the color binocular image containing the flame obtained by the binocular camera carried by the intelligent fire-fighting robot to obtain the real disparity;

[0038] Step A33, in the training of the deep binocular matching network based on matching cost, a series of color binocular images and lidar point cloud data of jets collected by the intelligent fire-fighting robot with calibrated binocular camera and lidar are used as training data, and the three-dimensional points of the jets in the lidar point cloud data are used as true values. The lidar three-dimensional points are projected into the color binocular image through the extrinsic parameters obtained by calibration, and the true depth of the jet in the color binocular image is determined. Then, the true depth is converted into true disparity through the baseline distance of the binocular camera to train the deep binocular matching network based on matching cost;

[0039] Step A34, during the training phase, an additional jet mask is input into the deep binocular matching network based on matching cost. The jet mask is used in the loss calculation phase. The loss outside the region of interest of the jet mask is 0. The jet mask is obtained through the color features and motion features of the jet.

[0040] The advantage of the deep binocular matching network based on matching cost adopted in the present invention is that traditional image features, whether SIFT or other statistical-based regional features, do not work well when performing binocular matching on jet areas where image features are relatively sparse, while the deep binocular matching network based on matching cost avoids the feature design in traditional binocular matching through learning, and obtains the depth information contained in the binocular image in an end-to-end manner.

[0041] The matching cost calculation method in the above step A32 may use methods such as GANet, CFNet, ACFNet, etc. For details, please refer to the matching cost calculation methods in GANet, CFNet, ACFNet, etc.

[0042] In the above step A34, at each jet pixel point, the mask of the jet is determined in the following manner. The mask calculation method is: mask A is calculated by the color model of the jet, mask B is calculated by frame difference, mask A and mask B are ANDed pixel by pixel, and then the jet mask is obtained by morphological opening operation.

[0043] Preferably, in step A33 of the above method, a series of color binocular images of jets and laser radar point cloud data collected by an intelligent firefighting robot with a calibrated binocular camera and laser radar are used as training data in the following manner, including:

[0044] Step A331, sensor system calibration: first calibrate the binocular camera carried by the intelligent fire-fighting robot, and then jointly calibrate the laser radar and binocular camera carried by the intelligent fire-fighting robot;

[0045] Step A332, data collection: using the calibrated binocular camera and lidar carried by the intelligent fire-fighting robot, collect a series of color binocular images of the jet and lidar point cloud data as training data.

[0046] Preferably, in step A331 of the above method, the sensor system is calibrated according to the following method, including:

[0047] Step A3311, camera selection: A binocular camera with a lens focal length of 6 to 8 mm is used as the binocular camera carried by the intelligent firefighting robot. The binocular camera is installed on the intelligent firefighting robot through a pan / tilt. The baseline length of the binocular camera is not less than 30 cm.

[0048] Step A3312, calibrate the binocular camera: capture the image of the calibration plate and match the feature points of the calibration plate in the left and right images, calculate the intrinsic parameters of the binocular camera through the feature points, and calculate the extrinsic parameters of the binocular camera through the matching relationship of the feature points;

[0049] Step A3313, joint calibration of the binocular camera and the lidar: use a calibration plate with a width of not less than 1m, find and match the corner points of the calibration plate in the image taken by the binocular camera and the lidar point cloud, calculate the external parameters of the binocular camera and the lidar, and keep the lidar relatively fixed with the binocular camera after installation so that the calibration parameters do not change.

[0050] Preferably, in step 3 of the above method, the current jet trajectory of the water cannon is measured in the following manner using a pre-built and trained deep binocular matching network based on matching cost, including:

[0051] Step B31, inputting a frame of binocular image to be measured acquired by the binocular camera carried by the intelligent fire-fighting robot into the trained deep binocular matching network based on matching cost, and obtaining the output binocular camera disparity;

[0052] Step B32, calculate the depth of each pixel in the jet area by using the calibrated external parameters of the binocular camera to obtain a depth map. The depth calculation formula of each pixel is:

[0053] ;

[0054] In the formula, It is depth information; for parallax; is the baseline length of the binocular camera; It is the scale factor of mapping the binocular camera coordinate system point to the image plane in the horizontal direction;

[0055] Step B33, using the depth map obtained in step B32, calculate the three-dimensional coordinates of each pixel in the jet area through similar triangles :

[0056] ;

[0057] in, p = [ X , Y , d ] is the three-dimensional coordinate, X and Y are the coordinate values ​​in the camera coordinate system; [ x , y ] is the pixel image coordinate of the depth map; [ c x , c y ] is the center point of the depth map; is the scale factor of the binocular camera coordinate system point mapped to the image plane in the vertical direction, and the scale factor is determined by the calibrated intrinsic parameters of the binocular camera;

[0058] Step B34: The three-dimensional coordinates of each pixel point in the jet region are Projected onto the jet center plane perpendicular to the ground, a cubic equation is fitted as the jet trajectory that gives the relationship between the jet height and distance. .

[0059] Preferably, in step 2 of the above method, the yaw angle of the water cannon is calculated in the following manner, including:

[0060] Step 21, placing the flame at the center of the binocular image by rotating the pan / tilt of the binocular camera so that the binocular camera faces the flame;

[0061] Step 22, calculate the yaw angle of the water cannon using the following formula , the calculation formula is:

[0062] ;

[0063] in, is the current rotation angle of the binocular camera gimbal; is the distance from the center of the binocular camera gimbal to the flame, , is the distance from the binocular camera to the flame, is the rotation radius of the pan / tilt of the binocular camera; is the distance between the water cannon and the binocular camera gimbal.

[0064] Preferably, in step 3 of the above method, the pitch angle of the water cannon is adjusted in the following manner, including:

[0065] Step C31, select an existing fuzzy controller, and adjust the pitch angle of the water cannon through the fuzzy control strategy of the fuzzy controller. First, the height difference between the jet trajectory and the flame at the distance where the flame is located is set to , the height difference It is divided into 7 fuzzy subsets, namely: positive large, positive middle, positive small, negative large, negative middle, negative small and zero;

[0066] Step C32, according to the membership function of the fuzzy controller, determine when the height difference When it falls into each fuzzy subset, the pitch angle of the water cannon is adjusted by the fuzzy controller according to the following formula:

[0067] ;

[0068] in, represents a fuzzy controller; It represents the adjustment amount of the water cannon pitch angle determined by the membership function and the height difference E. The adjustment amount of the water cannon pitch angle determined by the membership function of the fuzzy controller and the height difference E is:

[0069] The height difference E is in the range of [-4, -3], which is negative and large, and the elevation angle of the water cannon increases by 5°.

[0070] The height difference E is in the range of [-3, -2], which is negative, and the elevation angle of the water cannon increases by 2.5°.

[0071] When the height difference E is in the range of [-2, -0.75] and is negative, the elevation angle of the water cannon increases by 1°.

[0072] The height difference E is zero in the range of [-0.75, 0.75], and the elevation angle of the water cannon remains unchanged.

[0073] The height difference E is in the range of (-0.75, 2], which is positive and small, and the elevation angle of the water cannon decreases by 1°.

[0074] The height difference E is in the range of (2, 3], which is the middle. The elevation angle of the water cannon is reduced by 2.5°.

[0075] When the height difference E is in the range of (3, 4], it is positive, and the elevation angle of the water cannon is reduced by 5°;

[0076] Step C33, before the hit judgment result in step 4 is yes, repeat the jet measurement in step 3, the water cannon pitch angle adjustment in step C32 and the hit judgment in step 4.

[0077] Preferably, in the above method, the intelligent firefighting robot is an intelligent robot with a mechanical arm and a parallel gripper at the end of the mechanical arm. This intelligent robot can automatically dock the required firefighting pipe with the fire hydrant through the mechanical arm and the parallel gripper as needed, and can also install firefighting equipment, such as a water cannon, on the intelligent robot through a rotating mechanism that can adjust the angle.

[0078] The embodiment of the present invention further provides a processing device, including:

[0079] at least one memory for storing one or more programs;

[0080] At least one processor can execute one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor can implement the above method.

[0081] The embodiment of the present invention further provides a readable storage medium storing a computer program, which can implement the above method when the computer program is executed by a processor.

[0082] In summary, it can be seen that the automatic fire extinguishing method of the embodiment of the present invention enables the intelligent fire-fighting robot to measure the three-dimensional trajectory of the current jet of the water cannon by means of visual measurement, through a pre-built and trained deep binocular matching network based on matching cost, and adjust the pitch angle and yaw angle of the water cannon according to the measured three-dimensional trajectory of the jet as feedback, that is, the jet of the water cannon can be accurately aimed at the flame at the flame position to perform accurate automatic fire extinguishing operations. Since a complex jet physical model is not required, the measurement of a large number of physical quantities is avoided, and it can be conveniently applied to the automatic fire extinguishing task of the intelligent fire-fighting robot; the method fuses the angle quantity of the water cannon with the visual information to calculate the matching cost, overcomes the problem of sparse features in the jet image area, and improves the accuracy of the jet trajectory measurement.

[0083] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the solution provided by the embodiment of the present invention is described in detail with specific embodiments below.

[0084] Example 1

[0085] like Figure 1 As shown, an embodiment of the present invention provides an automatic fire extinguishing method for an intelligent fire-fighting robot based on binocular vision, which is used to control an intelligent fire-fighting robot equipped with a binocular camera and a laser radar to perform an automatic fire extinguishing task, comprising the following steps:

[0086] Step 1, flame detection: When the intelligent firefighting robot performs automatic fire extinguishing in an actual fire extinguishing scene, it uses the target detection model to obtain the flame position in the binocular image acquired by the binocular camera, and determines the three-dimensional position of the flame based on the detection frame coordinates of the detected flame in the two images of the binocular image;

[0087] Step 2, calculating the yaw angle of the water cannon: according to the three-dimensional position relationship between the current intelligent fire-fighting robot and the flame, the yaw angle of the water cannon is calculated to make the water cannon point to the flame position;

[0088] Step 3, adjust the pitch angle of the water cannon: measure the current jet trajectory of the water cannon by pre-building a deep binocular matching network based on matching cost and training data collected by the binocular camera and lidar of the intelligent firefighting robot. Use the gap between the current jet trajectory of the water cannon and the flame position as feedback to adjust the pitch angle of the water cannon so that the jet of the water cannon is constantly close to the flame position;

[0089] Step 4, hit judgment: when it is judged through the thermal imaging image that the jet hits the flame, that is, when the flame area at the flame position in the thermal imaging image is greatly reduced, maintain the current water cannon angle so that the jet continues to deliver the fire extinguishing medium until the flame area at the flame position in the binocular image is zero, then the automatic fire extinguishing task is completed.

[0090] Preferably, in step 3 of the above method, a deep binocular matching network based on matching cost is constructed and trained in the following manner, including:

[0091] Step A31, the design of the feature extractor. Since the texture features of the jet area in the image are not obvious, it is difficult to accurately match by only extracting features from the image, which leads to errors in parallax and depth calculation, that is, errors in the jet model. Considering that in the image captured by the camera, the shape of the jet trajectory is highly correlated with the pitch angle and yaw angle of the water cannon, rotating the water cannon at any angle will cause the jet trajectory to change in the image. Therefore, when using binocular images to measure the jet trajectory, it is necessary to fuse these two angles of information of the water cannon. The angle vector of the water cannon is introduced as input in the feature extractor, and the accuracy of binocular matching is improved by means of a feature extractor that fuses the water cannon angle information and image information. The pitch angle of the water cannon With yaw angle As the new input of the feature extractor. Since the angle is a scalar, the water cannon angle information , It is processed by a fully connected layer with 256 neurons, and the output of the fully connected layer is then flattened onto In the dimension, that is, the input one-dimensional vector is copied in the spatial dimension of the convolution feature map. Finally, the tiled vector is added to the input feature map in the sixth convolution block to complete the combination of angle information and image information;

[0092] Step A32, in order to make the deep binocular matching network maintain the robustness of disparity estimation when facing jet images under different photographic conditions, a tandem matching cost calculation method is used, preferably a CFNet-style matching cost generation method;

[0093] Step A33, in the training of the deep binocular matching network, the three-dimensional points of the jet in the lidar point cloud collected in step 2 are used as true values, and the three-dimensional points of the lidar are projected into the image through the external parameters of the binocular camera obtained by pre-calibration, so as to determine the true depth of the jet in the binocular image, and then the true distance is converted into the true disparity through the baseline distance of the binocular camera, so as to train the deep binocular matching network;

[0094] Step A34: In order to reduce the interference of irrelevant features in the scene on the model, during the training phase, the model inputs an additional jet mask, which is used in the loss calculation phase, and the loss outside the mask's region of interest is 0. The jet mask is obtained through the jet's color features and motion features.

[0095] Preferably, in the above method, the binocular camera and laser radar carried by the intelligent firefighting robot are calibrated in the following manner, including:

[0096] Step A3311, camera selection: The focal length of the binocular camera lens is preferably 6~8mm, so as to ensure that the camera field of view can just cover the entire jet. Then, fix the binocular camera on the intelligent firefighting robot, preferably on a rotating platform to ensure the flexibility of observation. The baseline length of the binocular camera should be no less than 30cm to ensure the ranging accuracy at a distance;

[0097] Step A3312, calibrating the binocular camera: capturing the image of the calibration plate and matching the feature points of the calibration plate in the left and right images. Subsequently, the intrinsic parameters of the binocular camera are calculated through the feature points, and the extrinsic parameters of the binocular camera are calculated through the matching relationship of the feature points.

[0098] Step A3313, joint calibration of camera and LiDAR: Use a calibration plate with a width of no less than 1m to find and match the corner points of the calibration plate in the image and LiDAR point cloud, so as to calculate the external parameters between the camera and LiDAR, that is, the rotation matrix and translation vector between the binocular camera and LiDAR coordinate systems. In order to prevent the calibration parameters from changing, the LiDAR should be relatively fixed with the binocular camera system after installation.

[0099] Preferably, in step 3 of the above method, the current jet trajectory of the water cannon is measured in the following manner using a pre-built and trained deep binocular matching network based on matching cost, including:

[0100] Step B31, inputting a frame of binocular image into the trained deep binocular matching network to obtain the disparity output by the deep binocular matching network;

[0101] Step B32, calculate the depth of each location of the jet using the external parameters of the binocular camera calibrated in step 12. The depth can be calculated as:

[0102]

[0103] In the formula, It is depth information; for parallax; is the baseline length of the binocular camera; is the scale factor of the camera coordinate system point mapped to the image plane in the horizontal direction;

[0104] Step B33, using the depth map, the three-dimensional coordinates of each pixel in the jet area Can be calculated by similar triangles:

[0105] ;

[0106] Among them, X and Y are the coordinate values ​​in the camera coordinate system. p = [ X , Y , d ] ; [ x , y ] is the pixel image coordinate of the depth map, [ c x , c y ] is the center point of the depth map, is the scale factor of the camera coordinate system point mapped to the image plane in the vertical direction, which is determined by the intrinsic parameters in step 12;

[0107] Step B34, jet 3D point Projected onto the jet center plane perpendicular to the ground, used to fit a cubic equation as the trajectory of the jet , which describes the relationship between the height of the jet and the distance.

[0108] Preferably, in step 2 of the above method, the yaw angle of the water cannon is calculated in the following manner, including:

[0109] Step 21, Yaw Angle of Water Cannon The direction of the jet is determined, and the water cannon is pointed in the direction of the flame so that the jet trajectory passes through the target point. First, the flame is placed in the center of the image by rotating the pan / tilt of the binocular camera. At this time, it can be considered that the binocular camera is facing the flame;

[0110] Step 22, then, the current rotation angle of the gimbal of the binocular camera is , the distance between the binocular camera and the flame is , the distance from the center of the PTZ to the flame is , the distance between the water cannon and the pan / tilt is , the rotation radius of the gimbal is . Water cannon yaw angle It can be calculated as:

[0111] ;

[0112] in,

[0113] ;

[0114] is the distance from the camera to the flame; is the rotation radius of the pan / tilt of the binocular camera.

[0115] Preferably, in step 3 of the above method, the pitch angle of the water cannon is adjusted and controlled in the following manner, including:

[0116] Step C31, select an existing fuzzy controller, and adjust the pitch angle of the water cannon through the fuzzy control strategy of the fuzzy controller ,First, the height difference between the jet trajectory and the target at the distance where the flame is located is defined as . Then, Divided into 7 fuzzy subsets, including positive large, positive middle, positive small, negative large, negative middle, negative small and zero;

[0117] Step C32, It represents the adjustment amount of the water cannon pitch angle determined by the membership function and the height difference E. The adjustment amount of the water cannon pitch angle determined by the membership function of the fuzzy controller and the height difference E is:

[0118] The height difference E is in the range of [-4, -3], which is negative and large, and the elevation angle of the water cannon increases by 5°.

[0119] The height difference E is in the range of [-3, -2], which is negative, and the elevation angle of the water cannon increases by 2.5°.

[0120] When the height difference E is in the range of [-2, -0.75] and is negative, the elevation angle of the water cannon increases by 1°.

[0121] The height difference E is zero in the range of [-0.75, 0.75], and the elevation angle of the water cannon remains unchanged.

[0122] The height difference E is in the range of (-0.75, 2], which is positive and small, and the elevation angle of the water cannon decreases by 1°.

[0123] The height difference E is in the range of (2, 3], which is the middle. The elevation angle of the water cannon is reduced by 2.5°.

[0124] When the height difference E is in the range of (3, 4], it is positive, and the elevation angle of the water cannon is reduced by 5°;

[0125] Step 63, before the hit judgment in step 4 occurs, repeat the jet measurement in step 3 and the water cannon pitch angle adjustment in step 62, as well as the hit judgment in step 7.

[0126] In the above step C32, the fuzzy controller can adopt an existing fuzzy controller, such as a basic fuzzy controller, a fuzzy PID controller, an adaptive fuzzy controller, etc. The present application preferably adopts a basic fuzzy controller. If the control of the water cannon motor can be more precise, a more precise fuzzy controller such as a fuzzy PID controller or an adaptive fuzzy controller can also be used.

[0127] In summary, the automatic fire extinguishing method according to the embodiment of the present invention has at least the following beneficial effects:

[0128] (1) The three-dimensional trajectory of the jet is obtained by visual measurement, and the pitch and yaw angles of the water cannon are adjusted based on the measured jet trajectory. This does not require a complex jet physical model, thus avoiding the measurement of a large number of physical quantities. It can be easily applied to the automatic fire-fighting task of the intelligent fire-fighting robot.

[0129] (2) This method combines the angle measurement of the water cannon with the visual information to calculate the matching cost, which overcomes the problem of sparse features in the jet image area and improves the accuracy of the jet trajectory measurement.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0131] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. An automatic fire extinguishing method of an intelligent fire fighting robot based on binocular vision, characterized in that: The method is used to control an intelligent firefighting robot equipped with a binocular camera, a laser radar and a thermal imaging camera to perform automatic firefighting tasks, including the following steps: Step 1, flame detection: When the intelligent fire-fighting robot performs an automatic fire-fighting task in an actual fire-fighting scene, the binocular camera carried by the intelligent fire-fighting robot obtains a color binocular image containing the flame, uses the target detection model to determine the flame position in the obtained color binocular image, and determines the three-dimensional position of the flame based on the detection frame of the flame position; Step 2, calculating the yaw angle of the water cannon: according to the three-dimensional position relationship between the current intelligent fire-fighting robot and the three-dimensional position of the flame, the yaw angle of the water cannon is calculated, and the yaw angle is adjusted to make the water cannon point to the flame position; Step 3, adjusting the pitch angle of the water cannon: using the pre-built and trained deep binocular matching network based on matching cost to measure the current jet trajectory of the water cannon, and using the gap between the current jet trajectory of the water cannon and the flame position as feedback, adjusting the pitch angle of the water cannon so that the jet of the water cannon is constantly close to the flame position; the deep binocular matching network based on matching cost is constructed and trained in the following manner, including: Step A31, construct a feature extractor: The feature extractor is composed of a fully connected layer containing 256 neurons and six convolutional blocks connected in parallel. With yaw angle As the input of the feature extractor, the water cannon pitch angle With yaw angle It is processed by a fully connected layer with 256 neurons, and the output of the fully connected layer is tiled to The input one-dimensional vector is copied on the spatial dimension of the convolution feature map to obtain an output vector, and the obtained output vector is added to the input feature map in the sixth convolution block to obtain a three-dimensional feature map that combines the angle information of the water cannon with the color binocular image information containing the jet corresponding to the feature map as the output feature of the feature extractor; Step A32, calculating the real disparity through the matching cost calculation network: using the output features of the feature extractor as input, using the matching cost calculation network to calculate the external parameters of the binocular camera through the matching relationship of the feature points by using the matching cost calculation method, and performing disparity estimation on the color binocular image containing the flame obtained by the binocular camera carried by the intelligent fire-fighting robot to obtain the real disparity; Step A33, in the training of the deep binocular matching network based on matching cost, a series of color binocular images and lidar point cloud data of jets collected by the intelligent fire-fighting robot with calibrated binocular camera and lidar are used as training data, and the three-dimensional points of the jets in the lidar point cloud data are used as true values. The lidar three-dimensional points are projected into the color binocular image through the extrinsic parameters obtained by calibration, and the true depth of the jet in the color binocular image is determined. Then, the true depth is converted into true disparity through the baseline distance of the binocular camera to train the deep binocular matching network based on matching cost; Step A34, during the training phase, an additional jet mask is input into the deep binocular matching network based on matching cost, and the jet mask is used in the loss calculation phase, and the loss outside the region of interest of the jet mask is 0, wherein the jet mask is obtained by the color feature and motion feature of the jet; The method of measuring the current jet trajectory of the water cannon by using a pre-built and trained deep binocular matching network based on matching cost in the following manner includes: Step B31, inputting a frame of binocular image to be measured acquired by the binocular camera carried by the intelligent fire-fighting robot into the trained deep binocular matching network based on matching cost, and obtaining the output binocular camera disparity; Step B32, calculate the depth of each pixel in the jet area by using the calibrated external parameters of the binocular camera to obtain a depth map. The depth calculation formula of each pixel is: ； In the formula, It is depth information; for parallax; is the baseline length of the binocular camera; It is the scale factor of mapping the binocular camera coordinate system point to the image plane in the horizontal direction; Step B33, using the depth map obtained in step B32, calculate the three-dimensional coordinates of each pixel in the jet area through similar triangles : ； in, is the three-dimensional coordinate, X and Y are the coordinate values ​​in the camera coordinate system; is the pixel image coordinate of the depth map; is the center point of the depth map; is the scale factor of the binocular camera coordinate system point mapped to the image plane in the vertical direction, and the scale factor is determined by the calibrated intrinsic parameters of the binocular camera; Step B34: The three-dimensional coordinates of each pixel point in the jet region are Projected onto the jet center plane perpendicular to the ground, a cubic equation is fitted as the jet trajectory that gives the relationship between the jet height and distance. ; Step 4, fire extinguishing hit judgment: when it is judged that the flame area of ​​the flame position in the thermal imaging image continuously acquired by the thermal imaging camera carried by the intelligent fire-fighting robot continues to decrease, it is confirmed that the jet hits the flame at the flame position, and the current yaw angle and pitch angle of the water cannon are maintained, so that the jet of the water cannon continues to transport the fire extinguishing medium until the flame area of ​​the flame position in the newly acquired thermal imaging image is judged to be zero, then the fire extinguishing task is terminated.

2. The automatic fire extinguishing method of the intelligent fire fighting robot based on binocular vision according to claim 1 is characterized in that: In step A33, a series of color binocular images of jets and laser radar point cloud data collected by the intelligent firefighting robot with calibrated binocular camera and laser radar are used as training data in the following manner, including: Step A331, sensor system calibration: first calibrate the binocular camera carried by the intelligent fire-fighting robot, and then jointly calibrate the laser radar and binocular camera carried by the intelligent fire-fighting robot; Step A332, data collection: using the calibrated binocular camera and lidar carried by the intelligent fire-fighting robot, collect a series of color binocular images of the jet and lidar point cloud data as training data.

3. The automatic fire extinguishing method of the intelligent fire fighting robot based on binocular vision according to claim 2 is characterized in that: In step A331, the sensor system is calibrated according to the following method, including: Step A3311, camera selection: A binocular camera with a lens focal length of 6 to 8 mm is used as the binocular camera carried by the intelligent firefighting robot. The binocular camera is installed on the intelligent firefighting robot through a pan / tilt. The baseline length of the binocular camera is not less than 30 cm. Step A3312, calibrate the binocular camera: capture the image of the calibration plate and match the feature points of the calibration plate in the left and right images, calculate the intrinsic parameters of the binocular camera through the feature points, and calculate the extrinsic parameters of the binocular camera through the matching relationship of the feature points; Step A3313, joint calibration of the binocular camera and the lidar: use a calibration plate with a width of not less than 1m, find and match the corner points of the calibration plate in the image taken by the binocular camera and the lidar point cloud, calculate the external parameters between the binocular camera and the lidar, and keep the lidar relatively fixed with the binocular camera after installation so that the calibration parameters do not change.

4. The automatic fire extinguishing method of an intelligent fire-fighting robot based on binocular vision according to any one of claims 1 to 3, characterized in that: In step 2, the yaw angle of the water cannon is calculated in the following manner, including: Step 21, placing the flame at the center of the binocular image by rotating the pan / tilt of the binocular camera so that the binocular camera faces the flame; Step 22, calculate the yaw angle of the water cannon using the following formula , the calculation formula is: ； in, is the current rotation angle of the binocular camera gimbal; is the distance from the center of the binocular camera pan / tilt to the flame, , is the distance from the binocular camera to the flame, is the rotation radius of the pan / tilt of the binocular camera; is the distance between the water cannon and the binocular camera gimbal.

5. The automatic fire extinguishing method of an intelligent fire-fighting robot based on binocular vision according to any one of claims 1 to 3, characterized in that: In step 3, the pitch angle of the water cannon is adjusted in the following manner, including: Step C31, select an existing fuzzy controller, and adjust the pitch angle of the water cannon through the fuzzy control strategy of the fuzzy controller. First, the height difference between the jet trajectory and the flame at the distance where the flame is located is set to , the height difference It is divided into 7 fuzzy subsets, namely: positive large, positive middle, positive small, negative large, negative middle, negative small and zero; Step C32, It represents the adjustment amount of the water cannon pitch angle determined by the membership function and the height difference E. The adjustment amount of the water cannon pitch angle determined by the membership function of the fuzzy controller and the height difference E is: The height difference E is in the range of [-4, -3], which is negative and large, and the elevation angle of the water cannon increases by 5°. The height difference E is in the range of [-3, -2], which is negative, and the elevation angle of the water cannon increases by 2.5°. When the height difference E is in the range of [-2, -0.75] and is negative, the elevation angle of the water cannon increases by 1°. The height difference E is zero in the range of [-0.75, 0.75], and the elevation angle of the water cannon remains unchanged. When the height difference E is in the range of (-0.75, 2] and is positive, the elevation angle of the water cannon decreases by 1°. The height difference E is in the range of (2, 3], which is the middle. The elevation angle of the water cannon is reduced by 2.5°. When the height difference E is in the range of (3, 4], it is positive, and the elevation angle of the water cannon is reduced by 5°; Step C33, before the hit judgment result in step 4 is yes, repeat the jet measurement in step 3, the water cannon pitch angle adjustment in step C32 and the hit judgment in step 4.

6. The automatic fire extinguishing method of an intelligent fire-fighting robot based on binocular vision according to any one of claims 1 to 3, characterized in that: The intelligent fire-fighting robot is an intelligent robot with a mechanical arm, and a parallel gripper at the end of the mechanical arm.

7. A processing device, characterized in that: include: at least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor is capable of implementing the method described in any one of claims 1 to 6.

8. A readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the method described in any one of claims 1 to 6 can be implemented.

Images