Double-arm intelligent fire-fighting robot based on remote motion capture and fire extinguishing method

Abstract

The invention provides a double-arm intelligent fire-fighting robot based on remote motion capture and a fire extinguishing method. The double-arm intelligent fire-fighting robot comprises a robot main body, a driving module, a prediction positioning module, a fire extinguishing module and a control module, the robot main body is used for bearing core parts of the four modules; the driving module is used for walking, turning, uphill and obstacle avoidance of the fire-fighting robot and performing real-time balance adjustment according to the field environment, and comprises three parts, namely a crawler belt, an obstacle avoidance sensor, a left mechanical arm and a right mechanical arm; the prediction positioning module is used for detecting and positioning a fire extinguishing point in a fire scene according to a multi-mode fire detection positioning algorithm; the fire extinguishing module is used for remotely simulating a personnel fire extinguishing posture of a fire extinguishing behavior based on remote action capture, and controlling a mechanical arm to move a water gun to spray water and extinguish fire through a personnel fire extinguishing posture mapping method; and the control module is used for coordinating other modules as a whole to ensure that various functions of the fire-fighting robot are smoothly realized. The robot can conduct high-precision intelligent fire extinguishing.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent fire-fighting robots, and in particular to a dual-arm intelligent fire-fighting robot and a fire-fighting method based on remote motion capture. Background Art

[0002] With the rapid development of artificial intelligence and robotics, a variety of innovative robots suitable for firefighting have been developed both domestically and internationally. For example, there are intelligent firefighting robots capable of autonomously navigating fire scenes, the LUF60 firefighting robot, which uses fans and water mist nozzles to effectively cool and extinguish fires, and the GARM firefighting robot, equipped with an infrared video surveillance system to detect and extinguish hidden fires. Despite significant technological advancements, these firefighting robots still face several challenges, including limited intelligent fire location capabilities and a lack of accurate fire location methods based on multimodal information; a lack of efficient and accurate firefighting methods, relying solely on remote control based on manual experience or single-image target detection guidance; and limited ability to maintain balance in complex environments. Summary of the Invention

[0003] In order to solve the technical problems existing in the above-mentioned prior art, the present invention provides a dual-arm intelligent firefighting robot and a fire extinguishing method based on remote motion capture. The technical solution is as follows:

[0004] On the one hand, a dual-arm intelligent firefighting robot based on remote motion capture is provided, wherein the firefighting robot includes a robot body, a driving module, a prediction and positioning module, a fire extinguishing module and a control module;

[0005] The robot body is used to carry the core components of the four modules and connect the complete functions of the robot;

[0006] The driving module is used for the firefighting robot to walk, turn, climb, avoid obstacles, and perform real-time balance adjustment according to the on-site environment. It includes three components: tracks, obstacle avoidance sensors, and left and right dual robotic arms. The tracks are located at the bottom of the firefighting robot, the obstacle avoidance sensors are arranged around the firefighting robot, and the left and right dual robotic arms are located at the front end of the robot body above the tracks.

[0007] The predictive positioning module is used to detect and locate fire extinguishing points at the fire scene based on a multimodal fire detection and positioning algorithm. It includes an intelligent camera and a temperature camera, both of which are deployed on the top of the robot body above the tracks. The intelligent camera uses a target detection model based on visual recognition technology to select and judge flames within the monitoring range. The temperature camera presents the temperature distribution within the fire scene in the form of a heat map, capturing the specific location of the flame based on different heat values. The multimodal fire detection and positioning algorithm uses the target detection model as its basis and adds flame-related temperature feature information in the heat map to the target detection model to improve the accuracy of visual flame detection;

[0008] The fire extinguishing module is used to remotely simulate the fire extinguishing behavior of firefighters based on remote motion capture, and manipulate the robot arm to move the water gun to spray water to extinguish the fire through the fire extinguishing posture mapping method. It includes a water tank and a water gun connected to a water spray pipe. The water tank is arranged at the rear end of the robot body above the crawler, and the water gun is arranged at the front end of the robot body above the crawler;

[0009] The control module is used to coordinate other modules to ensure that the various functions of the fire-fighting robot are smoothly implemented, including a core controller and corresponding control software and communication modules, which are all located in the main body of the fire-fighting robot.

[0010] Optionally, the driving module performs real-time balance adjustment according to the on-site environment and includes three parts:

[0011] Part 1:

[0012] When the robot rolls over to one side, the obstacle avoidance sensor monitors the situation in real time and automatically activates the robotic arm to support the side of the rollover ground;

[0013] Part II:

[0014] When the robot tilts to one side during operation, the obstacle avoidance sensor monitors the tilt direction of the robot in real time, and then the robotic arm automatically adjusts its posture to the opposite direction, achieving balance of the robot by redistributing the weight;

[0015] Part III:

[0016] When a climbing situation occurs, the force analysis without the participation of the robot arm begins:

[0017] Other resistances are negligible, and the resistance F 阻This includes the component of its own gravity along the slope and the friction resistance between the track wheels and the road. In addition, to ensure the smooth progress of the robot, the robot should also have a traction force F along the slope based on the drive motor. 牵 , where F 牵 Should be greater than F 阻 , the relevant analysis formula is as follows:

[0018] G=mg

[0019] F 阻 =Gsinθ+Gμcosθ

[0020] F 牵 ≥F 阻 =Gsinθ+Gμcosθ

[0021] Where θ is the angle between the upward slope and the plane, G is the gravity on the robot, m is the mass of the robot, g is the proportional coefficient, and μ is the dynamic friction factor. At this time, without considering the existence of the robotic arm, if the robot is to remain stable on the slope while maintaining the traction force, the uphill angle should have the highest threshold θ max The robot can rely on the obstacle avoidance sensor to determine the approximate range of the slope ahead. When the angle does not exceed θ max When you can move forward, θ max The relevant formula is as follows:

[0022]

[0023] When the robot determines that the slope ahead is lower than θ based on the obstacle avoidance sensor max When the robot moves forward, if the prediction is wrong, the slope is higher than θ max When the robot is tilted, it will become unstable and tend to slide down or overturn. At this time, the robot can adaptively activate the left and right mechanical arms to support the rear uphill surface respectively to prevent the robot from rolling and facilitate timely improvement of traction. The force analysis is shown below:

[0024] F 阻 =Gsinθ+(Gcosθ-F 臂 sinα)μ

[0025] F 牵 +F 臂 cosα+f≥F 阻 =Gsinθ+(Gcosθ-F 臂 sinα)μ

[0026] F 牵 +F 臂 cosα min +f=Gsinθ+Gμcosθ-F臂 μsinα min

[0027]

[0028] Where α is the angle between the robot arm and the slope when completing the support, α min is the minimum angle that the robot arm can support on the slope under the current robot equipment and traction conditions, F 臂 is the supporting force of the robot arm, and f is the upward friction force generated by the robot arm and the slope when supporting.

[0029] Optionally, the prediction and positioning module detects and locates the fire extinguishing point at the fire scene according to the multimodal fire detection and positioning algorithm, and includes two parts:

[0030] Part 1: Select the target object in the thermal map, that is, extract the characteristic information of the flame target part in the thermal map to prepare for the next step of fusion into the target detection model. Specifically:

[0031] When the target detection model completes the detection of the fire scene, it will generate multiple anchor frames for the image data. The anchor frames include specific flame targets. However, since the three-dimensional thermal map generated by the temperature camera is different from the visual image, the flame target anchor frame in the visual image cannot be directly transferred to the thermal map for flame selection. If you want to convert the two-dimensional anchor frame in the visual image into an anchor frame in the thermal map, you need to first convert the two-dimensional anchor frame in the visual image into coordinates in three-dimensional space, and then project the three-dimensional coordinates onto the two-dimensional plane of the thermal map through the rotation matrix, translation vector and the intrinsic parameter conversion of the temperature camera. In theory, the dimensional change from 2D to 3D requires the addition of depth information, but the visual image captured by the smart camera does not have depth information. Therefore, the feature weighted pixel value of each point is used as pseudo-depth information. The feature weighted pixel value of each point is obtained as pseudo-depth information as follows:

[0032] The original RGB image has three channels. Therefore, for an image with n pixels, the i-th pixel position will have three pixel values ​​C ranging from 0 to 255. iR 、C iG and C iB Then, the three pixel values ​​at each pixel are averaged to obtain the average pixel value C of this point. imean When n pixels complete the average pixel operation, a grayscale map is obtained. mean ;

[0033] Select the feature map output by the feature extraction network of the target detection model for convolution operation and convert it to the same dimension as the original image. The formula is as follows:

[0034]

[0035] In the formula, Conv() is the convolution operation, Map out is the feature map output by the feature extraction network, Map is the feature map, w ori ×h ori ×1 is the length, width and channel dimension of the original image, where the channel dimension is fixed to 1;

[0036] When you get Later, the Sigmoid function is used to process each value in the feature map into a decimal between 0 and 1. The converted feature map Map weight Assign weights to Map mean Among them, the result Map after empowerment fusion The semantic information obtained by model feature extraction is integrated to make up for the low sensitivity of pixel values. fusion The value of each pixel in is used as the depth information of the anchor box mapped to the 3D dimension. Based on the obtained depth information, the 2D anchor box coordinates in the visual image can be converted into 3D coordinates. Then, through the relevant operations of coordinate system conversion, the 3D anchor box is mapped to the heat map.

[0037] After completing the anchor frame mapping on the heat map, the area framed by the heat map anchor frame is cut out and converted into the same area as the Map by convolution operation. out Maps of the same size hot ;

[0038] Part II: Multimodal fusion of visual images and heat maps, specifically:

[0039] The anchor frame obtained by the target detection model is used to intercept the frame area in the original video image and convert it into the same frame area as the Map by convolution operation. out Maps of the same size ROI ;

[0040] Then select Map hot 、Map ROI and Map out The three feature maps are used as basic elements, and the data is screened and processed based on the Hodges-Lehmann principle. The three feature maps are paired in pairs, and the feature values ​​within each pair of feature maps are averaged, globally averaged, and median operations are performed to finally obtain the feature map of multimodal information fusion. multi , the formula is as follows:

[0041]

[0042] Map small1=GAP(Map mean1 )

[0043] Map small2 =GAP(Map mean2 )

[0044] Map small3 =GAP(Map mean3 )

[0045] Map multi :,:,c)

[0046] =I small1 (c)×Map mean1 (:,:,c)+I small2 (c)×Map mean2 (:,:,c)

[0047] +I small3 (c)×Map mean3 (:,:,c)

[0048]

[0049]

[0050] Map mean1 , Map mean2 and Map mean3 They are three different average feature maps after the mean operation; GAP is the global average pooling operation at the feature map spatial level; Map small1 , Map small2 and Map small3 is the compressed feature map after the three average feature maps are pooled; I is the indicator function, the output is 1 if the condition is met, otherwise it is 0; Median is the median calculation.

[0051] Optionally, the fire extinguishing module is used to remotely simulate the fire extinguishing behavior of firefighters based on remote motion capture, and to manipulate the robotic arm to move the water gun to spray water to extinguish the fire through the fire extinguishing posture mapping method, and includes two parts:

[0052] Part 1: Determination of key points of the human body and robotic arm;

[0053] The key points of the human body are the elbow and shoulder joints of the arms. The swing and rotation of these two joints can fully and specifically demonstrate the direction and displacement of the controlled water pipe.

[0054] The robotic arm has at least two freely movable joints;

[0055] Part II: Mapping equations of human posture data;

[0056] Considering the angle and displacement changes of the human arm joints and the robotic arm joints on the x-axis, y-axis, and z-axis in real space, and adding the influencing parameters of the environment and mechanical equipment, the mapping from human posture to robotic arm posture is not a completely linear relationship. Therefore, the constructed mapping equation integrates nonlinear operations and uses dynamic equations to perform secondary corrections on the calculation results to weaken the influence of physical conditions. Specifically:

[0057] First, assume that each robotic arm has only two joints, corresponding to the shoulder and elbow joints of the human arm;

[0058] Assume that the displacements of the operator's elbow joint in the x-axis, y-axis, and z-axis directions are d x d y and d z , the displacement change rates are and The shoulder joint of the human body cannot move independently, so the shoulder joint displacement is not calculated here. The angles of the shoulder joint of the human arm on the x-axis, y-axis and z-axis are θ x1 ,θ y1 and θ z1 , the angle change rates are and The angles of the elbow joint on the x-axis, y-axis, and z-axis are θx2, θ y2 and θ z2 , the angle change rates are and

[0059] The displacements of the robot arm elbow joint in the x-axis, y-axis and z-axis directions are s x 、s y and s z , the displacement change rates are and The angles of the robot shoulder joint on the x-axis, y-axis, and z-axis are α x1 , α y1 and α z1 , the angle change rates are and The angles of the robot arm elbow joint on the x-axis, y-axis and z-axis are α x2 , α y2 and α z2 , the angle change rates are and

[0060] The mapping calculation from the human body to the shoulder and elbow joints of the robotic arm is divided into two parts:

[0061] Part 1: Calculate the joint angle and displacement function to obtain the displacement distance and rotation angle of each joint of the robotic arm;

[0062] (1) Calculation of angles and displacements of each joint of the robotic arm

[0063] 1) The function of the robot arm shoulder joint angle in the x-axis direction is as follows:

[0064]

[0065] in k is the change angle of the robot shoulder joint in the x-axis direction based on the change of the human shoulder joint angle before the secondary correction; ij (i=1,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; L arm is the overall length of the robotic arm; μ joint is the friction coefficient of the robotic arm shoulder joint; C smoke Smoke density: When the smoke density at the fire scene is too high, it will affect the fire extinguishing judgment, so it should be taken into consideration;

[0066] 2) The x-axis angle function of the robot arm elbow joint is as follows:

[0067]

[0068] in k is the change angle of the robot arm elbow joint in the x-axis direction based on the change of the human arm elbow joint angle before the secondary correction; ij (i=2,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; L forearm is the length of the forearm; c joint Damping coefficient of the robotic arm elbow joint;

[0069] 3) The function of the robot arm shoulder joint angle in the y-axis direction is as follows:

[0070]

[0071] in k is the change angle of the robot shoulder joint in the y-axis direction based on the change of the human arm shoulder joint angle before the secondary correction; ij (i=3,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; m arm is the overall mass of the robotic arm; k joint is the stiffness of the robotic arm shoulder joint; T envThe ambient temperature of the scene. When the temperature is too high, the degree of flame burning in the fire scene may be much higher than the predicted location. Therefore, this factor needs to be considered in the scope of the fire extinguishing operation.

[0072] 4) The angle function of the robot arm elbow joint in the y-axis direction is as follows:

[0073]

[0074] in k is the change angle of the robot arm elbow joint in the y-axis direction based on the change angle of the human arm elbow joint before the secondary correction; ij (i=4,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; m forearm is the mass of the robotic arm; k elastic is the elastic coefficient of the elbow joint;

[0075] 5) The function of the robot arm shoulder joint angle in the z-axis direction is as follows:

[0076]

[0077] in k is the change angle of the robot shoulder joint in the z-axis direction based on the change of the human shoulder joint angle before the secondary correction; ij (i=5,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; P cg is the center of gravity of the entire robotic arm; k torque is the motor torque coefficient; V wind The wind speed at the scene. If the scene is not a completely enclosed scene and there is wind flow, the impact of wind speed on the direction of fire extinguishing water flow needs to be considered;

[0078] 6) The angle function of the robot arm elbow joint in the z-axis direction is as follows:

[0079]

[0080] in k is the change angle of the robot arm elbow joint in the z-axis direction based on the change angle of the human arm elbow joint before the secondary correction; ij (i=6,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; I forearm Moment of inertia of the robotic arm; μ ubrication Lubrication coefficient of the robotic arm elbow joint; V windThe wind speed at the scene. If the scene is not a completely enclosed scene and there is wind flow, the impact of wind speed on the direction of fire extinguishing water flow needs to be considered;

[0081] 7) The displacement function of the elbow joint of the robot arm in the x-axis direction is as follows:

[0082]

[0083] where k ij , i = 7, j = 1, 2, 3…7 are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0084] 8) The displacement function of the robot arm elbow joint in the y-axis direction is as follows:

[0085]

[0086] where k ij , i = 8, j = 1, 2, 3…7 are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0087] 9) The displacement function of the robot arm elbow joint in the z-axis direction is as follows:

[0088]

[0089] where k ij , i = 9, j = 1, 2, 3…7 are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0090] (2) Calculate the expected acceleration

[0091] The expected acceleration is obtained by taking the second derivative of the uncorrected angles of the shoulder and elbow joints of the robot arm obtained from the above formula and

[0092] Part 2: Secondary correction of joint angles based on dynamic equations;

[0093] (1) Solving the dynamic equation to obtain the dynamic torque

[0094] 1) x-axis direction

[0095] For the robotic arm shoulder joint:

[0096]

[0097] Among them, M x1 (q) is the element of the inertia matrix M(q) corresponding to the x-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G x1 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ x1 The dynamic torque solved for the x-axis direction of the shoulder joint of the robotic arm;

[0098] For the robotic arm elbow joint:

[0099]

[0100] Among them, M x2 (q) is the element of the inertia matrix M(q) corresponding to the x-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G x2 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ x2 The dynamic torque solved for the x-axis direction of the elbow joint of the robot arm;

[0101] 2) y-axis direction

[0102] For the robotic arm shoulder joint:

[0103]

[0104] Among them, M y1 (q) is the element of the inertia matrix M(q) corresponding to the y-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G y1 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ y1 The dynamic torque solved for the shoulder joint of the robot arm in the y-axis direction;

[0105] For the robotic arm elbow joint:

[0106]

[0107] Among them, M y2 (q) is the element of the inertia matrix M(q) corresponding to the y-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G y2 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ y2 The dynamic torque solved for the y-axis direction of the elbow joint of the robot arm;

[0108] 3) z-axis direction

[0109] For the robotic arm shoulder joint:

[0110]

[0111] Among them, M z1 (q) is the element of the inertia matrix M(q) corresponding to the z-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G z1 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ z1 The dynamic torque solved for the z-axis direction of the robot shoulder joint;

[0112] For the robotic arm elbow joint:

[0113]

[0114] Among them, M z2 (q) is the element of the inertia matrix M(q) corresponding to the y-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G z2 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ z2 The dynamic torque solved for the z-axis direction of the elbow joint of the robot arm;

[0115] (2) Secondary correction of the shoulder and elbow joint angles of the robotic arm

[0116] 1) x-axis direction

[0117] Robotic arm shoulder joint:

[0118]

[0119] where k τx1 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0120] Robotic arm elbow joint:

[0121]

[0122] where k τx2 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0123] 2) Y-axis direction:

[0124] Robotic arm shoulder joint:

[0125]

[0126] where k τy1 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0127] Robotic arm elbow joint:

[0128]

[0129] where k τy2 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0130] 3) z-axis direction:

[0131] Robotic arm shoulder joint:

[0132]

[0133] where k τz1 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0134] Robotic arm elbow joint:

[0135]

[0136] where k τz2 It is an assignable parameter and needs to be accurately determined based on the actual robot arm structure, sensor data and experiments.

[0137] In another aspect, a method for extinguishing a fire using the intelligent firefighting robot is provided, comprising:

[0138] S1. A firefighter with on-site firefighting experience is located in a remote control room away from the fire scene. They observe the real-time video feed from the intelligent firefighting robot at the fire scene. The firefighter wears motion capture sensors on the shoulder and elbow joints of both arms and holds a water gun model in both hands to simulate on-site firefighting.

[0139] S2. The intelligent firefighting robot detects fires based on on-site video footage, obtains reliable flame positioning through a multimodal fire detection and positioning algorithm, and ultimately transmits the detection results to the control room. Firefighters manually judge whether to extinguish the fire based on the results observed in the control room, thereby achieving human-machine integration. Once it is determined to extinguish the fire, the firefighters, relying on their own work experience, swing the firefighting posture of the holding water gun model in the control room. The robot will use the personnel firefighting posture mapping method to transmit the firefighter's arm posture to the mechanical arms on both sides, and manipulate the water gun to accurately extinguish the fire.

[0140] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0141] 1) The present invention can rely on the combination of sensors, tracks and robotic arms in the driving module, and use a self-developed adaptive balance adjustment method based on the robotic arm to achieve the phenomenon of robot imbalance caused by uphill or obstacles.

[0142] 2) This paper has developed a multimodal fire location method that integrates temperature and visual information. Based on convolutional neural networks and Hodges-Lehmann estimation, this method integrates, compares, and extracts multimodal information, avoiding inaccurate fire location caused by factors such as image interference and insensitivity to temperature differentiation.

[0143] 3) The present invention designs an intelligent firefighting robot with dual robotic arms that can be based on motion capture. Firefighters can make firefighting gestures in a remote control room, read the personnel posture information and map it to the robotic arm of the firefighting robot, so that the robotic arm can control the water gun to extinguish the fire. This method can not only make full use of the personal experience of firefighters to make up for the defects of the automatic firefighting algorithm that is not intelligent, but also solve the problem of insensitivity and inaccuracy of traditional remote control methods such as remote controls or button operations. In order to ensure that human posture behavior can effectively guide the robotic arm to complete the movement, the present invention has developed a personnel firefighting posture mapping method that integrates equipment factors and environmental factors to realize the transformation of personnel key points to robotic arm joints, and technically realizes the motion mapping of the dual-arm firefighting robot at the fire scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0144] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0145] Figure 1 This is a module block diagram of a dual-arm intelligent firefighting robot based on remote motion capture provided by an embodiment of the present invention;

[0146] Figure 2 This is a schematic structural diagram of a dual-arm intelligent firefighting robot based on remote motion capture provided by an embodiment of the present invention;

[0147] Figure 3 This is a simplified force diagram of a robot when climbing a slope, provided by an embodiment of the present invention;

[0148] Figure 4 This is a schematic diagram of three feature graphs provided by an embodiment of the present invention being expanded in the form of rows and columns and paired one by one;

[0149] Figure 5 This is a flow chart of a fire extinguishing method based on remote operation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0150] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0151] The embodiment of the present invention provides a dual-arm intelligent firefighting robot based on remote motion capture, such as Figure 1 and 2 As shown, the firefighting robot includes a robot body, a driving module, a prediction and positioning module, a fire extinguishing module and a control module;

[0152] The robot body is used to carry the core components of the four modules and connect the complete functions of the robot;

[0153] The driving module is used for the firefighting robot to walk, turn, climb, avoid obstacles, and perform real-time balance adjustment according to the on-site environment. It includes three components: tracks, obstacle avoidance sensors, and left and right dual robotic arms. The tracks are located at the bottom of the firefighting robot, the obstacle avoidance sensors are arranged around the firefighting robot, and the left and right dual robotic arms are located at the front end of the robot body above the tracks.

[0154] Based on a built-in global path planning algorithm, the firefighting robot can automatically drive its tracks to complete its operations within the firefighting scenario, or it can move based on remote commands from firefighters. The obstacle avoidance sensor can optionally utilize a smoke-resistant laser obstacle avoidance radar. This sensor returns point cloud data of obstacles ahead of the robot, enabling obstacle detection and avoidance. However, during actual firefighting operations, the robot may encounter areas where obstacles cannot be completely avoided or where the terrain is uneven. Passing through such areas is highly likely to cause the robot to lose balance. Therefore, an adaptive balance adjustment method based on a robotic arm is designed in this embodiment of the present invention. Uneven terrain, such as raised road surfaces ahead, may cause the robot to climb slopes or tilt during movement. Since the robot designed in this embodiment of the present invention is a dual-arm robot, both the left and right robotic arms can be used for adaptive stabilization when not engaged in firefighting operations.

[0155] Optionally, the driving module performs real-time balance adjustment according to the on-site environment and includes three parts:

[0156] Part 1:

[0157] When the robot rolls over to one side, the obstacle avoidance sensor monitors the situation in real time and automatically activates the robotic arm to support the side of the rollover ground;

[0158] Part II:

[0159] When the robot tilts to one side during operation, the obstacle avoidance sensor monitors the tilt direction of the robot in real time, and then the robotic arm automatically adjusts its posture to the opposite direction, achieving balance of the robot by redistributing the weight;

[0160] Part III:

[0161] When a climbing situation occurs, such as Figure 3 As shown in (a), the force analysis starts without the participation of the robotic arm:

[0162] Other resistances are negligible, and the resistance F 阻 This includes the component of its own gravity along the slope and the friction resistance between the track wheels and the road. In addition, to ensure the smooth progress of the robot, the robot should also have a traction force F along the slope based on the drive motor. 牵 , where F 牵 Should be greater than F 阻 , the relevant analysis formula is as follows:

[0163] G=mg

[0164] F 阻 =Gsinθ+Gμcosθ

[0165] F 牵 ≥F 阻 =Gsinθ+Gμcosθ

[0166] Where θ is the angle between the upward slope and the plane, G is the gravity on the robot, m is the mass of the robot, g is the proportional coefficient, and μ is the dynamic friction factor. At this time, without considering the existence of the robotic arm, if the robot is to remain stable on the slope while maintaining the traction force, the uphill angle should have the highest threshold θ max The robot can rely on the obstacle avoidance sensor to determine the approximate range of the slope ahead. When the angle does not exceed θ max When you can move forward, θ max The relevant formula is as follows:

[0167]

[0168] When the robot determines that the slope ahead is lower than θ based on the obstacle avoidance sensor max When the robot moves forward, if the prediction is wrong, the slope is higher than θ max When the robot is in a state of instability and tends to slide down or overturn, the robot can adaptively start the left and right mechanical arms to support the uphill surface at the rear to prevent the robot from rolling and to facilitate timely improvement of traction, such as Figure 3 As shown in (b), the force analysis is as follows:

[0169] F 阻 =Gsinθ+(Gcosθ-F 臂 sinα)μ

[0170] F 牵 +F 臂 cosα+f≥F 阻 =Gsinθ+(Gcosθ-F 臂 sinα)μ

[0171] F 牵 +F 臂 cosα min +f=Gsinθ+Gμcosθ-F 臂 μsinα min

[0172]

[0173] Where α is the angle between the robot arm and the slope when completing the support, α min is the minimum angle that the robot arm can support on the slope under the current robot equipment and traction conditions, F 臂is the supporting force of the robot arm, and f is the upward friction force generated by the robot arm and the slope when supporting.

[0174] In actual applications, users can determine the allowable support angle α of the robotic arm under different slopes based on the above formula through actual simulation tests, so as to determine the support posture under different slopes in advance. When the slope at the actual fire scene is too high and there is a tendency to slide and overturn, the robotic arm will be automatically supported according to the known α range to prevent support failure and equipment wear at the fire scene.

[0175] The predictive positioning module is used to detect and locate the fire extinguishing point (i.e., the flame position) at the fire scene according to the multimodal fire detection and positioning algorithm. It includes an intelligent camera and a temperature camera, both of which are arranged at the top of the robot body above the crawler. The intelligent camera uses a target detection model based on visual recognition technology to select and judge the flame within the monitoring range. The temperature camera presents the temperature distribution in the fire scene in the form of a heat map, and captures the specific position of the flame based on the difference in heat value. (However, when performing flame judgment alone, there are certain limitations. The target detection model is easily interfered by similar objects and has difficulty capturing the location of the fire extinguishing point when the flame amplitude is large. Although the temperature camera can determine the location of the fire extinguishing point by temperature difference, the high temperature environment and smoke interference at the fire scene will cause large errors in the presentation of the heat map. Therefore, the embodiment of the present invention designs a multimodal fire detection and positioning algorithm.) The multimodal fire detection and positioning algorithm uses the target detection model as its basis and adds the temperature feature information related to the flame in the heat map to the target detection model to improve the accuracy of visual flame detection.

[0176] Optionally, the prediction and positioning module detects and locates the fire extinguishing point at the fire scene according to the multimodal fire detection and positioning algorithm, and includes two parts:

[0177] Part 1: Select the target object in the thermal map, that is, extract the characteristic information of the flame target part in the thermal map to prepare for the next step of fusion into the target detection model. Specifically:

[0178] When the target detection model completes the detection of the fire scene, it will generate multiple anchor frames for the image data. The anchor frames include specific flame targets. However, since the three-dimensional thermal map generated by the temperature camera is different from the visual image, the flame target anchor frame in the visual image cannot be directly transferred to the thermal map for flame selection. If you want to convert the two-dimensional anchor frame in the visual image into an anchor frame in the thermal map, you need to first convert the two-dimensional anchor frame in the visual image into coordinates in three-dimensional space, and then project the three-dimensional coordinates onto the two-dimensional plane of the thermal map through the rotation matrix, translation vector and the intrinsic parameter conversion of the temperature camera. In theory, the dimensional change from 2D to 3D requires the addition of depth information, but the visual image captured by the smart camera does not have depth information. Therefore, the feature weighted pixel value of each point is used as pseudo-depth information. The feature weighted pixel value of each point is obtained as pseudo-depth information as follows:

[0179] The original RGB image has three channels. Therefore, for an image with n pixels, the i-th pixel position will have three pixel values ​​C ranging from 0 to 255. iR 、C iG and C iB Then, the three pixel values ​​at each pixel are averaged to obtain the average pixel value C of this point. imean When n pixels complete the average pixel operation, a grayscale map is obtained. mean ;

[0180] Select the feature map output by the feature extraction network of the target detection model for convolution operation and convert it to the same dimension as the original image. The formula is as follows:

[0181]

[0182] In the formula, Conv() is the convolution operation, Map out is the feature map output by the feature extraction network, Map is the feature map, w ori ×h ori ×1 is the length, width and channel dimension of the original image, where the channel dimension is fixed to 1;

[0183] When you get Later, the Sigmoid function is used to process each value in the feature map into a decimal between 0 and 1. The converted feature map Map weight Assign weights to Map mean Among them, the result Map after empowerment fusion The semantic information obtained by model feature extraction is integrated to make up for the low sensitivity of pixel values. fusionThe value of each pixel in is used as the depth information of the anchor box mapped to the 3D dimension. Based on the obtained depth information, the 2D anchor box coordinates in the visual image can be converted into 3D coordinates. Then, through the relevant operations of coordinate system conversion, the 3D anchor box is mapped to the heat map.

[0184] After completing the anchor frame mapping on the heat map, the area framed by the heat map anchor frame is cut out and converted into the same area as the Map by convolution operation. out Maps of the same size hot ;

[0185] Part II: Multimodal fusion of visual images and heat maps, specifically:

[0186] The anchor frame obtained by the target detection model is used to intercept the frame area in the original video image and convert it into the same frame area as the Map by convolution operation. out Maps of the same size ROI ;

[0187] Then select Map hot 、Map ROI and Map out The three feature maps are used as basic elements, and the data is screened and processed based on the Hodges-Lehmann principle. The three feature maps are paired in pairs, and the feature values ​​within each pair of feature maps are averaged, globally averaged, and median operations are performed to finally obtain the feature map of multimodal information fusion. multi , the formula is as follows:

[0188]

[0189]

[0190] Map small1 =GAP(Map mean1 )

[0191] Map small2 =GAP(Map mean2 )

[0192] Map small3 =GAP(Map mean3 )

[0193] Map multi (:,:,c)

[0194] =I samll1 (c)×Map mean1 (:,:,c)+I small2 (c)×Map mean2 (:,:,c)

[0195] +I small3 (c)×Map mean3 (:,:,c)

[0196]

[0197] Map mean1 , Map mean2 and Map mean3 They are three different average feature maps after the mean operation; GAP is the global average pooling operation at the feature map spatial level; Map small1 , Map small2 and Map small3 is the compressed feature map after the three average feature maps are pooled; I is the indicator function, the output is 1 if the condition is met, otherwise it is 0; Median is the median calculation.

[0198] Specifically, such as Figure 4 As shown, the embodiment of the present invention is to use the three shadow modules ([Map hot , Map ROI ][Map hot , Map out ][Map out , Map ROI ]) The list elements inside are averaged with each other, and then the three average feature maps obtained are globally averaged and pooled. After pooling, three compressed feature maps of size 1×1×C are obtained (where C is the number of channels of the feature map). Then, the three compressed feature maps are median-operated at each layer of the channel level. When the median of the i-th layer is determined to be a compressed feature map, the final output of the layer is the i-th layer of the average feature map corresponding to the compressed feature map. For the feature map, the feature values ​​of the corresponding points in the three feature maps are calculated to obtain a complete feature map of multimodal information fusion. multi .

[0199] Simply concatenating and fusing the flame features from the captured heat map with those from the visual image would likely result in a significant amount of redundant information. However, fusing the features from the two images using a convolution-based approach would inevitably eliminate noise from both the heat map and the visual image. This embodiment of the present invention avoids these two common approaches and instead leverages the unique noise filtering capabilities of the Hodges-Lehmann estimator to filter and interact with the original image information, deep feature information, and heat map information, achieving robust and effective fusion of visual and heat map features.

[0200] The multimodal fire detection and positioning algorithm of the embodiment of the present invention ensures the authenticity and reliability of flame target detection through self-designed target object selection and multimodal feature fusion methods, helping firefighting robots to capture fire extinguishing points more accurately.

[0201] The fire extinguishing module is used to remotely simulate the fire extinguishing behavior of firefighters based on remote motion capture, and manipulate the robot arm to move the water gun to spray water to extinguish the fire through the fire extinguishing posture mapping method. It includes a water tank and a water gun connected to a water spray pipe. The water tank is arranged at the rear end of the robot body above the crawler, and the water gun is arranged at the front end of the robot body above the crawler;

[0202] Optionally, the fire extinguishing module is used to remotely simulate the fire extinguishing behavior of firefighters based on remote motion capture, and to manipulate the robotic arm to move the water gun to spray water to extinguish the fire through the fire extinguishing posture mapping method, and includes two parts:

[0203] Part 1: Determination of key points of the human body and robotic arm;

[0204] The key points of the human body are the elbow and shoulder joints of the arms. The swing and rotation of these two joints can fully and specifically demonstrate the direction and displacement of the controlled water pipe. (The wrist joint has a smaller degree of control over the overall movement of the pipe and is not included in the calculation scope to avoid excessive calculation complexity.)

[0205] The robotic arm must have at least two freely movable joints (in principle, multiple joints can be designed to achieve more flexible control);

[0206] Part II: Mapping equation for human posture data (the mapping equation provided in the embodiment of the present invention is the basic formula for mapping of a dual-arm firefighting robot. The detailed variables and operations can be adjusted according to the actual situation of the robot and field experiments. The formula uses trigonometric functions, exponential operations, and square root operations to add nonlinear relationships to the mapping formula to further enhance the reliability of the results);

[0207] Considering the angle and displacement changes of the human arm joints and the robotic arm joints on the x-axis, y-axis, and z-axis in real space, and adding the influencing parameters of the environment and mechanical equipment, the mapping from human posture to robotic arm posture is not a completely linear relationship. Therefore, the constructed mapping equation integrates nonlinear operations and uses dynamic equations to perform secondary corrections on the calculation results to weaken the influence of physical conditions. Specifically:

[0208] First, assume that each robotic arm has only two joints, corresponding to the shoulder and elbow joints of the human arm;

[0209] Assume that the displacements of the operator's elbow joint in the x-axis, y-axis, and z-axis directions are dx d y and d z , the displacement change rates are and The shoulder joint of the human body cannot move independently, so the shoulder joint displacement is not calculated here. The angles of the shoulder joint of the human arm on the x-axis, y-axis and z-axis are θ x1 ,θ y1 and θ z1 , the angle change rates are and The angles of the elbow joint on the x-axis, y-axis, and z-axis are θ x2 ,θ y2 and θ z2 , the angle change rates are and

[0210] The displacements of the robot arm elbow joint in the x-axis, y-axis and z-axis directions are s x 、s y and s z , the displacement change rates are and The angles of the robot shoulder joint on the x-axis, y-axis, and z-axis are α x1 , α y1 and α z1 , the angle change rates are and The angles of the robot arm elbow joint on the x-axis, y-axis and z-axis are α x2 , α y2 and α z2 , the angle change rates are and

[0211] The mapping calculation from the human body to the shoulder and elbow joints of the robotic arm is divided into two parts:

[0212] Part 1: Calculate the joint angle and displacement function to obtain the displacement distance and rotation angle of each joint of the robotic arm;

[0213] (1) Calculation of angles and displacements of each joint of the robotic arm

[0214] 1) The function of the robot arm shoulder joint angle in the x-axis direction is as follows:

[0215]

[0216] in k is the change angle of the robot shoulder joint in the x-axis direction based on the change of the human shoulder joint angle before the secondary correction; ij(i=1,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; L arm is the overall length of the robotic arm; μ joint is the friction coefficient of the robotic arm shoulder joint; C smoke Smoke density: When the smoke density at the fire scene is too high, it will affect the fire extinguishing judgment, so it should be taken into consideration;

[0217] 2) The x-axis angle function of the robot arm elbow joint is as follows:

[0218]

[0219] in k is the change angle of the robot arm elbow joint in the x-axis direction based on the change of the human arm elbow joint angle before the secondary correction; ij (i=2,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; L forearm is the length of the forearm; c joint Damping coefficient of the robotic arm elbow joint;

[0220] 3) The function of the robot arm shoulder joint angle in the y-axis direction is as follows:

[0221]

[0222] in k is the change angle of the robot shoulder joint in the y-axis direction based on the change of the human arm shoulder joint angle before the secondary correction; ij (i=3,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; m arm is the overall mass of the robotic arm; k joint is the stiffness of the robotic arm shoulder joint; T env The ambient temperature of the scene. When the temperature is too high, the degree of flame burning in the fire scene may be much higher than the predicted location. Therefore, this factor needs to be considered in the scope of the fire extinguishing operation.

[0223] 4) The angle function of the robot arm elbow joint in the y-axis direction is as follows:

[0224]

[0225] in k is the change angle of the robot arm elbow joint in the y-axis direction based on the change angle of the human arm elbow joint before the secondary correction; ij(i=4,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; m forearm is the mass of the robotic arm; k elastic is the elastic coefficient of the elbow joint;

[0226] 5) The function of the robot arm shoulder joint angle in the z-axis direction is as follows:

[0227]

[0228] in k is the change angle of the robot shoulder joint in the z-axis direction based on the change of the human shoulder joint angle before the secondary correction; ij (i=5,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; P cg is the center of gravity of the entire robotic arm; k torque is the motor torque coefficient; V wind The wind speed at the scene. If the scene is not a completely enclosed scene and there is wind flow, the impact of wind speed on the direction of fire extinguishing water flow needs to be considered;

[0229] 6) The angle function of the robot arm elbow joint in the z-axis direction is as follows:

[0230]

[0231] in k is the change angle of the robot arm elbow joint in the z-axis direction based on the change angle of the human arm elbow joint before the secondary correction; ij (i=6,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; I forearm Moment of inertia of the robotic arm; μ ubrication Lubrication coefficient of the robotic arm elbow joint; V wind The wind speed at the scene. If the scene is not a completely enclosed scene and there is wind flow, the impact of wind speed on the direction of fire extinguishing water flow needs to be considered;

[0232] 7) The displacement function of the elbow joint of the robot arm in the x-axis direction is as follows:

[0233]

[0234] where k ij , i = 7, j = 1, 2, 3…7 are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0235] 8) The displacement function of the robot arm elbow joint in the y-axis direction is as follows:

[0236]

[0237] where k ij , i = 8, j = 1, 2, 3…7 are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0238] 9) The displacement function of the robot arm elbow joint in the z-axis direction is as follows:

[0239]

[0240] where k ij , i = 9, j = 1, 2, 3…7 are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0241] (2) Calculate the expected acceleration

[0242] The expected acceleration is obtained by taking the second derivative of the uncorrected angles of the shoulder and elbow joints of the robot arm obtained from the above formula and

[0243] Part 2: Secondary correction of joint angles based on dynamic equations;

[0244] (1) Solving the dynamic equation to obtain the dynamic torque

[0245] 1) x-axis direction

[0246] For the robotic arm shoulder joint:

[0247]

[0248] Among them, M x1 (q) is the element of the inertia matrix M(q) corresponding to the x-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G x1 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ x1 The dynamic torque solved for the x-axis direction of the shoulder joint of the robotic arm;

[0249] For the robotic arm elbow joint:

[0250]

[0251] Among them, M x2(q) is the element of the inertia matrix M(q) corresponding to the x-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G x2 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ x2 The dynamic torque solved for the x-axis direction of the elbow joint of the robot arm;

[0252] 2) y-axis direction

[0253] For the robotic arm shoulder joint:

[0254]

[0255] Among them, M y1 (q) is the element of the inertia matrix M(q) corresponding to the y-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G y1 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ y1 The dynamic torque solved for the shoulder joint of the robot arm in the y-axis direction;

[0256] For the robotic arm elbow joint:

[0257]

[0258] Among them, M y2 (q) is the element of the inertia matrix M(q) corresponding to the y-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G y2 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ y2 The dynamic torque solved for the y-axis direction of the elbow joint of the robot arm;

[0259] 3) z-axis direction

[0260] For the robotic arm shoulder joint:

[0261]

[0262] Among them, M z1(q) is the element of the inertia matrix M(q) corresponding to the z-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G z1 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ z1 The dynamic torque solved for the z-axis direction of the robot shoulder joint;

[0263] For the robotic arm elbow joint:

[0264]

[0265] Among them, M z2 (q) is the element of the inertia matrix M(q) corresponding to the y-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G z2 M(q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ z2 The dynamic torque solved for the z-axis direction of the elbow joint of the robot arm;

[0266] (2) Secondary correction of the shoulder and elbow joint angles of the robotic arm

[0267] 1) x-axis direction

[0268] Robotic arm shoulder joint:

[0269]

[0270] where k τx1 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0271] Robotic arm elbow joint:

[0272]

[0273] where k τx2 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0274] 2) Y-axis direction:

[0275] Robotic arm shoulder joint:

[0276]

[0277] where k τy1 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0278] Robotic arm elbow joint:

[0279]

[0280] where k τy2 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0281] 3) z-axis direction:

[0282] Robotic arm shoulder joint:

[0283]

[0284] where k τz1 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments;

[0285] Robotic arm elbow joint:

[0286]

[0287] where k τz2 It is an assignable parameter and needs to be accurately determined based on the actual robot arm structure, sensor data and experiments.

[0288] In addition to the angle and displacement of the human body movement, the parameters of the above formula need to be determined in advance through experiments with relevant requirements and the equipment's own parameters. After determining the values ​​of all parameters, the formula can calculate the corresponding angle (corrected angle) and displacement of each joint of the robotic arm in real time through the angle and displacement of each joint of the human body. Firefighters can therefore remotely control the water gun through the controller to perform firefighting operations. During the firefighting process, personnel can change their posture at any time, and the built-in algorithm can complete the transformation and mapping of human posture in real time. The remote operation mapping firefighting method based on motion capture can rely more on the experience of firefighters to extinguish fires, and is more convenient and efficient than simple remote control operation, and can complete the firefighting task as quickly and accurately as possible.

[0289] The control module is used to coordinate other modules to ensure that the various functions of the fire-fighting robot are smoothly implemented, including a core controller and corresponding control software and communication modules, which are all located in the main body of the fire-fighting robot.

[0290] The core controller is responsible for receiving data from various sensors, processing the data according to preset algorithms and programs, and issuing control instructions to various execution modules. The communication module is used to receive personnel posture information transmitted from the remote control room.

[0291] For robot movement, the core controller receives remote commands from firefighters or control signals generated by a built-in global path planning algorithm. These signals are then sent to the tracks of the driving module, driving the robot's movements, such as walking, turning, and climbing. When the obstacle avoidance sensor detects point cloud data of an obstacle ahead, it transmits this information to the core controller. Based on this data, the core controller uses an algorithm to determine the obstacle's location, size, and shape, calculates the optimal obstacle avoidance path, and adjusts the track's direction and speed to achieve obstacle avoidance.

[0292] When encountering situations that may cause the robot to lose balance, such as obstacles that cannot be completely avoided or uneven terrain, the core controller will call the adaptive balance adjustment method based on the robotic arm to adjust the robot's center of gravity through the movement of the robotic arm to maintain the robot's balance.

[0293] For the predictive positioning of fire extinguishing points, the core controller activates the temperature camera and the camera equipped with the fire visual recognition algorithm in the predictive positioning module, and starts the built-in multimodal fire detection and positioning algorithm. The temperature camera and the camera continuously transmit the on-site fire information to the core controller, and the core controller uses this information to perform multimodal fire judgment.

[0294] For on-site firefighting, when the robot arrives at the fire scene and completes the predicted positioning of the fire location, the core controller will enable the built-in personnel remote control posture mapping firefighting method according to the functional requirements of the firefighting module, and send the captured action information to the core controller. The core controller converts this action information into control instructions and drives the firefighting robot's arms to perform the corresponding firefighting actions through remote operation.

[0295] The process of modules driving in sequence:

[0296] Startup phase:

[0297] The control system module starts first, performing a self-test of the entire firefighting robot to ensure that all modules are functioning properly. This includes checking the functioning of the driving module's tracks, obstacle avoidance sensors, and robotic arms. It also checks the predictive positioning module's temperature and camera data collection capabilities. Furthermore, the firefighting module's posture mapping and dual-arm motion execution are functioning properly.

[0298] Mobile phase:

[0299] When a firefighter activates the robot, or the system activates it according to a pre-set mission, the control module's core controller sends a movement signal to the driving module based on the mission requirements (which can be a remote command or built-in global path planning), causing the robot to move toward the fire scene via its tracks. During this movement, the core controller continuously receives information from the obstacle avoidance sensors and adjusts the driving route in real time to ensure the robot reaches the fire scene safely.

[0300] Fire location phase:

[0301] When the robot arrives at the fire scene, the control module activates the predictive positioning module. The core controller coordinates the temperature camera and the camera to collect and analyze information from the scene, identifying and detecting the fire location using a multimodal fire detection algorithm.

[0302] Fire extinguishing stage:

[0303] Once the predictive positioning module locates the fire, the control module activates the fire extinguishing module. The core controller uses remote human motion capture technology to receive the firefighter's movement information and convert it into control commands, driving the firefighting robot's arms to perform firefighting operations.

[0304] Ending stage:

[0305] After completing the fire extinguishing task, the control module drives the driving module again, so that the robot can safely evacuate the fire scene according to a predetermined path or according to remote instructions.

[0306] like Figure 5 As shown, an embodiment of the present invention further provides a method for extinguishing a fire using the above-mentioned intelligent firefighting robot, comprising:

[0307] S1. A firefighter with on-site firefighting experience is located in a remote control room away from the fire scene. They observe the real-time video feed from the intelligent firefighting robot at the fire scene. The firefighter wears motion capture sensors on the shoulder and elbow joints of both arms and holds a water gun model in both hands to simulate on-site firefighting.

[0308] S2. The intelligent firefighting robot detects fires based on on-site video footage, obtains reliable flame positioning through a multimodal fire detection and positioning algorithm, and ultimately transmits the detection results to the control room. Firefighters manually judge whether to extinguish the fire based on the results observed in the control room, thereby achieving human-machine integration. Once it is determined to extinguish the fire, the firefighters, relying on their own work experience, swing the firefighting posture of the holding water gun model in the control room. The robot will use the personnel firefighting posture mapping method to transmit the firefighter's arm posture to the mechanical arms on both sides, and manipulate the water gun to accurately extinguish the fire.

[0309] Alternatively, if the firefighting area changes, firefighters can move forward, backward, left, and right within the control room, swinging their arms at various angles simultaneously. The firefighting posture mapping method accounts for changes in the front-to-back coordinates, thus accounting for the effects of the firefighter moving forward, backward, and at different angles, ensuring the water cannon remains aligned with the firefighter's control. When extinguishing the fire on-site is complete, firefighters can pause the mapping to reduce the burden on the communication channel.

[0310] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A dual-arm intelligent firefighting robot based on remote motion capture, characterized in that: The firefighting robot includes a robot body, a driving module, a prediction and positioning module, a fire extinguishing module and a control module; The robot body is used to carry the core components of the four modules and connect the complete functions of the robot; The driving module is used for the firefighting robot to walk, turn, climb, avoid obstacles, and perform real-time balance adjustment according to the on-site environment. It includes three components: tracks, obstacle avoidance sensors, and left and right dual robotic arms. The tracks are located at the bottom of the firefighting robot, the obstacle avoidance sensors are arranged around the firefighting robot, and the left and right dual robotic arms are located at the front end of the robot body above the tracks. The predictive positioning module is used to detect and locate fire extinguishing points at the fire scene based on a multimodal fire detection and positioning algorithm. It includes an intelligent camera and a temperature camera, both of which are deployed on the top of the robot body above the tracks. The intelligent camera uses a target detection model based on visual recognition technology to select and judge flames within the monitoring range. The temperature camera presents the temperature distribution within the fire scene in the form of a heat map, capturing the specific location of the flame based on different heat values. The multimodal fire detection and positioning algorithm uses the target detection model as its basis and adds flame-related temperature feature information in the heat map to the target detection model to improve the accuracy of visual flame detection; The fire extinguishing module is used to remotely simulate the fire extinguishing behavior of firefighters based on remote motion capture, and manipulate the robot arm to move the water gun to spray water to extinguish the fire through the fire extinguishing posture mapping method. It includes a water tank and a water gun connected to a water spray pipe. The water tank is arranged at the rear end of the robot body above the crawler, and the water gun is arranged at the front end of the robot body above the crawler; The control module is used to coordinate other modules to ensure that the various functions of the fire-fighting robot are smoothly implemented, including a core controller and corresponding control software and communication modules, which are all located in the main body of the fire-fighting robot.

2. The intelligent fire-fighting robot according to claim 1, characterized in that: The driving module performs real-time balance adjustment according to the on-site environment and includes three parts: Part 1: When the robot rolls over to one side, the obstacle avoidance sensor monitors the situation in real time and automatically activates the robotic arm to support the side of the rollover ground; Part II: When the robot tilts to one side during operation, the obstacle avoidance sensor monitors the tilt direction of the robot in real time, and then the robotic arm automatically adjusts its posture to the opposite direction, achieving balance of the robot by redistributing the weight; Part III: When a climbing situation occurs, the force analysis without the participation of the robot arm begins: Other resistances are negligible, and the resistance F 阻 This includes the component of its own gravity along the slope and the friction resistance between the track wheels and the road. In addition, to ensure the smooth progress of the robot, the robot should also have a traction force F along the slope based on the drive motor. 牵 , where F 牵 Should be greater than F 阻 , the relevant analysis formula is as follows: G=mg F 阻 =Gsinθ+Gμcosθ F 牵 ≥F 阻 =Gsinθ+Gμcosθ Where θ is the angle between the upward slope and the plane, G is the gravity on the robot, m is the mass of the robot, g is the proportional coefficient, and μ is the dynamic friction factor. At this time, without considering the existence of the robotic arm, if the robot is to remain stable on the slope while maintaining the traction force, the uphill angle should have the highest threshold θ max The robot can rely on the obstacle avoidance sensor to determine the approximate range of the slope ahead. When the angle does not exceed θ max When you can move forward, θ max The relevant formula is as follows: When the robot determines that the slope ahead is lower than θ based on the obstacle avoidance sensor max When the robot moves forward, if the prediction is wrong, the slope is higher than θ max When the robot is tilted, it will become unstable and tend to slide down or overturn. At this time, the robot can adaptively activate the left and right mechanical arms to support the rear uphill surface respectively to prevent the robot from rolling and facilitate timely improvement of traction. The force analysis is shown below: F 阻 =Gsinθ+(Gcosθ-F 臂 sinα)m F 牵 +F 臂 cosα+f≥F 阻 =Gsinθ+(Gcosθ-F 臂 sinα)m F 牵 +F 臂 cosα min +f=Gsinθ+Gμcosθ-F 臂 μsinα min Where α is the angle between the robot arm and the slope when completing the support, α min is the minimum angle that the robot arm can support on the slope under the current robot equipment and traction conditions, F 臂 is the supporting force of the robot arm, and f is the upward friction force generated by the robot arm and the slope when supporting.

3. The intelligent fire-fighting robot according to claim 1, characterized in that: The prediction and positioning module detects and locates the fire extinguishing point at the fire scene based on the multimodal fire detection and positioning algorithm. It consists of two parts: Part 1: Select the target object in the thermal map, that is, extract the characteristic information of the flame target part in the thermal map to prepare for the next step of fusion into the target detection model. Specifically: When the target detection model completes the detection of the fire scene, it will generate multiple anchor frames for the image data. The anchor frames include specific flame targets. However, since the three-dimensional thermal map generated by the temperature camera is different from the visual image, the flame target anchor frame in the visual image cannot be directly transferred to the thermal map for flame selection. If you want to convert the two-dimensional anchor frame in the visual image into an anchor frame in the thermal map, you need to first convert the two-dimensional anchor frame in the visual image into coordinates in three-dimensional space, and then project the three-dimensional coordinates onto the two-dimensional plane of the thermal map through the rotation matrix, translation vector and the intrinsic parameter conversion of the temperature camera. In theory, the dimensional change from 2D to 3D requires the addition of depth information, but the visual image captured by the smart camera does not have depth information. Therefore, the feature weighted pixel value of each point is used as pseudo-depth information. The feature weighted pixel value of each point is obtained as pseudo-depth information as follows: The original RGB image has three channels. Therefore, for an image with n pixels, the i-th pixel position will have three pixel values ​​C ranging from 0 to 255. iR 、C iG and C iB Then, the three pixel values ​​at each pixel are averaged to obtain the average pixel value C of this point. imean When n pixels complete the average pixel operation, a grayscale map is obtained. mean ; Select the feature map output by the feature extraction network of the target detection model for convolution operation and convert it to the same dimension as the original image. The formula is as follows: In the formula, Conv() is the convolution operation, Map out is the feature map output by the feature extraction network, Map is the feature map, w ori ×h ori ×1 is the length, width and channel dimension of the original image, where the channel dimension is fixed to 1; When you get Later, the Sigmoid function is used to process each value in the feature map into a decimal between 0 and 1. The converted feature map Map weight Assign weights to Map mean Among them, the result Map after empowerment fusion The semantic information obtained by model feature extraction is integrated to make up for the low sensitivity of pixel values. fusion The value of each pixel in is used as the depth information of the anchor box mapped to the 3D dimension. Based on the obtained depth information, the 2D anchor box coordinates in the visual image can be converted into 3D coordinates. Then, through the relevant operations of coordinate system conversion, the 3D anchor box is mapped to the heat map. After completing the anchor frame mapping on the heat map, the area framed by the heat map anchor frame is cut out and converted into the same area as the Map by convolution operation. out Maps of the same size hot ; Part II: Multimodal fusion of visual images and heat maps, specifically: The anchor frame obtained by the target detection model is used to intercept the frame area in the original video image and convert it into the same frame area as the Map by convolution operation. out Maps of the same size ROI ; Then select Map hot 、Map ROI and Map out The three feature maps are used as basic elements, and the data is screened and processed based on the Hodges-Lehmann principle. The three feature maps are paired in pairs, and the feature values ​​within each pair of feature maps are averaged, globally averaged, and median operations are performed to finally obtain the feature map of multimodal information fusion. multi , the formula is as follows: Map small1 =GAP(Map mean1 ) Map small2 =GAP(Map mean2 ) Map small3 =GAP(Map mean3 ) Map multi (:,:,c) =I small1 (c)×Map mean1 (:,:,c)+I small2 (c)×Map mean2 (:,:,c) +I small3 (c)×Map mean3 (:,:,c) Map mean1 , Map mean2 and Map mean3 They are three different average feature maps after the mean operation; GAP is the global average pooling operation at the feature map spatial level; Map small1 , Map small2 and Map small3 It is the compressed feature map after the three average feature maps are pooled; I is the indicator function, the output is 1 if the condition is met, otherwise it is 0; Median is the median calculation.

4. The intelligent fire-fighting robot according to claim 1, characterized in that: The fire extinguishing module is used to remotely simulate the firefighting behavior of firefighters based on remote motion capture. Through the firefighting posture mapping method, the robot arm is controlled to move the water gun to spray water to extinguish the fire. It consists of two parts: Part 1: Determination of key points of the human body and robotic arm; The key points of the human body are the elbow and shoulder joints of the arms. The swing and rotation of these two joints can fully and specifically demonstrate the direction and displacement of the controlled water pipe. The robotic arm has at least two freely movable joints; Part II: Mapping equations of human posture data; Considering the angle and displacement changes of the human arm joints and the robotic arm joints on the x-axis, y-axis, and z-axis in real space, and adding the influencing parameters of the environment and mechanical equipment, the mapping from human posture to robotic arm posture is not a completely linear relationship. Therefore, the constructed mapping equation integrates nonlinear operations and uses dynamic equations to perform secondary corrections on the calculation results to weaken the influence of physical conditions. Specifically: First, assume that each robotic arm has only two joints, corresponding to the shoulder and elbow joints of the human arm; Assume that the displacements of the operator's elbow joint in the x-axis, y-axis, and z-axis directions are d x d y and d z , the displacement change rates are and The shoulder joint of the human body cannot move independently, so the shoulder joint displacement is not calculated here. The angles of the shoulder joint of the human arm on the x-axis, y-axis and z-axis are θ x1 ,θ y1 and θ z1 , the angle change rates are and The angles of the elbow joint on the x-axis, y-axis, and z-axis are θ x2 ,θ y2 and θ z2 , the angle change rates are and The displacements of the robot arm elbow joint in the x-axis, y-axis and z-axis directions are s x 、s y and s z , the displacement change rates are and The angles of the robot shoulder joint on the x-axis, y-axis, and z-axis are α x1 , α y1 and α z1 , the angle change rates are and The angles of the robot arm elbow joint on the x-axis, y-axis and z-axis are α x2 , α y2 and α z2 , the angle change rates are and The mapping calculation from the human body to the shoulder and elbow joints of the robotic arm is divided into two parts: Part 1: Calculate the joint angle and displacement function to obtain the displacement distance and rotation angle of each joint of the robotic arm; (1) Calculation of angles and displacements of each joint of the robotic arm 1) The function of the robot arm shoulder joint angle in the x-axis direction is as follows: in k is the change angle of the robot shoulder joint in the x-axis direction based on the change of the human shoulder joint angle before the secondary correction; ij (i=1,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; L arm is the overall length of the robotic arm; μ joint is the friction coefficient of the robotic arm shoulder joint; C smoke Smoke density: When the smoke density at the fire scene is too high, it will affect the fire extinguishing judgment, so it should be taken into consideration; 2) The x-axis angle function of the robot arm elbow joint is as follows: in k is the change angle of the robot arm elbow joint in the x-axis direction based on the change of the human arm elbow joint angle before the secondary correction; ij (i=2,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; L forearm is the length of the forearm; c joint Damping coefficient of the robotic arm elbow joint; 3) The function of the robot arm shoulder joint angle in the y-axis direction is as follows: in k is the change angle of the robot shoulder joint in the y-axis direction based on the change of the human arm shoulder joint angle before the secondary correction; ij (i=3,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; m arm is the overall mass of the robotic arm; k joint is the stiffness of the robotic arm shoulder joint; T env The ambient temperature of the scene. When the temperature is too high, the degree of flame burning in the fire scene may be much higher than the predicted location. Therefore, this factor needs to be considered in the scope of the fire extinguishing operation. 4) The angle function of the robot arm elbow joint in the y-axis direction is as follows: in k is the change angle of the robot arm elbow joint in the y-axis direction based on the change angle of the human arm elbow joint before the secondary correction; ij (i=4,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; m forearm is the mass of the robotic arm; k elastic is the elastic coefficient of the elbow joint; 5) The function of the robot arm shoulder joint angle in the z-axis direction is as follows: in k is the change angle of the robot shoulder joint in the z-axis direction based on the change of the human shoulder joint angle before the secondary correction; ij (i=5,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; P cg is the center of gravity of the entire robotic arm; k torque is the motor torque coefficient; V wind The wind speed at the scene. If the scene is not a completely enclosed scene and there is wind flow, the impact of wind speed on the direction of fire extinguishing water flow needs to be considered; 6) The angle function of the robot arm elbow joint in the z-axis direction is as follows: in k is the change angle of the robot arm elbow joint in the z-axis direction based on the change angle of the human arm elbow joint before the secondary correction; ij (i=6,j=1,2,3…7) are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and a large number of experiments; I forearm Moment of inertia of the robotic arm; μ ubrication Lubrication coefficient of the robotic arm elbow joint; V wind The wind speed at the scene. If the scene is not a completely enclosed scene and there is wind flow, the impact of wind speed on the direction of fire extinguishing water flow needs to be considered; 7) The displacement function of the elbow joint of the robot arm in the x-axis direction is as follows: where k ij , i = 7, j = 1, 2, 3…7 are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and experiments; 8) The displacement function of the robot arm elbow joint in the y-axis direction is as follows: where k ij , i = 8, j = 1, 2, 3…7 are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and experiments; 9) The displacement function of the robot arm elbow joint in the z-axis direction is as follows: where k ij , i = 9, j = 1, 2, 3…7 are assignable parameters, which need to be accurately determined based on the actual robot arm structure, sensor data, and experiments; (2) Calculate the expected acceleration The expected acceleration is obtained by taking the second derivative of the uncorrected angles of the shoulder and elbow joints of the robot arm obtained from the above formula and Part 2: Secondary correction of joint angles based on dynamic equations; (1) Solving the dynamic equation to obtain the dynamic torque 1) x-axis direction For the robotic arm shoulder joint: Among them, M x1 (q) is the element of the inertia matrix M(q) corresponding to the x-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G x1 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ x1 The dynamic torque solved for the x-axis direction of the shoulder joint of the robotic arm; For the robotic arm elbow joint: Among them, M x2 (q) is the element of the inertia matrix M(q) corresponding to the x-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G x2 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ x2 The dynamic torque solved for the x-axis direction of the elbow joint of the robot arm; 2) y-axis direction For the robotic arm shoulder joint: Among them, M y1 (q) is the element of the inertia matrix M(q) corresponding to the y-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G y1 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ y1 The dynamic torque solved for the shoulder joint of the robot arm in the y-axis direction; For the robotic arm elbow joint: Among them, M y2 (q) is the element of the inertia matrix M(q) corresponding to the y-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G y2 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ y2 The dynamic torque solved for the y-axis direction of the elbow joint of the robot arm; 3) z-axis direction For the robotic arm shoulder joint: Among them, M z1 (q) is the element of the inertia matrix M(q) corresponding to the z-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G z1 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ z1 The dynamic torque solved for the z-axis direction of the shoulder joint of the robot arm; For the robotic arm elbow joint: Among them, M z2 (q) is the element of the inertia matrix M(q) corresponding to the y-axis direction, and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; is the Coriolis force and centrifugal force matrix The corresponding elements, the values ​​can be obtained by theoretical calculation based on the parameters of the actual robot arm; G z2 (q) is the corresponding element of the gravity vector G(q), and its value can be obtained by theoretical calculation based on the parameters of the actual manipulator; τ z2 The dynamic torque solved for the z-axis direction of the elbow joint of the robot arm; (2) Secondary correction of the shoulder and elbow joint angles of the robotic arm 1) x-axis direction Robotic arm shoulder joint: where k τx1 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments; Robotic arm elbow joint: where k τx2 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments; 2) Y-axis direction: Robotic arm shoulder joint: where k τy1 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments; Robotic arm elbow joint: where k τy2 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments; 3) z-axis direction: Robotic arm shoulder joint: where k τz1 It is an assignable parameter that needs to be accurately determined based on the actual robot arm structure, sensor data, and experiments; Robotic arm elbow joint: where k τz2 It is an assignable parameter and needs to be accurately determined based on the actual robot arm structure, sensor data and experiments.

5. A method for extinguishing a fire using the intelligent firefighting robot according to any one of claims 1 to 4, comprising: S1. A firefighter with on-site firefighting experience is located in a remote control room away from the fire scene. They observe the real-time video feed from the intelligent firefighting robot at the fire scene. The firefighter wears motion capture sensors on the shoulder and elbow joints of both arms and holds a water gun model in both hands to simulate on-site firefighting. S2. The intelligent firefighting robot detects fires based on on-site video footage, obtains reliable flame positioning through a multimodal fire detection and positioning algorithm, and ultimately transmits the detection results to the control room. Firefighters manually judge whether to extinguish the fire based on the results observed in the control room, thereby achieving human-machine integration. Once it is determined to extinguish the fire, the firefighters, relying on their own work experience, swing the firefighting posture of the holding water gun model in the control room. The robot will use the personnel firefighting posture mapping method to transmit the firefighter's arm posture to the mechanical arms on both sides, and manipulate the water gun to accurately extinguish the fire.

Images