Fire extinguishing robot control method and system based on multi-modal sensing and autonomous navigation

Abstract

The invention discloses a fire extinguishing robot control method and system based on multi-modal sensing and autonomous navigation, and the method comprises the steps: collecting environment temperature data according to a preset frequency through a thermal imager, and triggering early warning when an abnormal threshold value is reached; after early warning, the depth camera obtains a depth image, the depth image is processed through mirror reflection filtering and smoke filtering algorithms and input into a flame form recognition model to judge whether flame features exist or not, and if yes, a fire disaster is judged. And after the fire is confirmed, constructing a real-time model, planning an obstacle avoidance path and guiding the robot to a fire source according to a pre-stored three-dimensional / two-dimensional map in combination with a synchronous positioning and map construction technology. After reaching, adjusting the spraying angle of the fire extinguishing mechanism, and extinguishing fire by water mist / dry powder. During fire extinguishing, the real-time video of the fire scene and the fire source position data are uploaded to a fire-fighting command center in real time. The problems that in the prior art, autonomous fire extinguishing cannot be achieved, environment perception is poor, and early warning and fire extinguishing efficiency is low are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of fire extinguishing, and particularly relates to a control method and system for a fire extinguishing robot based on multi-modal perception and autonomous navigation. Background Art

[0002] In modern society, fires pose a serious threat to people's lives and property safety. With the development of technology, fire-fighting technology is also constantly advancing, but there are still certain defects in the existing fire-fighting related technologies.

[0003] Traditional smoke alarms have played a certain role in fire early warning. However, their functions are limited to simple smoke detection and alarm. Once smoke is detected, only an alarm signal can be issued, but no substantial intervention can be made on the fire. When a fire occurs in an unattended place, such as a home or a warehouse at night, the fire may spread rapidly before the arrival of rescue personnel, causing irreparable losses. This makes it difficult for traditional smoke alarms to meet the needs of comprehensive safety protection in practical applications and cannot effectively reduce the harm caused by fires.

[0004] Although existing fire-fighting robots have improved the fire-fighting efficiency to a certain extent, most fire-fighting robots rely on preset paths to move, which makes them unable to cope when facing complex and changeable dynamic environments. For example, at a fire scene, specular reflection will interfere with the robot's sensors, causing deviations in the environmental information obtained by the robot and resulting in positioning failure; and smoke, as a common companion of fire, will seriously block the robot's line of sight, also causing positioning difficulties. This lack of environmental adaptability makes the fire-fighting robot unable to accurately reach the fire source position at a critical moment, delaying the fire-fighting opportunity and reducing the fire-fighting effect.

[0005] In terms of fire alarm information, existing alarm systems often lack accurate location data. In large buildings, relying only on simple address descriptions, it is difficult for rescue personnel to quickly and accurately determine the specific floor and room location where the fire occurred, nor can they obtain the three-dimensional model information of the area. This results in rescue personnel spending a lot of time looking for the fire source after entering the scene, seriously delaying the rescue, allowing the fire to spread and expand for more time, and increasing the rescue difficulty and losses.

[0006] In summary, there is an urgent need for a new control method and system for a fire extinguishing robot based on multi-modal perception and autonomous navigation to solve the problems of the existing technology being unable to extinguish fires autonomously, having poor environmental perception ability, and low efficiency of early warning and fire extinguishing, so as to improve the efficiency of fire early warning and fire extinguishing and reduce fire losses. Summary of the Invention

[0007] Therefore, the present invention provides a control method and system for a fire extinguishing robot based on multi-modal perception and autonomous navigation to solve the problems of the existing technology being unable to extinguish fires autonomously, having poor environmental perception ability, and low efficiency of early warning and fire extinguishing.

[0008] To achieve the above object, the present invention provides the following technical solutions: A control method for a fire-fighting robot based on multi-modal perception and autonomous navigation, including:

[0009] Continuously collect ambient temperature data using an infrared thermal imager at a preset frequency. When the detected temperature reaches or exceeds the abnormal temperature threshold, trigger a temperature anomaly warning and send a temperature anomaly reminder. After triggering the temperature anomaly warning, continuously obtain the ambient depth image using a depth camera;

[0010] Process the ambient depth image using a specular reflection filtering algorithm to remove specular reflection interference; use a smoke filtering algorithm to process the ambient depth image to reduce smoke interference; input the processed ambient depth image into a flame morphology recognition model, and determine whether there are flame features based on the output result of the flame morphology recognition model. If so, it is determined that a fire has occurred;

[0011] After confirming a fire, based on a pre-stored three-dimensional / two-dimensional map, combined with the simultaneous localization and mapping technology with dynamic update, construct a real-time environment model and plan an obstacle avoidance path to guide the fire-fighting robot to move to the fire source location. When the fire-fighting robot reaches the fire source location, adjust the spraying angle of the fire-fighting actuator and perform fire-fighting operations through a water mist / dry powder spraying device;

[0012] During the fire-fighting process of the fire-fighting robot, obtain the real-time video of the fire scene, the pre-stored user address, the three-dimensional / two-dimensional map, and the internal structure data of the room, and upload them to the fire command center in real time.

[0013] As a preferred solution for the control method of the fire-fighting robot based on multi-modal perception and autonomous navigation, the specular reflection filtering algorithm adopts polarization compensation and multi-spectral fusion, and constructs a Mueller matrix model to decompose the polarization state of specular reflected light to suppress reflection interference. The formula is:

[0014] I corrected = I raw - k·(I parallel - I perpendicular )

[0015] In the formula, I corrected is the reflected light intensity after polarization compensation and multi-spectral fusion processing, I raw is the original reflected light intensity, I parallel and I perpendicular are the light intensities in the parallel and perpendicular polarization directions respectively, and k is the polarization compensation coefficient.

[0016] As an optimal solution of the control method for a fire - fighting robot based on multi - modal perception and autonomous navigation, the smoke filtering algorithm combines RGB - D image segmentation and transmittance model correction, and clears the image in a smoke environment by establishing the mapping relationship C 校 = C0·t(x), where C 校 is the corrected contrast, C0 is the original contrast, t(x)= e -βd(x) is the transmittance model, β is the smoke attenuation coefficient, d(x) is the scene depth, and the real - time value of β is obtained by least - squares fitting.

[0017] As an optimal solution of the control method for a fire - fighting robot based on multi - modal perception and autonomous navigation, the flame shape recognition model is constructed based on a convolutional neural network, and the cross - entropy loss function is used for model training. The expression of the cross - entropy loss function L is:

[0018]

[0019] In the formula, N is the number of samples, C is the number of categories, y ic is the true label that the sample i belongs to the category c, and p ic is the probability that the model predicts that the sample i belongs to the category c; the flame shape recognition model inputs the fused image of the thermal imaging and depth cameras, and outputs the flame position coordinates and morphological feature vectors, which are used to judge the fire source center and the spreading trend.

[0020] As an optimal solution of the control method for a fire - fighting robot based on multi - modal perception and autonomous navigation, during the process of planning the obstacle - avoidance path, based on the A* algorithm combined with the dynamic obstacle - avoidance strategy, an open list and a closed list are constructed. The open list is used to store the nodes to be evaluated, and the closed list is used to store the evaluated nodes; when calculating the node cost, the objective function f(n) used is:

[0021] f(n)= g(n)+ h(n)+ λ·s(n)

[0022] In the formula, g(n) is the actual cost from the starting point to the current node, h(n) is the heuristic cost from the current node to the end point, s(n) is the smoke concentration influence factor, and λ is the weight coefficient.

[0023] As an optimal solution of the control method for a fire - fighting robot based on multi - modal perception and autonomous navigation, during the process of confirming whether a fire has occurred, it also includes:

[0024] Using a microphone to collect the user's voice wake - up command in real time. If the voice wake - up command contains a preset keyword and the voiceprint is verified, after the verification passes, the response action of the fire - fighting robot is triggered.

[0025] As an optimal solution for the control method of a fire - fighting robot based on multi - modal perception and autonomous navigation, during the process of confirming whether a fire has occurred, it further includes:

[0026] Label the kitchen and microwave oven as safe fire source areas, set a spread threshold, and trigger an upgraded alarm when the fire spread speed reaches or exceeds the spread threshold; combine historical fire data and environmental monitoring data to dynamically adjust the scope of the safe fire source area and the spread threshold;

[0027] It further includes:

[0028] After the fire - fighting operation is completed, collect environmental temperature data again through a thermal imager, combine with the environmental depth image obtained by the depth camera, and input it into the flame shape recognition model for judgment; if the output result of the flame shape recognition model shows that there are no flame features and the environmental temperature drops to the preset normal range, it is determined that the fire - fighting is successful, and the feedback information of successful fire - fighting is uploaded to the fire command center and associated devices in real - time;

[0029] The fire - fighting robot is equipped with an automatic water supply and drainage system: the automatic water supply and drainage system includes an automatic water replenishment interface and a drainage device; when the water level in the water storage device of the fire - fighting robot is lower than the preset water level, based on the pre - stored map and the real - time environment model, the robot autonomously navigates to the preset water replenishment point and connects with the water replenishment point through the automatic water replenishment interface for automatic water replenishment; the wastewater generated during the fire - fighting process is discharged through the drainage device.

[0030] The present invention also provides a control system for a fire - fighting robot based on multi - modal perception and autonomous navigation, including:

[0031] A data acquisition module, which is used to continuously collect environmental temperature data at a preset frequency by using a thermal imager, trigger a temperature anomaly warning and send a temperature anomaly reminder when the detected temperature reaches or exceeds the abnormal temperature threshold; after triggering the temperature anomaly warning, continuously obtain the environmental depth image by using the depth camera;

[0032] A data processing module, which is used to process the environmental depth image by using the specular reflection filtering algorithm to remove specular reflection interference; process the environmental depth image by using the smoke filtering algorithm to reduce smoke interference;

[0033] A fire judgment module, which is used to input the processed environmental depth image into the flame shape recognition model, and judge whether there are flame features according to the output result of the flame shape recognition model. If there are, it is determined that a fire has occurred;

[0034] A fire - fighting navigation module, which is used to, after confirming that a fire has occurred, based on the pre - stored three - dimensional / two - dimensional map, combine with the dynamically updated simultaneous localization and mapping technology, construct a real - time environment model and plan an obstacle - avoidance path to guide the fire - fighting robot to move to the fire source location;

[0035] The fire extinguishing execution module is used to adjust the spraying angle of the fire extinguishing execution mechanism and perform fire extinguishing operations through the water mist / dry powder spraying device when the fire extinguishing robot reaches the fire source location;

[0036] The on-site data transmission module is used to obtain the real-time video of the fire scene, the pre-stored user address, the three-dimensional / two-dimensional map, and the internal structure data of the room during the fire extinguishing process of the fire extinguishing robot and upload them to the fire command center in real time.

[0037] As an optimal solution for the fire extinguishing robot control system based on multi-modal perception and autonomous navigation, in the data processing module:

[0038] The specular reflection filtering algorithm adopts polarization light compensation and multi-spectral fusion, and decomposes the polarization state of the specular reflection light through constructing a Mueller matrix model to suppress the reflection interference. The formula is:

[0039] I corrected =I raw -k·(I parallel -I perpendicular )

[0040] In the formula, I corrected is the reflected light intensity after polarization light compensation and multi-spectral fusion processing, I raw is the original reflected light intensity, I parallel and I perpendicular are the light intensities in the parallel and perpendicular polarization directions respectively, and k is the polarization compensation coefficient.

[0041] As an optimal solution for the fire extinguishing robot control system based on multi-modal perception and autonomous navigation, in the data processing module:

[0042] The smoke filtering algorithm combines RGB-D image segmentation and transmittance model correction, and performs image clarity processing in the smoke environment by establishing the mapping relationship C 校 =C0·t(x), where C 校 is the corrected contrast, C0 is the original contrast, t(x)=e -βd(x) is the transmittance model, β is the smoke attenuation coefficient, d(x) is the scene depth, and the real-time value of β is obtained by least squares fitting.

[0043] As an optimal solution for the fire extinguishing robot control system based on multi-modal perception and autonomous navigation, in the fire judgment module:

[0044] The flame morphology recognition model is constructed based on a convolutional neural network, and the cross-entropy loss function is used for model training. The expression of the cross-entropy loss function L is:

[0045]

[0046] Where N is the number of samples, C is the number of categories, and y ic is the true label that sample i belongs to category c, and p ic is the probability that the model predicts that sample i belongs to category c; the flame morphology recognition model inputs the fused image of the thermal imager and the depth camera, and outputs the flame position coordinates and the morphological feature vector, which are used to judge the fire source center and the spread trend.

[0047] As an optimal solution for the fire-fighting robot control system based on multi-modal perception and autonomous navigation, in the fire-fighting navigation module:

[0048] Based on the A* algorithm combined with the dynamic obstacle avoidance strategy, an open list and a closed list are constructed. The open list is used to store the nodes to be evaluated, and the closed list is used to store the evaluated nodes; when calculating the node cost, the objective function f(n) used is:

[0049] f(n) = g(n) + h(n) + λ·s(n)

[0050] Where g(n) is the actual cost from the starting point to the current node, h(n) is the heuristic cost from the current node to the end point, s(n) is the smoke concentration influence factor, and λ is the weight coefficient.

[0051] As an optimal solution for the fire-fighting robot control system based on multi-modal perception and autonomous navigation, it further includes:

[0052] A voice monitoring module, which is used to collect the user's voice wake-up command in real time by using a microphone. If the voice wake-up command contains a preset keyword and performs voiceprint verification, after the verification is passed, it triggers the response action of the fire-fighting robot.

[0053] As an optimal solution for the fire-fighting robot control system based on multi-modal perception and autonomous navigation, it further includes:

[0054] A safe fire source area marking module, which is used to mark the kitchen and the microwave oven as safe fire source areas, set a spread threshold, and trigger an upgraded alarm when the fire spread speed reaches or exceeds the spread threshold; combine historical fire data and environmental monitoring data to dynamically adjust the scope and spread threshold of the safe fire source area;

[0055] It further includes:

[0056] A fire-fighting feedback module, which is used to, after the fire-fighting operation is completed, collect the environmental temperature data again through a thermal imager, combine the environmental depth image obtained by the depth camera, and input it into the flame morphology recognition model for judgment; if the output result of the flame morphology recognition model shows that there are no flame features and the environmental temperature drops to the preset normal range, it is determined that the fire-fighting is successful, and the feedback information of the successful fire-fighting is uploaded to the fire control command center and related devices in real time;

[0057] The fire - fighting robot is equipped with an automatic water supply and drainage system: The automatic water supply and drainage system includes an automatic water replenishment interface and a drainage device; When the water level in the water storage device of the fire - fighting robot is lower than the preset water level, based on the pre - stored map and the real - time environment model, the robot autonomously navigates to the preset water replenishment point and connects with the water replenishment point through the automatic water replenishment interface for automatic water replenishment; The wastewater generated during the fire - fighting process is discharged through the drainage device.

[0058] The beneficial effects of the present invention are as follows:

[0059] First, by adopting the multi - modal perception method of a thermal imager and a depth camera, the false alarm situation easily generated by a single sensor is greatly reduced, and the accuracy and reliability of fire warning are improved. The collected image data is processed through a specular reflection filtering algorithm and a smoke filtering algorithm to effectively remove specular reflection and smoke interference. Then, by using a flame shape recognition model based on deep learning, the flame characteristics can be accurately identified, and it can be accurately judged whether a fire has occurred, avoiding misjudgment caused by environmental interference and providing a reliable basis for taking timely fire - fighting measures.

[0060] Second, based on the pre - stored map and the dynamically updated SLAM technology, an environment model can be quickly constructed and the best path can be planned. In a complex environment, obstacles can be avoided in real time, and the path can be flexibly adjusted, enabling the fire - fighting robot to quickly reach the fire source location. After reaching the fire source location, the fire - fighting execution mechanism can automatically adjust the spraying angle, flow rate, and time according to the actual position and size of the fire source to achieve precise fire - fighting. This intelligent control method can effectively improve the fire - fighting efficiency, reduce the waste of fire extinguishing agent, and reduce the impact on the environment.

[0061] Third, the real - time video of the fire scene and the accurate fire source location data determined based on the three - dimensional map can be uploaded to the fire - fighting command center in real time. Firefighters can comprehensively understand the situation of the fire scene based on this information, formulate more scientific and reasonable rescue plans, and improve the rescue efficiency and coordination. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only exemplary, and for those of ordinary skill in the art, without creative efforts, other implementation drawings can be obtained based on the provided drawings.

[0063] The structures, proportions, sizes, etc. shown in this specification are only used to match the content disclosed in the specification for those familiar with this technology to understand and read, and are not used to limit the implementation conditions of the present invention. Therefore, they do not have substantial technical significance. Any modification of the structure, change in the proportional relationship, or adjustment of the size, without affecting the efficacy that the present invention can produce and the purpose that can be achieved, should still fall within the scope covered by the technical content disclosed in the present invention.

[0064] Figure 1 It is a schematic flowchart of a control method for a fire-fighting robot based on multi-modal perception and autonomous navigation provided by an embodiment of the present invention;

[0065] Figure 2 It is a schematic diagram of the system architecture of a fire-fighting robot based on multi-modal perception and autonomous navigation provided by an embodiment of the present invention. Specific implementation manners

[0066] The following specific embodiments illustrate the implementation manners of the present invention. Those familiar with this technology can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope protected by the present invention.

[0067] Embodiment 1

[0068] See Figure 1 , an embodiment 1 of the present invention provides a control method for a fire-fighting robot based on multi-modal perception and autonomous navigation, including the following steps:

[0069] S1: Continuously collect ambient temperature data using an infrared thermal imager at a preset frequency. When the detected temperature reaches or exceeds the abnormal temperature threshold, trigger a temperature anomaly warning and send a temperature anomaly reminder; after triggering the temperature anomaly warning, continuously obtain ambient depth images using a depth camera.

[0070] The thermal imager can collect the ambient temperature at a preset frequency of 10 times per minute, and can accurately sense the infrared rays emitted by objects and convert them into temperature data. When the ambient temperature reaches or exceeds the set abnormal temperature threshold (such as 110 °C), it indicates that the environment may have an abnormal high temperature and there is a potential risk of fire. At this time, a temperature anomaly warning signal is immediately triggered. After receiving the temperature anomaly warning, the depth camera starts to work. By emitting and receiving specific light rays, based on the time-of-flight principle, it calculates the time difference between the emission and reception of the light rays, and then accurately obtains the distance information of the objects in the surrounding environment, generating an ambient depth image with a resolution of up to 640×480 pixels, which can provide accurate spatial dimension information for subsequent judgment of whether there is a fire and the location of the fire source, etc.

[0071] S2: Use the specular reflection filtering algorithm to process the ambient depth image to remove specular reflection interference; use the smoke filtering algorithm to process the ambient depth image to reduce smoke interference; input the processed ambient depth image into the flame morphology recognition model, and judge whether there are flame characteristics according to the output result of the flame morphology recognition model. If so, it is determined that a fire has occurred.

[0072] At the fire scene, specular reflection and smoke will seriously interfere with the image information obtained by the depth camera and affect the judgment of the fire. The specular reflection filtering algorithm uses polarization light compensation and multi-spectral fusion technology. By constructing a Mueller matrix model, it decomposes the polarization state of the specular reflection light to suppress the reflection interference. Specifically, this algorithm can reduce the interference of specular reflection light to less than 20% of the original. The smoke filtering algorithm combines RGB-D image segmentation and transmittance model correction. First, it uses RGB-D image segmentation technology to accurately separate the smoke area from the image, and then according to the transmittance model, by establishing the mapping relationship between the smoke concentration and the image contrast, it clarifies the image affected by smoke. After being processed by this algorithm, the contrast of the image can be increased by more than 30%. Finally, the processed ambient depth image is input into the flame morphology recognition model constructed based on the convolutional neural network. The flame morphology recognition model is trained with a large number of flame sample images and optimized using the cross-entropy loss function. Its recognition accuracy can reach more than 95%, and it can accurately identify whether there are flame characteristics in the image to determine whether a fire has occurred.

[0073] S3: When it is confirmed that a fire has occurred, based on the pre-stored three-dimensional / two-dimensional map, combined with the simultaneous localization and mapping technology with dynamic update, construct a real-time environment model and plan an obstacle avoidance path to guide the fire-fighting robot to move to the fire source location; when the fire-fighting robot reaches the fire source location, adjust the spraying angle of the fire-fighting actuator and perform fire-fighting operations through the water mist / dry powder spraying device.

[0074] Once a fire is determined to have occurred, the fire-fighting robot needs to quickly and accurately reach the fire source location to extinguish the fire. The fire-fighting robot relies on a pre-stored three-dimensional / two-dimensional map and combines simultaneous localization and mapping (SLAM) technology to construct and update the map of the surrounding environment at a frequency of once per second, determining its own position in the environment and the distribution of surrounding obstacles. Based on the A* algorithm combined with a dynamic obstacle avoidance strategy, it plans an obstacle avoidance path. The A* algorithm finds the optimal path by calculating the cost of each node and constructs open and closed lists to manage the search nodes. The objective function for calculating the node cost comprehensively considers the actual cost from the starting point to the current node, the heuristic cost from the current node to the end point, and the smoke concentration influence factor. The weight coefficient can be dynamically adjusted between 0.5 and 2 according to the actual scenario. During the movement, if it encounters dynamic obstacles, the robot will adjust the path in real time according to the dynamic obstacle avoidance strategy to ensure reaching the fire source location at the fastest speed, and the error of its path planning can be controlled within ±5 centimeters. After reaching the fire source location, the fire-fighting robot will automatically adjust the spraying angle of the fire extinguishing actuator with an accuracy of ±5° according to the actual position and size of the fire, and select a suitable water mist or dry powder spraying device to extinguish the fire. The water mist spraying flow rate can be adjusted between 5 and 20 liters per minute, and the dry powder spraying amount can be adjusted between 1 and 5 kilograms per minute to achieve the best fire extinguishing effect.

[0075] S4: During the fire extinguishing process of the fire-fighting robot, obtain the real-time video of the fire scene, the pre-stored user address, the three-dimensional / two-dimensional map, and the data of the internal structure of the room, and upload them to the fire command center in real time.

[0076] The fire-fighting robot is equipped with a high-definition camera that can take real-time videos of the fire scene at a frame rate of 30 frames per second, and its video resolution is as high as 1920×1080 pixels, which can clearly and smoothly present key situations such as the size of the fire, the spreading direction, and the surrounding environment at the scene. For example, firefighters can intuitively see the height and color of the flame through the video to judge the intensity of the fire; by observing the direction of the smoke, they can understand the spreading direction of the fire; and they can also see clearly whether there are flammable materials and obstacles around, providing an intuitive basis for formulating rescue strategies.

[0077] Meanwhile, the fire-fighting robot stores the user address information in advance. An accurate address can help firefighters quickly locate the specific location of the fire. Especially in some large communities and complex building areas, it can save a lot of time in searching for the fire scene, enabling the rescue force to reach the scene quickly. Using the pre-stored 3D / 2D map, the fire-fighting robot can clearly understand the overall layout of the building. At the same time, combined with its own positioning technology, it can accurately determine the position data of the fire source with an accuracy of ±10 cm. The internal structure data of the room further refines the information, allowing firefighters to understand the furniture arrangement, passage position, etc. in the room. This is of great significance for formulating rescue routes and judging possible trapped positions of people. For example, if there are special structures such as partitions and narrow passages in the room, firefighters can plan the rescue plan in advance to avoid obstacles during the rescue process. These collected real-time videos of the fire scene, pre-stored user addresses, 3D / 2D maps, and internal structure data of the room are uploaded to the fire command center in real time through a high-speed wireless communication module at a transmission rate of not less than 10 Mbps. High-speed and stable transmission can ensure that information reaches the fire command center in a timely and accurate manner, enabling firefighters to adjust the rescue strategy and allocate rescue resources in a timely manner according to the actual situation on the scene, greatly improving the rescue efficiency.

[0078] In this embodiment, in step S2, the specular reflection filtering algorithm uses polarization compensation and multispectral fusion. By constructing a Mueller matrix model, the polarization state of the specular reflection light is decomposed to suppress the reflection interference. The formula is:

[0079] I corrected =I raw -k·(I parallel -I perpendicular )

[0080] In the formula, I corrected is the reflected light intensity after polarization compensation and multispectral fusion processing, I raw is the original reflected light intensity, I parallel and I perpendicular are the light intensities in the parallel and perpendicular polarization directions respectively, and k is the polarization compensation coefficient.

[0081] Specifically, I raw is the original reflected light intensity containing specular reflection light collected by the depth camera. I parallel and I perpendicular are the light intensities in the parallel and perpendicular polarization directions obtained by analyzing the polarization characteristics of light, which can reflect the characteristics and intensity of the specular reflection light. k is the polarization compensation coefficient, and its value range is between 0.1 - 0.5. The specific value is determined through experiments according to the actual scene and device characteristics. Through this formula, the interference light part k·(I raw calculated from the polarization light characteristics is subtracted from the original reflected light intensity Iparallel -I perpendicular )), the reflected light intensity I after removing the specular reflection interference can be obtained corrected , enabling the image to more truly reflect the scene information and providing more reliable data for subsequent accurate fire judgment.

[0082] In this embodiment, in step S2, the smoke filtering algorithm combines RGB-D image segmentation and transmittance model correction, and clears the image in a smoke environment by establishing a mapping relationship C 校 = C0·t(x), where C 校 is the corrected contrast, C0 is the original contrast, t(x) = e -βd(x) is the transmittance model, β is the smoke attenuation coefficient, d(x) is the scene depth, and the real-time value of β is obtained by least squares fitting.

[0083] Specifically, smoke will make the image blurred, reduce the image contrast, and affect the fire judgment. The RGB-D image segmentation technology can use the color (RGB) and depth (D) information of the image to segment the smoke area from the entire image, and the segmentation accuracy can reach more than 90%. The transmittance model t(x) = e -βd(x) describes the attenuation of light in smoke, where β is the smoke attenuation coefficient, which is related to factors such as the concentration and particle size of smoke, and d(x) represents the depth at different positions in the scene. By establishing a mapping relationship C 校 = C0·t(x) between the smoke concentration and the image contrast, the image contrast can be adjusted according to the attenuation degree of light by smoke. Since the smoke attenuation coefficient β varies in different fire scenes, the real-time value of β is calculated by least squares fitting using a large amount of image data, and the calculation error can be controlled within ±5%, making the correction of the image contrast more in line with the actual smoke environment, thereby achieving the purpose of clearing the image in a smoke environment, improving the image quality, and facilitating the subsequent recognition of flame features.

[0084] In this embodiment, in step S2, the flame shape recognition model is constructed based on a convolutional neural network, and the cross-entropy loss function is used for model training. The expression of the cross-entropy loss function is:

[0085]

[0086] In the formula, N is the number of samples, C is the number of categories, y ic is the true label of sample i belonging to category c, and p ic is the probability that the model predicts that sample i belongs to category c; the flame shape recognition model inputs the fused image of the thermal imaging and depth cameras, and outputs the flame position coordinates and the shape feature vector, which are used to judge the fire source center and the spread trend.

[0087] Specifically, the convolutional neural network has powerful image feature extraction capabilities. Through multiple convolutional layers, pooling layers, and fully connected layers, it can automatically learn various features of the flame image. When training the flame morphology recognition model, the cross-entropy loss function is used to measure the difference between the model prediction result and the true label. Here, N represents the number of samples used to train the model, and no less than 10,000 samples are used for training. The more samples, the more comprehensive the features learned by the model. C is the number of categories. In flame recognition, it can usually be divided into two categories: with flame and without flame. y ic is the true label that sample i belongs to category c. If sample i indeed belongs to category c, then y ic = 1, otherwise y ic = 0. p ic is the probability that the model predicts that sample i belongs to category c, and its value range is between 0 and 1. The goal of model training is to minimize the value of the cross-entropy loss function L by continuously adjusting the network parameters, thereby improving the prediction accuracy of the model. In practical applications, the fused image of the thermal imaging and the depth camera is input into the model. The thermal imaging image can provide temperature information, and the depth camera image can provide spatial information. After the two are fused, the model can more comprehensively understand the situation of the flame. The flame position coordinates output by the model can accurately determine the position of the flame in the environment, and the positioning accuracy can reach ±10 cm. The morphological feature vector includes features such as the area, perimeter, aspect ratio, and flicker frequency of the flame. By analyzing these features, the fire source center and the spread trend of the fire can be judged, providing an important basis for fire extinguishing decisions.

[0088] In this embodiment, in step S3, during the process of planning the obstacle avoidance path, based on the A* algorithm combined with the dynamic obstacle avoidance strategy, an open list and a closed list are constructed. The open list is used to store the nodes to be evaluated, and the closed list is used to store the evaluated nodes; when calculating the node cost, the objective function used is:

[0089] f(n) = g(n) + h(n) + λ·s(n)

[0090] In the formula, g(n) is the actual cost from the starting point to the current node, h(n) is the heuristic cost from the current node to the end point, s(n) is the smoke concentration influence factor, and λ is the weight coefficient.

[0091] Specifically, the A* algorithm is a heuristic search algorithm. When planning a path, it manages search nodes by constructing an open list and a closed list. The open list stores those nodes that have been discovered but not yet evaluated, while the closed list stores the nodes that have been evaluated to avoid repeated evaluation. The objective function f(n) for calculating the node cost takes multiple factors into comprehensive consideration. g(n) represents the actual cost from the starting point to the current node n, which includes factors such as the distance the robot moves and the number of turns. For every 1-meter increase in the moving distance, g(n) increases by 1 unit, and for each turn, g(n) increases by 0.5 units. h(n) is the heuristic cost from the current node n to the end point, estimated using the Manhattan distance, and its calculation error can be controlled within ±10%. s(n) is the smoke concentration influence factor. Smoke concentration data is obtained in real time through a smoke sensor, and the smoke concentration is divided into three levels: low, medium, and high, corresponding to s(n) values of 0.5, 1, and 2 respectively. λ is the weight coefficient, whose value range is between 0.5 and 2 and is adjusted according to the degree of influence of smoke on the robot's movement in the actual scenario. Through this objective function, the algorithm can comprehensively consider various factors and find an optimal obstacle avoidance path from the starting point to the fire source location. When encountering dynamic obstacles, combined with the dynamic obstacle avoidance strategy, the robot can adjust the path in real time to ensure safe and fast arrival at the fire source location.

[0092] In a possible embodiment, during the process of confirming whether a fire has occurred, it further includes:

[0093] Using a microphone to collect the user's voice wake-up command in real time. If the voice wake-up command contains a preset keyword and voiceprint verification is performed, after the verification passes, the response action of the fire extinguishing robot is triggered.

[0094] Specifically, in addition to relying on sensor data to judge a fire, a voice interaction function is added. The microphone listens to the surrounding environment sounds in real time at a sampling frequency of 16,000 times per second. When the collected voice wake-up command contains preset keywords such as "There's a fire" or "Extinguish the fire", the system will further perform voiceprint verification. The accuracy rate of voiceprint verification can reach over 98%. Only the commands issued by authorized users who have pre-entered voiceprint information can pass the verification. After the verification passes, the system will immediately trigger the response action of the fire extinguishing robot, which provides a more direct and faster way for users who discover a fire to start the fire extinguishing robot, improving the response speed and practicality of the system.

[0095] In a possible embodiment, during the process of confirming whether a fire has occurred, it further includes:

[0096] Label the kitchen and microwave oven as safe fire source areas, set a spread threshold, and trigger an upgraded alarm when the fire spread speed reaches or exceeds the spread threshold; combine historical fire data and environmental monitoring data to dynamically adjust the scope of the safe fire source area and the spread threshold.

[0097] Specifically, in some scenarios, normal fire sources may appear near kitchens or microwaves, but this doesn't necessarily mean a fire has occurred. To avoid false alarms, these areas are marked as safe fire source areas and a spread threshold is set. The spread threshold is set such that the fire spread area per minute does not exceed 1 square meter. When the fire spread speed reaches or exceeds this threshold, it indicates that the fire may be out of control and an upgraded alarm needs to be triggered to notify higher-level management or the fire department. At the same time, combining historical fire data and environmental monitoring data, the system can dynamically adjust the scope of the safe fire source area and the spread threshold. For example, if the frequency of kitchen fires is high in a certain area, the scope of the kitchen safe fire source area can be reduced by 20%, and the spread threshold can be lowered to no more than 0.5 square meters of fire spread area per minute; if the environmental monitoring data shows that the current environment is dry and flammable, the threshold can also be adjusted accordingly to make the fire judgment more accurate and reasonable.

[0098] In a possible embodiment, it further includes:

[0099] After the fire extinguishing operation is completed, the environmental temperature data is collected again by the thermal imager, combined with the environmental depth image obtained by the depth camera, and input into the flame shape recognition model for judgment; if the output result of the flame shape recognition model shows that there are no flame features and the environmental temperature drops to the preset normal range, it is determined that the fire extinguishing is successful, and the feedback information of successful fire extinguishing is uploaded to the fire command center and associated devices in real time.

[0100] Specifically, the completion of the fire extinguishing operation doesn't mean that the fire hazard has been completely eliminated, so it is necessary to further confirm whether the fire has been truly extinguished. The thermal imager can accurately detect the temperature distribution in the environment. By collecting the environmental temperature data again, it can be judged whether there are still high-temperature areas at the scene, and these high-temperature areas may be unextinguished fire sources or smoldering points. The environmental depth image obtained by the depth camera provides the spatial information of the scene to help identify the locations where fire sources may be hidden, such as behind objects or in gaps. The flame shape recognition model has been trained extensively and has a powerful flame feature recognition ability. Inputting the data from the thermal imager and the depth camera into it can more comprehensively and accurately judge whether there are still flames at the scene. Only when the flame shape recognition model confirms that there are no flame features and at the same time the environmental temperature also drops to the preset normal range (for example, the normal temperature range for a general indoor environment is set to 20°C - 30°C, which can be adjusted according to the actual scenario), can it be determined that the fire has been successfully extinguished. Uploading the feedback information of successful fire extinguishing to the fire command center and associated devices (such as user mobile phones, property management terminals, etc.) can, on the one hand, enable firefighters to timely understand the fire extinguishing situation and reasonably arrange subsequent work, such as whether further on-site inspections are needed and whether rescue forces can be withdrawn; on the other hand, it can also reassure the relevant personnel, and at the same time facilitate the recording and subsequent analysis of the fire incident.

[0101] In a possible embodiment, the fire extinguishing robot is equipped with an automatic water supply and drainage system: the automatic water supply and drainage system includes an automatic water replenishment interface and a drainage device; when the water level in the water storage device of the fire extinguishing robot is lower than the preset water level, based on the pre-stored map and the real-time environment model, the robot autonomously navigates to the preset water replenishment point and connects to the water replenishment point through the automatic water replenishment interface for automatic water replenishment; the wastewater generated during the fire extinguishing process is discharged through the drainage device.

[0102] Specifically, during the fire extinguishing process, a preset water level is set (for example, 20% of the capacity of the water storage device is set as the preset water level, and the specific value can be adjusted according to the fire extinguishing ability and actual needs of the robot). When the water level is lower than this value, it means that the fire extinguishing endurance ability of the robot is threatened and water needs to be replenished in a timely manner. Relying on the pre-stored map and the real-time environment model, the fire extinguishing robot can clearly know its own position and the surrounding environment, including the position of the preset water replenishment point. Through navigation, a path can be planned to avoid obstacles and reach the water replenishment point efficiently. After reaching the water replenishment point, the automatic water replenishment interface will automatically connect to the water replenishment point to achieve the automatic water replenishment function, ensuring that the robot can continuously obtain sufficient fire extinguishing water. At the same time, wastewater is generated during the fire extinguishing process. If these wastewaters are not discharged in a timely manner, it will increase the load of the robot and affect its mobility and fire extinguishing efficiency. The drainage device is responsible for discharging these wastewaters to keep the robot in the best working state. The drainage device can automatically adjust the drainage flow according to the actual situation at the fire extinguishing site, such as the speed of wastewater generation and the surrounding drainage conditions, to ensure an efficient and smooth drainage process.

[0103] The application scenarios of the present invention are as follows:

[0104] Industrial scenarios

[0105] Factory workshops: There are usually a large number of mechanical equipment, electrical circuits and flammable and explosive items in factory workshops, with relatively large fire hazards. Once a fire occurs, it may cause serious consequences such as equipment damage, production stagnation and casualties. Through the present invention, the workshop can be monitored continuously for 24 hours, and the environmental changes can be sensed in real time through multi-modal sensors such as thermal imagers, depth cameras and microphones. When abnormal temperatures, flame characteristics are detected or a voice wake-up command is received, it can quickly judge whether a fire has occurred. Using the autonomous navigation function, the robot can quickly avoid obstacles in the complex workshop environment, such as piled-up raw materials, equipment, etc., and accurately reach the fire source location for fire extinguishing. At the same time, the real-time video of the fire scene and the fire source location data are uploaded to the fire command center, facilitating remote command and resource allocation by firefighters and minimizing fire losses to the greatest extent.

[0106] Commercial scenarios

[0107] Malls and supermarkets: Malls and supermarkets are densely populated with people and piled up with goods. Once a fire breaks out, it is easy to cause panic and congestion among people, increasing the difficulty of rescue. The present invention can be installed on each floor and area of the mall to monitor the environment in real time. In the initial stage of a fire, the robot can respond quickly, plan the best path through autonomous navigation, avoid crowds and obstacles, and reach the fire source to extinguish the fire. At the same time, the fire scene information is uploaded to the mall's monitoring center and the fire department in real time to help them understand the fire situation in a timely manner and formulate rescue plans. In addition, the voice interaction function of the robot can provide evacuation guidance for customers during a fire, stabilize the emotions of customers, and reduce casualties.

[0108] Public facility scenarios

[0109] Subway stations and railway stations: Subway stations and railway stations are transportation hubs with frequent personnel flow. Once a fire breaks out, it may lead to traffic paralysis and a large number of casualties. Through the present invention, it can be deployed in areas such as the waiting hall, platform, and passage of the station to monitor the environment in real time. When a fire is detected, the robot can respond quickly, use its autonomous navigation function to open a passage through the crowd, reach the fire source to extinguish the fire. At the same time, the fire scene information is uploaded to the station's command center and the fire department in real time to coordinate the evacuation of passengers and carry out rescue work. The robot can also be linked with other safety devices in the station, such as fire sprinkler systems and smoke alarms, to improve the ability to respond to fires.

[0110] Home scenarios

[0111] Ordinary residences: In ordinary residences, areas such as the kitchen and living room are high-risk areas for fires. Through the present invention, it can reuse the functions of daily service devices such as floor-sweeping robots in the home to reduce equipment costs. During non-fire periods, the robot can be used as a floor-sweeping robot for daily cleaning work; when a fire is detected, it quickly switches to the fire extinguishing mode. Through multi-modal perception technology, the robot can accurately judge the location and size of the fire, and select an appropriate fire extinguishing method to extinguish the fire. At the same time, the fire information is sent to the owner's mobile phone in real time, allowing the owner to understand the situation at home in a timely manner and remotely control the robot to carry out fire extinguishing operations.

[0112] Embodiment 2

[0113] See Figure 2 , Embodiment 2 of the present invention also provides a fire extinguishing robot control system based on multi-modal perception and autonomous navigation, including:

[0114] A data acquisition module 001, which is used to continuously collect ambient temperature data using an infrared thermal imager at a preset frequency. When it detects that the temperature reaches or exceeds the abnormal temperature threshold, it triggers an abnormal temperature warning and sends an abnormal temperature reminder; after triggering the abnormal temperature warning, it continuously obtains the ambient depth image using a depth camera;

[0115] The data processing module 002 is configured to process the environmental depth image by using a specular reflection filtering algorithm to remove specular reflection interference; and process the environmental depth image by using a smoke filtering algorithm to reduce smoke interference.

[0116] The fire judgment module 003 is configured to input the processed environmental depth image into a flame shape recognition model, and judge whether there are flame features according to the output result of the flame shape recognition model. If so, it is determined that a fire has occurred.

[0117] The fire extinguishing navigation module 004 is configured to, when it is confirmed that a fire has occurred, based on a pre-stored three-dimensional / two-dimensional map, combine the simultaneous localization and mapping technology with dynamic update to construct a real-time environmental model and plan an obstacle avoidance path, and guide the fire extinguishing robot to move to the fire source location.

[0118] The fire extinguishing execution module 005 is configured to, when the fire extinguishing robot reaches the fire source location, adjust the spraying angle of the fire extinguishing execution mechanism and perform a fire extinguishing operation through a water mist / dry powder spraying device.

[0119] The on-site data transmission module 006 is configured to, during the fire extinguishing process of the fire extinguishing robot, acquire real-time video of the fire scene, pre-stored user address, three-dimensional / two-dimensional map, and room internal structure data, and upload them to the fire command center in real time.

[0120] In this embodiment, in the data processing module 002:

[0121] The specular reflection filtering algorithm adopts polarization light compensation and multi-spectral fusion, and decomposes the polarization state of the specular reflection light by constructing a Mueller matrix model to suppress the reflection interference. The formula is:

[0122] I corrected = I raw - k·(I parallel - I perpendicular )

[0123] In the formula, I corrected is the reflected light intensity after polarization light compensation and multi-spectral fusion processing, I raw is the original reflected light intensity, I parallel and I perpendicular are the light intensities in the parallel and perpendicular polarization directions respectively, and k is the polarization compensation coefficient.

[0124] In this embodiment, in the data processing module 002:

[0125] The smoke filtering algorithm combines RGB-D image segmentation and transmittance model correction, and establishes a mapping relationship C between the smoke concentration and the image contrast 校= C0·t(x) for image enhancement processing in a smoky environment, where C 校 is the corrected contrast, C0 is the original contrast, and t(x) = e -βd(x) is the transmittance model, β is the smoke attenuation coefficient, d(x) is the scene depth, and the real-time value of β is obtained by least squares fitting.

[0126] In this embodiment, in the fire judgment module 003:

[0127] The flame morphology recognition model is constructed based on a convolutional neural network, and the cross-entropy loss function is used for model training. The expression of the cross-entropy loss function L is:

[0128]

[0129] In the formula, N is the number of samples, C is the number of categories, y ic is the true label of sample i belonging to category c, and p ic is the probability that the model predicts that sample i belongs to category c; the flame morphology recognition model inputs the fused image of the thermal imaging and depth cameras, and outputs the flame position coordinates and morphological feature vectors for judging the fire source center and the spread trend.

[0130] In this embodiment, in the fire extinguishing navigation module 004:

[0131] Based on the A* algorithm combined with the dynamic obstacle avoidance strategy, an open list and a closed list are constructed. The open list is used to store the nodes to be evaluated, and the closed list is used to store the evaluated nodes; when calculating the node cost, the objective function f(n) used is:

[0132] f(n) = g(n) + h(n) + λ·s(n)

[0133] In the formula, g(n) is the actual cost from the starting point to the current node, h(n) is the heuristic cost from the current node to the end point, s(n) is the smoke concentration influence factor, and λ is the weight coefficient.

[0134] In a possible embodiment, it further includes:

[0135] A voice monitoring module 007, which is used to collect the user's voice wake-up command in real time by using a microphone. If the voice wake-up command contains a preset keyword and voiceprint verification is performed, after the verification is passed, the response action of the fire extinguishing robot is triggered.

[0136] In a possible embodiment, it further includes:

[0137] The safe fire source area marking module 008 is used to mark the kitchen and microwave oven as safe fire source areas, set a spread threshold, and trigger an upgraded alarm when the fire spread speed reaches or exceeds the spread threshold; combining historical fire data and environmental monitoring data, dynamically adjust the scope of the safe fire source area and the spread threshold.

[0138] In a possible embodiment, it further includes:

[0139] The fire extinguishing feedback module 009 is used to, after the fire extinguishing operation is completed, collect environmental temperature data again through a thermal imager, combine the environmental depth image obtained by the depth camera, and input it into the flame morphology recognition model for judgment; if the output result of the flame morphology recognition model shows that there are no flame features and the environmental temperature drops to the preset normal range, it is determined that the fire extinguishing is successful, and the feedback information of successful fire extinguishing is uploaded to the fire command center and associated devices in real time.

[0140] In a possible embodiment, the fire extinguishing robot is equipped with an automatic water supply and drainage system: the automatic water supply and drainage system includes an automatic water replenishment interface and a drainage device; when the water level in the water storage device of the fire extinguishing robot is lower than the preset water level, based on the pre-stored map and the real-time environment model, the robot autonomously navigates to the preset water replenishment point and connects to the water replenishment point through the automatic water replenishment interface for automatic water replenishment; the wastewater generated during the fire extinguishing process is discharged through the drainage device.

[0141] It should be noted that for the information interaction, execution process, etc. between the above system modules, since they are based on the same concept as the method embodiment in Embodiment 1 of this application, the technical effects brought by them are the same as those of the method embodiment of this application. For the specific content, reference can be made to the description in the method embodiment shown above in this application, and details will not be repeated here.

[0142] Embodiment 3

[0143] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which program codes for the control method of a fire extinguishing robot based on multi-modal perception and autonomous navigation are stored, and the program codes include instructions for executing the control method of a fire extinguishing robot based on multi-modal perception and autonomous navigation in Embodiment 1 or any possible implementation manner thereof.

[0144] The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center integrating one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive (SolidState Disk, SSD)), etc.

[0145] Embodiment 4

[0146] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0147] The processor and the memory complete communication with each other through a bus; the memory stores program instructions executable by the processor, and the processor can execute the control method of the fire extinguishing robot based on multi-modal perception and autonomous navigation according to Embodiment 1 or any possible implementation manner thereof by invoking the program instructions.

[0148] Specifically, the processor can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading software code stored in the memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0149] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.).

[0150] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by program code executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order than here, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to implement. In this way, the present invention is not limited to any specific combination of hardware and software.

[0151] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it on the basis of the present invention, which will be obvious to those skilled in the art. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of protection required by the present invention.

Claims

1. A control method for a fire-fighting robot based on multimodal perception and autonomous navigation, characterized in that, Including: Continuously collect ambient temperature data using an infrared thermal imager at a preset frequency. When the detected temperature reaches or exceeds the abnormal temperature threshold, trigger a temperature anomaly warning and send a temperature anomaly reminder. After triggering the temperature anomaly warning, continuously obtain the ambient depth image using a depth camera. Process the ambient depth image using a specular reflection filtering algorithm to remove specular reflection interference. Process the ambient depth image using a smoke filtering algorithm to reduce smoke interference. Input the processed ambient depth image into a flame morphology recognition model, and determine whether there are flame features based on the output result of the flame morphology recognition model. If so, it is determined that a fire has occurred. After confirming that a fire has occurred, based on the pre-stored 3D / 2D map and combined with the simultaneous localization and mapping technology with dynamic update, construct a real-time environment model and plan an obstacle avoidance path to guide the fire-fighting robot to move to the fire source location. When the fire-fighting robot reaches the fire source location, adjust the spraying angle of the fire-fighting actuator and perform fire-fighting operations through a water mist / dry powder spraying device. During the fire-fighting process of the fire-fighting robot, obtain the real-time video of the fire scene, the pre-stored user address, the 3D / 2D map, and the internal structure data of the room, and upload them to the fire command center in real time.

2. The control method of the fire-fighting robot based on multi-modal perception and autonomous navigation according to claim 1, wherein, The specular reflection filtering algorithm adopts polarization compensation and multispectral fusion, and constructs a Mueller matrix model to decompose the polarization state of the specular reflection light to suppress the reflection interference. The formula is: I corrected = I raw - k·(I parallel - I perpendicular ) Where, I corrected is the reflected light intensity after polarization light compensation and multispectral fusion processing, I raw is the original reflected light intensity, I parallel and I perpendicular are the light intensities in the parallel and perpendicular polarization directions respectively, and k is the polarization compensation coefficient.

3. The control method of the fire-fighting robot based on multi-modal perception and autonomous navigation according to claim 1, characterized in that, The smoke filtering algorithm combines RGB-D image segmentation and transmittance model correction to clarify the image in a smoke environment by establishing a mapping relationship C between the smoke concentration and the image contrast 校 = C0·t(x), where C 校 is the corrected contrast, C0 is the original contrast, t(x) = e -βd(x) is the transmittance model, β is the smoke attenuation coefficient, d(x) is the scene depth, and the real-time value of β is obtained by least squares fitting.

4. The control method of the fire-fighting robot based on multi-modal perception and autonomous navigation according to claim 1, characterized in that, The flame morphology recognition model is constructed based on a convolutional neural network, and the cross-entropy loss function is used for model training. The expression of the cross-entropy loss function L is: where N is the number of samples, C is the number of classes, and y ic is the true label that sample i belongs to class c, and p ic is the probability that the model predicts that sample i belongs to class c; the flame morphology recognition model inputs the fused image of the thermal imaging and depth camera, outputs the flame position coordinates and the morphology feature vector, and is used to judge the fire source center and the spread trend.

5. The control method of the fire extinguishing robot based on multi-modal perception and autonomous navigation according to claim 1, wherein, During the process of planning the obstacle avoidance path, based on the A* algorithm combined with the dynamic obstacle avoidance strategy, construct an open list and a closed list. Store the nodes to be evaluated through the open list, and store the evaluated nodes through the closed list. When calculating the node cost, the objective function f(n) used is: f(n) = g(n) + h(n) + λ·s(n) In the formula, g(n) is the actual cost from the starting point to the current node, h(n) is the heuristic cost from the current node to the end point, s(n) is the smoke concentration influence factor, and λ is the weight coefficient.

6. The control method of the fire-fighting robot based on multi-modal perception and autonomous navigation according to claim 1, characterized in that During the process of confirming whether a fire has occurred, it also includes: Continuously collect the voice wake-up command of the user using a microphone. If the voice wake-up command contains a preset keyword and voiceprint verification is performed, after the verification is passed, trigger the response action of the fire-fighting robot.

7. The control method of the fire extinguishing robot based on multimodal perception and autonomous navigation according to claim 1, wherein During the process of confirming whether a fire has occurred, it also includes: Label the kitchen and microwave oven as safe fire source areas, set a spread threshold, and trigger an upgraded alarm when the fire spread speed reaches or exceeds the spread threshold. Combine historical fire data and environmental monitoring data to dynamically adjust the scope of the safe fire source area and the spread threshold. It also includes: After the fire-fighting operation is completed, collect the ambient temperature data again using an infrared thermal imager, combine the ambient depth image obtained by the depth camera, and input it into the flame morphology recognition model for judgment. If the output result of the flame morphology recognition model shows that there are no flame features and the ambient temperature drops to the preset normal range, it is determined that the fire-fighting is successful, and the feedback information of the successful fire-fighting is uploaded to the fire command center and associated devices in real time. The fire-fighting robot is equipped with an automatic water supply and drainage system: The automatic water supply and drainage system includes an automatic water replenishment interface and a drainage device; when the water level in the water storage device of the fire-fighting robot is lower than the preset water level, based on the pre-stored map and the real-time environment model, the robot autonomously navigates to the preset water replenishment point and connects to the water replenishment point through the automatic water replenishment interface for automatic water replenishment; the wastewater generated during the fire-fighting process is discharged through the drainage device.

8. A fire-fighting robot control system based on multi-modal perception and autonomous navigation, characterized in that, It includes: A data acquisition module, which is used to continuously collect ambient temperature data using an infrared thermal imager at a preset frequency. When the detected temperature reaches or exceeds the abnormal temperature threshold, it triggers a temperature anomaly warning and sends a temperature anomaly reminder. After triggering the temperature anomaly warning, use a depth camera to continuously obtain the ambient depth image. A data processing module, which is used to process the ambient depth image using a specular reflection filtering algorithm to remove specular reflection interference. Use a smoke filtering algorithm to process the ambient depth image to reduce smoke interference. A fire judgment module, which is used to input the processed ambient depth image into a flame shape recognition model, and judge whether there are flame characteristics based on the output result of the flame shape recognition model. If so, it is determined that a fire has occurred. A fire-fighting navigation module, which is used to, after confirming a fire, based on the pre-stored three-dimensional / two-dimensional map, combined with the dynamically updated simultaneous localization and mapping technology, construct a real-time environment model and plan an obstacle avoidance path to guide the fire-fighting robot to move to the fire source location. A fire-fighting execution module, which is used to, when the fire-fighting robot reaches the fire source location, adjust the spraying angle of the fire-fighting execution mechanism and perform fire-fighting operations through a water mist / dry powder spraying device. A field data transmission module, which is used to obtain the real-time video of the fire scene, the pre-stored user address, the three-dimensional / two-dimensional map, and the internal structure data of the room during the fire-fighting process of the fire-fighting robot, and upload them to the fire command center in real time.

9. The fire-fighting robot control system based on multi-modal perception and autonomous navigation according to claim 8, characterized in that, In the data processing module: The specular reflection filtering algorithm uses polarization compensation and multi-spectral fusion, and constructs a Mueller matrix model to decompose the polarization state of specular reflection light to suppress reflection interference. The formula is: I corrected = I raw - k·(I parallel - I perpendicular ) where I corrected is the reflected light intensity after polarization light compensation and multispectral fusion processing, I raw is the original reflected light intensity, I parallel and I perpendicular are the light intensities in the parallel and perpendicular polarization directions respectively, and k is the polarization compensation coefficient; In the data processing module: The described smoke filtering algorithm combines RGB-D image segmentation and transmittance model correction to clarify the image in a smoke environment by establishing a mapping relationship C between the smoke concentration and the image contrast 校 = C0·t(x), where C 校 is the corrected contrast, C0 is the original contrast, t(x) = e -βd(x) is the transmittance model, β is the smoke attenuation coefficient, d(x) is the scene depth, and the real-time value of β is obtained by least squares fitting; In the fire judgment module: The flame shape recognition model is constructed based on a convolutional neural network, and a cross-entropy loss function is used for model training. The expression of the cross-entropy loss function L is: where N is the number of samples, C is the number of categories, and y ic is the true label that sample i belongs to category c, and p ic is the probability that the model predicts that sample i belongs to category c; the flame morphology recognition model inputs the fused image of the thermal imaging and depth camera, outputs the flame position coordinates and the morphological feature vector, and is used to judge the fire source center and the spreading trend; In the fire-fighting navigation module: Based on the A* algorithm combined with a dynamic obstacle avoidance strategy, construct an open list and a closed list. Store the nodes to be evaluated through the open list, and store the evaluated nodes through the closed list; when calculating the node cost, the objective function f(n) used is: f(n) = g(n) + h(n) + λ·s(n) In the formula, g(n) is the actual cost from the starting point to the current node, h(n) is the heuristic cost from the current node to the end point, s(n) is the smoke concentration influence factor, and λ is the weight coefficient.

10. The fire-fighting robot control system based on multi-modal perception and autonomous navigation according to claim 8, characterized in that, It also includes: A voice monitoring module, which is used to use a microphone to continuously collect the user's voice wake-up command in real time. If the voice wake-up command contains a preset keyword and performs voiceprint verification, after the verification is passed, it triggers the response action of the fire-fighting robot. A safe fire source area marking module, which is used to mark the kitchen and microwave oven as safe fire source areas, set a spread threshold, and trigger an upgrade alarm when the fire spread speed reaches or exceeds the spread threshold; Combined with historical fire data and environmental monitoring data, dynamically adjust the scope and spread threshold of the safe fire source area; It also includes: A fire extinguishing feedback module, which is used to, after the fire extinguishing operation is completed, collect environmental temperature data again through a thermal imager, combine the environmental depth image obtained by the depth camera, and input it into the flame shape recognition model for judgment; if the output result of the flame shape recognition model shows that there are no flame characteristics and the environmental temperature drops to the preset normal range, it is determined that the fire extinguishing is successful, and the feedback information of the successful fire extinguishing is uploaded to the fire command center and associated devices in real time; The fire extinguishing robot is equipped with an automatic water supply and drainage system: the automatic water supply and drainage system includes an automatic water replenishment interface and a drainage device; when the water level in the water storage device of the fire extinguishing robot is lower than the preset water level, based on the pre-stored map and the real-time environment model, the robot autonomously navigates to the preset water replenishment point and connects with the water replenishment point through the automatic water replenishment interface for automatic water replenishment; the wastewater generated during the fire extinguishing process is discharged through the drainage device.

Images