An emergency robot intelligent control method and system based on multi-modal perception

Abstract

This invention provides an intelligent control method and system for emergency robots based on multimodal perception, relating to the field of robot intelligent control technology. The method includes: acquiring aerial images of a disaster area collected by a drone, dividing the images into grids, generating a gridded map, and planning initial travel paths for each emergency robot; receiving multimodal data uploaded in real time by each emergency robot as it moves along the initial travel path; mapping temperature values ​​and local wind direction to corresponding grids based on the current location; determining the prevailing wind direction of each grid based on the local wind direction of adjacent grids, and performing diffusion processing on the temperature values ​​of each grid based on the prevailing wind direction to form a global temperature distribution; selecting target grids for each emergency robot to be explored next based on the global temperature distribution and the prevailing wind direction of each grid, dynamically selecting target grids, and controlling the emergency robots to move towards unknown areas with exploration value.

Description

Technical Field

[0001] This invention relates to the field of robot intelligent control technology, specifically to an intelligent control method and system for emergency robots based on multimodal perception. Background Technology

[0002] In the event of disasters such as fires and explosions, quickly ascertaining the distribution of fire sources, the evolution of temperature fields, and the location of trapped personnel within the disaster area is crucial for improving rescue efficiency and reducing casualties. Traditional manual reconnaissance methods, limited by environmental hazards and the lag in information acquisition, are no longer sufficient to meet the timeliness and safety requirements of modern emergency response. Therefore, emergency robot systems have become the core equipment for disaster site reconnaissance.

[0003] Existing emergency robot reconnaissance methods typically employ two modes: First, relying on drones to survey the disaster area from the air and acquire global images using visible light or thermal imaging equipment. However, in a smoke-filled fire environment, smoke strongly scatters and absorbs optical wavelengths, leading to severe image degradation. Details of ground fire sources, structural boundaries, and thermal anomaly areas are difficult to identify, failing to provide high-precision spatial information suitable for path planning. Second, ground robots carrying sensors enter the site to collect data, but due to the lack of a global field of view, they often require traversing the entire area to complete the detection, resulting in low efficiency. Furthermore, static path planning can lead to repeated coverage of already detected areas, wasting limited energy and time resources.

[0004] To address the aforementioned issues, existing technologies have proposed a collaborative detection scheme involving drones and ground robots. For example, the drone first collects images of the disaster area to create a rough map, which then guides the ground robot's detection path. The ground robot moves along the path, collecting environmental data and transmitting it back. However, this method has the following shortcomings: First, the image quality of the drone deteriorates in dense smoke environments, making it impossible to provide the robot with accurate dynamic information such as wind direction and fire spread trends. Second, the robot passively executes the preset path, lacking the ability to actively adjust its detection direction based on changes in the on-site environment (such as wind direction and temperature distribution). Third, it fails to fully utilize the wind direction information inherent in the smoke itself, while wind direction is a crucial basis for determining the direction of fire spread and predicting dangerous areas. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent control method for emergency robots based on multimodal perception. In harsh environments, the emergency robot collects multimodal data such as temperature and images in real time. The direction of smoke drift is extracted from the images as the local wind direction. The data is mapped onto a gridded map. The prevailing wind direction is determined based on the consistency of wind direction between adjacent grids, and temperature diffusion is carried out to form a global temperature distribution. Then, the target grid for detection is dynamically selected, and the emergency robot is controlled to move to unknown areas with detection value.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following solution:

[0007] An intelligent control method for emergency robots based on multimodal perception includes:

[0008] The system acquires aerial images of the disaster area collected by drones, divides the images into grids, generates a gridded map, and plans initial travel paths for each emergency robot.

[0009] Receive multimodal data uploaded in real time by each emergency robot as it moves along its initial path; the multimodal data includes current position, temperature value, and local wind direction;

[0010] The local wind direction is determined by the emergency robot by analyzing the tilt direction of the edge of the smoke area in the image captured by the camera in its forward direction, which is either the left or right direction.

[0011] Map the temperature value and local wind direction to the corresponding grid based on the current location, and update the grid properties;

[0012] The prevailing wind direction of each grid is determined based on the local wind direction of adjacent grids, and the temperature value of each grid is diffused based on the prevailing wind direction to form a global temperature distribution.

[0013] Based on the global temperature distribution and the prevailing wind direction of each grid, the target grid to be explored next is selected for each emergency robot to update the initial travel path and send corresponding control commands to the emergency robot.

[0014] Furthermore, the process of determining the local direction is as follows:

[0015] The emergency robot analyzes continuous images captured by its forward-moving camera, extracts the smoke region from the continuous images, analyzes the tilt trend of the upper edge of the smoke region in the continuous images, and determines whether the smoke drifts to the left or right based on the analysis results, which serves as the local wind direction.

[0016] Furthermore, the process of determining the prevailing wind direction for each grid based on the local wind direction of adjacent grids is as follows:

[0017] For grids with local wind direction records:

[0018] If no wind direction is recorded in any of its adjacent grids, the local wind direction of that grid is used as the prevailing wind direction.

[0019] If the wind direction is the same for adjacent recorded grids, the local wind direction of that grid is used as the dominant wind direction.

[0020] If the wind directions of adjacent recorded grids are inconsistent, one of them will be selected as the prevailing wind direction according to the preset rules.

[0021] Furthermore, in the process of determining the local direction, the tilt change rate of the upper edge of the smoke area in continuous images is calculated, and the confidence level of the wind direction observation is estimated by the tilt change rate; where the smaller the tilt change rate, the higher the confidence level of the wind direction observation.

[0022] Furthermore, the preset rules include at least one of the following:

[0023] Compare the temperature values ​​of this grid with those of adjacent grids with records, and select the local wind direction corresponding to the maximum temperature value as the dominant wind direction of this grid;

[0024] Compare the confidence scores of wind direction observations between this grid and adjacent grids with recorded data, and select the local wind direction corresponding to the highest confidence score as the dominant wind direction of this grid.

[0025] Furthermore, based on the global temperature distribution and the prevailing wind direction of each grid, the process of selecting the next target grid for each emergency robot to be detected is as follows:

[0026] The main detection direction is determined based on the prevailing wind direction of the grid: if the prevailing wind direction is to the left, the main detection direction is to the left of the direction of travel; if the prevailing wind direction is to the right, the main detection direction is to the right of the direction of travel.

[0027] At least one candidate grid is selected from the main detection directions;

[0028] The target grid is selected by combining the temperature value of the candidate grids with whether they have been detected.

[0029] Furthermore, the process of selecting the target grid is as follows:

[0030] If there are unexplored grids among the candidate grids, the nearest unexplored grid will be selected first.

[0031] If all candidate grids have been explored, the grid with the highest temperature value is selected as the target grid.

[0032] Furthermore, the process of diffusion processing of temperature values ​​for each grid based on the prevailing wind direction is as follows:

[0033] For each grid with a prevailing wind direction, its temperature value is gradually transferred along the adjacent grids to which the prevailing wind direction points, and the transfer intensity decreases as the transfer distance increases;

[0034] For a grid covered by transmission from multiple source grids, the temperature values ​​from all transmission sources are taken and a weighted average is calculated. The weighted average value is then used as the temperature estimate for that grid.

[0035] In the weighted average calculation, the weights used are inversely proportional to the transmission distance.

[0036] Furthermore, the control command includes the location of the target grid and the location of the grid with the highest temperature value among the grids adjacent to the target grid, which is used to control the emergency robot to turn the camera to the location of the grid with the highest temperature value for observation while moving towards the target grid.

[0037] An intelligent control system for an emergency robot based on multimodal perception, employing an intelligent control method for the emergency robot based on multimodal perception, includes:

[0038] Initialization module: acquires aerial images of the disaster area collected by drones, divides the images into grids, generates a gridded map, and plans the initial travel paths for each emergency robot;

[0039] Data acquisition module: Receives multimodal data uploaded in real time by each emergency robot as it moves along its initial path; the multimodal data includes current position, temperature value, and local wind direction;

[0040] The local wind direction is determined by the emergency robot by analyzing the tilt direction of the edge of the smoke area in the image captured by the camera in its forward direction, which is either the left or right direction.

[0041] Grid data mapping module: Maps temperature values ​​and local wind direction to the corresponding grid based on the current location and updates grid properties;

[0042] Grid data processing module: Determines the prevailing wind direction of each grid based on the local wind direction of adjacent grids, and performs diffusion processing on the temperature values ​​of each grid based on the prevailing wind direction to form a global temperature distribution;

[0043] Emergency robot control module: Based on the global temperature distribution and the prevailing wind direction of each grid, select the target grid to be detected next for each emergency robot, update the initial travel path, and send corresponding control commands to the emergency robot.

[0044] The beneficial effects of this invention are:

[0045] This invention provides an intelligent control method for emergency robots based on multimodal perception, addressing the problem of existing technologies' inability to effectively detect unknown areas in harsh environments such as dense smoke. Compared to existing technologies, this invention, for the first time, uses smoke as an information carrier. By analyzing the tilting trend of the upper edge of the smoke in the forward-moving camera image, it achieves high-precision local wind direction estimation under conditions without wind speed sensors, completely eliminating the dependence on gas concentration sensors and mechanical anemometers. It maintains stable perception capabilities even in scenarios where traditional sensors fail, such as dense smoke, high temperatures, and strong turbulence. Based on this, after determining the prevailing wind direction based on the consistency of wind direction between adjacent grids, a distance attenuation and weighted averaging algorithm is used to spatially propagate the measured temperature along the prevailing wind direction. This ensures that the temperature estimation value of undetected areas conforms to the physical law of fire spreading with the wind, generating a dynamically consistent thermal distribution map, providing a scientific basis for the robot to select the detection direction.

[0046] Regarding target selection, this invention proposes a dual-level priority decision-making logic: first, undetected grids are prioritized to ensure full coverage without blind spots; when no undetected areas exist, the system switches to the grid with the highest temperature to accurately pinpoint potential fire sources. This mechanism maximizes detection depth while ensuring broad coverage, effectively avoiding resource waste caused by repeated robot detection and significantly improving information acquisition efficiency per unit time. Furthermore, control commands guide the robot to pre-position its camera in the direction of the maximum temperature gradient for preview during movement. Once the preview detects an abnormal temperature increase, the system immediately triggers path replanning, forming a closed-loop control from perception to decision-making to execution, significantly improving the robot's rapid response to environmental changes. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the steps of the intelligent control method for emergency robots in Embodiment 1 of the present invention.

[0048] Figure 2 This is a schematic diagram of the upper edge of the smoke area in Embodiment 1 of the present invention.

[0049] Figure 3 This is a schematic diagram of the initial travel path of the emergency robot in Embodiment 1 of the present invention.

[0050] Figure 4 This is a schematic diagram of the candidate mesh for the emergency robot in Embodiment 1 of the present invention.

[0051] Figure 5 This is a schematic diagram of the updated travel path of the emergency robot in Embodiment 1 of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0054] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0055] Furthermore, for clarity and brevity, descriptions of well-known structures, functions, and configurations may have been omitted. Those skilled in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of this disclosure.

[0056] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0057] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0058] The present invention will now be described in detail with reference to the accompanying drawings and embodiments:

[0059] Example 1

[0060] In this embodiment, an intelligent control method for emergency robots based on multimodal perception is provided, such as... Figure 1 As shown, the intelligent control method for emergency robots includes the following steps:

[0061] Step S1: Obtain aerial images of the disaster area collected by drones, divide the images into grids, generate a gridded map, and plan the initial travel paths for each emergency robot;

[0062] Step S2: Receive multimodal data uploaded in real time by each emergency robot as it moves along the initial path; the multimodal data includes the current position, temperature value, and local wind direction;

[0063] Step S3: Map the temperature value and local wind direction to the grid based on the current location, and update the grid properties;

[0064] Step S4: Determine the prevailing wind direction of each grid based on the local wind direction of adjacent grids, and perform diffusion processing on the temperature values ​​of each grid based on the prevailing wind direction to form a global temperature distribution;

[0065] Step S5: Based on the global temperature distribution and the prevailing wind direction of each grid, select the target grid to be detected next for each emergency robot, update the initial travel path, and send control commands to the emergency robot.

[0066] In step S1, the drone is a platform equipped with a high-definition camera, used to collect aerial images of the disaster area and transmit them to the cloud. The cloud employs an intelligent control method for emergency robots, responsible for functions such as gridded map management, data fusion, determining the prevailing wind direction, temperature diffusion, target grid selection, and multi-robot task allocation. The emergency robot is equipped with a temperature sensor, a visible light camera, a positioning module, and an edge computing unit to collect environmental data in real time and perform preliminary data processing in the edge computing unit, avoiding excessive computational load on the cloud. The cloud platform aggregates grid data from all emergency robots in real time, including the detection status, temperature value, and wind direction of each grid. Based on the detection status of each grid, it can dynamically adjust the travel path and target grid of each robot, preventing multiple emergency robots from repeatedly entering the same grid area. The purpose is to avoid redundant detection through multi-robot collaboration, significantly improving the overall working efficiency of the multi-robot system; through centralized scheduling, optimal coverage of the detection area can be achieved, shortening the time required for comprehensive reconnaissance and providing more timely on-site information for rescue decisions.

[0067] In step S2, the process of determining the local direction is as follows:

[0068] The emergency robot analyzes continuous images captured by its forward-moving camera, extracts the smoke region from the continuous images, analyzes the tilt trend of the upper edge of the smoke region in the continuous images, and determines whether the smoke drifts to the left or right based on the analysis results, which serves as the local wind direction.

[0069] In analyzing the tilt trend of the upper edge of the smoke region in continuous images, the tilt change rate of the upper edge of the smoke region in continuous images can be calculated, and the confidence level of wind direction observation can be estimated by the tilt change rate; among them, the smaller the tilt change rate, the higher the confidence level of wind direction observation.

[0070] Specifically, when the emergency robot reaches grid (5,8) along the initial path, it controls the camera to continuously capture multiple images in the direction of travel. Then, through edge computing, it analyzes the smoke region in the continuously captured images and finds that: the upper edge of the smoke region in the continuous images is tilted to the left (i.e., the left side is higher than the right side), and the rate of change of tilt is stable (i.e., the tilt angle of the left tilt in the continuous images does not differ by more than a threshold). Therefore, it is determined that the local wind direction is to the left. At the same time, the temperature sensor measures the current temperature to be 42℃. The emergency robot packages and uploads the "current position (5,8), temperature 42℃, local wind direction to the left" data to the cloud.

[0071] The process of edge computing unit performing smoke region analysis on an image is as follows: First, the continuous grayscale images captured by the camera are preprocessed. Then, a lightweight semantic segmentation network is used to extract the smoke regions in the image, resulting in... Figure 2 The smoke region segmentation result is shown in (a) above; then, edge detection is performed on the segmented smoke region to extract its contour and locate the upper edge, as shown below. Figure 2 As shown in (b) in the image; finally, by analyzing the tilt trend of the upper edge in the continuous images (tilting to the left or right).

[0072] The edge computing unit performs smoke region analysis on continuously captured images as follows: In a 10-frame image sequence, the upper edge tilt angles are -32°, -31°, -33°, -32°, -31°, -32°, -33°, -32°, -31°, and -32° (negative values ​​indicate a leftward tilt). Then, the tilt angle change rate is calculated: the maximum absolute value of the angle difference between adjacent frames is 2°, and the average change rate is approximately 0.3° / frame. It can be seen that the tilt angle change rate is very small, indicating stable smoke flow; therefore, the confidence level for wind direction observation can be set to 0.9. If, in another scene, the tilt angle fluctuates drastically, such as -35°, -20°, -40°, -25°, and -30°, with an average change rate of 5° / frame, then the confidence level for wind direction observation can be set to 0.4.

[0073] When the cloud receives "current location (5,8), temperature is 42℃, local wind direction is left", it maps the current location (5,8) to the grid position, maps the temperature of 42℃ and local wind direction to the grid, and updates the grid attributes; the grid attributes include the temperature value and local wind direction recorded by the grid.

[0074] In step S4, the process of determining the prevailing wind direction of each grid based on the local wind direction of adjacent grids is as follows:

[0075] For grids with local wind direction records:

[0076] If no wind direction is recorded in any of its adjacent grids, the local wind direction of that grid is used as the prevailing wind direction.

[0077] If the wind direction is the same for adjacent recorded grids, the local wind direction of that grid is used as the dominant wind direction.

[0078] If the wind directions of adjacent recorded grids are inconsistent, one of them will be selected as the prevailing wind direction according to the preset rules.

[0079] Specifically, the process of determining the prevailing wind direction can be divided into the following three scenarios:

[0080] The first scenario is: no records in adjacent grids; for example, if grid A (3,5) has a local wind direction record of "left", and its 8 adjacent grids have no wind direction records, the cloud directly uses "left" as the main wind direction of grid A;

[0081] The second method is: adjacent grids have the same wind direction; for example, grid B (4,5) has a local wind direction record of "left", and its adjacent grids (3,5) and (5,5) also have wind direction records, and both are "left". All adjacent grids with records have the same wind direction, so "left" is taken as the main wind direction of grid B.

[0082] Scenario 3: Wind directions of adjacent grids are inconsistent; grid C(6,8) has a local wind direction record of "left", while its adjacent grid (5,7) is recorded as "right" and (7,9) is recorded as "left". Therefore, the wind directions of adjacent grids are inconsistent. The dominant wind direction can be selected according to the preset rules, which are designed by relevant staff based on their experience and knowledge.

[0083] The preset rules include at least one of the following:

[0084] Compare the temperature values ​​of this grid with those of adjacent grids with recorded values, and select the local wind direction corresponding to the highest temperature value as the main wind direction of this grid; the purpose is to prioritize heat sources, as areas with higher temperatures are likely to be heat sources.

[0085] The confidence levels of wind direction observations for this grid are compared with those of adjacent grids with recorded data. The local wind direction corresponding to the highest confidence level is selected as the prevailing wind direction for this grid. The purpose is to prioritize reliability and use more reliable observation information. The above preset rules can be flexibly selected or combined according to actual conditions to ensure the rationality and accuracy of the prevailing wind direction determination results.

[0086] In step S4, the process of diffusion processing of the temperature values ​​of each grid based on the prevailing wind direction is as follows:

[0087] For each grid with a prevailing wind direction, its temperature value is gradually transferred along the adjacent grids to which the prevailing wind direction points, and the transfer intensity decreases as the transfer distance increases;

[0088] For a grid covered by transmission from multiple source grids, the temperature values ​​from all transmission sources are taken and a weighted average is calculated. The weighted average value is then used as the temperature estimate for that grid.

[0089] In the weighted average calculation, the weights used are inversely proportional to the transmission distance.

[0090] In step S5, the process of selecting the target grid for the next detection step for each emergency robot based on the global temperature distribution and the prevailing wind direction of each grid is as follows:

[0091] The main detection direction is determined based on the prevailing wind direction of the grid: if the prevailing wind direction is to the left, the main detection direction is to the left of the direction of travel; if the prevailing wind direction is to the right, the main detection direction is to the right of the direction of travel.

[0092] At least one candidate grid is selected from the main detection directions;

[0093] The target grid is selected by combining the temperature value of the candidate grids with whether they have been detected.

[0094] The process of selecting the target grid is as follows:

[0095] If there are unexplored grids among the candidate grids, the nearest unexplored grid will be selected first.

[0096] If all candidate grids have been explored, the grid with the highest temperature value is selected as the target grid.

[0097] In step S5, the control command includes the position of the target grid and the position of the grid with the highest temperature value among the grids adjacent to the target grid, which is used to control the emergency robot to turn the camera to the grid with the highest temperature value for observation while moving towards the target grid.

[0098] Specifically, the control commands sent to the robot include not only the target grid location but also the location of the grid with the highest temperature value among the adjacent grids. This allows the robot to turn its camera towards the high-temperature direction for advance observation as it moves toward the target, thereby acquiring visual information about the area in advance. If the preview reveals an area with greater detection value, the plan can be dynamically adjusted.

[0099] In the current grid map, the initial path planned by the emergency robot is as follows: Figure 3 As shown; at this time, the camera equipped on the emergency robot faces the direction of travel, capturing continuous images. After data processing by the edge computing unit, the multimodal data is uploaded to the cloud. The cloud analyzes the uploaded multimodal data to determine the main direction of the grid where the emergency robot is located. If the main direction is to the right, then the main detection direction is to the right of the direction of travel, i.e., downwards, as shown. Figure 4As shown, in the main detection direction, the following three candidate grids can be selected. Combining the temperature value of the candidate grids with whether they have already been detected, the target grid is selected from these. When the target grid is the rightmost grid among the three candidate grids, the initial travel path is updated based on the target grid, resulting in the updated travel path, as shown below. Figure 5 As shown, this method can guide emergency robots to unknown grids that have exploration value. The update of the initial travel path based on the target grid is obtained through path optimization using the initial travel path and the geographical environment. This path optimization is a standard technique used by those skilled in the art and will not be elaborated upon further here.

[0100] Example 2

[0101] An intelligent control system for an emergency robot based on multimodal perception, employing an intelligent control method for the emergency robot based on multimodal perception, includes:

[0102] Initialization module: acquires aerial images of the disaster area collected by drones, divides the images into grids, generates a gridded map, and plans the initial travel paths for each emergency robot;

[0103] Data acquisition module: Receives multimodal data uploaded in real time by each emergency robot as it moves along its initial path; the multimodal data includes current position, temperature value, and local wind direction;

[0104] The local wind direction is determined by the emergency robot by analyzing the tilt direction of the edge of the smoke area in the image captured by the camera in its forward direction, which is either the left or right direction.

[0105] Grid data mapping module: Maps temperature values ​​and local wind direction to the corresponding grid based on the current location and updates grid properties;

[0106] Grid data processing module: Determines the prevailing wind direction of each grid based on the local wind direction of adjacent grids, and performs diffusion processing on the temperature values ​​of each grid based on the prevailing wind direction to form a global temperature distribution;

[0107] Emergency robot control module: Based on the global temperature distribution and the prevailing wind direction of each grid, select the target grid to be detected next for each emergency robot, update the initial travel path, and send corresponding control commands to the emergency robot.

[0108] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Based on the technical essence of the present invention, any simple modifications, equivalent substitutions, and improvements made to the above embodiments within the spirit and principles of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An intelligent control method for emergency robots based on multimodal perception, characterized in that, include: The system acquires aerial images of the disaster area collected by drones, divides the images into grids, generates a gridded map, and plans initial travel paths for each emergency robot. Receive multimodal data uploaded in real time by each emergency robot as it moves along its initial path; the multimodal data includes current position, temperature value, and local wind direction; The local wind direction is determined by the emergency robot by analyzing the tilt direction of the edge of the smoke area in the image captured by the camera in its forward direction, which is either the left or right direction. Map the temperature value and local wind direction to the corresponding grid based on the current location, and update the grid properties; The prevailing wind direction of each grid is determined based on the local wind direction of adjacent grids, and the temperature value of each grid is diffused based on the prevailing wind direction to form a global temperature distribution. The process of diffusion processing of temperature values ​​in each grid based on the prevailing wind direction is as follows: For each grid with a prevailing wind direction, its temperature value is gradually transferred along the adjacent grids to which the prevailing wind direction points, and the transfer intensity decreases as the transfer distance increases; For a grid covered by transmission from multiple source grids, the temperature values ​​from all transmission sources are taken and a weighted average is calculated. The weighted average value is then used as the temperature estimate for that grid. In the weighted average calculation, the weights used are inversely proportional to the transmission distance; Based on the global temperature distribution and the prevailing wind direction of each grid, the target grid to be explored next is selected for each emergency robot to update the initial travel path and send corresponding control commands to the emergency robot.

2. The intelligent control method for an emergency robot based on multimodal perception according to claim 1, characterized in that, The process of determining the local direction is as follows: The emergency robot analyzes continuous images captured by its forward-moving camera, extracts the smoke region from the continuous images, analyzes the tilt trend of the upper edge of the smoke region in the continuous images, and determines whether the smoke drifts to the left or right based on the analysis results, which serves as the local wind direction.

3. The intelligent control method for an emergency robot based on multimodal perception according to claim 2, characterized in that, The process of determining the prevailing wind direction of each grid based on the local wind direction of adjacent grids is as follows: For grids with local wind direction records: If no wind direction is recorded in any of its adjacent grids, the local wind direction of that grid is used as the prevailing wind direction. If the wind direction is the same for adjacent recorded grids, the local wind direction of that grid is used as the dominant wind direction. If the wind directions of adjacent recorded grids are inconsistent, one of them will be selected as the prevailing wind direction according to the preset rules.

4. The intelligent control method for an emergency robot based on multimodal perception according to claim 3, characterized in that, In the process of determining the local direction, the tilt change rate of the upper edge of the smoke area in continuous images is calculated, and the confidence level of the wind direction observation is estimated by the tilt change rate; where the smaller the tilt change rate, the higher the confidence level of the wind direction observation.

5. The intelligent control method for an emergency robot based on multimodal perception according to claim 4, characterized in that, The preset rules include at least one of the following: Compare the temperature values ​​of this grid with those of adjacent grids with records, and select the local wind direction corresponding to the maximum temperature value as the dominant wind direction of this grid; Compare the confidence scores of wind direction observations between this grid and adjacent grids with recorded data, and select the local wind direction corresponding to the highest confidence score as the dominant wind direction of this grid.

6. The intelligent control method for an emergency robot based on multimodal perception according to claim 1, characterized in that, Based on the global temperature distribution and the prevailing wind direction of each grid, the process of selecting the target grid for the next step of detection for each emergency robot is as follows: The main detection direction is determined based on the prevailing wind direction of the grid: if the prevailing wind direction is to the left, the main detection direction is to the left of the direction of travel; if the prevailing wind direction is to the right, the main detection direction is to the right of the direction of travel. At least one candidate grid is selected from the main detection directions; The target grid is selected by combining the temperature value of the candidate grids with whether they have been detected.

7. The intelligent control method for an emergency robot based on multimodal perception according to claim 6, characterized in that, The process of selecting the target grid is as follows: If there are unexplored grids among the candidate grids, the nearest unexplored grid will be selected first. If all candidate grids have been explored, the grid with the highest temperature value is selected as the target grid.

8. The intelligent control method for an emergency robot based on multimodal perception according to claim 1, characterized in that, The control command includes the location of the target grid and the location of the grid with the highest temperature value among the adjacent grids. It is used to control the emergency robot to turn the camera towards the grid with the highest temperature value for observation while moving towards the target grid.

9. An intelligent control system for an emergency robot based on multimodal perception, characterized in that, The application of the multimodal perception-based intelligent control method for emergency robots as described in any one of claims 1-8 includes: Initialization module: acquires aerial images of the disaster area collected by drones, divides the images into grids, generates a gridded map, and plans the initial travel paths for each emergency robot; Data acquisition module: Receives multimodal data uploaded in real time by each emergency robot as it moves along its initial path; the multimodal data includes current position, temperature value, and local wind direction; The local wind direction is determined by the emergency robot by analyzing the tilt direction of the edge of the smoke area in the image captured by the camera in its forward direction, which is either the left or right direction. Grid data mapping module: Maps temperature values ​​and local wind direction to the corresponding grid based on the current location and updates grid properties; Grid data processing module: Determines the prevailing wind direction of each grid based on the local wind direction of adjacent grids, and performs diffusion processing on the temperature values ​​of each grid based on the prevailing wind direction to form a global temperature distribution; Emergency robot control module: Based on the global temperature distribution and the prevailing wind direction of each grid, select the target grid to be detected next for each emergency robot, update the initial travel path, and send corresponding control commands to the emergency robot.

Images