Motorized unmanned fire fighting vehicle linkage multi-height precise intelligent fire extinguishing operation system

Abstract

The application discloses a motorized unmanned fire-fighting vehicle linkage multi-height precise intelligent fire extinguishing operation system and relates to the technical field of intelligent fire-fighting operation. The system comprises a multi-source sensing module which collects images with depth information, toxic and harmful gas concentration and operation posture pressure data of a fire site. A situation analysis module constructs a three-dimensional thermal distribution model of the fire site by improving a stereo vision matching algorithm, identifies three-dimensional coordinates and scale grades of a fire source, and generates a dynamic diffusion risk map by analyzing the gas concentration data in space and time. A linkage decision module inputs situation data into a man-vehicle linkage decision model to generate a multi-height fire source cooperative fire extinguishing operation instruction set. A closed-loop scheduling module issues control instructions through an adaptive operation scheduler based on the instruction set to drive the unmanned fire-fighting vehicle, the high-altitude injection system and the fire unmanned aerial vehicle to cooperatively operate and complete closed-loop control by relying on real-time updating data.

Description

Technical Field

[0001] This invention belongs to the field of intelligent firefighting operation technology, specifically a multi-height precision intelligent fire extinguishing operation system that links mobile unmanned firefighting vehicles and personnel. Background Technology

[0002] Fire scenes are complex and ever-changing. Traditional emergency response methods rely on fragmented on-site perception channels, making it difficult to simultaneously collect in-depth information such as fire scene images, toxic and hazardous gas concentrations, and the posture and pressure of operational equipment. The lack of comprehensive data collection dimensions prevents the creation of a complete fire scene environmental dataset, forcing reliance on localized, superficial information for disaster assessment. Fire scene information processing methods are rudimentary, failing to utilize stereoscopic vision matching for refined image processing, hindering the construction of three-dimensional thermal distribution models, and preventing accurate location of fire source coordinates and classification of fire scale levels. Furthermore, the lack of spatiotemporal correlation analysis mechanisms for toxic and hazardous gas data prevents the generation of dynamic diffusion risk maps that can predict evolution trends.

[0003] During fire suppression, various types of fire situation information cannot be centrally integrated, and there is no well-established model architecture to coordinate multi-dimensional information input. Furthermore, standardized collaborative operation instructions cannot be generated for fire sources at different altitudes. Unmanned firefighting equipment often operates on a standalone basis, making unified scheduling across different devices impossible.

[0004] The lack of a dedicated adaptive scheduling architecture in routine operations makes it difficult to generate a coherent and orderly sequence of equipment control commands, hindering effective coordination between unmanned fire trucks, aerial ladder systems, and firefighting drones. The absence of real-time data feedback and dynamic update mechanisms during operation execution prevents the construction of a complete closed-loop control system, making it difficult to meet the actual operational requirements of simultaneous and precise firefighting at multiple heights in complex fire scenes. Summary of the Invention

[0005] This invention aims to solve at least one of the technical problems existing in the prior art;

[0006] Therefore, this invention proposes a mobile unmanned firefighting vehicle-human linkage multi-height precision intelligent fire extinguishing operation system, comprising: The multi-source sensing module acquires and aggregates images of the fire scene containing depth information, data on the concentration of toxic and harmful gases, and real-time operational posture and pressure data at the fire-fighting operation site. The situation analysis module performs an improved stereo vision matching algorithm on the fire scene image containing depth information, constructs a three-dimensional thermal distribution model of the fire scene and identifies the three-dimensional coordinates and scale level of key fire sources, performs spatiotemporal correlation analysis on the concentration data of toxic and harmful gases, and generates a dynamic diffusion risk map. The linkage decision module inputs the three-dimensional thermal distribution model of the fire site, the three-dimensional coordinates and scale level of the key fire source points, the dynamic diffusion risk map, and the real-time operation posture and pressure data into the human-vehicle linkage decision model to generate a set of collaborative fire extinguishing operation instructions for fire sources at different heights. The closed-loop scheduling module, based on the collaborative firefighting operation instruction set, generates and issues specific equipment control instruction sequences through the adaptive operation scheduler, drives unmanned fire trucks, aerial ladder spray systems and firefighting drones to perform collaborative actions, and forms closed-loop control based on feedback real-time and updated data.

[0007] Furthermore, an improved stereo vision matching algorithm is applied to the fire scene image containing depth information to construct a three-dimensional thermal distribution model of the fire scene and identify the three-dimensional coordinates and scale level of key fire sources, including: The improved stereo vision matching algorithm optimizes the matching cost calculation process based on the non-rigid and dynamic characteristics of flame texture; Correction and alignment are performed on multiple fire scene images containing depth information collected by the reconnaissance drone in different poses; Using the improved stereo vision matching algorithm, the disparity of each pixel is calculated among multiple images. When calculating the matching cost, the improved stereo vision matching algorithm integrates the cost based on gradient information, the cost based on Census transform, and the cost based on the color invariance feature of the flame region, and assigns adaptive weights to the dynamic flame region. Based on the camera intrinsic parameters and the parallax, a dense three-dimensional point cloud of the fire scene is calculated; In the dense three-dimensional point cloud, point cloud clusters belonging to flames are extracted based on the characteristics of high-temperature regions in infrared or visible light images; For each flame cloud cluster, spatial clustering and fitting are performed to estimate its three-dimensional bounding box, centroid coordinates, volume and average temperature. The centroid coordinates are the three-dimensional coordinates of the key fire source points. The scale level is determined by combining the volume and average temperature. All flame cloud clusters and their attributes are integrated in three-dimensional space to form the three-dimensional thermal distribution model of the fire scene.

[0008] Furthermore, spatiotemporal correlation analysis is performed on the concentration data of the toxic and harmful gases to generate a dynamic diffusion risk map, including: Acquire carbon monoxide, carbon dioxide, and oxygen concentration data collected by gas sensors at different locations on an unmanned fire truck at continuous timestamps; The concentration data of each sensor is associated with its three-dimensional coordinates on the unmanned fire truck and the collection timestamp to form spatiotemporal concentration data points; Using the Kriging spatial interpolation method, the concentration estimate of unsampled locations is interpolated in a four-dimensional spatiotemporal volume consisting of the fire area and the time dimension. Based on preset dangerous concentration thresholds for toxic and harmful gases, safe zones, warning zones, and danger zones are divided in the four-dimensional spatiotemporal volume. At fixed time slices, the danger zone and warning zone are projected onto a two-dimensional planar map, and their diffusion direction and trend are marked to form the dynamic diffusion risk map.

[0009] Furthermore, the three-dimensional thermal distribution model of the fire scene, the three-dimensional coordinates and scale level of the key fire source points, the dynamic diffusion risk map, and the real-time operational posture and pressure data are input into the human-vehicle linkage decision-making model to generate a set of coordinated firefighting operation instructions for fire sources at different heights, including: The human-vehicle linkage decision model receives input data and, based on a preset fire source height threshold, classifies the key fire source points into low-altitude fire sources, medium-altitude fire sources, and high-altitude fire sources. For low-altitude fire sources, the decision model prioritizes dispatching the vehicle-mounted vertical lift-type aerial spray system of unmanned fire trucks to extinguish the fire. Based on the three-dimensional coordinates, scale level, and dynamic diffusion risk map of the fire source, the target elevation angle, azimuth angle, spray pressure, and extinguishing agent type and mixing ratio of the fire monitor are calculated. For fire sources at medium to high altitudes, the decision model dispatches firefighting drones to extinguish the fire. Based on the three-dimensional coordinates, scale level, and dynamic diffusion risk map of the fire source, the model plans the take-off point, flight path, hovering operation point, and timing of water hose delivery for the drones. The decision-making model also needs to coordinate the operation sequence and spatial avoidance of vehicles and drones to ensure that the operating radius of the fire monitor does not conflict with the flight airspace of the drone. Based on the above decisions, a set of collaborative firefighting operation instructions is generated, which includes equipment identification, action type, target parameters, and execution sequence.

[0010] Furthermore, based on the aforementioned collaborative firefighting operation instruction set, a specific sequence of equipment control instructions is generated and issued through an adaptive operation scheduler, including: The adaptive job scheduler parses each instruction in the collaborative firefighting operation instruction set; For instructions involving the vehicle-mounted vertical lift-type spray system, the adaptive work scheduler converts them into specific control instructions for the lifting mechanism telescopic motor, rotary motor, water pump frequency converter, foam pump driver, flow regulating valve servo controller and air compressor controller, including target position, target speed, target pressure and valve opening. For instructions involving firefighting drones, the adaptive operation scheduler converts them into flight control instructions and mission payload control instructions for the drones through the drone ground control station interface, including takeoff, waypoint flight, hovering, hose dropping, and spraying initiation. The adaptive job scheduler arranges all control commands into a time-synchronized command sequence based on the timing dependencies between commands and the device response time. The command sequence is transmitted in real time to the corresponding execution device controller via the CAN bus, wireless data transmission link and MESH network in the control unit of the unmanned fire truck.

[0011] Furthermore, the improved stereo vision matching algorithm optimizes the matching cost calculation process based on the non-rigid and dynamic characteristics of flame textures, including: Acquire a sequence of fire scene images containing depth information collected by a reconnaissance drone in consecutive time frames; For the current image frame to be processed, detect the highlighted and moving areas to preliminarily determine the potential flame areas; Within the initially determined potential flame area, texture analysis is performed on the image blocks to calculate the directional consistency and randomness measure of local textures in order to identify flame texture areas with non-rigid deformation characteristics. In the cost calculation stage of the stereo vision matching algorithm, a dynamic smoothing constraint term is introduced for the pixels belonging to the flame texture region; The dynamic smoothing constraint term is adaptively adjusted according to the degree of disruption of the disparity continuity of adjacent pixels in the previous frame matching result in the current frame. The greater the degree of disruption, the weaker the constraint, so as to allow non-rigid deformation of the flame area. The total cost function after incorporating the dynamic smoothing constraint term will be used for subsequent disparity optimization calculations to adapt to the dynamic and non-rigid changes of the flame during the matching process.

[0012] Furthermore, the improved stereo vision matching algorithm, when calculating the matching cost, integrates the cost based on gradient information, the cost based on Census transform, and the cost based on the color invariance features of the flame region, and assigns adaptive weights to the dynamic flame region, including: For each pixel in the image to be matched, within its supported window, calculate the sum of the absolute differences of the gradients with the candidate matching points and the Hamming distance after Census transform. For areas that may belong to flames, additional feature vector distances are calculated in a defined color invariant space, which can reduce the interference of brightness changes on flame color judgment. Normalize the gradient cost, Census cost, and color invariance cost; A dynamic weight is estimated for the current pixel, which is determined based on the magnitude of the inter-frame difference in the image sequence of the region where the pixel is located. Regions with large inter-frame differences are considered dynamic flame regions and are given a higher weight due to the cost of color invariance. The three normalized costs are linearly weighted and summed using the dynamic weights to obtain the final matching cost for the pixel.

[0013] Furthermore, using the Kriging spatial interpolation method, concentration estimates for unsampled locations are interpolated within a four-dimensional spatiotemporal volume comprised of the fire area and the time dimension, including: The concentration data collected by each gas sensor is combined with its three-dimensional spatial coordinates on the unmanned fire truck and the collection timestamp to form a four-dimensional spatiotemporal data point; Calculate the semivariogram among all known four-dimensional spatiotemporal data points to quantify the correlation of concentration in the three spatial dimensions and the temporal dimension; Based on the semi-variogram, a Kriging interpolation model is constructed to describe the spatial distribution structure characteristics of concentration in a four-dimensional spatiotemporal volume. The Kriging interpolation model is applied to each unknown node of the four-dimensional spatiotemporal grid defined by the geographical extent of the fire and the time window. The Kriging interpolation model uses a weighted linear combination of known four-dimensional spatiotemporal data points to calculate the concentration estimate for each unknown node. The weights are determined by the semi-variogram values ​​between the node and all known points.

[0014] Furthermore, for low-altitude fire sources, the decision model prioritizes dispatching the vehicle-mounted vertical lift aerial spray system of the unmanned fire truck for fire suppression, including: Based on the three-dimensional coordinates of the fire source and the real-time positioning coordinates of the unmanned fire truck, the horizontal distance and height difference between the fire source and the fire monitor base are calculated. Based on the kinematic model of the vertical lifting and firing system, the boom extension length, elevation angle, and rotation angle required to align the muzzle with the fire source are solved inversely. Based on the size, level, and type of the fire source, the system queries the preset fire extinguishing strategy database for recommended spray pressure, flow rate, and extinguishing agent mixing ratio. The calculated boom extension length, pitch angle, slewing angle, spray pressure, flow rate, and mixing ratio are encapsulated into a specific elevated spraying operation instruction and added to the coordinated firefighting operation instruction set.

[0015] Furthermore, the aforementioned system enables the unmanned fire truck, aerial ladder fire system, and fire-fighting drone to perform coordinated actions and form a closed-loop control based on real-time and updated feedback data, specifically including: The equipment control command sequence drives the unmanned fire truck, the vehicle-mounted vertical lift aerial spray system, and the fire-fighting drone to perform coordinated fire-fighting actions. During execution, the real-time operational posture and pressure data, as well as the updated fire scene images containing depth information and toxic and harmful gas concentration data are continuously collected to form an updated multi-source perception data set, which is fed back to the human-vehicle linkage decision model to form closed-loop control.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This system collects fire scene images with depth information, toxic and hazardous gas concentration data, and operational posture and pressure data from firefighting operations, achieving unified aggregation and integration of multi-dimensional on-site information. An improved stereoscopic vision matching algorithm is used to process the fire scene images, constructing a three-dimensional thermal distribution model of the fire scene and identifying the three-dimensional coordinates and scale level of key fire sources. Spatiotemporal correlation analysis is conducted on the toxic and hazardous gas concentration data to generate a dynamic diffusion risk map, achieving digital standardization of the fire scene's spatial structure, fire source status, and gas evolution trends. This addresses the shortcomings of single-sensor data in representing the overall fire situation, providing comprehensive basic data support for subsequent decision-making.

[0017] The three-dimensional thermal distribution model of the fire scene, the three-dimensional coordinates and scale level of key fire sources, the dynamic diffusion risk map, and real-time operational posture and pressure data are all integrated into the human-vehicle linkage decision-making model to generate a set of collaborative firefighting operation instructions adapted to the needs of fire sources at different heights. Based on the adaptive operation scheduler, the collaborative operation instruction set is parsed to generate a standardized equipment control instruction sequence, driving unmanned fire trucks, aerial ladder spray systems, and firefighting drones to complete collaborative operations according to a unified logic.

[0018] It receives real-time operational and updated data from the field, dynamically adjusts the operation process, and forms a complete closed-loop control flow. It establishes a standardized scheduling logic for the coordinated operation of multiple equipment, breaks the independent operation mode of a single piece of equipment, continuously corrects the operation status based on real-time data from the field, adapts to the operation logic of simultaneously dealing with fire sources at different heights in complex fire scenes, and improves the overall operation mechanism of unmanned firefighting equipment linkage operation. Attached Figure Description

[0019] Figure 1 This is a timing diagram of the mobile unmanned firefighting vehicle-human linkage multi-height precision intelligent fire extinguishing operation system described in this invention; Figure 2 A flowchart for constructing a three-dimensional thermal distribution model of a fire scene and identifying fire sources; Figure 3 This is a flowchart of the process for generating instructions for human-vehicle linkage decision-making and coordinated firefighting operations. Detailed Implementation

[0020] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] See Figure 1 This invention provides a mobile, unmanned firefighting vehicle-machine linkage multi-height precision intelligent firefighting operation system, and other systems include: The multi-source sensing module acquires and aggregates fire scene images containing depth information, toxic and harmful gas concentration data, and real-time operational posture and pressure data from the firefighting operation site. The situation analysis module processes the fire scene images containing depth information using an improved stereoscopic vision matching algorithm, constructs a three-dimensional thermal distribution model of the fire scene, identifies the three-dimensional coordinates and scale levels of key fire sources, and performs spatiotemporal correlation analysis on the toxic and harmful gas concentration data to generate a dynamic diffusion risk map. The coordinated decision-making module inputs the three-dimensional thermal distribution model of the fire scene, the three-dimensional coordinates and scale levels of the key fire sources, the dynamic diffusion risk map, and the real-time operational posture and pressure data into the human-vehicle coordinated decision-making model to generate a set of coordinated firefighting operation instructions for fire sources at different heights. Based on the coordinated firefighting operation instruction set, the closed-loop scheduling module generates and issues specific equipment control instruction sequences through an adaptive operation scheduler, driving unmanned fire trucks, aerial ladder fire systems, and firefighting drones to perform coordinated actions, and forming closed-loop control based on feedback real-time and updated data.

[0022] In one embodiment of the present invention, when performing an improved stereo vision matching algorithm on the fire scene image containing depth information to construct a three-dimensional thermal distribution model of the fire scene and identify the three-dimensional coordinates and scale level of key fire sources, refer to... Figure 2The improved stereo vision matching algorithm optimizes the matching cost calculation process based on the non-rigid and dynamic characteristics of flame texture. Multiple fire scene images containing depth information, acquired by a reconnaissance UAV at different poses, are corrected and aligned. Using the improved stereo vision matching algorithm, the disparity of each pixel is calculated among multiple images. When calculating the matching cost, the improved stereo vision matching algorithm integrates costs based on gradient information, costs based on Census transform, and costs based on the color invariance characteristics of flame regions, and assigns adaptive weights to dynamic flame regions. Based on camera intrinsic parameters and the disparity, a dense 3D point cloud of the fire scene is calculated. In the dense 3D point cloud, point cloud clusters belonging to flames are extracted based on the high-temperature region characteristics in infrared or visible light images. Spatial clustering and fitting are performed on each flame point cloud cluster to estimate its 3D bounding box, centroid coordinates, volume, and average temperature. The centroid coordinates are the 3D coordinates of the key fire source points, and the scale level is determined by combining the volume and average temperature. All flame point cloud clusters and their attributes are integrated in 3D space to form the 3D thermal distribution model of the fire scene.

[0023] In practical implementation, targeting a multi-story warehouse fire scene, a reconnaissance drone equipped with a binocular camera and an infrared thermal imager acquired multiple fire scene images containing depth information from various hovering points around and above the warehouse. Epipolar correction and image alignment were performed on the acquired images to ensure that the projections of the same spatial point in multiple images lie on the same epipolar line. When executing the improved stereo vision matching algorithm, the matching cost calculation process was optimized based on the non-rigid and dynamic characteristics of flame texture. For each image pair to be matched, a cost based on gradient information was calculated at each pixel location, i.e., the sum of the absolute differences in the gradients of the left and right images in the horizontal and vertical directions; simultaneously, a cost based on Census transform was calculated, encoding the brightness comparison results within the pixel neighborhood into a bit string and calculating the Hamming distance; additionally, for flame pixel regions that appear orange-red in the bright areas, a cost based on the color invariance characteristics of the flame region was calculated. This feature converts the RGB three-channel values ​​to a color space insensitive to illumination changes before calculating the Euclidean distance of the feature vector. An adaptive weight is assigned to the current pixel, determined based on the average grayscale difference between corresponding pixel blocks in the previous and current frames. Regions with an average grayscale difference exceeding a preset threshold are identified as dynamic flame regions. In these regions, the weights for color invariance cost, gradient cost, and census cost are set to 0.6, 0.2, and 0.2, respectively. In non-dynamic flame regions, the weights for the three costs are set to 0.3, 0.4, and 0.3, respectively. The normalized costs are then weighted and linearly summed to obtain the final matching cost for each pixel. Subsequently, a semi-global matching algorithm is used for cost aggregation and disparity optimization calculations to obtain the disparity value for each pixel. Based on the known binocular camera intrinsic matrix, baseline length, and the calculated disparity map, the coordinates of the 3D spatial point corresponding to each pixel are calculated using the triangulation principle, generating a dense 3D point cloud of the fire scene containing hundreds of thousands of spatial points. Within the dense 3D point cloud, point cloud clusters belonging to flames are extracted based on the spatial points corresponding to pixel regions with temperatures above 280 degrees Celsius in the infrared thermal image. Each point cloud cluster contains thousands of spatial points. Each flame cloud cluster is spatially segmented using Euclidean clustering. A three-dimensional bounding box is fitted to each segmented cluster using the least squares method. The geometric center coordinates of the bounding box represent the three-dimensional coordinates of the key fire source. The product of the bounding box volume and the average temperature of points within the cluster (obtained from infrared radiance inversion) is used as the scale-level quantification value. The three-dimensional bounding boxes, centroid coordinates, volume, and average temperature attributes of all flame cloud clusters are integrated into a single three-dimensional coordinate system to form a three-dimensional thermal distribution model of the fire scene.

[0024] In some embodiments, a reconnaissance drone flies over a fire scene in an outdoor parking lot, and the fire images it collects, containing depth information, show flame regions exhibiting obvious non-rigid fluctuations and shape changes. When processing this scenario, the improved stereo vision matching algorithm uses frame differencing to detect bright pixel regions with dynamically changing grayscale values ​​as potential flame regions in the current frame of a sequence of fire images containing depth information collected by the reconnaissance drone over consecutive time frames. Within these potential flame regions, the local binary pattern texture features of each image patch are calculated, and the histogram of texture directions is statistically analyzed. When the entropy value of the direction histogram is greater than 2.5 and the percentage of peak values ​​in the main direction is less than 30%, the image patch is determined to have flame textures with non-rigid deformation characteristics. For pixels identified as flame texture regions, a dynamic smoothing constraint term is introduced into the stereo matching cost function.

[0025] Optionally, in a high-rise building facade fire scenario, the fire image, which contains depth information and is captured by a reconnaissance drone, shows flames spreading rapidly vertically. The improved stereo vision matching algorithm, when calculating the matching cost, calculates the absolute difference of the gradients and the Hamming distance after Census transform between each pixel to be matched in the image and the candidate matching point within its supported 11×11 pixel window. For pixel regions determined to belong to flames based on the color histogram (satisfying a red channel value greater than 150 and a red component at least 30 higher than the green component), an additional feature vector distance in a defined color invariant space is calculated. This color invariant space uses normalized rg chromaticity coordinates, i.e. , Ignoring the luminance component, the feature vector distance is the Euclidean distance between two pixels in the (r,g) 2D space. The gradient cost is divided by the maximum possible gradient value (255), the Census cost by the number of bits (32), and the color invariance cost by the maximum possible Euclidean distance (2.0), and then normalized to the appropriate interval. A dynamic weight is estimated for the current pixel, equal to the average absolute difference in grayscale values ​​of the 5×5 region containing the current pixel between two consecutive frames divided by 255. The dynamic weight is 0.12 when the average absolute difference in grayscale values ​​is 30, and 0.47 when the average absolute difference in grayscale values ​​is 120. Regions with large inter-frame differences are identified as dynamic flame regions. For pixels in dynamic flame regions with a dynamic weight higher than 0.35, a color invariance cost weight of 0.7, a gradient cost weight of 0.1, and a Census cost weight of 0.2 are assigned. For non-dynamic regions with a dynamic weight lower than or equal to 0.35, a color invariance cost weight of 0.2, a gradient cost weight of 0.4, and a Census cost weight of 0.4 are assigned. The three normalized costs are linearly weighted and summed using the dynamic weights to obtain the final matching cost for each pixel.

[0026] In one embodiment of the present invention, regarding the specific implementation of the improved stereo vision matching algorithm for optimizing the matching cost calculation process based on the non-rigid and dynamic characteristics of flame texture, a sequence of fire scene images containing depth information collected by a reconnaissance drone in consecutive time frames is acquired. For the current image frame to be processed, highlighted and moving regions are detected to preliminarily determine potential flame regions. Within the preliminarily determined potential flame regions, texture analysis is performed on image blocks to calculate the directional consistency and randomness measure of local textures, in order to identify flame texture regions with non-rigid deformation characteristics. In the cost calculation stage of the stereo vision matching algorithm, a dynamic smoothing constraint term is introduced for pixels belonging to the flame texture regions. The dynamic smoothing constraint term is adaptively adjusted according to the degree of disruption of the disparity continuity of adjacent pixels in the previous frame matching result in the current frame; the greater the degree of disruption, the weaker the constraint force, so as to allow non-rigid deformation of the flame region. The total cost function after introducing the dynamic smoothing constraint term is used for subsequent disparity optimization calculation to adapt to the dynamic and non-rigid changes of the flame during the matching process.

[0027] The improved stereo vision matching algorithm integrates gradient-based costs, Census transform-based costs, and costs based on the color invariance features of flame regions when calculating the matching cost, and assigns adaptive weights to dynamic flame regions. For each pixel to be matched in the image, within its supported window, the sum of the absolute differences of gradients and the Hamming distance after Census transform are calculated between it and the candidate matching point. For regions that may belong to flames, the feature vector distance in a defined color invariance space is additionally calculated. This color invariance space can reduce the interference of brightness changes on the flame color judgment. The gradient cost, Census cost, and color invariance cost are normalized. A dynamic weight is estimated for the current pixel. This weight is determined based on the magnitude of the inter-frame differences in the image sequence of the region where the point is located. Regions with large inter-frame differences are considered dynamic flame regions and are assigned higher weights to the color invariance cost. The three normalized costs are linearly weighted and summed using the dynamic weight to obtain the final matching cost for the pixel.

[0028] In practical implementation, for an open-air stack fire scenario, a reconnaissance drone hovered at a height of 10 meters above the fire, acquiring a sequence of fire scene images containing depth information at a rate of 30 frames per second. For the current image frame to be processed, an adaptive threshold segmentation method was used to extract pixel regions with brightness higher than 200 (0-255 grayscale range). Combined with the inter-frame difference method (the absolute value of the grayscale change between three consecutive frames is greater than 25) to extract motion regions, the intersection of the brightness threshold and the motion detection results was initially identified as potential fire regions. Within the initially identified potential fire regions, local binary pattern texture features were calculated for each 16×16 pixel image block, and texture orientation histograms were statistically analyzed. Orientation consistency and randomness measures were calculated: orientation consistency was taken as the proportion of the maximum peak value in the histogram, and randomness measure was taken as the ratio of the histogram entropy value to the maximum possible entropy value. When the orientation consistency was lower than 0.35 and the randomness measure was higher than 0.7, the corresponding image block was identified as a fire texture region with non-rigid deformation characteristics. In the cost calculation stage of the stereo vision matching algorithm, a dynamic smoothing constraint term is introduced for each pixel belonging to the flame texture region.

[0029] In some embodiments, for the same open-air stack fire scenario, the flame region in the fire scene image containing depth information collected by the reconnaissance drone exhibits strong flickering and distortion. The improved stereo vision matching algorithm performs the following operations when calculating the matching cost. For each pixel to be matched in the image, within its supported 21×21 pixel window, the sum of the absolute difference of the gradient (the sum of the absolute values ​​of the gradients in the horizontal and vertical directions) and the Hamming distance after Census transformation are calculated respectively with the candidate matching point. (The Census transformation window size is 9×7 pixels, and the bit string length is 63 bits). For the flame region determined based on the red, green, and blue three-channel features (the red channel intensity is greater than the green channel intensity, and the green channel intensity is greater than the blue channel intensity, while the red channel intensity is greater than 140), the feature vector distance in a defined color invariance space is additionally calculated. This color invariance space uses the hue component and saturation component in the hue-saturation-lightness space, ignoring the lightness component. The feature vector distance is the Euclidean distance between the two pixels on the hue-saturation two-dimensional plane (hue value ranges from 0 to 360, and saturation value ranges from 0 to 1). The gradient cost is divided by the maximum possible gradient value of 510 (255 in each direction), the Census cost is divided by the number of bits of 63, and the color invariance cost is divided by the maximum possible Euclidean distance (maximum hue difference 360, maximum saturation difference 1, the maximum distance after normalization is approximately 360.001), and then normalized to the interval of 0 to 1. A dynamic weight is estimated for the current pixel, which is equal to the average absolute difference of gray levels in the 9×9 pixel region where the current pixel is located between two consecutive frames divided by 255. Regions with large inter-frame differences (average absolute difference of gray levels greater than 30) are identified as dynamic flame regions. For pixels in dynamic flame regions, the weighting coefficient for color invariance cost is 0.65, the weighting coefficient for gradient cost is 0.15, and the weighting coefficient for Census cost is 0.20. The three normalized costs are linearly weighted and summed using the dynamic weights: the final matching cost equals the dynamic flame region determination flag multiplied by (0.65 × color invariance cost + 0.15 × gradient cost + 0.20 × Census cost) plus the non-flame region determination flag multiplied by (0.30 × gradient cost + 0.35 × Census cost + 0.35 × color invariance cost), where the dynamic flame region determination flag is 1 when the average absolute difference of grayscale is greater than 30 and 0 otherwise, and the non-flame region determination flag is 1 minus the dynamic flame region determination flag.

[0030] Optionally, in an oil tank fire scenario, the flames in the fire scene images containing depth information from consecutive time frames collected by a reconnaissance drone exhibit rapid rolling motion. When initially identifying potential flame areas, a visual saliency detection-based method is used, fusing color contrast and motion contrast to generate a saliency map. Areas with saliency values ​​higher than 1.8 times the global mean are identified as potential flame areas. Within these potential flame areas, the gray-level co-occurrence matrix is ​​used to calculate the contrast, correlation, and energy of each image patch. When the contrast is greater than 50, the correlation is less than 0.2, and the energy is less than 0.3, flame texture areas with non-rigid deformation characteristics are identified. For pixels in the flame texture area, the calculation method for the degree of disparity continuity disruption between adjacent pixels in the dynamic smoothing constraint is changed: calculate the disparity gradient maps along the horizontal and vertical directions in the previous frame's matching result, and calculate the disparity gradient map of the current frame's intermediate matching result (without smoothing optimization). When the root mean square error between the two gradient maps is greater than 3 pixels, the disruption degree is considered high, and the dynamic adjustment factor α is set to 0.1.

[0031] In one embodiment of the present invention, the specific implementation method for performing spatiotemporal correlation analysis on the concentration data of toxic and harmful gases to generate a dynamic diffusion risk map is as follows: Carbon monoxide, carbon dioxide, and oxygen concentration data collected at continuous time stamps from gas sensors at different locations on an unmanned fire truck are acquired. The concentration data of each sensor is correlated with its three-dimensional coordinates on the unmanned fire truck and the collection time stamp to form spatiotemporal concentration data points. Using the Kriging space interpolation method, the concentration estimate of unsampled locations is interpolated in a four-dimensional spatiotemporal volume composed of the fire area and the time dimension. Based on a preset dangerous concentration threshold for toxic and harmful gases, safe areas, warning areas, and dangerous areas are divided in the four-dimensional spatiotemporal volume. On a fixed time slice, the dangerous areas and warning areas are projected onto a two-dimensional planar map, and their diffusion direction and trend are marked to form the dynamic diffusion risk map.

[0032] The detailed steps for interpolating concentration estimates for unsampled locations in a four-dimensional spatiotemporal volume using the Kriging spatial interpolation method are as follows: Concentration data collected by each gas sensor is combined with its three-dimensional spatial coordinates on the unmanned fire truck and the collection timestamp to form a four-dimensional spatiotemporal data point. The semi-variogram is calculated among all known four-dimensional spatiotemporal data points to quantify the correlation of concentration in the three spatial dimensions and the temporal dimension. Based on the semi-variogram, a Kriging interpolation model describing the spatial distribution structure of concentration in the four-dimensional spatiotemporal volume is constructed. The Kriging interpolation model is applied to each unknown node in the four-dimensional spatiotemporal grid defined by the fire scene's geographical extent and time window. The Kriging interpolation model uses a weighted linear combination of known four-dimensional spatiotemporal data points to calculate the concentration estimate for each unknown node; the weights are determined by the semi-variogram values ​​between that node and all known points.

[0033] In the specific implementation, for a fire in a chemical plant's tank area, an unmanned fire truck was deployed 20 meters upwind of the tank area. Four electrochemical gas sensors were installed at different locations on the top of the unmanned fire truck. The three-dimensional coordinates of the four sensors were (2.5m, 1.2m, 1.8m), (3.0m, 1.5m, 2.0m), (2.8m, 1.0m, 2.2m), and (3.2m, 1.3m, 1.9m), respectively, with the origin being the projection of the unmanned fire truck's center of mass onto the ground. The four sensors collected carbon monoxide, carbon dioxide, and oxygen concentration data at continuous timestamps (starting from 14:30:00, collected every 2 seconds). Table 1 shows some of the spatiotemporal concentration data collected by the sensors. The carbon monoxide concentration data collected by each sensor was correlated with the sensor's three-dimensional coordinates on the unmanned fire truck and the collection timestamp to form spatiotemporal concentration data points. Each data point contained five components (x, y, z, t, C), where C represents the carbon monoxide concentration value in ppm.

[0034] Table 1: Spatiotemporal concentration data collected by toxic and harmful gas sensors

[0035] In the specific implementation, the Kriging spatial interpolation method is used to interpolate the estimated carbon monoxide concentration at unsampled locations in a four-dimensional spatiotemporal volume composed of the fire area and the time dimension. The carbon monoxide concentration data collected by each gas sensor is combined with the three-dimensional spatial coordinates of the sensor on the unmanned fire truck and the collection timestamp to form a four-dimensional spatiotemporal data point, that is, the concentration value at point P(x,y,z,t) is C(x,y,z,t). The semi-variogram between all known four-dimensional spatiotemporal data points is calculated, and a spherical model is used to fit the semi-variogram γ(h)=c0+c1·(1.5·(h / a)-0.5·(h / a)^3), where h is the Euclidean distance between two four-dimensional spatiotemporal data points (including the three spatial dimensions and the time dimension, the spatial distance is in meters, the time distance is in seconds, and the time distance is multiplied by the wind speed). (Converting a velocity of 2 m / s to equivalent spatial distance), c0 is the nugget constant of 5, c1 is the partial sill value of 45, and a is the range of 12. A Kriging interpolation model describing the spatial distribution structure of carbon monoxide concentration in a four-dimensional spatiotemporal volume is constructed based on the semi-variogram. The Kriging interpolation model is applied to each unknown node (1-meter interval in the x-direction, 1-meter interval in the y-direction, 0-meter interval in the z-direction, and 1-second interval in the t-direction) of the four-dimensional spatiotemporal grid defined by the geographical range of the fire site (x-direction -5 m to 15 m, y-direction -3 m to 8 m, z-direction 0 m to 5 m) and the time window (14:30:00 to 14:31:00). The Kriging interpolation model calculates the estimated carbon monoxide concentration for each unknown node using a weighted linear combination of known four-dimensional spatiotemporal data points. The formula for calculating the estimated value is:

[0036] in: Represents four-dimensional spacetime position Estimated carbon monoxide concentration at [location] This represents the total number of known four-dimensional spatiotemporal data points (here, n is 32, which is the number of data points collected by 4 sensors at 8 timestamps). This indicates that the data is assigned to the i-th known four-dimensional spatiotemporal data point. Kriging weight coefficients, , Represents known four-dimensional spacetime data points Measured carbon monoxide concentration at the location; Kriging weighting coefficient It is obtained by solving the Kriging equations, which are constructed using the semivariogram values ​​between known points and between unknown points and known points.

[0037] In some embodiments, based on the carbon monoxide concentration data at the aforementioned chemical plant tank area fire site, and using preset hazardous gas concentration thresholds (carbon monoxide hazardous concentration threshold of 50 ppm and warning concentration threshold of 30 ppm), a safe zone (carbon monoxide concentration less than 30 ppm), a warning zone (carbon monoxide concentration greater than or equal to 30 ppm and less than 50 ppm) and a danger zone (carbon monoxide concentration greater than or equal to 50 ppm) are divided in the four-dimensional spatiotemporal volume. At a fixed time slice (e.g., 14:30:10), the danger zone and the warning zone are projected onto a two-dimensional planar map (xy plane, i.e., ground projection map). Based on the displacement vector of the centroid of the danger zone in two adjacent time slices (14:30:08 and 14:30:10), the diffusion direction (from the centroid of the danger zone at 14:30:08 to the centroid of the danger zone at 14:30:10) and the diffusion speed (displacement distance divided by 2 seconds) are calculated. Arrows are used to mark the diffusion direction and trend on the projection map, forming a dynamic diffusion risk map.

[0038] Optionally, for carbon dioxide concentration data (hazard threshold of 5000 ppm and warning threshold of 3000 ppm) and oxygen concentration data (hazard threshold of 18% and warning threshold of 19.5%), the same spatiotemporal correlation analysis and Kriging interpolation processing are performed respectively to generate carbon dioxide dynamic diffusion risk maps and oxygen dynamic diffusion risk maps.

[0039] In some embodiments, for a fire scene in the same chemical plant tank area, during the movement of an unmanned fire truck (the unmanned fire truck travels at a speed of 0.5 m / s along the positive x direction), the three-dimensional coordinates of the gas sensor change over time. The real-time positioning coordinates of the sensor at each timestamp (obtained through the unmanned fire truck's onboard differential GPS) are associated with the collected timestamps and concentration data to form a dynamically updated set of four-dimensional spatiotemporal data points. In the calculation of the semivariogram, the time distance is directly added to the four-dimensional Euclidean distance calculation in seconds, without equivalent spatial conversion. The scale factors of the four dimensions are set to a spatial scale factor of 1 (meter) and a time scale factor of 0.5 (second), respectively. The exponential model semivariogram γ(h) = c0 + c1·(1-exp(-h / a)) is adopted, where c0 = 3 (nuclear constant), c1 = 38 (partial sill value), and a = 8 (range). In the application of the Kriging interpolation model on the four-dimensional spatiotemporal grid nodes, only known data points within 30 seconds before the current timestamp are used to participate in the weight calculation to reduce the amount of computation.

[0040] In one embodiment of the present invention, when the three-dimensional thermal distribution model of the fire scene, the three-dimensional coordinates and scale level of the key fire source points, the dynamic diffusion risk map, and the real-time operation posture and pressure data are input into the human-vehicle linkage decision model to generate a set of coordinated firefighting operation instructions for fire sources at different heights, refer to [reference needed]. Figure 3 The human-vehicle linkage decision-making model receives input data and, based on a preset fire source height threshold, classifies key fire sources into low-altitude, mid-altitude, and high-altitude fire sources. For low-altitude fire sources, the decision-making model prioritizes dispatching the vehicle-mounted vertical lift-type aerial spray system of the unmanned fire truck for fire suppression. Based on the three-dimensional coordinates, scale level, and dynamic diffusion risk map of the fire source, it calculates the target elevation angle, azimuth angle, spray pressure, and extinguishing agent type and mixing ratio of the fire monitor. For mid- and high-altitude fire sources, the decision-making model dispatches fire-fighting drones for fire suppression. Based on the three-dimensional coordinates, scale level, and dynamic diffusion risk map of the fire source, it plans the drone's takeoff point, flight path, hovering operation point, and hose delivery timing. The decision-making model also needs to coordinate the operation sequence and spatial avoidance of the vehicle and drone to ensure that the fire monitor's operating radius does not conflict with the drone's flight airspace. Based on the above decisions, a collaborative fire suppression operation instruction set containing equipment identification, action type, target parameters, and execution sequence is generated.

[0041] For low-altitude fire sources, the decision model prioritizes the use of the unmanned fire truck's onboard vertical lift system for fire suppression. The specific implementation involves: calculating the horizontal distance and height difference between the fire source and the fire monitor base based on the fire source's three-dimensional coordinates and the unmanned fire truck's real-time positioning coordinates. Using the kinematic model of the vertical lift system, the required boom extension length, pitch angle, and slewing angle for aligning the nozzle with the fire source are determined. Based on the fire source's scale and type, recommended spray pressure, flow rate, and extinguishing agent mixing ratio are retrieved from a pre-set fire suppression strategy library. The calculated boom extension length, pitch angle, slewing angle, spray pressure, flow rate, and mixing ratio are then encapsulated into a specific vertical lift operation command and added to the coordinated fire suppression operation command set.

[0042] In practice, for a fire in a five-story commercial building, an unmanned fire truck was deployed 15 meters in front of the building. A reconnaissance drone provided a three-dimensional thermal distribution model of the fire scene, the three-dimensional coordinates and scale level of key fire sources, a dynamic spread risk map, and real-time operational attitude and pressure data from the air. The human-vehicle linkage decision model received the above input data and, based on preset fire source height thresholds (low-altitude fire source height threshold less than 3 meters, medium-altitude fire source height threshold greater than or equal to 3 meters and less than 8 meters, and high-altitude fire source height threshold greater than or equal to 8 meters), classified the two identified key fire sources into a low-altitude fire source (located at the window position on the second floor of the building, at a height of 2.8 meters, with a scale level of medium) and a high-altitude fire source (located at the window position on the fifth floor of the building, at a height of 13.5 meters, with a scale level of large). Refer to Table 2 for the attribute data of the two key fire sources.

[0043] Table 2: Key Ignition Source Attribute Data Table

[0044] In practical implementation, for low-altitude fire sources (fire source A), the human-vehicle linkage decision-making model prioritizes dispatching the vehicle-mounted vertical lift aerial spray system of the unmanned fire truck for fire extinguishing; based on the three-dimensional coordinates of fire source A (12.5 meters, 3.2 meters, 2.8 meters) and the real-time positioning coordinates of the unmanned fire truck (0, 2.0 meters, 0), the horizontal distance (12.8 meters) and height difference (2.0 meters) of the fire source relative to the fire monitor base are calculated; combined with the kinematic model of the vehicle-mounted vertical lift aerial spray system (initial boom length 3 meters, extension range 3 to 12 meters, pitch angle range -20 degrees to 80 degrees, slewing angle range -150 degrees to 150 degrees)... The calculations yielded that the required boom extension length to align the gun muzzle with the fire source was 8.5 meters, the elevation angle was 13.2 degrees, and the slewing angle was 22.5 degrees. Based on the scale level (medium) and type (solid combustion) of fire source A, the recommended spray pressure was 0.8 MPa, the flow rate was 25 liters / second, and the extinguishing agent mixing ratio was 9:1 (water to foam) from the preset fire extinguishing strategy library. The calculated boom extension length of 8.5 meters, elevation angle of 13.2 degrees, slewing angle of 22.5 degrees, spray pressure of 0.8 MPa, flow rate of 25 liters / second, and mixing ratio of 9:1 were packaged into a specific elevated spraying operation instruction and added to the coordinated fire extinguishing operation instruction set.

[0045] In practical implementation, for high-altitude fire sources (fire source B), the human-vehicle linkage decision-making model dispatches firefighting drones for fire suppression. Based on the three-dimensional coordinates (12.8m, 3.5m, 13.5m), scale level (large), and dynamic diffusion risk map of fire source B (showing smoke spreading northeastward with a wind speed of 3m / s), the drone's takeoff point is planned to be on the top platform of the unmanned fire truck (coordinates 0, 2.5m, 2.2m). The flight path is to vertically climb to a height of 20m and then fly horizontally to directly above fire source B (12.8m, 3.5m, 20m). The hovering operation point coordinates are (12.8m, 3.5m, 16m). The timing for releasing the fire hose is set so that the drone can release the fire hose tied to the unmanned fire truck when it reaches a height of 10 meters. The human-vehicle linkage decision-making model also coordinates the operation sequence and spatial avoidance of the unmanned fire truck and the fire-fighting drone to ensure that there is no conflict between the operating radius of the unmanned fire truck's onboard vertical lift system (within a horizontal range of 12.8 meters and a vertical range of 2 to 8 meters) and the flight airspace of the fire-fighting drone (above 10 meters in height and within a horizontal range of 10 to 15 meters). The specific avoidance rule is: when the lift system is in operation, the drone's flight height should be no less than 10 meters and it should not enter the range of 5 meters directly above the lift system.

[0046] In some embodiments, for scenarios where fire source A and fire source B coexist in the same fire scene, the collaborative firefighting operation instruction set generated by the human-vehicle linkage decision model includes equipment identification, action type, target parameters, and execution sequence; for unmanned fire trucks (equipment identification FD-01), the action type is elevated spraying, the target parameters include boom extension length of 8.5 meters, pitch angle of 13.2 degrees, slewing angle of 22.5 degrees, spray pressure of 0.8 MPa, flow rate of 25 liters / second, and mixing ratio of 9:1, and the execution sequence is start time 0 seconds; for firefighting drones (equipment identification UAV-... 03), the action type is flight path planning and hovering jetting, the target parameters include the takeoff point (0,2.5,2.2), the flight path point sequence [(0,2.5,2.2),(0,2.5,10),(12.8,3.5,10),(12.8,3.5,16)], the hovering operation point (12.8,3.5,16), the water hose throwing timing is the second point of the flight path (at an altitude of 10 meters), the execution sequence is the start time 0 seconds, and the jetting starts after the UAV reaches the operation point; the rule expression for coordinating the operation sequence of the vehicle and the UAV in the decision model is:

[0047] in: Indicates the drone takeoff delay time. This indicates the distance (taken as 6 meters) that the unmanned fire truck will travel from its current location to the designated work location. This indicates the speed of the unmanned fire truck (taken as 1.5 m / s). This indicates the length of the flight path of the firefighting drone from the takeoff point to the hovering work point (taken as 18 meters). The speed of the firefighting drone is given as 5 m / s; the calculation yields... Second, seconds, therefore The delay is 0.4 seconds, meaning the drone takes off 0.4 seconds after receiving the command, to ensure that the unmanned fire truck arrives at the work position first and completes the boom deployment.

[0048] In one embodiment of the present invention, when generating and issuing specific equipment control command sequences through an adaptive operation scheduler based on the collaborative firefighting operation command set, the adaptive operation scheduler parses each command in the collaborative firefighting operation command set. For commands involving vehicle-mounted vertical lift spray systems, the adaptive operation scheduler converts them into specific control commands for the extension motor, rotary motor, water pump frequency converter, foam pump driver, flow regulating valve servo controller, and air compressor controller of the lift mechanism, including target position, target speed, target pressure, and valve opening. For commands involving firefighting drones, the adaptive operation scheduler converts them into flight control commands and mission payload control commands for the drone through the drone ground control station interface, including takeoff, waypoint flight, hovering, hose dropping, and spray initiation. The adaptive operation scheduler arranges all control commands into a time-synchronized command sequence based on the timing dependencies between commands and equipment response time. The command sequence is then sent to the corresponding execution device controller in real time through the CAN bus, wireless data transmission link, and MESH network within the unmanned fire truck control unit.

[0049] The specific implementation of driving the unmanned fire truck, the elevated spray system, and the fire-fighting drone to perform coordinated actions and forming a closed-loop control based on feedback real-time and updated data is as follows: The equipment control command sequence drives the unmanned fire truck, the vehicle-mounted vertical lift spray system, and the fire-fighting drone to perform coordinated fire-fighting actions. During the execution process, the real-time operating posture and pressure data, as well as updated fire scene images containing depth information and toxic and harmful gas concentration data are continuously collected to form an updated multi-source perception data set, which is fed back to the human-vehicle linkage decision model to form a closed-loop control.

[0050] In practical implementation, for a fire scene in a five-story commercial building, the adaptive operation dispatcher received a set of coordinated firefighting operation instructions. This set of instructions contained two instructions: the first instruction was for device FD-01, the action type was elevated spraying, and the target parameters included boom extension length of 8.5 meters, pitch angle of 13.2 degrees, slewing angle of 22.5 degrees, spray pressure of 0.8 MPa, flow rate of 25 liters / second, water to foam mixing ratio of 9:1, and the execution sequence was start time 0 seconds; the second instruction was for device... Identified as UAV-03, the action type is flight path planning and hovering jet. The target parameters include the takeoff point (0,2.5,2.2), the flight path point sequence [(0,2.5,2.2),(0,2.5,10),(12.8,3.5,10),(12.8,3.5,16)], the hovering operation point (12.8,3.5,16), the water hose throwing timing is the second point of the flight path, the execution sequence is the start time 0 seconds, and the UAV takeoff delay is 0.4 seconds. The adaptive operation scheduler parses each instruction in the collaborative firefighting operation instruction set. For the instruction with equipment identification FD-01 and action type of elevated spray, it converts it into control instructions for the extension motor of the elevated mechanism (target position 8.5 meters), control instructions for the slewing motor (target slewing angle 22.5 degrees), control instructions for the pitch motor of the elevated mechanism (target pitch angle 13.2 degrees), control instructions for the water pump frequency converter (target speed 1450 rpm corresponding to output pressure 0.8 MPa), control instructions for the foam pump driver (target speed corresponding to a water flow rate of 22.5 liters / second and a foam mixing ratio of 9:1 of 2.5 liters / second), control instructions for the flow regulating valve servo controller (valve opening 65%), and control instructions for the air compressor controller (target pressure 0.6 MPa). For equipment identification UAV-03 and action type flight path planning and hovering jetting commands, the adaptive job scheduler converts them into UAV flight control commands and mission payload control commands through the UAV ground control station interface: take-off command (take off from the ground to an altitude of 2.2 meters), waypoint flight command (fly through two waypoints in sequence, (0,2.5,10) and (12.8,3.5,10)), hovering command (hover at coordinates (12.8,3.5,16)), water hose drop command (executed when flying to the second waypoint, i.e., at an altitude of 10 meters), and jetting start command (start the jetting pump after reaching the hovering work point).

[0051] In practical implementation, the adaptive job scheduler arranges all control commands into a time-synchronized command sequence based on the timing dependencies between commands and the equipment response time. The equipment response time parameters are as follows: telescopic motor response time of the lifting mechanism 0.3 seconds, slewing motor response time 0.2 seconds, pitch motor response time 0.2 seconds, water pump frequency converter response time 0.1 seconds, foam pump driver response time 0.1 seconds, flow regulating valve servo controller response time 0.05 seconds, air compressor controller response time 0.1 seconds, UAV ground control station command transmission delay 0.05 seconds, and UAV flight control response time 0.2 seconds. Based on the above parameters, the adaptive job scheduler sets the control commands for each actuator of the aerial spray system to be issued simultaneously at time reference 0 seconds. However, due to the difference in response time of each actuator, the actual sequence of actions reaching the target value is as follows: the flow regulating valve reaches 65% opening after 0.05 seconds; the water pump inverter, foam pump driver, and air compressor controller reach the target state after 0.1 seconds; the slewing motor and pitch motor reach the target angle after 0.2 seconds; and the telescopic motor reaches the target length of 8.5 meters after 0.3 seconds. For UAV control commands, due to a 0.4-second takeoff delay, the adaptive job scheduler sets the UAV takeoff command issuance time to 0.4 seconds, the waypoint flight command to be issued 0.5 seconds after the takeoff command, the hose dropping command to be embedded in the waypoint flight command as a conditional trigger command (automatically executed when the UAV altitude sensor feedback value reaches 10 meters ± 0.5 meters), and the hovering command and start-up spray command to be issued after the UAV reaches coordinates (12.8, 3.5, 16) and its speed returns to zero. The timing arrangement formula is expressed as:

[0052] in: Indicates the first The actual time of issuance of the control command. Indicates the first instruction in the coordinated firefighting operation instruction set The baseline execution timing of the instruction (0 seconds or 0.4 seconds). Indicates the first Response time of each execution device Indicates the first The device completes the action until the [number]th [device]. The dependency interval required for each device to start operating (extracted from the fire suppression strategy, the dependency interval between each actuator inside the aerial spray system is 0 seconds, and there is no dependency between the completion of the aerial spray system's operation and the drone's takeoff operation, so the dependency interval is 0).

[0053] In practical implementation, the control commands of the vehicle-mounted vertical lift-type jet system are sent in real time to the corresponding actuator controllers: the telescopic motor controller of the lifting mechanism receives the command for a target position of 8.5 meters, the slewing motor controller receives the command for a target angle of 22.5 degrees, the pitch motor controller receives the command for a target angle of 13.2 degrees, the water pump frequency converter receives the command for a target speed of 1450 rpm, the foam pump driver receives the command for flow distribution corresponding to the target speed, the flow regulating valve servo controller receives the command for a valve opening of 65%, and the air compressor controller receives the command for a target pressure of 0.6 MPa. The UAV control commands are then sent to the UAV ground control station via a wireless data transmission link and a MESH network. The wireless data transmission link operates at a frequency of 2.4 GHz and a transmission rate of 250 kbit / s. The MESH network uses a self-organizing network protocol and supports up to 32 nodes.

[0054] In some embodiments, a sequence of equipment control commands drives the unmanned fire truck, the vehicle-mounted vertical lift jet system, and the fire-fighting drone to perform coordinated fire-fighting actions. The chassis of the unmanned fire truck remains stationary, the vehicle-mounted vertical lift jet system unfolds its boom and adjusts its rotation and pitch angles according to the commands, and the water pump and foam pump reach the set pressure and flow rate within 0.1 seconds after starting. The fire monitor begins to spray extinguishing agent at the low-altitude fire source. The fire-fighting drone takes off from the top platform of the unmanned fire truck after a delay of 0.4 seconds, flies along the planned path, and drops a water hose at a height of 10 meters (one end of the water hose is fixed to the water hose interface of the unmanned fire truck, and the other end rises with the drone). After the drone reaches the hovering work point (12.8, 3.5, 16), it starts the jet pump to spray extinguishing agent at the high-altitude fire source. During operation, the unmanned fire truck's onboard sensors continuously collect real-time operational attitude and pressure data: pitch and slewing angles are collected via the boom angle encoder, telescopic length is collected via the telescopic displacement sensor, jet pressure is collected via the water pump outlet pressure sensor, and actual flow rate is collected via the flow meter. Flight attitude, position coordinates, and jet pump pressure data are transmitted back via the UAV data link. Simultaneously, a reconnaissance UAV continuously collects updated images of the fire scene, including depth information, and the unmanned fire truck's gas sensors continuously collect updated data on the concentration of toxic and harmful gases. All updated data forms an updated multi-source sensing data set. This updated multi-source sensing data set is fed back to the human-vehicle collaborative decision-making model via a wireless data transmission link. Based on the updated changes in the fire source location (displacement or size changes due to firefighting actions) and changes in the dynamic diffusion risk map, the human-vehicle collaborative decision-making model recalculates and outputs a new set of collaborative firefighting operation instructions, forming a closed-loop control system.

[0055] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A mobile, unmanned firefighting vehicle-managed, multi-height, precise, and intelligent firefighting operation system, characterized in that: The system includes: The multi-source sensing module acquires and aggregates images of the fire scene containing depth information, data on the concentration of toxic and harmful gases, and real-time operational posture and pressure data at the fire-fighting operation site. The situation analysis module performs an improved stereo vision matching algorithm on the fire scene image containing depth information, constructs a three-dimensional thermal distribution model of the fire scene and identifies the three-dimensional coordinates and scale level of key fire sources, performs spatiotemporal correlation analysis on the concentration data of toxic and harmful gases, and generates a dynamic diffusion risk map. The linkage decision module inputs the three-dimensional thermal distribution model of the fire site, the three-dimensional coordinates and scale level of the key fire source points, the dynamic diffusion risk map, and the real-time operation posture and pressure data into the human-vehicle linkage decision model to generate a set of collaborative fire extinguishing operation instructions for fire sources at different heights. The closed-loop scheduling module, based on the collaborative firefighting operation instruction set, generates and issues specific equipment control instruction sequences through the adaptive operation scheduler, drives unmanned fire trucks, aerial ladder spray systems and firefighting drones to perform collaborative actions, and forms closed-loop control based on feedback real-time data and updated data.

2. The mobile unmanned fire-fighting vehicle-mobile multi-height precision intelligent fire-fighting operation system according to claim 1, characterized in that, An improved stereo vision matching algorithm is applied to the fire scene image containing depth information to construct a three-dimensional thermal distribution model of the fire scene and identify the three-dimensional coordinates and scale level of key fire sources, including: The improved stereo vision matching algorithm optimizes the matching cost calculation process based on the non-rigid and dynamic characteristics of flame texture; Correction and alignment are performed on multiple fire scene images containing depth information collected by the reconnaissance drone in different poses; Using the improved stereo vision matching algorithm, the disparity of each pixel is calculated among multiple images. When calculating the matching cost, the improved stereo vision matching algorithm integrates the cost based on gradient information, the cost based on Census transform, and the cost based on the color invariance feature of the flame region, and assigns adaptive weights to the dynamic flame region. Based on the camera intrinsic parameters and the parallax, a dense three-dimensional point cloud of the fire scene is calculated; In the dense three-dimensional point cloud, point cloud clusters belonging to flames are extracted based on the characteristics of high-temperature regions in infrared or visible light images; For each flame cloud cluster, spatial clustering and fitting are performed to estimate its three-dimensional bounding box, centroid coordinates, volume and average temperature. The centroid coordinates are the three-dimensional coordinates of the key fire source points. The scale level is determined by combining the volume and average temperature. All flame cloud clusters and their attributes are integrated in three-dimensional space to form the three-dimensional thermal distribution model of the fire scene.

3. The mobile unmanned fire-fighting vehicle-mobile multi-height precision intelligent fire-fighting operation system according to claim 1, characterized in that, Spatiotemporal correlation analysis was performed on the concentration data of the toxic and harmful gases to generate a dynamic diffusion risk map, including: Acquire carbon monoxide, carbon dioxide, and oxygen concentration data collected by gas sensors at different locations on an unmanned fire truck at continuous timestamps; The concentration data of each sensor is associated with its three-dimensional coordinates on the unmanned fire truck and the collection timestamp to form spatiotemporal concentration data points; Using the Kriging spatial interpolation method, the concentration estimate of unsampled locations is interpolated in a four-dimensional spatiotemporal volume consisting of the fire area and the time dimension. Based on preset dangerous concentration thresholds for toxic and harmful gases, safe zones, warning zones, and danger zones are divided in the four-dimensional spatiotemporal volume. At fixed time slices, the danger zone and warning zone are projected onto a two-dimensional planar map, and their diffusion direction and trend are marked to form the dynamic diffusion risk map.

4. The mobile unmanned fire-fighting vehicle-mobile multi-height precision intelligent fire-fighting operation system according to claim 1, characterized in that, The three-dimensional thermal distribution model of the fire scene, the three-dimensional coordinates and scale level of the key fire source points, the dynamic diffusion risk map, and the real-time operational posture and pressure data are input into the human-vehicle linkage decision-making model to generate a set of collaborative firefighting operation instructions for fire sources at different heights, including: The human-vehicle linkage decision model receives input data and, based on a preset fire source height threshold, classifies the key fire source points into low-altitude fire sources, medium-altitude fire sources, and high-altitude fire sources. For low-altitude fire sources, the decision model prioritizes dispatching the vehicle-mounted vertical lift-type aerial spray system of unmanned fire trucks to extinguish the fire. Based on the three-dimensional coordinates, scale level, and dynamic diffusion risk map of the fire source, the target elevation angle, azimuth angle, spray pressure, and extinguishing agent type and mixing ratio of the fire monitor are calculated. For fire sources at medium to high altitudes, the decision model dispatches firefighting drones to extinguish the fire. Based on the three-dimensional coordinates, scale level, and dynamic diffusion risk map of the fire source, the model plans the take-off point, flight path, hovering operation point, and timing of water hose delivery for the drones. The decision-making model also needs to coordinate the operation sequence and spatial avoidance of vehicles and drones to ensure that the operating radius of the fire monitor does not conflict with the flight airspace of the drone. Based on the above decisions, a set of collaborative firefighting operation instructions is generated, which includes equipment identification, action type, target parameters, and execution sequence.

5. The mobile unmanned fire-fighting vehicle-mobile multi-height precision intelligent fire-fighting operation system according to claim 1, characterized in that, Based on the aforementioned collaborative firefighting operation instruction set, a specific sequence of equipment control instructions is generated and issued through the adaptive operation scheduler, including: The adaptive job scheduler parses each instruction in the collaborative firefighting operation instruction set; For instructions involving the vehicle-mounted vertical lift-type spray system, the adaptive work scheduler converts them into specific control instructions for the lifting mechanism telescopic motor, rotary motor, water pump frequency converter, foam pump driver, flow regulating valve servo controller and air compressor controller, including target position, target speed, target pressure and valve opening. For instructions involving firefighting drones, the adaptive operation scheduler converts them into flight control instructions and mission payload control instructions for the drones through the drone ground control station interface, including takeoff, waypoint flight, hovering, hose dropping, and spraying initiation. The adaptive job scheduler arranges all control commands into a time-synchronized command sequence based on the timing dependencies between commands and the device response time. The command sequence is transmitted in real time to the corresponding execution device controller via the CAN bus, wireless data transmission link and MESH network in the control unit of the unmanned fire truck.

6. The mobile unmanned fire-fighting vehicle-mobile multi-height precision intelligent fire extinguishing system according to claim 2, characterized in that, The improved stereo vision matching algorithm optimizes the matching cost calculation process based on the non-rigid and dynamic characteristics of flame textures, including: Acquire a sequence of fire scene images containing depth information collected by a reconnaissance drone in consecutive time frames; For the current image frame to be processed, detect the highlighted and moving areas to preliminarily determine the potential flame areas; Within the initially determined potential flame area, texture analysis is performed on the image blocks to calculate the directional consistency and randomness measure of local textures in order to identify flame texture areas with non-rigid deformation characteristics. In the cost calculation stage of the stereo vision matching algorithm, a dynamic smoothing constraint term is introduced for the pixels belonging to the flame texture region; The dynamic smoothing constraint term is adaptively adjusted according to the degree of disruption of the disparity continuity of adjacent pixels in the previous frame matching result in the current frame. The greater the degree of disruption, the weaker the constraint, so as to allow non-rigid deformation of the flame area. The total cost function after incorporating the dynamic smoothing constraint term will be used for subsequent disparity optimization calculations to adapt to the dynamic and non-rigid changes of the flame during the matching process.

7. The mobile unmanned fire-fighting vehicle-mobile multi-height precision intelligent fire-fighting operation system according to claim 2, characterized in that, The improved stereo vision matching algorithm, when calculating the matching cost, integrates the cost based on gradient information, the cost based on Census transform, and the cost based on the color invariance characteristics of the flame region, and assigns adaptive weights to the dynamic flame region, including: For each pixel in the image to be matched, within its supported window, calculate the sum of the absolute differences of the gradients with the candidate matching points and the Hamming distance after Census transform. For areas that may belong to flames, additional feature vector distances are calculated in a defined color invariant space, which can reduce the interference of brightness changes on flame color judgment. Normalize the gradient cost, Census cost, and color invariance cost; A dynamic weight is estimated for the current pixel, which is determined based on the magnitude of the inter-frame difference in the image sequence of the region where the pixel is located. Regions with large inter-frame differences are considered dynamic flame regions and are given a higher weight due to the cost of color invariance. The three normalized costs are linearly weighted and summed using the dynamic weights to obtain the final matching cost for the pixel.

8. The mobile unmanned fire-fighting vehicle-mobile multi-height precision intelligent fire-fighting operation system according to claim 3, characterized in that, Using the Kriging spatial interpolation method, concentration estimates for unsampled locations are interpolated within a four-dimensional spatiotemporal volume composed of the fire area and the time dimension, including: The concentration data collected by each gas sensor is combined with its three-dimensional spatial coordinates on the unmanned fire truck and the collection timestamp to form a four-dimensional spatiotemporal data point; Calculate the semivariogram among all known four-dimensional spatiotemporal data points to quantify the correlation of concentration in the three spatial dimensions and the temporal dimension; Based on the semi-variogram, a Kriging interpolation model is constructed to describe the spatial distribution structure characteristics of concentration in a four-dimensional spatiotemporal volume. The Kriging interpolation model is applied to each unknown node of the four-dimensional spatiotemporal grid defined by the geographical extent of the fire and the time window. The Kriging interpolation model uses a weighted linear combination of known four-dimensional spatiotemporal data points to calculate the concentration estimate for each unknown node. The weights are determined by the semi-variogram values ​​between the node and all known points.

9. The mobile unmanned fire-fighting vehicle-mobile multi-height precision intelligent fire-fighting operation system according to claim 4, characterized in that, For low-altitude fire sources, the decision model prioritizes dispatching the vehicle-mounted vertical lift aerial spray system of unmanned fire trucks for fire suppression, including: Based on the three-dimensional coordinates of the fire source and the real-time positioning coordinates of the unmanned fire truck, the horizontal distance and height difference between the fire source and the fire monitor base are calculated. Based on the kinematic model of the vertical lifting and firing system, the boom extension length, elevation angle, and rotation angle required to align the muzzle with the fire source are solved inversely. Based on the size, level, and type of the fire source, the system queries the preset fire extinguishing strategy database for recommended spray pressure, flow rate, and extinguishing agent mixing ratio. The calculated boom extension length, pitch angle, slewing angle, spray pressure, flow rate, and mixing ratio are encapsulated into a specific elevated spraying operation instruction and added to the coordinated firefighting operation instruction set.

10. The mobile unmanned fire-fighting vehicle-mobile multi-height precision intelligent fire-fighting operation system according to claim 5, characterized in that, The system enables the unmanned fire truck, aerial ladder fire system, and fire-fighting drone to perform coordinated actions and forms a closed-loop control based on real-time feedback data and updates, specifically including: The equipment control command sequence drives the unmanned fire truck, the vehicle-mounted vertical lift aerial spray system, and the fire-fighting drone to perform coordinated fire-fighting actions. During execution, the real-time operational posture and pressure data, as well as the updated fire scene images containing depth information and toxic and harmful gas concentration data are continuously collected to form an updated multi-source perception data set, which is fed back to the human-vehicle linkage decision model to form closed-loop control.

Images