A three-vision virtual alignment situation awareness equipment and method for smoke fire scene

Abstract

The present application relates to fire rescue situation awareness technical field, propose a kind of three visual virtual alignment situation awareness equipment and method for smoke fire scene, including three visual sensing head, synchronous acquisition unit, edge inference unit and output interface.Equipment synchronously obtains visible light, thermal infrared and millimeter wave data, based on rough installation prior art, multi-modal feature is mapped to virtual reference domain, residual offset field and confidence mask are determined, and thermal infrared and millimeter wave features are aligned;Combined with time synchronization reliability, modal data quality and confidence information weighted fusion, output fire situation information and risk cost chart.The present application reduces the influence of cross-modal spatial dislocation and low-quality information error fusion caused by equipment vibration, high-temperature deformation and thick smoke, water mist and other factors on the determination of fire source direction, obstacle boundary and traffic risk, improves the accuracy and reliability of fire rescue decision information.

Description

Technical Field

[0001] This invention relates to the field of fire rescue situational awareness technology, and in particular to a three-vision virtual alignment situational awareness equipment and method for dense smoke and fire scenes. Background Technology

[0002] Multimodal visual perception is a technology that integrates data from multiple sensors, such as visible light, thermal infrared, and millimeter-wave radar, to comprehensively identify and characterize elements such as fire sources, obstacles, and passable areas in complex scenes. It has advantages such as information complementarity, strong anti-interference ability, and adaptability to harsh environments. It can provide real-time and reliable situational information for firefighting robots and individual firefighter equipment in fire environments such as dense smoke, high temperature, and low visibility, and is the core direction of intelligent fire rescue equipment.

[0003] In existing technologies, when firefighting robots or individual firefighters operate in dense smoke and fire scenes, visible light cameras, thermal infrared cameras, and millimeter-wave radars are typically installed independently on the equipment body. Each sensor projects coordinates according to fixed extrinsic parameters calibrated at the factory, and the results of each mode recognition are directly superimposed and output. However, due to factors such as equipment movement, mechanical vibration, and high-temperature deformation in the fire scene, the actual relative positions between the sensors change over time. The pixel-level misalignment caused by the fixed extrinsic parameter projection cannot be corrected, resulting in the visible light recognition of fire sources and obstacle outlines not accurately aligning with the thermal infrared and millimeter-wave detection results in space. When dense smoke obscures the visible light image, there is a significant discrepancy between the location of the high-temperature fire source determined by the operator based on the thermal infrared image and the actual location of obstacles and passages in the visible light image. This causes firefighters or fire robots to make inaccurate judgments about the location of the fire source and the boundaries of obstacles. When dense smoke, water mist, or high temperature cause a certain modal data to fail locally or its quality to degrade, the existing equipment still directly superimposes and outputs the modal data in a fixed manner, failing to identify and remove erroneous information from the failed area. This further exacerbates the inaccuracy of fire situation assessment and affects the timeliness and safety of fire rescue decisions. Summary of the Invention

[0004] In view of this, the present invention proposes a three-vision virtual alignment situational awareness equipment and method for dense smoke fires, which solves the problem that the spatial misalignment of cross-modal information and the participation of failure information in the fusion caused by the relative pose changes of the three-modal sensors and the degradation of local data quality in dense smoke fires, resulting in inaccurate fire situation information and risk cost maps.

[0005] The technical solution of the present invention is implemented as follows: On the one hand, the present invention provides a three-vision virtual alignment situational awareness equipment for dense smoke and fire scenes, including a three-vision sensor head, a synchronous acquisition unit, an edge inference unit and an output interface; The three-vision sensor head includes a visible light camera, a thermal infrared camera, and a millimeter-wave radar, which are used to collect visible light data, thermal infrared data, and millimeter-wave data, respectively. The synchronous acquisition unit is connected to the three vision sensor head and the edge inference unit respectively, and is used to send visible light data, thermal infrared data and millimeter wave data corresponding to the same inference time to the edge inference unit. The edge inference unit is connected to the output interface and is used to construct three visual features based on visible light data, thermal infrared data, and millimeter wave data. Based on the sensor coarse installation prior, thermal infrared and millimeter-wave features are mapped to a virtual reference domain with visible light features as a reference, and the residual offset field and the corresponding confidence mask are determined within the local search range. Based on the residual offset field, thermal infrared features and millimeter wave features are sampled and aligned. The fusion weights are determined according to the time synchronization reliability, data quality of each mode, and confidence mask to fuse visible light features, thermal infrared aligned features, and millimeter wave aligned features to generate fire situation information and risk cost map. The output interface is used to output fire situation information and risk cost map.

[0006] In some embodiments, the three-vision sensor head further includes a mounting reference component, on which the visible light camera, the thermal infrared camera, and the millimeter-wave radar are fixedly mounted; The sensor coarse installation prior includes the intrinsic parameters of the visible light camera, the thermal infrared camera, and the millimeter-wave radar, as well as the extrinsic parameters or coordinate transformation relationships among the visible light camera, the thermal infrared camera, and the millimeter-wave radar.

[0007] In some embodiments, the synchronous acquisition unit includes data buffers for caching the visible light data, the thermal infrared data, and the millimeter wave data, respectively; The synchronous acquisition unit selects the corresponding visible light data, thermal infrared data, and millimeter wave data from the data buffer based on the target inference time and the timestamps or trigger signals corresponding to the visible light data, thermal infrared data, and millimeter wave data, respectively. The time deviation is determined based on the timestamps or trigger signals corresponding to the selected visible light data, thermal infrared data, and millimeter wave data, and the time synchronization reliability is determined based on the time deviation. The synchronous acquisition unit sends the selected visible light data, thermal infrared data, millimeter wave data, and the time synchronization reliability to the edge inference unit.

[0008] In some embodiments, the edge inference unit determines the local search range based on the sensor coarse-mount prior, and constructs cross-modal local correlations between the visible light feature and the thermal infrared feature, and between the visible light feature and the millimeter-wave feature, respectively, within the virtual reference domain; The thermal infrared residual offset field, the millimeter-wave residual offset field, and the corresponding confidence mask are determined based on the cross-modal local correlator. The offset of the thermal infrared residual offset field and the millimeter-wave residual offset field are constrained by the local search range. The confidence mask is a pixel-level continuous confidence mask.

[0009] In some embodiments, the edge inference unit determines the sampling position based on the thermal infrared residual offset field and the millimeter-wave residual offset field, and uses bilinear deformable sampling to sample the thermal infrared feature and the millimeter-wave feature respectively to obtain the thermal infrared alignment feature and the millimeter-wave alignment feature; For thermal infrared features or millimeter-wave features whose confidence values ​​at spatial locations are lower than a preset threshold, the corresponding bilinear deformable sampling is suppressed, or the fusion weight of the corresponding mode at the spatial location is reduced.

[0010] In some embodiments, the edge inference unit determines the quality of each modal data based on the clarity and / or smoke obscuration of the visible light data, the effective pixel ratio and / or saturation ratio of the thermal infrared data, and the signal-to-noise ratio and / or dot density of the millimeter wave data. When the quality of any modal data is lower than a preset quality threshold, the corresponding modality is downweighted or masked, or inference is performed based on one or two unmasked modalities to generate the fire situation information and the risk cost map, and the corresponding modal downgrade status is sent to the output interface.

[0011] On the other hand, the present invention also provides a three-vision virtual alignment situational awareness method for dense smoke and fire scenes, the method comprising: Acquire visible light data, thermal infrared data, and millimeter wave data corresponding to the same inference time, and determine the time synchronization reliability based on the visible light data, thermal infrared data, and millimeter wave data; Visible light features, thermal infrared features, and millimeter wave features are constructed based on the visible light data, thermal infrared data, and millimeter wave data. Based on the sensor coarse installation prior, the thermal infrared features and the millimeter-wave features are mapped to a virtual reference domain with the visible light features as a reference, and the residual offset field and the corresponding confidence mask are determined within the local search range; The thermal infrared feature and the millimeter-wave feature are sampled and aligned according to the residual offset field to obtain thermal infrared aligned features and millimeter-wave aligned features; The quality of each modal data is determined based on the visible light data, thermal infrared data, and millimeter wave data. The fusion weight is determined based on the time synchronization reliability, the quality of each modal data, and the confidence mask. The visible light features, thermal infrared alignment features, and millimeter wave alignment features are fused based on the fusion weight to generate fire situation information and risk cost map.

[0012] In some embodiments, determining the residual offset field and the corresponding confidence mask within the local search range includes: determining a local search center based on the sensor coarse installation prior and the residual offset field at a low-resolution feature scale; and within the local search range surrounding the local search center, calculating the cross-modal correlations between the visible light feature and the thermal infrared feature, and between the visible light feature and the millimeter-wave feature, respectively, to obtain the candidate displacement probability distribution. The initial displacement is determined based on the candidate displacement probability distribution, and the thermal infrared residual offset field and millimeter wave residual offset field are determined based on the initial displacement and the bounded correction amount. Pixel-level continuous confidence masks corresponding to the thermal infrared residual offset field and the millimeter-wave residual offset field are determined based on at least two of the following: initial effective mask, maximum correlation value, candidate displacement probability distribution entropy, mask logic value, and compensated cross-modal consistency error.

[0013] In some embodiments, the sampling alignment of the thermal infrared feature and the millimeter-wave feature based on the residual offset field includes: Based on the thermal infrared residual offset field and the millimeter-wave residual offset field, bilinear deformable sampling is used to sample the thermal infrared feature and the millimeter-wave feature respectively to obtain the thermal infrared alignment feature and the millimeter-wave alignment feature; Based on the confidence mask, the thermal infrared alignment feature and the millimeter wave alignment feature are respectively determined as high-confidence alignment feature, medium-confidence alignment feature, or low-confidence alignment feature; The high-confidence alignment features are preserved, the medium-confidence alignment features are attenuated, and the low-confidence alignment features are rejected from fusion. The time synchronization reliability is determined based on the time deviation between the visible light data, the thermal infrared data, and the millimeter wave data. The fusion weight is determined based on the time synchronization reliability, the quality of each modal data, and the confidence mask.

[0014] In some embodiments, the fire situation information includes the fire source status, obstacle status, and free space; The fire source state, the obstacle state, and the free space are transformed into the robot coordinate system, distance estimation and risk aggregation are performed, and the risk cost map is generated. When the data quality of any modality is lower than the first preset quality threshold for multiple consecutive inference cycles, the upper limit of the fusion weight of the corresponding modality is limited and the modality degradation status is output. When the quality of all modal data is lower than the second preset quality threshold or the time synchronization reliability is lower than the preset synchronization reliability threshold, the perception unavailable state is output. The fire situation information, the risk cost map, the modal degradation state, or the perception unavailability state are sent as a unified perception data frame to the robot control system.

[0015] The three-vision virtual alignment situational awareness equipment and method for dense smoke and fire scenes of the present invention have the following advantages over the prior art: (1) By synchronously acquiring visible light data, thermal infrared data and millimeter wave data, based on the sensor coarse installation prior, the thermal infrared features and millimeter wave features are mapped to a virtual reference domain with visible light features as the reference, and the residual offset field and corresponding confidence mask are determined within the local search range to sample and align the thermal infrared features and millimeter wave features; at the same time, the fusion weight is determined by combining the time synchronization reliability, the data quality of each modality and the confidence mask, and the reliable three visual features are fused to generate fire situation information and risk cost map, thereby reducing the impact of cross-modal spatial misalignment and low-quality information error fusion caused by equipment vibration, high temperature deformation and dense smoke, water mist and other factors on the judgment of fire source location, obstacle boundary and passage risk, and improving the accuracy and reliability of fire rescue decision information; (2) By acquiring visible light data, thermal infrared data and millimeter wave data at the same inference time, the corresponding three-vision features are constructed, and the residual offset field and confidence mask are determined in the virtual reference domain based on the sensor coarse installation prior. The thermal infrared features and millimeter wave features are sampled and aligned. Furthermore, the fusion weight is determined by combining the time synchronization reliability, the data quality of each modality and the confidence mask. The visible light features, thermal infrared aligned features and millimeter wave aligned features are fused to generate fire situation information and risk cost map, thereby reducing the cross-modal spatial misalignment caused by fire equipment vibration and high temperature deformation, as well as the impact of low-quality data caused by dense smoke, water mist, high temperature, etc. on the fire source location, obstacle boundary and passage risk judgment, and improving the accuracy and reliability of fire situation perception results. (3) The local search center is determined by coarse installation prior and residual offset field of low resolution feature scale. The cross-modal correlation between visible light features and thermal infrared features and millimeter wave features is calculated only within the constrained local search range. The initial displacement is determined based on the candidate displacement probability distribution. The thermal infrared residual offset field and millimeter wave residual offset field are obtained by combining the bounded correction amount. At the same time, a pixel-level continuous confidence mask is generated based on at least two of the initial effective mask, maximum correlation value, candidate displacement probability distribution entropy, mask logic value and compensated cross-modal consistency error. This enables the reliable alignment position and corresponding reliability of each mode to be determined for local spatial misalignment caused by equipment vibration, high temperature deformation and different sensor imaging mechanisms. This avoids the common field of view, occlusion or lack of effective echo area being mistakenly regarded as an alignable area. It provides spatially consistent and reliable input for three-modal feature fusion and improves the accuracy of fire source location, obstacle boundary and passage risk judgment. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a structural diagram of a three-vision virtual alignment situational awareness equipment for dense smoke and fire scenes according to the present invention. Figure 2 This is a flowchart of a three-vision virtual alignment situational awareness method for dense smoke fire scenes according to the present invention. Figure 3 This is a schematic diagram of the three-vision data synchronization and virtual vision tensor construction process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the three-vision virtual alignment situational awareness network according to an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the execution steps and sequential relationships of the online situational awareness method according to an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the training, compression, deployment, and version verification process of the three-vision virtual alignment situational awareness network according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] Existing firefighting robots or individual firefighter equipment typically acquire fire scene information using visible light cameras, thermal infrared cameras, and millimeter-wave radar, and then project and overlay the data from each modality based on pre-calibrated sensor installation relationships. However, in fire scene environments with dense smoke, high temperatures, water mist, and equipment vibration, the actual relative positions between sensors may change, causing the fixed installation relationships to fail to accurately reflect the actual spatial correspondence of each modality's data. Furthermore, different modal data may exhibit varying quality due to smoke obstruction, heat saturation, or weakened echoes. Directly fusing misaligned or low-quality modal data can easily lead to errors in determining the location of the fire source, obstacle boundaries, and navigable areas.

[0020] Please see Figure 1 The present invention provides a three-vision virtual alignment situational awareness device for dense smoke and fire scenes, including a three-vision sensor head, a synchronous acquisition unit, an edge inference unit and an output interface.

[0021] The three-vision sensor head includes a visible light camera, a thermal infrared camera, and a millimeter-wave radar, which are used to collect visible light data, thermal infrared data, and millimeter-wave data, respectively.

[0022] It is understood that visible light data is used to characterize the color, texture, scene semantics, and visible obstacle boundaries of the fire scene; thermal infrared data is used to characterize temperature distribution, high-temperature areas, and temperature rise trends; and millimeter-wave data is used to characterize target distance, orientation, velocity, obstacles, and spatial structure. The three sensors can be fixedly mounted on the same mounting reference, ensuring that the effective sensing areas of the three sensors at least partially overlap.

[0023] The synchronous acquisition unit is connected to the three vision sensor head and the edge inference unit respectively, and is used to send visible light data, thermal infrared data and millimeter wave data corresponding to the same inference time to the edge inference unit.

[0024] For example, the synchronous acquisition unit can cache the data of each modality and select data frames with small time differences based on the target inference time; for sensors with different sampling frequencies, data frames adjacent to the target inference time can be selected to form a three-modal data group for the same inference cycle.

[0025] The edge inference unit is connected to the output interface and is used to construct three visual features based on visible light data, thermal infrared data, and millimeter wave data.

[0026] For example, the edge inference unit converts visible light data, thermal infrared data, and millimeter wave data into input data suitable for feature extraction, and extracts visible light features, thermal infrared features, and millimeter wave features to retain the characterization information of each modality for fire source, obstacle, free space, and smoke risk.

[0027] The edge inference unit, based on the sensor coarse-mount prior, maps thermal infrared and millimeter-wave features to a virtual reference domain with visible light features as a reference, and determines the residual offset field and the corresponding confidence mask within the local search range.

[0028] It can be understood that the coarse-mount prior of the sensors includes the intrinsic parameters of each sensor and the initial extrinsic parameters or coordinate transformation relationships between the sensors. The virtual reference domain is used to establish a unified spatial reference among the three modal features, the local search range is used to limit the search positions of the thermal infrared features and millimeter-wave features relative to the visible light features, the residual offset field is used to characterize the local spatial offset that still exists after the coarse-mount prior mapping, and the confidence mask is used to characterize the alignment reliability of the corresponding positions.

[0029] The edge inference unit samples and aligns thermal infrared and millimeter-wave features based on the residual offset field, and determines the fusion weights based on time synchronization reliability, data quality of each modality, and confidence mask, so as to fuse visible light features, thermal infrared aligned features, and millimeter-wave aligned features to generate fire situation information and risk cost map.

[0030] It is understandable that time synchronization reliability is used to characterize the degree of correspondence between the three modal data in time; the quality of each modal data is used to characterize the usability of the corresponding modal data; and the confidence mask is used to characterize the spatial alignment reliability of the corresponding location. Fire situation information may include fire source status, obstacle status, free space, smoke risk characterization, and uncertainty, and the risk cost map is used to characterize the passage risk corresponding to different locations in the fire.

[0031] The output interface is used to output fire situation information and risk cost map.

[0032] It is understandable that the output interface is connected to the fire robot control system, individual soldier terminal or remote monitoring terminal so that the fire situation information and risk cost map can serve as the basis for rescuers to judge the location of the fire source, the boundary of obstacles and the route.

[0033] This embodiment synchronously acquires three-modal data, performs residual offset compensation on thermal infrared and millimeter-wave features within a virtual reference domain, and combines time synchronization reliability, data quality of each modality, and confidence mask for weighted fusion. This reduces spatial misalignment and erroneous information fusion caused by equipment vibration, high-temperature deformation, and local modal failures, thereby improving the reliability of fire situation information and risk cost maps.

[0034] In some embodiments, based on the above embodiments, the three-vision sensor head further includes a mounting reference component, on which the visible light camera, the thermal infrared camera, and the millimeter-wave radar are fixedly mounted.

[0035] It is understood that the mounting reference component is used to provide a common mechanical mounting reference for the visible light camera, thermal infrared camera, and millimeter-wave radar, enabling a relatively stable mounting relationship among the three sensors. The effective sensing areas of the three sensors at least partially overlap, allowing visible light data, thermal infrared data, and millimeter-wave data to detect the same fire area. The mounting reference component can be determined based on the installation position and sensing direction of the three sensors, fixing them under the same mounting reference.

[0036] In this embodiment, the three sensors are fixedly mounted on the same mounting reference, and it is not required that the three sensors maintain a pixel-level spatial correspondence at all times during equipment operation. Since the movement of the equipment, mechanical vibration, and high temperatures in the fire will cause slight installation offsets in the sensors, the common installation relationship provided by the mounting reference is mainly used to form a virtual reference domain and limit the local search range. The remaining local spatial offset is further compensated by the edge inference unit to determine the residual offset field.

[0037] The sensor coarse installation prior includes the intrinsic parameters of the visible light camera, the thermal infrared camera, and the millimeter-wave radar, as well as the extrinsic parameters or coordinate transformation relationships among the visible light camera, the thermal infrared camera, and the millimeter-wave radar.

[0038] For example, the sensor coarse installation prior is determined based on the factory calibration results of the three sensors, the mechanical dimensions of the mounting reference component, and the sensor installation relationship in the fire robot configuration file, and is pre-stored in the edge inference unit or a storage component connected to the edge inference unit. The intrinsic parameters are used to describe the data acquisition coordinate relationship of each sensor itself, and the extrinsic parameters or coordinate transformation relationship are used to describe the relative positional relationship and coordinate transformation relationship between different sensors.

[0039] After reading the sensor coarse-mount prior, the edge inference unit maps the thermal infrared and millimeter-wave features to a virtual reference domain with visible light features as a reference, and determines the initial search area corresponding to the thermal infrared and millimeter-wave features based on the sensor coarse-mount prior. The sensor coarse-mount prior is used to limit the physically feasible search range of the thermal infrared and millimeter-wave features relative to the visible light features, without requiring pixel-level precise alignment between the thermal infrared, millimeter-wave, and visible light features directly through the sensor coarse-mount prior.

[0040] For example, the coarse sensor installation prior is obtained from factory calibration, or it can be determined based on the mechanical dimensions of the mounting reference component and the installation positions of the three sensors, or it can be provided by the configuration file corresponding to the fire-fighting robot. When sensors are replaced, sensor installation positions are adjusted, or the fire-fighting robot platform is replaced, the coarse sensor installation prior can be updated according to the new sensor installation relationship so that the local search range corresponds to the current installation state of the three vision sensor heads.

[0041] In this embodiment, the coarse sensor installation prior is only used to provide the initial spatial correspondence and local search constraints among the three sensors. The edge inference unit also determines the residual offset field and confidence mask based on the local correspondence between visible light features, thermal infrared features, and millimeter-wave features, and performs subsequent sampling alignment based on the residual offset field. Therefore, when vibration or high-temperature deformation during device operation causes the actual relative position of the sensors to deviate from the initial installation relationship, the deviation can still be locally compensated, and the confidence mask can be used to distinguish between reliable alignment areas and low-reliability areas.

[0042] This embodiment provides a common mechanical mounting reference for visible light cameras, thermal infrared cameras, and millimeter-wave radar by installing a reference component. It also establishes an initial spatial correspondence between the three sensors through a coarse-grained installation prior consisting of intrinsic parameters, extrinsic parameters, or coordinate transformation relationships. Based on this, it limits the subsequent residual offset search range, thereby providing input basis for virtual alignment of thermal infrared and millimeter-wave features, determination of confidence masks, and multimodal fusion.

[0043] In some embodiments, based on the above embodiments, the synchronous acquisition unit includes data buffers for caching the visible light data, the thermal infrared data, and the millimeter wave data, respectively.

[0044] It is understood that the data buffer includes a visible light buffer, a thermal infrared buffer, and a millimeter-wave buffer, with the acquisition threads of the three types of sensors operating independently. When each data frame enters the corresponding data buffer, it saves the sensor's native timestamp, the edge inference unit's received timestamp, the frame sequence number, the data validity flag, and the device status word. The synchronous acquisition unit can establish a unified time reference using at least one of hardware triggering, a precision time protocol, a pulse-per-second signal, or a unified industrial control computer clock to reduce the time deviation between the three types of data.

[0045] The synchronous acquisition unit selects the corresponding visible light data, thermal infrared data, and millimeter wave data from the data buffer based on the target inference time and the timestamps or trigger signals corresponding to the visible light data, thermal infrared data, and millimeter wave data, respectively.

[0046] For example, the synchronous acquisition unit uses the target inference time corresponding to the current inference cycle as an index to select the data frame with the smallest time difference from the target inference time from the visible light buffer, the thermal infrared buffer, and the millimeter-wave buffer, respectively, to form a three-modal data group corresponding to the same inference time. For millimeter-wave radar with a sampling frequency lower than the inference frequency, neighboring frames can be selected for millimeter-wave data adjacent to the target inference time; for the temperature matrix continuously output by the thermal infrared camera, time interpolation can be performed to form auxiliary input data, but the highest temperature of the fire source, the alarm temperature, and the extreme value of temperature rise are still determined by the most recent valid measured frame, and the interpolation result does not replace the original temperature measurement value.

[0047] The time deviation is determined based on the timestamps or trigger signals corresponding to the selected visible light data, thermal infrared data, and millimeter wave data, and the time synchronization reliability is determined based on the time deviation.

[0048] It is understood that the synchronous acquisition unit determines the time deviation based on the time difference between the selected visible light data, thermal infrared data, and millimeter wave data and the target inference time, as well as the time difference between each modal data. The time synchronization reliability characterizes the degree of correspondence between the three modal data sets in the time dimension; a smaller time deviation corresponds to higher time synchronization reliability, and a larger time deviation corresponds to lower time synchronization reliability. The time synchronization reliability serves as input for determining the fusion weights, thereby reducing the bias in judging the fire source location, obstacle location, and free space caused by the fusion of data from different time points.

[0049] The synchronous acquisition unit sends the selected visible light data, thermal infrared data, millimeter wave data, and the time synchronization reliability to the edge inference unit.

[0050] For example, after receiving the trimodal data set and the time synchronization reliability, the edge inference unit performs preprocessing, feature construction, virtual alignment, and fusion processing on the visible light data, the thermal infrared data, and the millimeter-wave data. The time synchronization reliability, along with the subsequently obtained quality of each modality's data and the confidence mask, is used to determine the fusion weights.

[0051] This embodiment enables the edge inference unit to distinguish the time correspondence of the three modal data by separately caching the three modal data, selecting the data frame corresponding to the time based on the target inference time, calculating the time deviation and generating time synchronization reliability, and reducing the impact of time inconsistency data on fire situation information and risk cost map during the fusion process.

[0052] In some embodiments, based on the above embodiments, the edge inference unit determines the local search range according to the sensor coarse installation prior, and constructs cross-modal local correlations between the visible light feature and the thermal infrared feature, and between the visible light feature and the millimeter wave feature, respectively, within the virtual reference domain.

[0053] For example, the edge inference unit uses the virtual reference domain where the visible light features are located as a unified reference. At multiple feature scales, it maps the thermal infrared features and millimeter-wave features to corresponding coarsely predicted positions based on the sensor's coarse installation prior. For each visible light feature position, the coarsely predicted position and the residual offset obtained from upsampling at the low-resolution feature scale are used as the local search center, and a bounded local search window is established around the local search center. The edge inference unit calculates the normalized cross-modal correlations between the visible light features and the thermal infrared features, and between the visible light features and the millimeter-wave features, at each candidate position, and organizes the correlations corresponding to each candidate position into the cross-modal local correlation body according to spatial location and candidate displacement.

[0054] The multiple feature scales include three feature scales ranging from low resolution to high resolution. The residual offset field of the low-resolution feature scale serves as input for determining the local search center at the higher-resolution feature scale, enabling the higher-resolution feature scale to continue determining the residual offset within the corresponding local search range. For the lowest resolution feature scale, the initial residual offset can be set to zero.

[0055] The local search range is defined by the coarse-mounted prior of the sensor, which limits the candidate displacements to within the physically feasible range of thermal infrared or millimeter-wave features relative to visible light features. By constructing a cross-modal local correlation volume within the local search range, rather than searching for candidate positions across the entire virtual reference domain, the possibility of erroneous correspondences between different fire targets can be reduced, and the computational cost of the residual offset field can be decreased.

[0056] The thermal infrared residual offset field, the millimeter-wave residual offset field, and the corresponding confidence mask are determined based on the cross-modal local correlator. The offset of the thermal infrared residual offset field and the millimeter-wave residual offset field are constrained by the local search range. The confidence mask is a pixel-level continuous confidence mask.

[0057] For example, the edge inference unit normalizes the correlation within the local search range into a candidate displacement probability distribution, and determines the initial displacement based on the candidate displacement probability distribution. The maximum correlation value of the candidate displacement probability distribution is used to characterize the matching strength of the candidate positions, and the entropy of the candidate displacement probability distribution is used to characterize the concentration of candidate displacements. When multiple candidate positions have similar correlations, the entropy of the candidate displacement probability distribution increases, and the cross-modal correspondence of the corresponding positions becomes unclear.

[0058] The edge inference unit inputs at least two of the following into the offset mask predictor: the cross-modal local correlator, visible light features, thermal infrared or millimeter-wave features, initial displacement, coarse installation displacement, residual offset at the low-resolution feature scale, initial effective mask, and millimeter-wave distance prior. The offset mask predictor is composed of depthwise separable convolutions and outputs bounded corrections and mask logic values ​​corresponding to the thermal infrared features, and bounded corrections and mask logic values ​​corresponding to the millimeter-wave features. The edge inference unit combines the initial displacement, residual offset at the low-resolution feature scale, and bounded corrections to obtain the thermal infrared residual offset field and the millimeter-wave residual offset field, and restricts the offset to the corresponding local search range.

[0059] The initial effective mask is used to mark locations outside the common field of view, locations of invalid thermal infrared pixels, and locations with no effective millimeter-wave echoes. The edge inference unit generates a compensated pre-confidence mask based on the initial effective mask, mask logic value, maximum correlation value, and candidate displacement probability distribution entropy, so that locations with no physical correspondence or unclear candidate correspondence have lower confidence values.

[0060] After initial sampling of thermal infrared and millimeter-wave features based on the residual offset fields of thermal infrared and millimeter-wave, respectively, the edge inference unit calculates the consistency error between visible light features and thermal infrared sampled features, and the consistency error between visible light features and millimeter-wave sampled features. Based on these consistency errors, the pre-compensation confidence mask is attenuated to obtain the confidence mask. The confidence mask outputs continuous confidence values ​​at each spatial location, characterizing the reliability of the effective correspondence between the thermal infrared or millimeter-wave features and the visible light features at the corresponding locations.

[0061] The training process of the offset mask predictor includes acquiring three-vision synchronized samples labeled with fire source status, temperature information, obstacle occupancy, free space, and risk cost icons. The samples are then modified by adding smoke degradation, low illumination, thermal infrared temperature drift or saturation, missing or multipath virtual points on millimeter waves, cross-modal temporal offset, and spatial misalignment. The processed three-vision synchronized samples are input into a three-vision virtual alignment situational awareness network including the offset mask predictor. The task loss is calculated based on fire source classification and localization, temperature regression, obstacle occupancy, free space, and risk prediction results. This loss is then combined with cross-modal consistency loss, offset smoothing loss, and constraints on areas without corresponding regions to perform a reverse update of the offset mask predictor.

[0062] Through the above training process, the thermal infrared residual offset field and the millimeter-wave residual offset field are jointly constrained by fire source classification and localization, temperature regression, obstacle occupancy, free space, and risk prediction tasks. The confidence mask is jointly constrained by the initial effective mask, local correlation, and post-compensation consistency. For flame-transparent areas, radar-reflection-free areas, obstructed boundaries, or repetitive structure areas, the offset mask predictor allows the output of low confidence values ​​without forcibly determining the location as a valid corresponding location.

[0063] This embodiment defines the local search range through a coarse sensor installation prior, obtains candidate displacement information using cross-modal local correlation volume, and determines the thermal infrared residual offset field, millimeter-wave residual offset field, and pixel-level continuous confidence mask constrained by the local search range through an offset mask predictor. Then, the confidence mask is constrained by the initial effective mask, local correlation, and post-compensation consistency, providing spatial correspondence and reliability basis for the sampling alignment and fusion weight determination of thermal infrared and millimeter-wave features.

[0064] In some embodiments, based on the above embodiments, the edge inference unit determines the sampling position according to the thermal infrared residual offset field and the millimeter wave residual offset field, and uses bilinear deformable sampling to sample the thermal infrared feature and the millimeter wave feature respectively to obtain the thermal infrared alignment feature and the millimeter wave alignment feature.

[0065] It can be understood that the edge inference unit uses the visible light feature positions within the virtual reference domain as reference positions, and determines the target sampling position in the thermal infrared or millimeter-wave features based on the thermal infrared residual offset field or millimeter-wave residual offset field corresponding to the reference position. Since the determined target sampling position can be located between feature grids, the edge inference unit performs bilinear interpolation based on the adjacent feature positions around the target sampling position and their corresponding feature values ​​to obtain the thermal infrared or millimeter-wave aligned feature corresponding to the visible light feature position.

[0066] Specifically, the thermal infrared residual offset field is used to locally correct the initial thermal infrared mapping position obtained based on the sensor's coarse installation prior, while the millimeter-wave residual offset field is used to locally correct the initial millimeter-wave mapping position obtained based on the sensor's coarse installation prior. By sampling based on the thermal infrared residual offset field and the millimeter-wave residual offset field respectively, a correspondence can be established between the thermal infrared feature, the millimeter-wave feature, and the visible light feature within the virtual reference domain, without the need to directly superimpose the three modal features at fixed positions.

[0067] For thermal infrared features or millimeter-wave features whose confidence values ​​at spatial locations are lower than a preset threshold, the corresponding bilinear deformable sampling is suppressed, or the fusion weight of the corresponding mode at the spatial location is reduced.

[0068] For example, the edge inference unit reads the thermal infrared confidence mask corresponding to the thermal infrared features and the millimeter-wave confidence mask corresponding to the millimeter-wave features, and compares the confidence value of each spatial location with a preset threshold. The preset threshold is used to distinguish between locations with a relatively reliable cross-modal spatial correspondence and locations with an unreliable cross-modal spatial correspondence.

[0069] When the confidence value of the thermal infrared confidence mask or millimeter-wave confidence mask at a certain spatial location is lower than a preset threshold, it indicates that there are common out-of-field regions, invalid thermal infrared pixels, no effective millimeter-wave echoes, occlusion boundaries, repetitive structures, or low consistency after compensation at that spatial location. In this case, the edge inference unit does not use the thermal infrared or millimeter-wave features corresponding to that spatial location for alignment feature generation, in order to suppress low-reliability sampling results from entering the subsequent fusion process.

[0070] In another embodiment, the edge inference unit can still generate thermal infrared alignment features or millimeter-wave alignment features corresponding to low-confidence spatial locations, but reduces the fusion weight of the corresponding mode at that spatial location when determining the fusion weight. Thus, while retaining other reliable modal information, the impact of low-confidence thermal infrared features or millimeter-wave features on the fire source location, obstacle boundaries, free space, and risk cost map can be reduced.

[0071] The bilinear deformable sampling, confidence mask determination, and fusion weight adjustment are executed sequentially within each inference cycle. The edge inference unit sends the sampled thermal infrared alignment features and millimeter-wave alignment features, along with the confidence information corresponding to the spatial location, to the subsequent fusion processing as input for generating fire situation information and risk cost maps.

[0072] This embodiment determines the adaptive sampling position based on the thermal infrared residual offset field and the millimeter-wave residual offset field. It obtains thermal infrared alignment features and millimeter-wave alignment features corresponding to the visible light feature space through bilinear deformable sampling. It also suppresses low-reliability sampling or reduces the fusion weight of the corresponding mode based on pixel-level continuous confidence mask, thereby reducing the situation judgment deviation caused by local misalignment, lack of effective correspondence and low-reliability feature participation in fusion.

[0073] In some embodiments, based on the above embodiments, the edge inference unit determines the quality of each modal data according to the clarity and / or smoke obscuration of the visible light data, the effective pixel ratio and / or saturation ratio of the thermal infrared data, and the signal-to-noise ratio and / or dot density of the millimeter wave data.

[0074] For example, the edge inference unit constructs quality description information for visible light data, thermal infrared data, and millimeter-wave data, respectively. The quality description information for visible light data includes at least image sharpness and smoke obscuration. Image sharpness characterizes the discernibility of scene edges, textures, and target contours in the image, while smoke obscuration characterizes the degree to which smoke obscures visible light scene information. The quality description information for thermal infrared data includes at least the effective pixel ratio and saturation ratio. The effective pixel ratio characterizes the proportion of pixels that can provide effective temperature information, while the saturation ratio characterizes the proportion of pixels in the thermal infrared image that exceed the effective temperature measurement or effective response range. The quality description information for millimeter-wave data includes at least the signal-to-noise ratio (SNR) and point density. The SNR characterizes the strength of the effective echo relative to noise, while the point density characterizes the distribution of effective millimeter-wave detection points per unit space.

[0075] The edge inference unit inputs the quality description information corresponding to each modality into the modal quality gating network, and combines the feature information of each modality to obtain visible light quality scores, thermal infrared quality scores, and millimeter wave quality scores. The modal quality gating network includes a quality description information input part, a quality score calculation part, and a weight output part. Its input is the quality description information of each modality and the corresponding modal features, and its output is the original quality score characterizing the usability of each modality.

[0076] Furthermore, the edge inference unit combines the time synchronization reliability and the spatial alignment reliability reflected by the confidence mask with the original quality score to obtain a corrected quality score for each modality, and determines the fusion weight of each modality based on the corrected quality score. Specifically, when the visible light modality has not undergone cross-modal sampling alignment, its spatial alignment reliability can be handled according to the effective correspondence. For modalities with low time synchronization reliability, low modal quality, or low confidence values ​​corresponding to the confidence mask, the edge inference unit reduces their corrected quality score to minimize the impact of that modality on the fusion result.

[0077] The training process of the modal quality gating network includes acquiring simultaneous three-vision samples containing visible light data, thermal infrared data, and millimeter-wave data, and obtaining annotation information corresponding to fire source status, temperature information, obstacle occupancy, free space, and risk cost map. The training samples cover smokeless, light smoke, medium smoke, and heavy smoke fire environments, and also include data degradation scenarios such as visible light smoke obstruction, low illumination or overexposure, thermal infrared temperature drift or saturation, and missing millimeter-wave points or reduced signal-to-noise ratio.

[0078] Degraded three-vision synchronous samples are input into a three-vision virtual alignment situational awareness network, whereby the samples sequentially undergo three-vision feature construction, virtual alignment, modal quality gating, and fire situational output processing. The task loss is calculated based on fire source classification and location results, temperature information, obstacle occupancy results, free space results, and risk cost map. A quality gating loss is calculated based on the quality gating constraint that degraded modalities should receive lower fusion weights. The parameters of the modal quality gating network are updated based on the task loss and quality gating loss, enabling the network to learn the correspondence between data quality and fusion weights for each modality under different fire degradation conditions.

[0079] When the quality of any modal data is lower than a preset quality threshold, the corresponding modality is downweighted or masked, or inference is performed based on one or two unmasked modalities to generate the fire situation information and the risk cost map, and the corresponding modal downgrade status is sent to the output interface.

[0080] It is understandable that the edge inference unit compares the corrected quality score of each modality with a preset quality threshold. When the corrected quality score of any modality is lower than the preset quality threshold, the edge inference unit reduces the fusion weight of that modality or stops inputting the modality's features into subsequent fusion processing. Therefore, when visible light is severely obscured by dense smoke, thermal infrared light exhibits large-area saturation, or millimeter-wave effective echoes are insufficient, low-quality information from the corresponding modality is avoided from being used as reliable information in the fusion process.

[0081] In one specific embodiment, when the modal data quality of a certain modality is lower than a preset quality threshold for multiple consecutive inference cycles, the edge inference unit limits the upper limit of the fusion weight of that modality and outputs the corresponding degraded state. When the modal data quality recovers to above the recovery threshold for multiple consecutive inference cycles, the fusion weight of that modality is gradually restored. The quality threshold for entering the degraded state can be different from the quality threshold for recovering to the normal state, thereby reducing frequent switching of the degraded state caused by fluctuations in modal data quality around the threshold.

[0082] When one modality is masked, the edge inference unit performs inference based on the remaining two unmasked modalities. When two modalities are masked, the edge inference unit performs inference based on the remaining unmasked modality, and retains the corresponding availability or low confidence markers in the fire situation information and risk cost map. When the data quality of all modalities is lower than a preset quality threshold, or the time synchronization reliability is lower than a preset reliability threshold, the edge inference unit outputs a perceived unavailability state to avoid outputting unreliable situation results as valid evidence.

[0083] The output interface receives fire situation information, risk cost map, and modal degradation status, and sends them to the fire robot control system, individual soldier terminal, or remote monitoring terminal. The modal degradation status indicates the modality that is currently being downgraded or masked, enabling users to determine the location of the fire source, obstacle boundaries, passable areas, and the reliability of the risk cost map based on the currently valid modalities.

[0084] This embodiment determines the data quality of each modality based on visible light clarity and smoke obscuration, effective pixel ratio and saturation ratio of thermal infrared, millimeter wave signal-to-noise ratio and point density, and determines the fusion weight by combining time synchronization reliability and spatial alignment reliability. When the modal quality deteriorates, weight reduction, shielding, and single-mode or dual-mode inference are implemented to reduce erroneous information fusion caused by dense smoke obscuration, thermal infrared saturation, and weakened millimeter wave echoes. At the same time, the modal degradation status is output, providing a reliable basis for fire situation assessment and path decision-making.

[0085] Please see Figure 2 This invention also provides a three-vision virtual alignment situational awareness method for dense smoke fires. The method is executed by a three-vision virtual alignment situational awareness network installed in a firefighting robot, individual firefighter equipment, or edge inference device. The network includes a feature construction part that constructs visible light features, thermal infrared features, and millimeter-wave features respectively; a virtual alignment part for predicting residual offset fields and confidence masks; a quality gating part for determining the fusion weights of each modality; and a fusion and situational output part for outputting fire situational information and a risk cost map. The method includes:

[0086] Acquire visible light data, thermal infrared data, and millimeter wave data corresponding to the same inference time, and determine the time synchronization reliability based on the visible light data, thermal infrared data, and millimeter wave data.

[0087] It can be understood that, using the target inference time as an index, data that is temporally closest to the target inference time is selected from visible light data, thermal infrared data, and millimeter-wave data to form a three-modal data set. Based on the timestamp or trigger signal corresponding to each modal data, the time deviation between each modal data and the target inference time, as well as the time deviation between modalities, are determined, and the time synchronization reliability is determined based on the time deviation. The smaller the time deviation, the higher the time synchronization reliability.

[0088] Visible light features, thermal infrared features, and millimeter wave features are constructed based on the visible light data, thermal infrared data, and millimeter wave data.

[0089] For example, visible light data is processed into image data to obtain visible light features for characterizing color, contours, flame textures, obstacle boundaries, and scene semantics. Thermal infrared data is processed into temperature information to obtain thermal infrared features for characterizing temperature distribution, high-temperature regions, temperature gradients, and temperature rise changes. Millimeter-wave data is processed into spatial data to obtain millimeter-wave features for characterizing target distance, orientation, velocity, obstacles, and spatial structures. These three types of features can be output at multiple feature scales to adapt to the characterization needs of fire sources, obstacles, and passageways at different spatial scales.

[0090] Based on the sensor coarse installation prior, the thermal infrared features and the millimeter-wave features are mapped to a virtual reference domain with the visible light features as a reference, and the residual offset field and the corresponding confidence mask are determined within the local search range.

[0091] It can be understood that the coarse sensor installation prior includes the intrinsic parameters of each sensor and the extrinsic parameters or coordinate transformation relationships between sensors. Based on the coarse sensor installation prior, thermal infrared and millimeter-wave features are mapped to a virtual reference domain with visible light features as a reference, obtaining the coarse predicted positions of the thermal infrared and millimeter-wave features. Based on the coarse predicted positions, within the local search range defined by the coarse sensor installation prior, cross-modal local correlations between visible light and thermal infrared features, and between visible light and millimeter-wave features, are constructed respectively.

[0092] Based on the correlation of candidate positions in the cross-modal local correlator, candidate displacement information of thermal infrared and millimeter-wave features relative to visible light features is determined, and thermal infrared residual offset fields and millimeter-wave residual offset fields are obtained accordingly. The residual offset fields are used to characterize the local spatial offset that still exists after coarse-mounted a priori mapping. Thermal infrared confidence masks and millimeter-wave confidence masks are determined based on at least two of the following: the initial effective mask, the correlation of candidate positions, the distribution of candidate displacements, and the consistency after compensation. The confidence masks are pixel-level continuous confidence masks used to characterize the cross-modal alignment reliability of corresponding spatial positions.

[0093] The thermal infrared feature and the millimeter-wave feature are sampled and aligned based on the residual offset field to obtain thermal infrared aligned features and millimeter-wave aligned features.

[0094] For example, using the visible light feature position within the virtual reference domain as the reference position, the target sampling position of the thermal infrared feature is determined based on the thermal infrared residual offset field, and the target sampling position of the millimeter-wave feature is determined based on the millimeter-wave residual offset field. For the target sampling position located between feature grids, bilinear deformable sampling is performed based on the feature values ​​of its surrounding adjacent feature positions to obtain thermal infrared aligned features and millimeter-wave aligned features that have a spatial correspondence with the visible light features.

[0095] For spatial locations with low confidence values ​​in the confidence mask, the sampling alignment of the corresponding modal features can be suppressed, or the sampling results can be retained and the fusion weight of the corresponding modality can be reduced during subsequent fusion. This avoids forcibly identifying areas outside the common field of view, invalid pixel areas in thermal infrared, areas without effective echoes in millimeter waves, and occlusion boundary areas as the corresponding spatial regions.

[0096] The quality of each modal data is determined based on the visible light data, thermal infrared data, and millimeter wave data. The fusion weight is determined based on the time synchronization reliability, the quality of each modal data, and the confidence mask. The visible light features, thermal infrared alignment features, and millimeter wave alignment features are fused based on the fusion weight to generate fire situation information and risk cost map.

[0097] It is understandable that visible light data quality is determined based on the sharpness and smoke obstruction of visible light data, thermal infrared data quality is determined based on the effective pixel ratio and saturation ratio of thermal infrared data, and millimeter-wave data quality is determined based on the signal-to-noise ratio and point density of millimeter-wave data. The data quality of each modality, the reliability of time synchronization, and the spatial alignment reliability reflected by the confidence mask are all used as the basis for determining the fusion weights, so that modes with lower time correspondence, lower data quality, or lower spatial alignment reliability receive lower fusion weights.

[0098] At each spatial location, visible light features, thermal infrared alignment features, and millimeter wave alignment features are fused according to their corresponding fusion weights to obtain fused features. Based on the fused features, fire situation information such as fire source status, obstacle status, free space, smoke risk characterization, and uncertainty is output. Risk aggregation is performed according to the spatial distribution of fire source, obstacle, free space, and smoke risk to generate a risk cost map.

[0099] When the data quality of any modality falls below a preset quality threshold, the fusion weight of that modality is reduced or it is masked. If one modality is masked, inference is performed based on the remaining two modalities. If two modalities are masked, inference is performed based on the remaining modality, and the corresponding modality degradation status is output. When the data quality or time synchronization reliability of all modalities falls below a preset threshold, a perceived unavailable status is output.

[0100] The training process of the three-vision virtual aligned situational awareness network includes collecting and labeling three-vision synchronized samples, and dividing the samples into training, validation, and test sets according to independent spaces. Smoke degradation, low illumination, thermal infrared temperature drift or saturation, missing or multipath virtual points on millimeter waves, cross-modal time shift, and spatial misalignment are randomly added to the training samples to form training samples under different fire conditions.

[0101] The training samples are input into the three-vision virtual alignment situational awareness network, which sequentially performs feature construction, virtual alignment, quality gating, feature fusion, and fire situational awareness output. The task loss is calculated based on fire source detection results, temperature information, obstacle occupancy results, and risk cost map results. This loss is then combined with cross-modal consistency loss, offset smoothing loss, quality gating constraints, and knowledge distillation loss for joint training to update the network parameters. After training, the network can undergo channel pruning, structural reparameterization, and mixed precision quantization to adapt to the computational resources of edge inference devices.

[0102] This embodiment acquires three-modal data synchronously, constructs multimodal features, determines the residual offset field and confidence mask in the virtual reference domain, and completes the sampling alignment of thermal infrared features and millimeter-wave features based on the residual offset field. Then, it performs weighted fusion by combining time synchronization reliability, modal data quality and spatial alignment reliability, which can reduce the erroneous fusion caused by changes in the relative position of sensors, local spatial misalignment and modal data degradation, and improve the reliability of fire situation information and risk cost map.

[0103] In some embodiments, based on the above embodiments, determining the residual offset field and the corresponding confidence mask within the local search range includes: The local search center is determined based on the sensor coarse installation prior and the residual offset field at a low-resolution feature scale.

[0104] It is understandable that the edge inference unit first maps the thermal infrared and millimeter-wave features to a virtual reference domain with visible light features as a reference, based on the sensor's coarse installation prior, to obtain the coarse predicted position corresponding to each spatial location. In the feature scale processing from low to high resolution, the residual offset field determined by the low-resolution feature scale is mapped to the higher-resolution feature scale, and together with the coarse predicted position, determines the local search center of the higher-resolution feature scale. For the lowest resolution feature scale, the initial value of the residual offset field can be set to zero, and the coarse predicted position can be used as the local search center.

[0105] The local search center is used to provide initial positional constraints for cross-modal feature matching at the current feature scale. By using the residual offset field of the low-resolution feature scale as the search basis for the higher-resolution feature scale, the approximate offset direction can be determined first in a larger spatial range, and then a more refined local offset can be determined in a smaller spatial range, thereby reducing the computational cost and incorrect matching caused by global search in the entire virtual reference domain.

[0106] Within the local search range around the local search center, the cross-modal correlations between the visible light feature and the thermal infrared feature, and between the visible light feature and the millimeter wave feature, are calculated to obtain the candidate displacement probability distribution.

[0107] For example, the edge inference unit uses the locations of each visible light feature as a reference and selects multiple candidate locations for thermal infrared features and multiple candidate locations for millimeter-wave features within a local search range around the corresponding local search center. The edge inference unit calculates the cross-modal correlation between the visible light feature and each thermal infrared candidate feature, as well as the cross-modal correlation between the visible light feature and each millimeter-wave candidate feature, and normalizes the correlation corresponding to each candidate location to obtain the thermal infrared candidate displacement probability distribution and the millimeter-wave candidate displacement probability distribution.

[0108] The candidate displacement probability distribution is used to characterize the relative probability that each candidate displacement forms a valid cross-modal correspondence with the current visible light feature position. The higher the correlation between candidate positions, the greater the probability of the candidate displacement corresponding to that position. When multiple candidate positions have close correlations, it indicates that the cross-modal correspondence of that spatial position is not clear enough, and the confidence level of that position can be judged by combining the candidate displacement probability distribution entropy.

[0109] The initial displacement is determined based on the candidate displacement probability distribution, and the thermal infrared residual offset field and millimeter-wave residual offset field are determined based on the initial displacement and the bounded correction amount.

[0110] It is understood that the edge inference unit determines the initial thermal infrared displacement based on the probability distribution of candidate thermal infrared displacements, and determines the initial millimeter-wave displacement based on the probability distribution of candidate millimeter-wave displacements. The initial displacement can be the candidate displacement with the highest probability, or it can be determined based on each candidate displacement and its corresponding probability.

[0111] The edge inference unit inputs cross-modal local correlations, visible light features, thermal infrared features or millimeter-wave features, initial displacement, coarse installation displacement, residual offset fields at low-resolution feature scales, and the initial effective mask into the offset mask predictor. The offset mask predictor performs feature processing on the input information, outputting bounded correction values ​​and mask logic values ​​corresponding to the thermal infrared features, and bounded correction values ​​and mask logic values ​​corresponding to the millimeter-wave features, respectively. The edge inference unit determines the thermal infrared residual offset field and the millimeter-wave residual offset field based on the initial displacement and bounded correction values, ensuring that the offset of the residual offset field is within the corresponding local search range.

[0112] The bounded correction amount is used to further locally correct the initial displacement, preventing the residual offset field from exceeding the physically feasible range defined by the prior knowledge of the sensor coarse installation. The mask logic value is used to characterize whether the corresponding spatial location has the conditions to form a valid cross-modal correspondence, and serves as the input for determining the pixel-level continuous confidence mask.

[0113] The training process of the offset mask predictor includes acquiring simultaneous training samples from three vision systems, including visible light data, thermal infrared data, and millimeter-wave data, and obtaining annotation information corresponding to fire source status, temperature information, obstacle occupancy, free space, and risk cost maps. Degradation conditions such as cross-modal spatial misalignment, cross-modal temporal shift, smoke obstruction, low illumination, thermal infrared temperature drift or saturation, and missing or multipath virtual points on millimeter-wave data are added to the training samples to form training data under different fire conditions.

[0114] Training data is input into a three-vision virtual alignment situational awareness network containing an offset mask predictor. Feature construction, cross-modal local correlation, residual offset prediction, confidence mask determination, sampling alignment, quality gating, and feature fusion are performed sequentially. The task loss is determined based on fire source classification and localization results, temperature information, obstacle occupancy results, free space results, and risk cost map results. The offset mask predictor is then trained using cross-modal consistency loss, offset smoothing loss, and constraints for areas without corresponding regions to update its parameters.

[0115] Pixel-level continuous confidence masks corresponding to the thermal infrared residual offset field and the millimeter-wave residual offset field are determined based on at least two of the following: initial effective mask, maximum correlation value, candidate displacement probability distribution entropy, mask logic value, and compensated cross-modal consistency error.

[0116] For example, the initial effective mask is used to mark locations outside the common field of view, invalid thermal infrared pixel locations, and locations with no valid millimeter-wave echoes. The maximum correlation value is used to characterize the matching strength of the optimal candidate location within the local search range. The candidate displacement probability distribution entropy is used to characterize the concentration of candidate displacements. The mask logic value is used to characterize the valid correspondence conditions determined by the offset mask predictor. The compensated cross-modal consistency error is used to characterize the degree of difference between visible light features and thermal infrared sampling features, and between visible light features and millimeter-wave sampling features, after the initial sampling is completed based on the residual offset field.

[0117] The edge inference unit determines a thermal infrared confidence mask and a millimeter-wave confidence mask based on at least two selected pieces of information. For locations where the initial effective mask indicates the absence of valid data, locations with low maximum correlation values, locations with high candidate displacement probability distribution entropy, locations with low mask logic values, or locations with large compensated cross-modal consistency errors, a lower confidence value is determined. The confidence mask outputs continuous confidence values ​​at each spatial location to characterize the reliability of the spatial correspondence between the thermal infrared or millimeter-wave features and the visible light features at that location.

[0118] This embodiment determines the local search center by using the sensor coarse installation prior and the residual offset field at a low-resolution feature scale. Within the local search range, cross-modal correlation is constructed and the candidate displacement probability distribution is obtained. Then, the thermal infrared residual offset field and the millimeter-wave residual offset field are determined by the initial displacement and the bounded correction amount. Finally, the pixel-level continuous confidence mask is determined by combining the initial effective mask, candidate matching information, mask logic value and the compensated consistency error. This provides spatially constrained alignment results and a reliable basis for sampling alignment and fusion weight determination.

[0119] In some embodiments, based on the above embodiments, the step of sampling and aligning the thermal infrared feature and the millimeter-wave feature according to the residual offset field includes: Based on the thermal infrared residual offset field and the millimeter-wave residual offset field, bilinear deformable sampling is used to sample the thermal infrared feature and the millimeter-wave feature respectively to obtain the thermal infrared alignment feature and the millimeter-wave alignment feature.

[0120] It can be understood that the edge inference unit uses the visible light feature position within the virtual reference domain as the reference position, determines the target sampling position of the thermal infrared feature based on the thermal infrared residual offset field corresponding to the reference position, and determines the target sampling position of the millimeter-wave feature based on the millimeter-wave residual offset field corresponding to the reference position. The target sampling position is jointly determined by the coarse prediction position corresponding to the sensor coarse installation prior and the residual offset field, in order to compensate for the local position deviation that still exists after coarse installation mapping.

[0121] When the target sampling location is located between thermal infrared feature grids or millimeter-wave feature grids, the edge inference unit performs bilinear interpolation based on the feature values ​​of adjacent feature locations around the target sampling location to obtain the feature value corresponding to the target sampling location. The resulting thermal infrared alignment feature and millimeter-wave alignment feature are in the same virtual reference domain as the visible light feature, and can be used as the fusion input for subsequent spatial location correspondence.

[0122] The bilinear deformable sampling does not use a fixed sampling position, but determines the sampling position based on the thermal infrared residual offset field or millimeter-wave residual offset field corresponding to each spatial position. For local misalignment caused by sensor relative pose changes, mechanical vibration, or high-temperature deformation, compensation is made by the sampling offset corresponding to different spatial positions, reducing cross-modal spatial non-overlap caused by fixed extrinsic parameter projection.

[0123] Based on the confidence mask, the thermal infrared alignment feature and the millimeter-wave alignment feature are respectively determined as high-confidence alignment feature, medium-confidence alignment feature, or low-confidence alignment feature.

[0124] For example, the edge inference unit reads the continuous confidence values ​​of the thermal infrared confidence mask and the millimeter-wave confidence mask at various spatial locations, and performs confidence classification according to preset high confidence thresholds and low confidence thresholds. Alignment features with confidence values ​​greater than or equal to the high confidence threshold are determined as high confidence alignment features, alignment features with confidence values ​​lower than the high confidence threshold but greater than the low confidence threshold are determined as medium confidence alignment features, and alignment features with confidence values ​​less than or equal to the low confidence threshold are determined as low confidence alignment features.

[0125] The high-confidence alignment feature indicates that the corresponding position has a relatively clear cross-modal correspondence. The medium-confidence alignment feature indicates that the corresponding position has a certain correspondence, but it may be affected by smoke obstruction, temperature saturation, insufficient effective echo, local repetitive structures, or reduced consistency after compensation. The low-confidence alignment feature indicates that the cross-modal correspondence at the corresponding position is unreliable, or that there is no effective thermal infrared data or millimeter-wave data at the corresponding position.

[0126] The high-confidence alignment features are preserved, the medium-confidence alignment features are attenuated, and the low-confidence alignment features are rejected from fusion.

[0127] It is understandable that for high-confidence aligned features, the edge inference unit retains the corresponding thermal infrared or millimeter-wave aligned features and participates in subsequent feature fusion according to the corresponding fusion weights. For medium-confidence aligned features, the edge inference unit attenuates the feature value or fusion weight according to the corresponding confidence value, so that such features can retain some effective information while reducing their impact on the fusion result.

[0128] For low-confidence alignment features, the edge inference unit refuses to include them as valid cross-modal alignment features in the fusion process. In this case, the edge inference unit can perform fusion based on visible light features at the same spatial location and other unrejected modal features, avoiding erroneous information from common out-of-view regions, invalid thermal infrared pixel regions, millimeter-wave regions without effective echoes, or regions with unclear spatial correspondences from affecting the identification results of fire sources, obstacles, and free space.

[0129] The high-confidence threshold and low-confidence threshold are determined based on the confidence mask distribution and fire situation output results in the training samples. During training, three-vision synchronous samples with different smoke obscuration levels, thermal infrared saturation levels, millimeter-wave echo quality, and cross-modal spatial misalignment levels are input into the three-vision virtual aligned situational awareness network. Based on the output results of fire source status, obstacle occupancy, free space, and risk cost map, the feature retention, attenuation, and rejection fusion strategies corresponding to the confidence levels are jointly adjusted.

[0130] The time synchronization reliability is determined based on the time deviation between the visible light data, the thermal infrared data, and the millimeter wave data. The fusion weight is determined based on the time synchronization reliability, the quality of each modal data, and the confidence mask.

[0131] For example, the edge inference unit determines the time deviation between each modal data and the target inference time, as well as the time deviation between each modal data, based on the timestamps or trigger signals corresponding to the visible light data, thermal infrared data, and millimeter wave data. The edge inference unit determines the time synchronization reliability based on the time deviations; a smaller time deviation corresponds to higher time synchronization reliability, and a larger time deviation corresponds to lower time synchronization reliability.

[0132] The edge inference unit further determines the quality of each modal data based on the clarity and / or smoke obscuration of the visible light data, the effective pixel ratio and / or saturation ratio of the thermal infrared data, and the signal-to-noise ratio and / or point density of the millimeter-wave data. Time synchronization reliability is used to characterize whether each modal data corresponds to the same fire scene state, modal data quality is used to characterize the usability of each modal data itself, and confidence mask is used to characterize the alignment reliability of thermal infrared features and millimeter-wave features at corresponding spatial locations.

[0133] The edge inference unit uses time synchronization reliability, modal data quality, and confidence mask as the criteria for determining fusion weights. For modal features with low time synchronization reliability, low modal data quality, low confidence level, or those identified as low-confidence alignment features, their fusion weight is reduced or they are rejected from participating in fusion. For modal features with high time synchronization reliability, high modal data quality, and belonging to high-confidence alignment features, their fusion weight is increased.

[0134] This embodiment performs bilinear deformable sampling based on the thermal infrared residual offset field and the millimeter-wave residual offset field to obtain thermal infrared alignment features and millimeter-wave alignment features corresponding to the visible light feature space. Then, based on the confidence mask, the alignment features are subjected to high confidence retention, medium confidence attenuation and low confidence rejection fusion processing. Finally, the fusion weight is determined by combining the time synchronization reliability and the data quality of each mode, thereby reducing the deviation in fire situation judgment caused by local misalignment, time inconsistency and low reliability mode information participating in fusion.

[0135] In some embodiments, based on the above embodiments, the fire situation information includes the fire source status, obstacle status, and free space; As can be understood, the fire source state characterizes the location of the fire source and its high temperature or combustion state, the obstacle state characterizes the location and boundaries of obstacles affecting the robot's passage, and the free space characterizes the area that the robot can pass through. The edge inference unit outputs the above fire situation information based on the fused features and retains the fused reliability information corresponding to each spatial location for use in risk convergence and control decisions.

[0136] The fire source state, obstacle state, and free space are transformed into the robot coordinate system, distance estimation and risk convergence are performed, and the risk cost map is generated.

[0137] For example, the edge inference unit transforms the fire source state, obstacle state, and free space in the virtual reference domain to the robot coordinate system based on the sensor coarse installation prior and the robot's body coordinate relationship. Distance estimation is then performed on the transformed fire source position, obstacle position, and free space position to obtain the spatial distance and orientation relationship of each fire scene element relative to the robot.

[0138] The edge reasoning unit aggregates risks based on the fire source status, obstacle status, free space, and corresponding distance information. Areas with closer fire sources, obstacles, and non-free space correspond to higher risk costs, while passable free space areas far from fire sources and obstacles correspond to lower risk costs, thus generating a risk cost map characterizing the robot's passage risks at different locations.

[0139] The situational output component, used to output fire situational information and risk cost maps, can be trained together with the feature construction component, virtual alignment component, and fusion component during training. During training, synchronous samples containing fire source status, obstacle occupancy, free space, and risk cost map annotations are input into the three-vision virtual alignment situational awareness network. The task loss is determined based on the output results of fire source status, obstacle status, free space, and risk cost map, and the network parameters are updated according to the task loss to ensure that the output fire situational information and risk cost map correspond to the annotation information.

[0140] When the data quality of any modality is lower than the first preset quality threshold for multiple consecutive inference cycles, the upper limit of the fusion weight of the corresponding modality is limited and the modality degradation status is output.

[0141] It is understood that the edge inference unit records the data quality of each modality during continuous inference cycles. When the data quality of any modality among visible light data, thermal infrared data, or millimeter wave data is continuously lower than a first preset quality threshold, it is determined that the modality has entered a degraded state, and the upper limit of the fusion weight of the modality in the subsequent fusion process is limited.

[0142] The modal degradation state is used to characterize that the corresponding mode is in a continuously low-quality state. For situations where visible light is continuously blocked by dense smoke, thermal infrared has a large range of saturation, or millimeter waves have insufficient effective echoes, limiting the upper limit of the fusion weight of the corresponding mode can prevent the mode's quality from having an excessive impact on the fusion result when it recovers from short-term fluctuations.

[0143] When the quality of all modal data is lower than the second preset quality threshold or the time synchronization reliability is lower than the preset synchronization reliability threshold, the perception is unavailable state is output.

[0144] For example, when the quality of visible light data, thermal infrared data, and millimeter wave data are all below a second preset quality threshold, the edge inference unit determines that none of the current three modalities can provide reliable fire scene perception information. When the time synchronization reliability is below the second preset quality threshold, the edge inference unit determines that there is a lack of time correspondence among the current three modalities that meets the fusion requirements.

[0145] When outputting a perception unavailable state, the edge inference unit does not use the current fire situation information and risk cost map as normal perception results, or marks the perception unavailable state in the corresponding results. This avoids using unreliable fire situation information as the basis for robot movement or obstacle avoidance when all three modal data are invalid or severely misaligned in time.

[0146] The fire situation information, the risk cost map, the modal degradation state, or the perception unavailability state are sent as a unified perception data frame to the robot control system.

[0147] It is understood that a unified perception data frame includes at least the fire source state, obstacle state, free space, risk cost map, and modal availability information corresponding to the current inference moment. Modal availability information includes normal state, modal degradation state, and perception unavailable state, which is used to enable the robot control system to distinguish the availability of current fire situation information.

[0148] After receiving unified perception data frames, the robot control system can determine the travel area and obstacle avoidance area based on the risk cost map, and adjust the way it uses fire situation information based on modal degradation or perception unavailability. When a modal degradation state exists, the robot control system makes travel and obstacle avoidance decisions based on the fire situation information corresponding to the undegraded mode. When a perception unavailability state exists, the robot control system avoids making high-risk travel decisions based solely on the current perception results.

[0149] This embodiment transforms the fire source state, obstacle state, and free space into the robot coordinate system, generates a risk cost map by combining distance estimation and risk convergence, outputs the modal degradation state by judging the modal quality of continuous inference cycles, and outputs the perception unavailable state when the quality of all modalities or the reliability of time synchronization is insufficient. Finally, the relevant information is sent to the robot control system as a unified perception data frame, thereby providing the robot with a perception basis with an available state identifier for passage, obstacle avoidance and risk control in dense smoke and fire.

[0150] Please see Figure 3 This is a schematic diagram of the three-vision data synchronization and virtual vision tensor construction process according to an embodiment of the present invention. The three vision sensors acquire visible light raw frames, thermal infrared raw frames, and millimeter-wave raw data, respectively. The visible light raw frame includes RGB data, exposure parameters, and a raw timestamp; the thermal infrared raw frame includes a temperature matrix, status words, and a raw timestamp; and the millimeter-wave raw data includes point clouds, distance-angle maps, signal-to-noise ratio, velocity, and a timestamp. The synchronous acquisition unit uses the target inference time... Based on this, neighboring frame selection, temporal interpolation or motion compensation, and outlier frame removal were performed on the raw trimodal data, and the confidence level of the visible light state was determined. Confidence of thermal infrared state Millimeter wave state confidence and overall synchronization reliability Subsequently, the visible light data was normalized for brightness, processed for contrast, and characterized for smoke quality, to construct an RGB visual tensor of size H x W x 3. Normalize the thermal infrared data for temperature, temperature difference, and temperature rise correlation processing, and construct a structure with a size of [size missing]. take Thermal infrared visual tensor multiplied by 3 Extract nearest distance, signal-to-noise ratio, radial velocity, occupancy probability, forward-looking grid, and bird's-eye view grid information from millimeter-wave data to construct a millimeter-wave virtual visual tensor. Ultimately, , and and unified time stamp , , , , Equipment status and rough installation prior Together they form a unified three-vision data group The data is then input into the TVA-SANet three-vision virtual alignment situational awareness network for subsequent feature construction, residual offset compensation, confidence mask determination, fusion processing, and generation of fire situation information and risk cost maps.

[0151] Please see Figure 4 This is a schematic diagram of the structure of the three-vision virtual alignment situational awareness network (TVA-SANet) according to an embodiment of the present invention. The TVA-SANet includes an RGB lightweight encoder, a thermal infrared encoder, a radar sparse encoder, a virtual alignment module, a modal quality gating module, a lightweight fusion backbone, and multiple situational output heads. (RGB visual tensor) Processed by an RGB lightweight encoder, it outputs visible light features at one-quarter, one-eighth, and one-sixteenth scales; thermal infrared visual tensor. Processed by a thermal infrared encoder, the output contains multi-scale thermal features with temperature holding information; millimeter-wave virtual vision tensor. After processing by the radar sparse encoder, forward-looking and bird's-eye view features are output. The virtual alignment module, based on the sensor coarse-fit prior, calculates cross-modal local correlations within the local search range, determines the thermal infrared residual offset field and millimeter-wave residual offset field, and generates thermal infrared and millimeter-wave confidence masks. Spatially aligned thermal infrared and millimeter-wave features are then obtained through deformable bilinear sampling. The modal quality gating module outputs dynamic fusion weights for each mode based on smoke quality, thermal saturation, millimeter-wave point density and multipath quality, time synchronization reliability, and spatial alignment confidence. The lightweight fusion backbone includes a Rep-DW Block, a bidirectional multi-scale fusion structure, a structure reparameterization module, and an INT8-friendly operator, used to fuse visible light features, thermal infrared alignment features, and millimeter-wave alignment features according to the dynamic fusion weights. The fusion results are input into the fire source situational awareness head, obstacle situational awareness head, and risk situational characterization head, respectively, and output the fire source type, center, boundary, highest temperature, average temperature, temperature rise trend, distance, and confidence level. The outputs also include the obstacle occupancy grid, free space, distance, orientation, radial velocity, unknown area, and uncertainty, as well as risk situational information including smoke, dynamic targets, uncertainty, initial risk grid, and sub-item costs.

[0152] Please see Figure 5This diagram illustrates the execution steps and relationships of the online situational awareness method according to an embodiment of the present invention. The online situational awareness method includes steps such as equipment initialization and self-testing, simultaneous acquisition of three vision systems, data preprocessing and tensor quantization, establishment of a virtual reference domain and coarse priors, three-way multi-scale feature encoding, learnable virtual alignment, modal quality assessment and dynamic gating, lightweight fusion and multi-task situational reasoning, coordinate recovery and risk convergence, and protocol encapsulation and output. First, in step S101, the robot configuration file, model, calibration priors, and protocol configuration are loaded. The visible light camera, thermal infrared camera, millimeter-wave radar, and edge inference device are initialized and self-tested to obtain a set of usable sensors. Then, in step S102, visible light data, thermal infrared data, and millimeter-wave data are matched based on the target inference time to generate corresponding three-vision data sets, and the time synchronization reliability is determined. In step S103, the three-vision data are preprocessed and tensor quantized to construct visible light virtual vision tensors, thermal infrared virtual vision tensors, and millimeter-wave virtual vision tensors, which serve as inputs to the three-vision virtual alignment situational awareness network.

[0153] In step S104, a virtual reference domain is established based on the sensor coarse installation prior, using visible light features as a reference, and the coarse predicted positions and local search ranges of thermal infrared and millimeter-wave features are determined. In step S105, multi-scale feature encoding is performed on the visible light, thermal infrared, and millimeter-wave data respectively to obtain features containing color texture, temperature changes, distance orientation, and spatial structure information. In step S106, the thermal infrared residual migration field, millimeter-wave residual migration field, and corresponding confidence masks are predicted based on cross-modal local correlation, and the thermal infrared and millimeter-wave features are aligned to the virtual reference domain through bilinear deformable sampling.

[0154] In one specific embodiment, step S106 includes: Establish a local correlation volume. For the modes to be aligned... Feature scale and reference position According to rough installation prior Obtain the coarse predicted position of the feature to be aligned in the reference domain. If the current scale is not the coarsest scale, the residual migration field obtained from the previous low-resolution scale is upsampled and converted to the current scale as an additional correction to the coarse predicted position. A radius is established centered on the corrected coarse predicted position. Local search window The reference eigenvector and candidate eigenvector are normalized by L2, and then the cosine correlation is calculated by channel group and averaged to reduce the influence of the difference in characteristic amplitude of visible light, thermal infrared and millimeter wave on the correlation value.

[0155] ; in, Indicates the thermal infrared values ​​of the modes to be aligned. or millimeter wave , Indicates the feature scale. Indicates the reference position in the virtual reference domain. This represents the candidate displacement within the local search window. Representing visible light reference features at scale Location The feature vector at that location, Representing modes In scale The features to be aligned below Indicates based on rough installation priors The obtained modes In scale The rough predicted location is below. This represents the L2 norm normalization operation. To prevent small positive numbers with a denominator of zero.

[0156] S106.2, obtain the candidate displacement probability and initial displacement. Then, categorize the relevant values ​​within the local search window according to the temperature parameter. conduct Normalization to obtain candidate shifts At reference location probability distribution at And calculate the initial displacement according to the expected probability. The maximum relevance value reflects the matching strength of the optimal candidate, and the probability distribution entropy. It reflects whether candidate positions are concentrated; when multiple candidates have similar correlation values, the entropy increases, indicating that the correspondence at that position is unclear.

[0157] ; ; ; in, for The temperature parameter used for normalization This indicates that the coarsely predicted location is the center and the radius is... Local search window, Indicates candidate displacement At reference location The probability distribution at that location, This represents the initial displacement calculated based on the expected probability. Entropy represents the probability distribution of candidate displacements.

[0158] S106.3 predicts the bounded residual migration field. The local correlated volume, reference feature, feature to be aligned, initial displacement, coarse mounting displacement, previous-scale residual migration, initial effective mask, and millimeter-wave distance prior are input into a migration-mask predictor composed of depth-separable convolutions. The predictor outputs a two-dimensional unconstrained migration. and mask logic value The residual offset is constrained to a physically feasible search radius at the current scale using hyperbolic tangent and truncation functions. Within; estimates are made from coarse to fine in the order of 1 / 16, 1 / 8, and 1 / 4, with the lower resolution scale responsible for the overall orientation and the higher resolution scale only compensating for local boundary errors.

[0159] ; in, Representing modes In scale The residual offset field below, This is the scaling factor for converting the residual migration field from the previous scale to the current scale. Indicates an upsampling operation. This represents the residual offset field obtained at the previous lower resolution scale. The physically feasible search radius at the current scale. This represents the two-dimensional unconstrained offset output by the offset-mask predictor. It is the hyperbolic tangent function. This indicates that the value within the parentheses will be truncated and restricted to negative values. To Zheng Within the range.

[0160] S106.4, Generate and validate the confidence mask. The initial effective mask obtained in step S104 is then used... The mask logic value, maximum correlation value, and correlation distribution entropy output from the offset-mask predictor are combined to obtain the compensated pre-confidence mask. The initial effective mask is used to directly exclude locations outside the common field of view, thermal infrared dead pixels, and millimeter-wave locations without effective echoes; the higher the maximum correlation value and the lower the correlation distribution entropy, the clearer the correspondence. After completing the initial sampling, the consistency error between the reference feature and the sampled feature is calculated. The confidence mask is attenuated twice to avoid pseudo-matches where the correlation peak is high but the semantics or spatial structure is still inconsistent after compensation.

[0161] ; ; ; in, This represents the initial effective mask obtained in step S104. This represents the Sigmoid activation function. This represents the mask logic value output by the offset mask predictor. and These are the weighting coefficients for the maximum correlation term and the distribution entropy term, respectively. This indicates a compensation pre-confidence mask. This represents the consistency error between the compensated reference feature and the sampled feature. This represents the cosine similarity operation. The adjustment coefficient for the second-order attenuation of the confidence mask is used to address the consistency error. This represents the final generated confidence mask.

[0162] S106.5, Perform residual offset compensation. Add the coarse installation mapping position to the residual offset field to obtain the continuously sampled coordinates. Bilinear sampling is performed using the product of the horizontal and vertical distances of the four surrounding integer grid points as weights. For positions where the confidence mask is above the first threshold, the compensated alignment features are retained; for positions between the first and second thresholds, the confidence mask is continuously attenuated; for positions below the second threshold, they are marked as having no corresponding or unknown states and do not participate in strong fusion. The residual offset field is simultaneously propagated to the next higher resolution scale to narrow the search range at the next scale; in adjacent inference cycles, only high-confidence residual offsets are used as optional initializations for the local search in the next cycle, and low-confidence offsets are not propagated over time.

[0163] ; ; in, Representing modes In scale Below, reference position The continuous sampling coordinates at the location are obtained from the coarse installation mapping position. With the residual offset field at that location Add them together to get; The product of the horizontal and vertical distances of the four nearest integer grid points is used as the interpolation weight for the modality. Feature map We perform a weighted summation to obtain the sampling features; This indicates the confidence mask obtained in step S106.4 at that location; Representing modes In scale Below, position The final output alignment feature, which is the result of element-wise multiplication of the confidence mask and the sampled feature, is used to suppress or reject low-confidence location features in subsequent fusion.

[0164] S106.6, the offset-mask predictor is trained using a task-driven approach. The virtual alignment does not output a three-color fusion map that "appears to be completely overlapping." Instead, the residual offset field is updated inversely by fire source classification and localization, temperature regression, obstacle occupancy, free space, and risk prediction losses. Unstable offsets and false high-confidence masks are jointly suppressed by cross-modal consistency loss, offset smoothing loss, and constraints on uncorresponding regions. Effective alignment is considered complete as long as the compensated features improve the situational awareness assessment. For fire-transparent regions, regions without radar reflection, occlusion boundaries, or regions with repetitive structures, low confidence is allowed without forced correspondence.

[0165] In step S107, the fusion weights of each mode are determined by comprehensively considering smoke masking, thermal infrared saturation, millimeter-wave point cloud quality, time synchronization reliability, and spatial alignment confidence. Low-quality or low-confidence modes are subject to weight reduction, masking, or downgrading.

[0166] In one specific embodiment, step S107 includes: S107.1, the visible light quality description vector includes local contrast, underexposure ratio, overexposure ratio, image sharpness, and smoke occlusion probability; the thermal infrared quality description vector includes saturated pixel ratio, bad pixel ratio, temperature drift, effective temperature range, and thermal gradient stability; the millimeter wave quality description vector includes dot density, average SNR, multipath anomaly ratio, empty grid ratio, and continuous frame stability.

[0167] S107.2, input the three quality descriptions and encoder global features into a small gated network to obtain the original quality score. , and Then, the time confidence scores for each modality are calculated. Alignment confidence with average Introduced to form a corrected mass fraction , and When visible light does not undergo cross-modal sampling, it is taken as... The dynamic weights are normalized using Softmax; when modalities are missing, severely out of sync, or alignment confidence is close to zero, the corresponding corrected quality score is significantly reduced.

[0168] ; ; in, Represents the modal identifier, with values... Represents visible light, Represents thermal infrared, Represents millimeter waves, This represents the original quality score of the corresponding modality obtained by inputting the quality description vector and the encoder's global features into a small gated network. This indicates the time synchronization confidence level of the corresponding mode. This represents the average alignment confidence of the corresponding modes, taken when visible light does not undergo cross-modal sampling. Equals 1, This is the weighting factor for the corrected quality score based on the time synchronization confidence level. This is the weighting factor for the adjusted quality score based on the average alignment confidence. To prevent small positive numbers with zero or negative values ​​from being generated in logarithmic operations, This represents the corrected quality score obtained by combining the original quality score, time synchronization confidence, and alignment confidence. , and These represent the corrected mass fractions for the visible light, thermal infrared, and millimeter wave modes, respectively. Represents an exponential function. Indicates the corresponding mode through The dynamic fusion weights obtained after normalization , and These are the visible light fusion weight, the thermal infrared fusion weight, and the millimeter wave fusion weight, respectively, and the sum of the three is 1.

[0169] S107.3, local control is performed at each spatial location using a confidence mask, and the fused features are: ; in, Indicates the characteristics of visible light. and These represent the thermal infrared alignment features and the millimeter-wave alignment features, respectively, which are the sampling features obtained after residual offset compensation in step S106. and These represent the confidence masks for thermal infrared and millimeter wave, respectively, with symbols... This represents element-wise multiplication, used to perform local control on alignment features at each spatial location using a confidence mask. , and For the aforementioned dynamic fusion weights, This indicates the fusion feature obtained after weighted fusion, which is used to generate fire situation information and risk cost map in the subsequent process.

[0170] S107.4: When the quality of a certain modality is below a first threshold for N consecutive frames, its weight upper limit is limited to a preset value and a degraded state is output; when it recovers and exceeds the recovery threshold for N consecutive frames, the weight is smoothly restored. Different entry and recovery thresholds are used to create hysteresis, avoiding frequent state jitter. When the overall synchronization confidence level... C sync When the value is below the second threshold, the usability level of the entire frame situation result is directly reduced instead of relying solely on relative weights for judgment.

[0171] In step S108, the visible light features, thermal infrared alignment features, and millimeter-wave alignment features are weighted and fused to output fire situation information such as fire source status, obstacle status, free space, smoke risk characterization, and uncertainty. In step S109, the fire situation information is converted to the robot coordinate system, distance estimation and risk convergence are performed, and a risk cost map is generated. In step S110, the fire situation information, risk cost map, time synchronization reliability, modal degradation status, or perception unavailability status are encapsulated into a unified perception data frame and sent to the robot control system to support subsequent navigation, obstacle avoidance, path planning, and risk linkage processing, and then proceed to the next inference cycle.

[0172] In conclusion, Figure 5 This invention demonstrates an online closed-loop processing procedure that integrates three-vision data synchronous acquisition, virtual alignment, quality gating, situational reasoning, risk convergence, and unified data output. This enables the device to output fire situation information and risk cost maps with reliability indicators even under conditions of dense smoke obscuring, local misalignment, thermal infrared saturation, or reduced millimeter-wave echo quality.

[0173] Please see Figure 6 This diagram illustrates the training, compression, deployment, and version verification process of a three-vision virtual alignment situational awareness network according to an embodiment of the present invention. The process includes data acquisition, annotation and scene segmentation, virtual misalignment enhancement, joint training, distillation and pruning, structural reparameterization, hybrid precision quantization, and edge verification and release. In step T201, multi-scene three-vision synchronization sequences are acquired. These sequences include visible light data, thermal infrared data, and millimeter-wave data, covering fire environments such as smokeless, light smoke, medium smoke, and heavy smoke. In step T202, fire sources, temperature, obstacle occupancy, and risk information in the samples are labeled, and the training, validation, and test sets are divided according to independent spaces to reduce validation bias caused by adjacent data from the same scene entering different datasets simultaneously.

[0174] In step T203, the training samples are enhanced with virtual misalignment, including random translation or rotation, cross-modal time offset, smoke occlusion, thermal infrared saturation or temperature drift, millimeter-wave point loss or multipath virtual points, etc., to form training data containing cross-modal spatial misalignment, temporal inconsistency, and modal degradation. In step T204, the enhanced training samples are input into the three-vision virtual alignment situational awareness network, and the following processes are executed sequentially: three-vision feature construction, virtual alignment, residual offset field and confidence mask determination, quality gating, weighted fusion, and multi-task situational output processing. The task loss is calculated based on the results of fire source detection, temperature information, obstacle occupancy, free space, and risk cost map. At the same time, joint training is performed by combining cross-modal consistency loss, offset smoothing loss, no corresponding region constraint, and quality gating constraint.

[0175] In step T205, the network that has completed joint training undergoes distillation and pruning, allowing a smaller-scale network to learn the output representation from the joint training network and removing network channels or branches that have little impact on the fire situation output. In step T206, the pruned network undergoes structural reparameterization, transforming the network structure from the training phase into a structure suitable for edge inference while maintaining the main features extraction, virtual alignment, quality gating, and situation output functions. In step T207, the reparameterized network undergoes mixed-precision quantization. The main calculation process is performed using INT8 precision, while outputs requiring precision, such as offset and temperature regression, are processed using FP16 precision, thus balancing edge inference speed, storage resource consumption, and critical output accuracy.

[0176] In step T208, the quantized network is deployed to an edge inference device to verify the model's accuracy, inference speed, power consumption, and output consistency across different network versions. The verified network is then published as a verified model to the edge industrial control computer or robot's on-site operating environment. Newly collected samples are only used for offline retraining, version verification, and updates in a secure environment; the on-site operating model is not directly modified online using unverified samples.

[0177] In conclusion, Figure 6 The complete processing flow of the three-vision virtual aligned situational awareness network is shown, from multi-scene data acquisition, augmented training, joint optimization, model compression, edge deployment to version verification and release. This enables the network to meet the accuracy requirements of fire situational awareness while adapting to the computing resources of edge inference devices, and ensuring the verifiability and stability of the on-site running model.

[0178] In one specific embodiment, a three-vision virtual alignment situational awareness device for dense smoke and fire scenes is provided. The device includes a three-vision sensor head, a synchronous acquisition unit, an edge inference unit, an output interface, and a protective mounting structure. The three-vision sensor head includes a visible light camera, a thermal infrared camera, and a millimeter-wave radar, used to acquire visible light data, thermal infrared data, and millimeter-wave data corresponding to the same inference time.

[0179] In this embodiment, the three-vision sensor head is installed on the upper front of the tracked firefighting robot to meet the needs of observation range of the forward fire area, obstacle recognition range, and perception of the robot's passage area. The three-vision sensor head can also be installed on the top of a wheeled robot, the torso of a legged robot, a track inspection platform, or an electric telescopic pole. The specific installation position can be determined according to the structure of the work platform, the direction of travel, and the observation requirements.

[0180] In one example, the center mounting height of the three-vision sensor head is 1.05 meters, the depression angle is 8 degrees, and it is connected to the robot via a quick-release flange. The quick-release flange is equipped with a locating pin and an anti-reverse insertion structure to limit the installation position and direction of the sensor head after repeated disassembly and assembly, so that the coarse installation prior changes after repeated installation are within the spatial misalignment adaptation range of the three-vision virtual alignment situational awareness network.

[0181] The visible light camera has a resolution of 1920 x 1080 and an output frequency of 30 frames per second, featuring automatic exposure and wide dynamic range. It is used to acquire flame color texture, scene semantics, visible obstacle boundaries, and other visible light scene information, and to construct visible light features.

[0182] The thermal infrared camera is an uncooled thermal infrared camera, operating in the 8 to 14 micrometer wavelength range, outputting a 640x512 radiation temperature matrix at a frequency of 25 frames per second. The thermal infrared camera is used to obtain temperature distribution, high-temperature regions, temperature gradients, and temperature rise trends, and to construct thermal infrared features to supplement visible light mode scene information under conditions of dense smoke obscuring or low illumination.

[0183] The millimeter-wave radar employs a frequency-modulated continuous wave (FMCV) system, operating in the 60-64 GHz or 76-81 GHz frequency band, with a horizontal field of view of no less than 100 degrees and an output frequency of 20 frames per second. The millimeter-wave radar is used to obtain information on target range, azimuth, velocity, obstacles, and spatial structure, and to construct millimeter-wave features based on millimeter-wave echo data.

[0184] The resolution, operating band, output frequency, and field of view parameters of the visible light camera, thermal infrared camera, and millimeter-wave radar described above are exemplary parameters of this embodiment and do not constitute a limitation of the present invention. In other embodiments, sensors with different resolutions, bands, output frequencies, or field of view ranges can be used depending on the robot's operating speed, fire environment, detection distance, edge reasoning capability, and task requirements.

[0185] The three sensors are fixed to a rigid reference member made of aluminum alloy or high-temperature resistant composite material. The rigid reference member is used to maintain the relative mounting relationship between the visible light camera, the thermal infrared camera, and the millimeter-wave radar, and to provide a structural basis for establishing the coarse mounting prior of the sensors.

[0186] The sensor head housing features partitioned windows adapted to different modes. The visible light window uses a transparent protective material, the thermal infrared window uses a long-wave infrared transmitting material, and the corresponding area for millimeter-wave radar uses a low-dielectric-loss transmissive material. By using appropriate window materials for different sensors, the impact of the protective housing on the acquisition of visible light, thermal infrared, and millimeter-wave data can be reduced while protecting the sensors.

[0187] The sensor head housing also includes a heat insulation layer, a heat-conducting plate, an air channel, and a temperature sensor. The heat insulation layer reduces the direct thermal radiation impact on internal components from the high-temperature environment of the fire scene, the heat-conducting plate and air channel dissipate internal heat, and the temperature sensor acquires the internal temperature of the sensor head.

[0188] When the internal temperature reaches the first threshold, the equipment increases the cooling airflow or reduces the edge inference frequency to reduce the risk of continuous temperature rise. When the internal temperature reaches the second threshold, the equipment sends an over-temperature degradation status to the robot control system, enabling the robot control system to adjust its operating strategy or stop corresponding high-load sensing processing based on this status.

[0189] The edge inference unit can be an edge inference industrial control computer, including a multi-core central processing unit, a graphics processing unit or a neural network processor, more than 16 gigabytes of memory, non-volatile memory, a gigabit Ethernet interface, a USB3 interface, a CAN FD interface, and an isolated power supply interface. The edge inference unit is used to run a three-vision virtual alignment situational awareness network, completing three-vision data synchronization, feature construction, virtual alignment, confidence mask determination, modal quality assessment, weighted fusion, fire situational inference, risk cost map generation, protocol adaptation, and equipment health monitoring.

[0190] Visible light cameras and thermal infrared cameras can be connected to the edge inference industrial control computer via gigabit Ethernet or USB3 interfaces, while millimeter-wave radar can be connected via Ethernet or CAN FD interfaces. The synchronous acquisition unit can use hardware triggering, precise time protocols, second pulse signals, or a unified clock method to form a unified inference time based on the timestamps or trigger signals of each modality's data, and determine the time synchronization reliability of the three vision data.

[0191] In one embodiment, the edge inference industrial control computer and the three-vision sensor head are integrated into a single device. In another embodiment, the edge inference industrial control computer is installed inside the sealed cabin of the fire-fighting robot, and the three-vision sensor head is connected to the edge inference industrial control computer via a communication interface, thereby enabling the sensor head to perform data acquisition functions and the edge inference industrial control computer to perform calculation, control, and output functions.

[0192] The output interface encapsulates fire situation information, risk cost maps, modal degradation states, over-temperature degradation states, or perception unavailability states into unified perception data frames and sends them to the robot control system. The robot control system can obtain information on the fire source status, obstacle status, free space, and risk areas based on the unified perception data frames to perform navigation, obstacle avoidance, path planning, or risk linkage processing.

[0193] In summary, this embodiment constructs a three-vision situational awareness system suitable for deployment in dense smoke and fire environments by fixing a visible light camera, a thermal infrared camera, and a millimeter-wave radar to a rigid reference component, and combining them with quick-release positioning, protective partition windows, heat insulation and heat dissipation structures, synchronous acquisition units, and edge inference industrial control computers. This system can provide a multimodal data foundation for the three-vision virtual aligned situational awareness network, including coarse installation priors, temporal correspondences, and equipment operating status information, and outputs fire situation information and risk cost maps.

[0194] In one specific embodiment, the processing procedure of the three-vision virtual alignment situational awareness equipment is explained using an online inference cycle after a fire-fighting robot enters a smoke-filled warehouse as an example. In this embodiment, the edge inference unit operates with a target inference cycle of 30 milliseconds, and the target inference time corresponding to the k-th inference cycle is... The 30-millisecond target inference cycle matches the end-to-end inference latency of the three-vision virtual alignment situational awareness network deployed with FP16 precision or hybrid INT8 precision, and reserves processing margin for the encapsulation, transmission and subsequent inference task scheduling of unified perception data frames.

[0195] In step S102, the edge inference unit selects the time closest to the target inference time from the visible light cache, thermal infrared cache, and millimeter-wave cache. The data shows that the nearest visible light frame differs from the target inference time by 3 milliseconds, the thermal infrared frame by 8 milliseconds, and the millimeter-wave frame by 12 milliseconds, all less than the 20-millisecond synchronization threshold. Therefore, the edge inference unit forms a complete three-vision data set containing visible light, thermal infrared, and millimeter-wave data, and calculates the overall time synchronization reliability. It is 0.82.

[0196] In step S103, the edge inference unit scales the visible light frame to 640x384 visible light input data. The edge inference unit converts the thermal infrared radiation temperature matrix into three-channel thermal infrared input data: temperature, temperature gradient, and temperature rise, to characterize high-temperature regions, temperature changes, and temperature rise changes.

[0197] The edge inference unit rasterizes the millimeter-wave detection points into 160x96 forward-looking virtual vision data and 200x200 bird's-eye view grid data. The forward-looking virtual vision data stores the nearest distance, signal-to-noise ratio, velocity, and occupancy probability corresponding to each grid position, which is used to construct millimeter-wave forward-looking features. The bird's-eye view grid data is used to provide spatial structure information around the robot for obstacle state recognition and spatial risk convergence.

[0198] In step S104, the edge inference unit maps the thermal infrared and millimeter-wave features to a virtual reference domain with visible light features as a reference, based on the sensor coarse-mount prior in the robot configuration file, and limits the candidate correspondence range of the thermal infrared and millimeter-wave features to a local search range around the visible light reference features. The coarse-mount prior is used to provide the initial spatial mapping relationship and local search constraints, and does not require the thermal infrared and millimeter-wave features to completely coincide with the visible light features after coarse mapping.

[0199] In step S105, the edge inference unit outputs visible light features, thermal infrared features, and millimeter-wave features at scales of one-quarter, one-eighth, and one-sixteenth, respectively. The low-resolution feature scale is used to determine the initial residual offset over a larger range, while the high-resolution feature scale further determines the fine local offset based on the residual offset at the low-resolution feature scale.

[0200] In step S106, the edge inference unit predicts the thermal infrared residual migration field, the millimeter-wave residual migration field, and the corresponding confidence mask based on cross-modal local correlation. At the 1 / 8 feature scale, the predicted average residual migration of the thermal infrared features is 2.4 feature units in the horizontal direction and 0.8 feature units in the vertical direction, while the average residual migration of the millimeter-wave features is -1.6 feature units in the horizontal direction and 1.1 feature units in the vertical direction.

[0201] The edge inference unit performs bilinear deformable sampling on the thermal infrared and millimeter-wave features based on the residual offset field to obtain thermal infrared and millimeter-wave aligned features. For spatial locations with low local correlation, where the confidence value corresponding to the confidence mask is less than 0.2, the edge inference unit does not use the thermal infrared or millimeter-wave aligned features at that location as strong fusion features in subsequent fusion, in order to reduce the impact of erroneous spatial correspondences.

[0202] In step S107, the edge inference unit determines the fusion weights of each mode based on the degree of visible light smoke obstruction, thermal infrared data quality, millimeter-wave data quality, time synchronization reliability, and confidence mask. In this embodiment, the visible light data is heavily obstructed by smoke, and the visible light fusion weights... The thermal infrared fusion weight is 0.17. The millimeter-wave fusion weight is 0.48. It is 0.35.

[0203] Therefore, thermal infrared and millimeter-wave features serve as the primary sources of situational awareness information within this inference cycle, while visible light features retain some flame texture and scene contour information. The fusion weights reflect the relative availability of each modality of data under the current fire conditions, rather than permanently masking any particular modality of data.

[0204] In step S108, the edge inference unit performs weighted fusion based on visible light features, thermal infrared alignment features, and millimeter wave alignment features to output fire situation information. In this embodiment, a real fire source candidate is identified on the right side in front of the robot, and the highest temperature corresponding to the fire source candidate is 286 degrees Celsius. At the same time, a fallen box obstacle is identified 2.3 meters in front of the robot.

[0205] In step S109, the edge reasoning unit converts the fire source direction and obstacle position to the robot base coordinate system, and performs distance estimation and risk convergence based on the fire source, obstacle, and their spatial positions. In this embodiment, risk expansion is performed within a 1.2-meter radius around the fire source and a 0.45-meter radius around the obstacle within the risk cost map to form a risk cost region reflecting the thermal risk of the fire source and the passage risk of the obstacle.

[0206] In step S110, the edge inference unit publishes fire source target information, occupied grid, risk cost map, and diagnostic information through the second version of the robot operating system. The robot path planning module identifies the passable area on the left based on the risk cost map and performs path planning accordingly.

[0207] This equipment is used to provide fire scene perception results and risk information, and does not directly determine the robot's specific movement path. The robot's path planning, motion control, and execution strategy are determined by the robot control system based on the unified perception data frame, the robot's own state, and task requirements.

[0208] In summary, this embodiment demonstrates the online inference process of a firefighting robot completing a three-vision data synchronization, preprocessing, virtual alignment, confidence control, dynamic fusion, fire source and obstacle identification, risk cost map generation, and unified perception data output in a smoke-filled warehouse. By increasing the fusion weights of thermal infrared and millimeter-wave modes when visible light is severely obscured by smoke, and suppressing strong feature fusion of low-confidence spatial locations, the reliability of fire source candidate, obstacle location, and passable area determination can be improved.

[0209] In one specific embodiment, the training data composition, training phase, model size, and edge deployment method of the three-vision virtual alignment situational awareness network are described.

[0210] A dataset was constructed comprising six types of spaces: shopping malls, warehouses, workshops, exhibition halls, hotel corridors, and single-level residential buildings. The dataset includes 60 independent spaces in a simulation environment and 20 independent spaces in a controllable fire-fighting experimental field, for a total of 80 independent spaces.

[0211] Visible light data, thermal infrared data, and millimeter-wave data were collected in each independent space under smoke-free, light smoke, medium smoke, and heavy smoke conditions, and three-vision synchronized samples were formed according to a unified inference time. This resulted in 42,800 sets of three-vision synchronized samples, which were used to train and validate the fire situational awareness capabilities of the three-vision virtual aligned situational awareness network under different spatial types and smoke levels.

[0212] Training, validation, and test samples are divided into independent spaces. Specifically, 56 independent spaces are used for the training space, 12 for the validation space, and 12 for the test space, corresponding to 70%, 15%, and 15% of the training, validation, and test sets, respectively.

[0213] By using independent spatial partitioning instead of randomly dividing adjacent frames into datasets, data from the same space or nearby acquisition locations are prevented from simultaneously entering the training, validation, or test sets. This reduces scene leakage caused by repetition of scene background, structural layout, and information from adjacent time points, and improves the reliability of performance verification for spaces not included in the training.

[0214] The network input size for each three-vision synchronous sample is set to 640 x 384. The initial number of channels for the visible light encoder is 24, the initial number of channels for the thermal infrared encoder is 16, the initial number of channels for the millimeter-wave encoder is 16, and the maximum number of channels for the fusion backbone is 192.

[0215] Visible light encoders are used to extract color textures, scene outlines, flame appearance, and visible obstacle boundary features. Thermal infrared encoders are used to extract temperature distribution, high-temperature regions, temperature gradients, and temperature rise variation features. Millimeter-wave encoders are used to extract distance, orientation, velocity, occupancy probability, and spatial structure features. The fusion backbone combines visible light features, thermal infrared alignment features, and millimeter-wave alignment features to output fused features for use in generating fire source states, obstacle states, free space, and risk cost maps.

[0216] The training process uses the AdamW optimizer, with an initial learning rate set to... The batch size was set to 16, and the number of training rounds was set to 160. During training, the network parameters were updated based on the task loss of the fire source state, temperature information, obstacle occupancy, free space, and risk cost graph, and were jointly optimized by cross-modal consistency loss, offset smoothing loss, no corresponding region constraint, quality gating constraint, and knowledge distillation loss.

[0217] In the first 20 rounds of training, the visible light encoder, thermal infrared encoder, millimeter-wave encoder, fusion backbone, and each task head are trained to enable the network to learn the basic correspondence between the three modal data and the fire situation output. Virtual alignment and full quality gating processing are not enabled in this stage to avoid affecting the construction of basic features and task outputs when residual offset prediction and quality weight learning are not yet stable.

[0218] Virtual alignment processing was enabled during training rounds 21 to 60. The network predicted the thermal infrared residual migration field, millimeter-wave residual migration field, and confidence mask based on sensor coarse installation priors, cross-modal local correlations, initial displacements, and bounded corrections, and obtained thermal infrared alignment features and millimeter-wave alignment features based on the residual migration field.

[0219] In training rounds 61 and onwards, full quality gating and failure enhancement are enabled. The quality gating component combines visible light sharpness and smoke obscuration, effective and saturated pixel ratios in thermal infrared, millimeter-wave signal-to-noise ratio and point density, time synchronization reliability, and a confidence mask to learn the fusion weights corresponding to each modality. Failure enhancement enables the network to learn to reduce, attenuate, or reject fusion of low-quality or low-reliability modal features under conditions such as smoke obscuration, thermal infrared temperature drift or saturation, missing or multipath-based points in millimeter-wave data, cross-modal time shifts, and spatial misalignment.

[0220] In the knowledge distillation training, the distillation temperature of the teacher network is set to 3. The student network learns the representation results of the teacher network in the fire situation output to obtain network parameters suitable for subsequent model compression and edge deployment.

[0221] The trained three-vision virtual aligned situational awareness network has 3.86M parameters and a theoretical computational cost of 5.9 GFLOPs. These parameter and computational costs represent the network size in this embodiment, used to achieve a balance between fire situational awareness accuracy and edge inference resources.

[0222] After training, the network is reparameterized and a hybrid precision deployment version is generated. In this hybrid precision deployment version, the ordinary backbone operator uses INT8 precision, while the residual migration prediction and temperature regression use FP16 precision, in order to maintain the accuracy of residual migration prediction and temperature information output while reducing computational resource consumption.

[0223] The model package stores both the FP16 version and the mixed-precision version. During device startup self-test or input check during operation, if the input data range is found to exceed the quantization calibration range of the mixed-precision version, the edge inference unit switches to the FP16 version for inference to avoid the quantization input range mismatch from adversely affecting the output of fire situation information and risk cost map.

[0224] In summary, this embodiment uses synchronized samples covering different building spaces, smoke levels, and fire data degradation scenarios to conduct phased joint training of the three-vision virtual aligned situational awareness network. Furthermore, by employing knowledge distillation, structural reparameterization, and hybrid precision deployment to adapt to edge inference devices, it improves the model's deployment adaptability in different operating environments while ensuring its fire situational awareness, spatial alignment, and risk cost map generation capabilities.

[0225] Please refer to Table 1 below, which is a comparison table of experimental results of three-vision fusion and virtual alignment ablation in embodiments of the present invention. Table 1 shows that after the addition of millimeter-wave virtual vision, the obstacle occupancy IoU increased from 64.1% of direct RGB-T stitching to 73.8%; after further adding LVA (Learnable Virtual Alignment) virtual alignment, the mAP of fire sources increased by 2.0 percentage points, and the false negative rate of heavy smoke decreased by 2.8 percentage points. Fixed projection can only compensate for the overall installation relationship, and is inferior to learnable virtual alignment when facing random parallax and slight rotation.

[0226] Table 1 Comparison of Experimental Results of Three-Vision Fusion and Virtual Alignment Ablation ; Please refer to Table 2 below, which is a comparison table of robustness experimental results under different cross-modal alignment perturbation conditions of the embodiments of the present invention. Table 2 shows that the feature center error is calculated by the high response center distance of the same target in the reference feature and the aligned feature. Even with an input of 18 pixels horizontal misalignment and 12 pixels vertical misalignment, the present invention still maintains 91.6% mAP@0.5, indicating that the network can learn residual alignment within the coarse installation prior constraints. This table uses a specially constructed perturbation test subset, so the 92.8% without additional perturbation differs from the 92.6% in the overall test set ablation table by 0.2 percentage points, which does not constitute a contradiction under the same statistical caliber.

[0227] Table 2. Comparison of robustness experimental results under different cross-modal alignment perturbation conditions. ; Please refer to Table 3 below, which is a performance comparison table of different models in this invention deployed on edge inference devices. Table 3 shows that the hybrid INT8 version has a 0.5 percentage point decrease in mAP@0.5 compared to FP16, a steady-state throughput of 57.4 FPS, and a single-frame end-to-end latency of 20.3 ms. After retaining FP16 for offset prediction and temperature regression, there was no significant amplification in alignment accuracy and temperature error. In the table, "speed" refers to the throughput during stable pipeline operation, and "end-to-end latency" refers to the average latency from synchronous framing to structured result output for a single frame. Since acquisition, inference, and transmission can be executed in parallel pipelined, they do not need to be strictly reciprocals of each other.

[0228] Table 3. Performance Comparison of Different Models on Edge Inference Devices ; Please refer to Table 4 below, which is a comparison table of situational awareness performance and system degradation status under different smoke levels and single-mode failure conditions in this embodiment of the invention. Table 4 shows that under heavy smoke conditions, the scene-level fire source detection rate is still 87.9%, and the average error of the nearest obstacle distance is 0.18m. When the RGB lens is blocked, the millimeter wave is disconnected, or the thermal infrared is saturated, the equipment can issue a mode degradation status and continue to output the remaining reliable results; when the millimeter wave is disconnected, the accurate obstacle distance error is not counted, and the relevant distance fields are marked as low confidence or invalid. If all three modes are of low quality at the same time, the situational awareness unavailable status is output according to step S110.

[0229] Table 4. Comparison of Situational Awareness Performance and System Degradation State under Different Smoke Levels and Single-Mode Failure Conditions ; Please refer to Table 5 below, which is a comparison table of situational information transmission delays under different communication output methods in this embodiment of the invention. Furthermore, continuous operation testing and the description in Table 5 show that the equipment ran continuously for 8 hours without any abnormal exits of the model process, with an average power consumption of 28.6W and a peak internal temperature of 62.4℃. The protocol interface and model inference are decoupled; expired continuous map frames are discarded when any network transmission queue becomes congested, but fire source alarms, faults, and heartbeat messages are retained.

[0230] Table 5 Comparison of Situational Information Transmission Delay under Different Communication Output Methods ; This invention does not limit the use of visible light features as the sole reference domain. In fixed scenarios where visible light is unavailable for extended periods, a pre-defined robot-forward virtual grid or thermal infrared features can also be used as the reference domain. It does not limit millimeter-wave data to point clouds; distance-angle maps, micro-Doppler maps, target lists, or grid tensors composed of these can be used. It does not limit the brand of edge chips; they can be deployed on heterogeneous platforms such as GPUs, NPUs, FPGAs, and CPUs.

[0231] The three-vision sensor head can be integrated with the industrial control computer, or it can adopt a separate structure of sensor head and computing box; it can be fixedly installed, or installed on an electric telescopic pole or gimbal. The robot can provide odometry and IMU for map motion compensation, or it can output only the local results in the sensor head coordinate system when the robot state is missing.

[0232] The following is the pseudocode of the online three-vision virtual alignment situational awareness algorithm according to an embodiment of the present invention: Input: Visible light frame I v Thermal infrared frame I t Millimeter-wave frame R, target time t k Rough installation prioritization P 0 Output: Fire, Obs, Free, Risk, Health D k , C v , C t , C r , C sync ←SyncAcquire( I v , I t ,R, t k ) X v ←BuildRGBTensor( D k .visible) X t ←Stack(NormalizeTemp,TempGradient,TempRise) X r ←RadarToVirtualVision( D k .radar,channels=[range,snr,velocity,occupancy]) { F v s }←RGBEncoder( X v ) { F t s }←ThermalEncoder( X t ) { F r s }, F r BEV ←RadarEncoder( X r ) for each scale s do Ω t , Ω r ←BuildLocalSearchWindow( P 0,s) Corr t , Corr r ←LocalCorrelation( F v s , F t s , F r s , Ω t , Ωr ) Δp t , Δp r , M t , M r ←OffsetMaskPredictor( Corr t , Corr r ) F t a ←DeformSample( F t s , Δp t ) F r a ←DeformSample( F r s , Δp r ) W v , W t , W r ←QualityGate( X v , X t , X r , C v , C t , C r , C sync , M t , M r ) F f s ←Wv· F v s+Wt ·( M t ⊙ F t a )+Wr·(M r ⊙ F r a ) end for P←LightweightFusionBackbone({ F f s}, F r BEV ) Fire←FireStateHead(P);Obs,Free←OccupancyHead(P);RiskComp←RiskComponentHead(P) Fire,Obs,Free,RiskComp←TransformToRobotFrame(Fire,Obs,Free,RiskComp) Risk←BuildRiskMap(Fire,Obs,Free,RiskComp) Health←BuildHealthState( C sync , W v , W t , W r DeviceStatus) return Fire,Obs,Free,Risk,Health

[0233] Specifically, this means that within an online inference cycle, visible light frames, thermal infrared frames, and millimeter-wave frames are acquired simultaneously, and data matching is performed according to the target time to obtain the synchronization confidence of each modality and the overall synchronization confidence. Subsequently, the visible light frames are constructed as a visible light input tensor, the thermal infrared data is converted into a thermal infrared input tensor containing temperature, temperature gradient, and temperature rise information, and the millimeter-wave data is converted into a virtual visual tensor containing distance, signal-to-noise ratio, velocity, and occupancy probability.

[0234] Each input tensor is fed into a three-vision virtual alignment situational awareness network to obtain multi-scale visible light features, thermal infrared features, and millimeter-wave features. Based on coarse-mounted priors, a local search region is constructed within the virtual reference domain. The residual offset fields and confidence masks corresponding to the thermal infrared and millimeter-wave features are predicted through cross-modal local correlation. The residual offset fields are then used to deformably sample the features to be aligned, mapping them to the virtual reference domain.

[0235] Furthermore, by combining time synchronization reliability, visible light clarity and smoke obscuration, effective pixel ratio and saturation ratio of thermal infrared, millimeter-wave signal-to-noise ratio and point density, and confidence mask, the fusion weights of each mode are determined. High-confidence features are retained, medium-confidence features are attenuated, and low-confidence features are rejected for strong fusion. Based on the fused features, fire source information, obstacle information, free space information, risk cost map, and equipment health status are output, and the results are converted to the robot coordinate system and encapsulated for output. When any mode continues to be of low quality or fails, degradation inference is performed according to the available modes and the corresponding state information is output.

[0236] The following is pseudocode for the TVA-SANet training and edge deployment algorithm according to an embodiment of the present invention: Input: Spatially partitioned dataset D, Teacher network, Target edge platform E Output: Validated edge model package D train , D val , D test ←SplitByScene(D) Repeat for each epoch B←SampleBatch( D train ) B aug ←SmokeThermalRadarDegrade(B) B aug ←RandomSpatialTemporalMisalignment( B aug ) Pred,Offset,Mask,Gate←TVASANet( B aug ) L← L fire +0. 6L mask +0. 4L temp +0. 8L occ +0. 5L risk +0. 2L align +0. 1L smooth+0. 2L gate L←L+0. 3L distill (Pred,Teacher(B)) BackPropagateAndUpdate(L) until validation metric no longer improves ModelPruned←ChannelPruneAndRecover(TVASANet) ModelRep←Reparameterize(ModelPruned) ModelMix←Quantize(INT8 backbone, FP16 offset / temp layers) Result←Verify(ModelMix,E,metrics=[accuracy,risk,FPS,latency,power,stability]) if Result passes thresholds then SignAndPackage(ModelMix) elseRollBackAndRetrain return Package

[0237] Specifically, this means dividing the three-vision synchronization samples into training, validation, and test sets according to independent spaces. During training, a batch of samples are obtained from the training set, and smoke degradation, thermal infrared degradation, millimeter-wave degradation, and random spatial and temporal misalignment enhancements are applied to the samples to simulate modal quality changes, installation errors, and synchronization errors in a dense smoke fire scene.

[0238] The enhanced samples are input into a three-vision virtual alignment situational awareness network to obtain results for fire source, temperature, obstacle occupancy, risk cost map, residual offset field, confidence mask, and modal quality gating. A joint loss is constructed based on fire source detection, fire source region, temperature regression, obstacle occupancy, risk characterization, cross-modal alignment consistency, residual offset field smoothness, and quality gating. A distillation loss is constructed by combining the teacher network output. The network parameters are updated through backpropagation until the validation metric no longer improves.

[0239] After training, the network undergoes channel pruning and retraining, followed by structural reparameterization. Specifically, the backbone network is quantized using INT8, while FP16 accuracy is preserved for residual offset prediction, deformable sampling, and temperature regression-related components, resulting in a mixed-precision edge model. This edge model is then deployed to the target edge platform to verify perception accuracy and risk cost. Figure 1 Consistency, frame rate, end-to-end latency, power consumption, and long-term operational stability are evaluated. If the validation is successful, a model package containing model files, normalized parameters, quantization scale, compatible version, and rollback version is generated. If the validation fails, the model is rolled back and retrained.

[0240] The following is the pseudocode for the local correlation-bounded residual offset-confidence mask-deformable compensation algorithm according to an embodiment of the present invention: Input: Multi-scale reference features {F_v^s}, features to be aligned {F_i^s}, coarse mounting prior P_0, initial effective mask {V_i^s}, i∈{t,r} Output: Residual offset field {ΔP_i^s}, confidence mask {M_i^s}, alignment feature {F_i^{a,s}} ΔP_i^{s+1}←0 for s=1 / 16,1 / 8,1 / 4do p0←Π_i^s(P_0,p);d_coarse←κ_s·Up(ΔP_i^{s+1}) Ω_i^s(p)←LocalWindow(p0+d_coarse,R_s) Corr_i^s(p,δ)←GroupedCosine(F_v^s(p),F_i^s(p0+d_coarse+δ)) P_i^s(δ|p)←Softmax(Corr_i^s / θ) d_i^s←Σ_δP_i^s(δ|p)·δ;H_i^s←-Σ_δP_i^sln(P_i^s+ε) Z_i^s,B_i^s←OffsetMaskNet(Corr_i^s,F_v^s,F_i^s,d_i^s,RangePrior) ΔP_i^s←d_coarse+Clip(d_i^s+R_s·tanh(Z_i^s),[-R_s,R_s]) M_pre←V_i^s·σ(B_i^s+a1·max(Corr_i^s)-a2·H_i^s) q_i^s←p0+ΔP_i^s;F_sample←BilinearSample(F_i^s,q_i^s) E_post←1-Cosine(F_v^s,F_sample) M_i^s←M_pre·exp(-γE_post) F_i^{a,s}←M_i^s⊙F_sample end for return {ΔP_i^s},{M_i^s},{F_i^{a,s}}

[0241] Specifically, this means that for thermal infrared and millimeter-wave features, visible light features or a preset virtual reference grid are used as virtual reference domains at each scale. The coarse predicted position of the feature to be aligned is determined based on the coarse mounting prior, and a bounded local search window is established around the coarse predicted position by combining the residual offset field of the previous lower resolution scale.

[0242] Within the local search window, the normalized correlation between the reference feature and each candidate feature to be aligned is calculated. The correlation values ​​are converted into a candidate displacement probability distribution, and the initial displacement and correlation distribution entropy are obtained based on the probability distribution. The local correlated volume, initial displacement, reference feature, feature to be aligned, coarse installation displacement, initial effective mask, and millimeter-wave distance prior are input into the offset-mask predictor, which outputs the residual offset field and mask logic value. The residual offset field is constrained by the current scale search radius and is progressively refined from low resolution to high resolution.

[0243] The initial effective mask, mask logic value, maximum correlation value, and correlation distribution entropy are combined to obtain a pre-compensation confidence mask. Then, based on the consistency error between the compensated reference features and the sampled features, the confidence mask is verified and attenuated. Subsequently, the coarse installation mapping position is superimposed with the residual offset field to obtain continuous sampled coordinates, and bilinear sampling is used to obtain alignment features. For high-confidence positions, the compensated alignment features are retained; for medium-confidence positions, the features are attenuated according to the confidence mask; and for low-confidence positions, they are marked as having no corresponding feature or an unknown state and do not participate in strong fusion.

[0244] It should be noted that the definitions of various terms in the embodiments of the present invention specifically include: The unified three-vision data set refers to the data object consisting of the visible light vision tensor, thermal infrared vision tensor, and millimeter-wave virtual vision tensor corresponding to the same target inference time, as well as the submodal state confidence, overall synchronization reliability, equipment status, and coarse installation prior.

[0245] The virtual reference domain is a unified feature coordinate space established for the three-vision virtual alignment situational awareness network. It uses visible light features as a reference to establish a unified spatial correspondence between the three modal features and is not the same as the fused image viewed by a human.

[0246] The coarse installation prior includes the intrinsic parameters of each sensor and the initial extrinsic parameters or coordinate transformation relationships between sensors. It is used to limit the local search range of thermal infrared features and millimeter-wave features relative to visible light features, without requiring pixel-level accuracy.

[0247] The residual offset field is used to characterize the local spatial offset that still exists after coarse installation prior mapping. It is determined by cross-modal local correlation, candidate displacement probability distribution and initial displacement, and is used to sample and align thermal infrared features and millimeter wave features.

[0248] The confidence mask outputs continuous confidence values ​​at each spatial location, which characterize the reliability of the spatial correspondence between the thermal infrared or millimeter-wave features and the visible light features at that location. This is an important basis for determining the fusion weights.

[0249] Fire situation information and risk cost map refers to the structured results output by the edge inference unit after fusing visible light features, thermal infrared alignment features and millimeter wave alignment features. It includes the location and temperature trend of the fire source, the status of obstacles and free space, smoke risk and comprehensive risk cost, as well as modal quality and equipment health status.

[0250] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A three-vision virtual alignment situational awareness equipment for dense smoke and fire scenes, characterized in that, It includes a three-vision sensor head, a synchronous acquisition unit, an edge inference unit, and an output interface; The three-vision sensor head includes a visible light camera, a thermal infrared camera, and a millimeter-wave radar, which are used to collect visible light data, thermal infrared data, and millimeter-wave data, respectively. The synchronous acquisition unit is connected to the three vision sensor head and the edge inference unit respectively, and is used to send visible light data, thermal infrared data and millimeter wave data corresponding to the same inference time to the edge inference unit. The edge inference unit is connected to the output interface and is used to construct three visual features based on visible light data, thermal infrared data, and millimeter wave data. Based on the sensor coarse installation prior, thermal infrared and millimeter-wave features are mapped to a virtual reference domain to determine the residual offset field and the corresponding confidence mask within the local search range. Based on the residual offset field, thermal infrared features and millimeter wave features are sampled and aligned. The fusion weights are determined according to the time synchronization reliability, data quality of each mode, and confidence mask to fuse visible light features, thermal infrared aligned features, and millimeter wave aligned features to generate fire situation information and risk cost map. The output interface is used to output fire situation information and risk cost map.

2. The three-vision virtual alignment situational awareness equipment for dense smoke and fire scenes as described in claim 1, characterized in that, The three-vision sensor head also includes a mounting reference component, on which the visible light camera, the thermal infrared camera, and the millimeter-wave radar are fixedly mounted; The sensor coarse installation prior includes the intrinsic parameters of the visible light camera, the thermal infrared camera, and the millimeter-wave radar, as well as the extrinsic parameters or coordinate transformation relationships among the visible light camera, the thermal infrared camera, and the millimeter-wave radar.

3. The three-vision virtual alignment situational awareness equipment for dense smoke and fire scenes as described in claim 1, characterized in that, The synchronous acquisition unit includes data buffer areas for caching the visible light data, the thermal infrared data, and the millimeter wave data, respectively; The synchronous acquisition unit selects the corresponding visible light data, thermal infrared data, and millimeter wave data from the data buffer based on the target inference time and the timestamps or trigger signals corresponding to the visible light data, thermal infrared data, and millimeter wave data, respectively. The time deviation is determined based on the timestamps or trigger signals corresponding to the selected visible light data, thermal infrared data, and millimeter wave data, and the time synchronization reliability is determined based on the time deviation. The synchronous acquisition unit sends the selected visible light data, thermal infrared data, millimeter wave data, and the time synchronization reliability to the edge inference unit.

4. The three-vision virtual alignment situational awareness equipment for dense smoke and fire scenes as described in claim 1, characterized in that, The edge inference unit determines the local search range based on the sensor coarse installation prior, and constructs cross-modal local correlations between the visible light feature and the thermal infrared feature, and between the visible light feature and the millimeter wave feature, respectively, within the virtual reference domain; The thermal infrared residual offset field, the millimeter-wave residual offset field, and the corresponding confidence mask are determined based on the cross-modal local correlator. The offset of the thermal infrared residual offset field and the millimeter-wave residual offset field are constrained by the local search range. The confidence mask is a pixel-level continuous confidence mask.

5. The three-vision virtual alignment situational awareness equipment for dense smoke and fire scenes as described in claim 4, characterized in that, The edge inference unit determines the sampling position based on the thermal infrared residual offset field and the millimeter wave residual offset field, and uses bilinear deformable sampling to sample the thermal infrared feature and the millimeter wave feature respectively to obtain the thermal infrared alignment feature and the millimeter wave alignment feature; For thermal infrared features or millimeter-wave features whose confidence values ​​at spatial locations are lower than a preset threshold, the corresponding bilinear deformable sampling is suppressed, or the fusion weight of the corresponding mode at the spatial location is reduced.

6. The three-vision virtual alignment situational awareness equipment for dense smoke and fire scenes as described in claim 1, characterized in that, The edge inference unit determines the quality of each modal data based on the clarity and / or smoke obscuration of the visible light data, the effective pixel ratio and / or saturation ratio of the thermal infrared data, and the signal-to-noise ratio and / or dot density of the millimeter wave data. When the quality of any modal data is lower than a preset quality threshold, the corresponding modality is downweighted or masked, or inference is performed based on one or two unmasked modalities to generate the fire situation information and the risk cost map, and the corresponding modal downgrade status is sent to the output interface.

7. A three-vision virtual alignment situational awareness method for dense smoke fire scenes, characterized in that, The method includes: Acquire visible light data, thermal infrared data, and millimeter wave data corresponding to the same inference time, and determine the time synchronization reliability based on the visible light data, thermal infrared data, and millimeter wave data; Visible light features, thermal infrared features, and millimeter wave features are constructed based on the visible light data, thermal infrared data, and millimeter wave data. Based on the sensor coarse installation prior, the thermal infrared features and the millimeter-wave features are mapped to a virtual reference domain, and the residual offset field and the corresponding confidence mask are determined within the local search range. The thermal infrared feature and the millimeter-wave feature are sampled and aligned according to the residual offset field to obtain thermal infrared aligned features and millimeter-wave aligned features. The quality of each modal data is determined based on the visible light data, thermal infrared data, and millimeter wave data. The fusion weight is determined based on the time synchronization reliability, the quality of each modal data, and the confidence mask. The visible light features, thermal infrared alignment features, and millimeter wave alignment features are fused based on the fusion weight to generate fire situation information and risk cost map.

8. The three-vision virtual alignment situational awareness method for dense smoke fire scenes as described in claim 7, characterized in that, The step of determining the residual offset field and the corresponding confidence mask within the local search range includes: determining the local search center based on the sensor coarse installation prior and the residual offset field at the low-resolution feature scale; and calculating the cross-modal correlations between the visible light feature and the thermal infrared feature, and between the visible light feature and the millimeter-wave feature, respectively, within the local search range around the local search center to obtain the candidate displacement probability distribution. The initial displacement is determined based on the candidate displacement probability distribution, and the thermal infrared residual offset field and millimeter wave residual offset field are determined based on the initial displacement and the bounded correction amount. Pixel-level continuous confidence masks corresponding to the thermal infrared residual offset field and the millimeter-wave residual offset field are determined based on at least two of the following: initial effective mask, maximum correlation value, candidate displacement probability distribution entropy, mask logic value, and compensated cross-modal consistency error.

9. The three-vision virtual alignment situational awareness method for dense smoke fire scenes as described in claim 8, characterized in that, The step of sampling and aligning the thermal infrared feature and the millimeter-wave feature based on the residual offset field includes: Based on the thermal infrared residual offset field and the millimeter-wave residual offset field, bilinear deformable sampling is used to sample the thermal infrared feature and the millimeter-wave feature respectively to obtain the thermal infrared alignment feature and the millimeter-wave alignment feature; Based on the confidence mask, the thermal infrared alignment feature and the millimeter wave alignment feature are respectively determined as high-confidence alignment feature, medium-confidence alignment feature, or low-confidence alignment feature; The high-confidence alignment features are preserved, the medium-confidence alignment features are attenuated, and the low-confidence alignment features are rejected from fusion. The time synchronization reliability is determined based on the time deviation between the visible light data, the thermal infrared data, and the millimeter wave data. The fusion weight is determined based on the time synchronization reliability, the quality of each modal data, and the confidence mask.

10. The three-vision virtual alignment situational awareness method for dense smoke fire scenes as described in claim 7, characterized in that, The fire situation information includes the fire source status, obstacle status, and free space; The fire source state, the obstacle state, and the free space are transformed into the robot coordinate system, distance estimation and risk aggregation are performed, and the risk cost map is generated. When the data quality of any modality is lower than the first preset quality threshold for multiple consecutive inference cycles, the upper limit of the fusion weight of the corresponding modality is limited and the modality degradation status is output. When the quality of all modal data is lower than the second preset quality threshold or the time synchronization reliability is lower than the preset synchronization reliability threshold, the perception unavailable state is output. The fire situation information, the risk cost map, the modal degradation state, or the perception unavailability state are sent as a unified perception data frame to the robot control system.

Images