Fire-fighting three-dimensional situation construction method and system based on visual feedback correction

Abstract

This invention relates to a method and system for constructing a 3D fire situation based on visual feedback correction, belonging to the interdisciplinary field of digital twins and fire protection IoT. It includes: acquiring a 3D building model and monitoring video stream; constructing a mapping benchmark based on the intrinsic and extrinsic parameters of the monitoring equipment; back-projecting the video stream onto the surface of the 3D model to generate a dynamic projection texture, while simultaneously mapping the corner points of the detection frame onto the 3D model to construct a closed bounding box as a physical benchmark for the fire situation; comparing the spatial deviation between the dynamic projection texture and the physical benchmark for the fire situation to calculate the reprojection error; if the error exceeds the tolerance, iteratively optimizing the extrinsic parameters to generate a mapping correction benchmark; transmitting the mapping correction benchmark back and re-performing projection and positioning; obtaining the diffusion situation based on the corrected closed bounding box combined with the building material properties; and outputting a 3D fire situation scene. This invention achieves continuous alignment between monitoring video and the 3D model and the construction of the physical space of the fire situation, and can be widely applied in the field of intelligent fire situation visualization.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of digital twins and fire protection Internet of Things, specifically involving a method and system for constructing a three-dimensional fire situation based on visual feedback correction. Background Technology

[0002] The mainstream technical approaches for visualizing building fire safety conditions currently fall into two categories: First, there's the visualization method based on two-dimensional floor plans. Most fire protection IoT platforms still present the fire situation by overlaying alarm points onto floor plans or CAD drawings. Alarm points are only marked with icons on the floor plan, without spatial correlation to the actual monitoring footage. On-duty personnel need to manually switch to view the video, making it impossible to accurately map the flame location in the video to the building space. Two-dimensional floor plans cannot present the building's height dimension information, making it impossible to intuitively display the vertical spread path of smoke between floors in high-rise building fires. Furthermore, sensor alarms, video footage, and building structural information are scattered across different pages, requiring on-duty personnel to manually correlate and comprehensively judge, resulting in long response times and low accuracy.

[0003] Second, there's the visualization method based on 3D digital twins. In recent years, 3D fire protection digital twin systems based on BIM+GIS have gradually emerged, allowing monitoring videos to be mapped onto the surface of a 3D model using texture mapping. Existing video projection solutions disclose basic methods for mapping video image pixels onto the surface of a 3D model. However, existing solutions still have the following technical shortcomings: Camera calibration parameters (especially extrinsic parameters) can drift over long-term use due to factors such as equipment vibration and structural thermal expansion and contraction caused by temperature changes. This causes the video projection position to gradually deviate from the actual spatial position, with a projection error rate often exceeding 2%, affecting the accurate judgment of flame location. Although existing technologies include methods to minimize reprojection errors by adjusting camera extrinsic parameters through bundle adjustment and video fusion methods based on 3D scenes, these all rely on static calibration objects (such as checkerboard calibration boards) or pre-calibrated reference points for parameter correction, and cannot achieve online automatic correction of calibration parameters in fire emergency scenarios.

[0004] The accumulation of projection errors lacks effective detection and correction methods. The drift of camera calibration parameters is a gradual, cumulative process, difficult to detect in conventional monitoring scenarios. However, in fire emergency scenarios, projection deviations can cause the flame position to shift by several meters in the 3D model, leading on-duty personnel to formulate rescue plans based on incorrect spatial locations, with serious consequences. In existing solutions, reprojection errors are typically calculated only during the calibration phase, lacking a mechanism for continuous monitoring and automatic correction based on dynamically detected targets during runtime.

[0005] In existing video projection solutions, the flame / smoke detection frame is only displayed on the video screen and is not physically linked to the 3D model space. The pixel-level information of the detection frame cannot be directly converted into physical fire information in the building space. On-duty personnel still need to "guess the location from the video" and cannot directly obtain precise information about which building, floor, or room the flame is in from the 3D model. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a method for constructing a three-dimensional fire situation based on visual feedback correction. The objective of this invention can be achieved through the following technical solutions: A method for constructing a three-dimensional fire situation based on visual feedback correction includes the following steps: Obtain the 3D model of the building and the monitoring video stream, and construct a mapping benchmark based on the intrinsic and extrinsic parameters of the monitoring equipment; Based on the mapping reference, the video stream is back-projected onto the surface of the three-dimensional model and the texture coordinates are corrected according to the surface differential geometry properties to generate dynamic projection texture; at the same time, the corner points of the flame or smoke detection box in the video stream are mapped to the three-dimensional model to construct a closed bounding box as the physical reference of the fire. The spatial deviation between the dynamic projection texture and the fire physical reference is compared, the reprojection error accumulated over multiple frames is calculated, the intrinsic parameters are locked when the error exceeds the preset tolerance, and the extrinsic parameters are iteratively optimized under the physical constraints of the installation pose of the monitoring equipment to generate a mapping correction reference. The mapping correction reference is returned to update the mapping reference, and the projection and positioning are re-executed. Based on the spatial position and range of the corrected enclosed box and the building material properties, the diffusion pattern of flames and smoke is obtained, and a three-dimensional fire situation scene is output.

[0007] Specifically, the process of constructing the mapping benchmark includes: The installation location coordinates and orientation angle of the monitoring equipment are marked in the three-dimensional model as initial external parameters; the pixel coordinates of at least four sets of known building structure feature points in the monitoring equipment screen are obtained, and the three-dimensional spatial coordinates corresponding to each set of feature points are marked in the three-dimensional model; Using pixel coordinates and 3D spatial coordinates as input, the intrinsic and extrinsic parameter matrices of the monitoring device are obtained. The inverse of the intrinsic and extrinsic parameter matrices is used as the mapping reference from the pixel coordinate system to the world coordinate system.

[0008] Specifically, the generation process of the dynamic projection texture includes: Calculate the principal curvature and Gaussian curvature of the surface where the hit point is located, and determine the surface type based on the sign of the Gaussian curvature; For convex surfaces, the ratio of arc length to chord length is calculated as the distortion compensation coefficient; for saddle surfaces, the independent arc-chord ratios along the two principal curvature directions are calculated as the distortion compensation coefficients; for planes, the distortion compensation coefficient is set to 1; the texture coordinates are reverse-corrected using the distortion compensation coefficient to generate the dynamic projection texture.

[0009] Specifically, the process of mapping the corner points of the flame or smoke detection box in the video stream to the 3D model includes: Obtain the pixel coordinates of the four corner points of the detection box. Starting from the optical center of the monitoring device, perform ray projection along the ray direction corresponding to the four corner points respectively, and find the intersection with the surface of the three-dimensional model to obtain four three-dimensional hit points. The spatial region enclosed by the four three-dimensional hit points is defined as a closed bounding box, and the six faces of the closed bounding box are parallel to the coordinate axis plane of the three-dimensional model. The spatial coordinate range of the closed bounding box in the three-dimensional model is extracted as the physical reference of the fire situation.

[0010] Specifically, the calculation method for the reprojection error includes: For each frame, the pixel distance between the color centroid position of the region corresponding to the detection box in the dynamic projection texture and the projection center position of the closed bounding box in the fire physical reference projected onto the image plane is taken as the single-frame projection deviation of the frame, and the weighted average of the projection deviations of consecutive single frames is taken as the cumulative reprojection error of the multiple frames.

[0011] Specifically, the iterative optimization process for external parameters includes: The intrinsic parameter matrix of the fixed monitoring equipment remains unchanged, and the installation position coordinates and orientation angle of the monitoring equipment are used as variables to be optimized; the optimization boundary of the installation position coordinates is set as a spatial sphere range with a preset radius centered on the initial calibration position, and the optimization boundary of the orientation angle is set as a cone range with a preset angle centered on the initial calibration orientation. Within the optimization boundary, the extrinsic parameters are iteratively solved with the goal of minimizing the reprojection error. After each iteration, it is determined whether the optimization variable exceeds the optimization boundary. If it does, the value is truncated to the boundary value.

[0012] Specifically, after updating the mapping benchmark, the process also includes a multi-channel video time synchronization step: The system acquires video streams output by monitoring equipment, with each frame of each video stream carrying a capture timestamp. Using the latest common timestamp of all video streams as the synchronization benchmark, video frames with timestamps earlier than the synchronization benchmark are buffered and waited for, while video frames with timestamps later than the synchronization benchmark are discarded. At the synchronization benchmark time, the multiple video streams are synchronously projected onto the surface of the three-dimensional model.

[0013] Specifically, the process of acquiring the diffusion situation includes: The spatial location of the enclosed box is taken as the starting position of the fire source, and the spatial range of the enclosed box is taken as the initial size of the fire source; the structural information of the area where the fire source is located is obtained from the three-dimensional model, including the wall material type and ventilation condition parameters; The first correction factor for the horizontal spread rate is determined based on the wall material type, and the second correction factor for the horizontal spread rate is determined based on the ventilation condition parameters. The actual horizontal spread rate is obtained, and the flame spread boundary at the preset time step is obtained. The flame spread boundary at the preset time step is visualized in the 3D model using 3D bounding boxes of different colors.

[0014] Specifically, after obtaining the spread situation, the process also includes a rescue path planning step: The space of the three-dimensional model is discretized into a three-dimensional mesh; the mesh area covered by the flame spread boundary and the smoke diffusion path is used as the restricted area; the mesh nodes corresponding to corridors and stair passages are assigned a first passage cost, and the mesh nodes corresponding to narrow passages are assigned a second passage cost, with the second passage cost being greater than the first passage cost; the path with the minimum passage cost is searched from the building entrance as the starting point and the spatial location of the enclosed box as the ending point as the optimal rescue path.

[0015] The beneficial effects of this invention are as follows: This invention establishes a visual feedback closed-loop correction mechanism of "projection → detection → comparison → correction → reprojection," utilizing the dynamically detected flame or smoke itself as a reference benchmark for calibration parameter correction, rather than relying on static calibration objects. When the reprojection error between the projected texture and the physical benchmark mapped to the detection box exceeds the tolerance, the system automatically triggers online correction of the calibration parameters. Compared with existing technologies that rely on static calibration objects or offline recalibration, this invention achieves automatic detection and dynamic correction of calibration parameters during operation, enabling the video projection to continuously maintain high-precision alignment with the architectural 3D model.

[0016] This invention proposes a correction strategy of "locking intrinsic parameters and iteratively optimizing under the physical constraints of extrinsic parameters." The intrinsic parameters of the camera (focal length, principal point coordinates, etc.) are determined by factory calibration and are less affected by environmental changes; while the extrinsic parameters (installation position and orientation angle) are significantly affected by factors such as equipment vibration and temperature changes, and are the main source of calibration drift. This invention keeps the intrinsic parameters unchanged, optimizes only the extrinsic parameters, and performs iterative solutions under the physical boundary constraints of the installation pose. This avoids singular solutions that may arise from unconstrained optimization and improves the convergence speed and reliability of the correction process.

[0017] This invention maps the corner points of a flame or smoke detection frame onto the surface of a 3D model using ray projection, generating a closed bounding box representing the physical extent of the fire. This transforms the pixel-level fire information from the 2D detection frame into physical-level fire information in the 3D model space. On-duty personnel can directly obtain the floor and specific spatial location of the flame in the 3D situation map, eliminating the need to "guess the location from a video," significantly improving the efficiency and accuracy of fire situation assessment.

[0018] This invention uses a modified enclosed box as the fire source input and combines the combustion characteristic parameters of the wall material in the 3D model to deduce the flame spread boundary and smoke diffusion path, realizing a complete technical chain from "precise positioning" to "situational prediction". Compared with the traditional approach where video projection and situational prediction are independent, this invention directly drives the spread prediction with the modified spatial positioning results, making the prediction results based on the precise physical location of the fire source and improving the reliability of situational prediction. Attached Figure Description

[0019] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0020] Figure 1 This is a schematic diagram of the overall process of a fire-fighting three-dimensional situation construction method based on visual feedback correction according to the present invention; Figure 2 This is a flowchart of the closed-loop process for dynamic correction of calibration parameters in this invention; Figure 3 This is a diagram illustrating the rescue route planning effect in this invention. Detailed Implementation

[0021] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0022] Please see Figures 1-3 A method for constructing a three-dimensional fire situation based on visual feedback correction includes the following steps: Obtain the 3D model of the building and the monitoring video stream, and construct a mapping benchmark based on the intrinsic and extrinsic parameters of the monitoring equipment; Based on the mapping reference, the video stream is back-projected onto the surface of the three-dimensional model and the texture coordinates are corrected according to the surface differential geometry properties to generate dynamic projection texture; at the same time, the corner points of the flame or smoke detection box in the video stream are mapped to the three-dimensional model to construct a closed bounding box as the physical reference of the fire. The spatial deviation between the dynamic projection texture and the fire physical reference is compared, the reprojection error accumulated over multiple frames is calculated, the intrinsic parameters are locked when the error exceeds the preset tolerance, and the extrinsic parameters are iteratively optimized under the physical constraints of the installation pose of the monitoring equipment to generate a mapping correction reference. The mapping correction reference is returned to update the mapping reference, and the projection and positioning are re-executed. Based on the spatial position and range of the corrected enclosed box and the building material properties, the diffusion pattern of flames and smoke is obtained, and a three-dimensional fire situation scene is output.

[0023] A 3D model needs to be built for the target building, and a mapping benchmark needs to be established between the video pixels of the monitoring equipment and the 3D model space. For new building projects, a parametric rapid modeling method is adopted: a template matching the target building type is selected from the built-in building template library, and key building parameters, including the number of floors, floor height, area per floor, corridor width, and stair location parameters, are input. The template engine automatically generates a 3D model of the building based on these parameters, including structural elements such as floor slabs, exterior walls, corridors, stairwells, elevator shafts, and room partitions. For building projects with existing BIM models, the BIM model is lightweighted by removing detailed components irrelevant to fire safety, retaining core elements such as building structure, fire escape routes, and fire protection facilities. At the same time, the geometric surfaces of the BIM model are merged into simplified patches, compressing the model size to within 30MB, ensuring a frame rate of no less than 30 frames per second when rendering on the web.

[0024] After the 3D model is constructed, spatial registration and calibration of each monitoring device are performed within the model. For each camera, the approximate 3D coordinates and orientation of the camera's installation location are marked in the 3D model. Then, a set of known building structure reference points from the actual camera footage are acquired. These reference points can be feature points that are easily identifiable and marked in both the video footage and the 3D model, such as wall corners, window frame corners, or door frame corners. The corresponding 3D spatial coordinates of each set of reference points are marked in the 3D model. Using the pixel coordinates and 3D spatial coordinates of the reference points as input, the camera's intrinsic and extrinsic parameter matrices are solved using a perspective n-point positioning algorithm. The intrinsic parameter matrix includes the lateral focal length, longitudinal focal length, principal point lateral coordinates, and principal point longitudinal coordinates, representing the camera's imaging parameters. The extrinsic parameter matrix includes the camera's 3D position coordinates and 3D orientation angle in the world coordinate system, representing the camera's installation pose in the building space. The inverse of the intrinsic and extrinsic parameter matrices is used as a mapping reference from the pixel coordinate system to the world coordinate system. This mapping reference provides a forward mapping relationship from 3D spatial coordinates to pixel coordinates, and a reverse mapping relationship from pixel coordinates to the ray direction in the 3D model space.

[0025] After initial calibration, the three-dimensional spatial coordinates of each group of feature points are reprojected to the pixel coordinate system using the current intrinsic and extrinsic parameter matrices to obtain the reprojected pixel coordinates. The Euclidean distance between the reprojected pixel coordinates of each group of feature points and their actual pixel coordinates is calculated. The root mean square value of the Euclidean distance of all feature points is taken as the reprojection error. With the goal of minimizing the reprojection error, the Levenberg-Marquardt iterative algorithm is used to jointly optimize the intrinsic and extrinsic parameter matrices. After each iteration, the intrinsic and extrinsic parameters are updated and the reprojection error is recalculated until the number of iterations reaches a preset upper limit or the change in the reprojection error is less than a preset threshold. The optimized mapping benchmark is then output.

[0026] After the mapping reference is constructed, the system receives video streams from various monitoring devices in real time. For each frame of each video stream, the system performs an inverse projection transformation based on the mapping reference: for each pixel in the video frame, the normalized ray direction is obtained through inverse transformation of the intrinsic parameter matrix, and then the ray direction in the world coordinate system is obtained through inverse transformation of the rotation matrix of the extrinsic parameter matrix. Starting from the spatial position of the camera, the ray is projected along the ray direction in the world coordinate system, and the intersection with the surface of the 3D model is obtained to obtain the hit point. After obtaining the hit point, the system calculates the curvature parameters of the surface where the hit point is located, including the principal curvature and the Gaussian curvature. The surface type is determined according to the sign of the Gaussian curvature: when the Gaussian curvature is positive, it is determined to be a convex surface; when the Gaussian curvature is negative, it is determined to be a saddle surface; when the Gaussian curvature is zero, it is determined to be a plane. For a convex surface, the ratio of arc length to chord length is calculated as the distortion compensation coefficient; for a saddle surface, the independent arc-chord ratio along the two principal curvature directions is calculated as the distortion compensation coefficient; for a plane, the distortion compensation coefficient is set to 1. The texture coordinates are reverse-corrected using the aforementioned distortion compensation coefficient. Then, the corrected texture coordinates are used to sample the video texture, and the video color values ​​are written to the corresponding positions on the surface of the 3D model to generate a dynamic projection texture without stretching or deformation. The dynamic projection texture carries spatial location information—each pixel on the surface of the 3D model is associated with its 3D spatial coordinates in the camera coordinate system.

[0027] The system acquires the detection results output by the flame or smoke detection model. These results include the corner pixel coordinates of the detection box in the video frame, the detection confidence score, and the category label. For each detection box, the system performs a transformation from the detection box to three-dimensional space based on the mapping reference: starting from the optical center of the monitoring device, ray projection is performed along the ray directions corresponding to the four corner points, intersecting with the surface of the three-dimensional model to obtain four three-dimensional hit points. The spatial region enclosed by these four hit points is defined as a closed bounding box, with its six faces parallel to the coordinate axes of the three-dimensional model. This closed bounding box represents the actual physical extent of the flame or smoke in the three-dimensional space of the building. The spatial coordinate range of the closed bounding box in the three-dimensional model is extracted, including the minimum and maximum values ​​in the horizontal direction, the minimum and maximum values ​​in the width direction, and the minimum and maximum values ​​in the height direction, serving as the physical reference for the fire situation.

[0028] After the bounding box is constructed, the repair confidence level corresponding to the bounding box is calculated: the product of the detection confidence level of the detection frame and the raycasting hit rate is used as the original confidence level of the bounding box; the raycasting hit rate is the ratio of the number of corner points that successfully hit the surface of the 3D model to the total number of corner points; when the hit rate is lower than a preset threshold, the frame is marked as a low-confidence frame, and the bounding box of the frame is not included in the subsequent calculation of reprojection error accumulation. This confidence evaluation mechanism ensures that only reliable detection results can be used for subsequent calibration parameter correction, avoiding erroneous corrections caused by false detections or projection failures.

[0029] After the dynamic projection texture and the fire physical reference are generated separately in the same frame, their spatial positions are compared to calculate the projection deviation. Specifically, for each frame, the pixel distance between the color centroid position of the region corresponding to the detection box in the dynamic projection texture and the projection center position of the bounding box in the fire physical reference projected onto the image plane is taken as the single-frame projection deviation for that frame. The weighted average of the single-frame projection deviations of multiple consecutive frames is taken as the cumulative reprojection error, where the weight of each frame is determined by the repair confidence of the bounding box of that frame—frames with higher repair confidence have a larger weight in the cumulative error calculation.

[0030] When the accumulated reprojection error across multiple frames exceeds a preset tolerance, the system determines that the current mapping reference has drifted and needs correction. The correction process employs the following strategy: the intrinsic parameter matrix of the fixed monitoring device remains unchanged, while only the extrinsic parameter matrix is ​​iteratively optimized. This is because in engineering practice, the camera's intrinsic parameters (focal length, principal point coordinates, etc.) are determined by factory calibration, are less affected by temperature changes, and remain stable in the short term; while the extrinsic parameters (installation position and orientation angle) are significantly affected by factors such as equipment vibration and structural thermal expansion and contraction caused by temperature changes, and are the main source of calibration drift. During iterative optimization of the extrinsic parameters, physical constraints are set for the monitoring device's installation pose: the optimization boundary for the installation position coordinates is set to a sphere with a preset radius centered on the initial calibration position, and the optimization boundary for the orientation angle is set to a cone with a preset angle centered on the initial calibration orientation. These physical constraints ensure that the optimization results do not deviate from the actual physical installation pose, avoiding singular solutions that may arise from unconstrained optimization. Within the optimization boundary, the Levenberg-Marquardt iterative algorithm is used to iteratively solve for the extrinsic parameters with the objective of minimizing the accumulated reprojection error across multiple frames. After each iteration, it is determined whether the optimization variables exceed the optimization boundary; if so, the values ​​are truncated to the boundary values. After the iteration is complete, the optimized extrinsic parameter matrix is ​​output, generating a mapping correction benchmark.

[0031] When multiple monitoring devices are connected, there may be time discrepancies between the video streams. To solve this problem, time synchronization is performed on the multiple video streams: each frame of each video stream carries a capture timestamp. The latest common timestamp of all video streams is used as the synchronization benchmark. Video frames with timestamps earlier than the synchronization benchmark are buffered and waited for, while video frames with timestamps later than the synchronization benchmark are discarded. At the synchronization benchmark time, the multiple video streams are synchronously projected onto the surface of the 3D model, eliminating the projection misalignment caused by the time discrepancies between the multiple video streams.

[0032] During the re-execution of projection and positioning, the current rendering load of the graphics processor is monitored, and dynamic frame rate rendering is performed: when the rendering load is below a first preset threshold, projection rendering is performed at the original resolution of the video stream; when the rendering load is between the first and second preset thresholds, the projection rendering resolution is reduced to a preset percentage of the original resolution; when the rendering load is above the second preset threshold, the area corresponding to the flame or smoke detection box is rendered at the original resolution, while the resolution of non-detection areas is reduced. This intelligent region-priority rendering strategy ensures that key information for situational awareness remains clear at all times, while guaranteeing that the overall rendering frame rate is not less than 30 frames per second.

[0033] The corrected spatial location and extent of the enclosed box are used as the fire source input, and the situation is extrapolated by combining the building material properties. The specific extrapolation process is as follows: the spatial location of the enclosed box is taken as the starting position of the fire source, and the spatial extent of the enclosed box is taken as the initial size of the fire source; the structural information of the area where the fire source is located is read from the 3D model, including the wall material type and ventilation condition parameters; a first correction coefficient for the horizontal spread rate is determined according to the wall material type (e.g., the correction coefficient is smaller for concrete walls and larger for wooden partitions), and a second correction coefficient for the horizontal spread rate is determined according to the ventilation condition parameters (the correction coefficient is larger under natural ventilation conditions and smaller under enclosed space conditions); the actual horizontal spread rate is obtained by multiplying the preset baseline spread rate by the first correction coefficient and the second correction coefficient, and the flame spread boundary for multiple future time steps is extrapolated using the actual horizontal spread rate; the flame spread boundary for multiple time steps is visualized in the 3D model using 3D enclosed boxes of different colors.

[0034] After acquiring the spread of flames and smoke, a rescue path planning step is executed: the space of the 3D model is discretized into a 3D mesh. The mesh area covered by the flame spread boundary and the smoke spread path is designated as a restricted area. Mesh nodes corresponding to corridors and stairwells are assigned lower passage costs, while mesh nodes corresponding to narrow passages are assigned higher passage costs. Starting from the building entrance and ending at the spatial location of the enclosed box, a 3D A* algorithm is used to search for the path with the minimum passage cost as the optimal rescue path, which is then visualized in the 3D model. Simultaneously, the system acquires real-time monitoring data from smoke and temperature sensors deployed within the building. Using the installation location of each sensor as the center and a preset effective monitoring radius as the influence range, the sensor monitoring values ​​are mapped to color distributions within the influence range. The color distributions of multiple sensors on the same floor are weighted, superimposed, and fused to generate a semi-transparent heatmap overlay. This heatmap overlay is then rendered on the corresponding floor plane of the 3D model.

[0035] The system sends back the mapping correction reference to update the mapping reference, re-executes projection and positioning, and infers the spread of flames and smoke based on the spatial position and range of the corrected bounding box and the building material properties, ultimately outputting a 3D fire situation scene. Throughout the process, video projection and flame enhancement are updated in real time every frame, sensor thermal maps are updated every 5 seconds, and spread prediction and path planning are updated every 30 seconds. All update operations are automatically executed in the graphics processor rendering loop, without the need for manual refresh.

[0036] This embodiment uses a standard floor of a high-rise residential building as an example. The standard floor plan is assumed to be a rectangular area, its extent in the 3D model space determined by the minimum X-axis value X_min, maximum X-axis value X_max, minimum Y-axis value Y_min, maximum Y-axis value Y_max, minimum Z-axis value Z_min, and maximum Z-axis value Z_max. A surveillance camera is deployed on this floor, pre-installed on the ceiling at the end of the corridor. The origin of the world coordinate system, O_w, is set at the geometric center of the building's ground floor, with the positive Z-axis pointing vertically upwards.

[0037] In the 3D modeling phase, a parametric rapid modeling approach is employed. Operators select a residential template from the building template library, inputting the following parameters: number of floors N, floor height H, standard floor plan dimensions Lx and Ly, corridor width W_c, and stairwell location Sx and Sy. The template engine generates a 3D model based on these parameters, including floor slabs, exterior walls, corridors, stairwells, and room partition structures. During generation, a material attribute identifier is assigned to each geometric facet of the 3D model. The wall material identifier M_id corresponds to concrete walls, brick walls, and wooden partitions, respectively. Each material is associated with a combustion characteristic parameter, specifically a horizontal spread rate correction coefficient M_mat. Corridor areas in the floor plan are marked as passageways, and stairwell areas are marked as vertical passageways. These attributes are used for subsequent path planning calculations.

[0038] The camera spatial registration and calibration process is as follows. In the 3D model, the operator marks the initial installation position coordinates P_c0 of the camera, which are (X_c0, Y_c0, Z_c0), where Z_c0 corresponds to the ceiling height, equal to the floor height H multiplied by the current floor index minus one plus the floor height. Simultaneously, the initial orientation angles of the camera are marked, including the horizontal yaw angle θ_yaw0 and the vertical pitch angle θ_pitch0. In the actual camera view, the operator selects at least four sets of building structure reference points. The spatial coordinates of these reference points in the 3D model are P_ref_i, where i ranges from 1 to n, and n is not less than 4. The pixel coordinates of each set of reference points in the video frame are p_ref_i, i.e., (u_ref_i, v_ref_i). Substituting each set of 3D spatial coordinates P_ref_i and the corresponding pixel coordinates p_ref_i into the perspective n-point localization algorithm equations, the camera intrinsic parameter matrix K and extrinsic parameter matrix [R|t] are obtained. The intrinsic parameter matrix K has the following form: K=[[f_x,0,c_x],[0,f_y,c_y],[0,0,1]] Where f_x and f_y are the lateral and longitudinal focal lengths, respectively, and c_x and c_y are the lateral and longitudinal coordinates of the principal point. The extrinsic parameter matrix consists of a rotation matrix R and a translation vector t, where R is a 3×3 rotation matrix and t is a 3×1 translation vector. The rotation matrix R is determined by three Euler angles: the pitch angle about the X-axis of the world coordinate system, the yaw angle about the Y-axis, and the roll angle about the Z-axis. The three components of the translation vector t are the X, Y, and Z coordinates of the camera in the world coordinate system, respectively.

[0039] The inverse of the intrinsic parameter matrix K and the extrinsic parameter matrix [R|t] is used as the mapping reference M_ref from the pixel coordinate system to the world coordinate system. This mapping reference includes mapping relationships in two directions: the forward mapping F_forward maps the 3D space coordinate P to the pixel coordinate p, i.e., p = K·[R|t]·P; the inverse mapping F_inverse maps the pixel coordinate p to the ray direction vector d in 3D space, i.e., d = R -1 ·K -1 ·[u, v, 1]ᵀ, where u and v are pixel coordinate values, [u, v, 1]ᵀ is the homogeneous pixel coordinate vector, and R -1 Let K be the inverse of the rotation matrix R. -1 Let K be the inverse of the intrinsic parameter matrix K.

[0040] After initial calibration, the system verifies the reprojection error of the mapping reference. For each set of reference points P_ref_i, the reprojected pixel coordinates p_proj_i = F_forward(P_ref_i) are calculated through forward mapping. The Euclidean distance δ_i between the reprojected pixel coordinates p_proj_i and the actual pixel coordinates p_ref_i is calculated. The root mean square value of all n sets of reference points is taken as the reprojection error RMSE, RMSE = sqrt((1 / n) × Σ||p_proj_i - p_ref_i|| 2 With the goal of minimizing RMSE, the Levenberg-Marquardt iterative algorithm is used to jointly optimize the intrinsic parameter matrix K and the extrinsic parameter matrix [R|t]. After each iteration, K and [R|t] are updated and the RMSE is recalculated until the change in RMSE is less than the preset threshold ε_r or the number of iterations reaches the preset upper limit N_iter. The optimized mapping benchmark M_ref_opt is then output.

[0041] The system receives the video stream output from the camera in real time. Each frame is a color image with a resolution of W×H, where W is the number of pixels in the width direction and H is the number of pixels in the height direction. For each pixel coordinate (u, v) in the current frame, the system performs the following projection texture mapping operation: The ray direction vector d_pixel = F_inverse(u, v) in 3D space is calculated through inverse mapping. Starting from the camera's spatial position P_cam, a ray is projected along the ray direction d_pixel, intersecting with the surface of the 3D model. During ray projection, the ray equation can be expressed as P_ray(t) = P_cam + t × d_pixel, where t is a real number parameter greater than zero, representing the propagation distance along the ray direction. All triangular faces in the 3D model are traversed sequentially, checking whether the ray intersects with a face. The hit point with the smallest positive t value is taken as the hit point P_hit of the pixel on the 3D model surface. The face containing the hit point P_hit has preset texture coordinate parameters. Mapping the hit point P_hit to the texture coordinate space of the face yields the texture coordinates (s_tex, t_tex).

[0042] The system further calculates surface distortion correction parameters at the hit point P_hit. It obtains the first principal curvature κ1 and the second principal curvature κ2 of the surface containing the hit point, and calculates the Gaussian curvature G = κ1 × κ2. The surface type of the hit point is determined based on the sign of the Gaussian curvature G: when G > 0, it is classified as a convex surface; when G < 0, it is classified as a saddle surface; and when G approaches zero, it is classified as a plane. For convex surfaces, the geometric patch parameters of the hit point are read from the 3D model, and the arc length parameter s_arc and chord parameter s_chord along the texture coordinate direction are calculated. The distortion compensation coefficient α is the ratio of s_arc to s_chord, i.e., α = s_arc / s_chord. For saddle surfaces, two independent distortion compensation coefficients α1 and α2 are calculated along the first and second principal curvature directions, respectively. For planes, the distortion compensation coefficient is set to 1. The texture coordinates are reverse-corrected using the distortion compensation coefficient: the corrected texture coordinates (s_corr, t_corr) = (s_tex × α, t_tex × α_corr), where α_corr is the distortion compensation coefficient corresponding to the direction of the second texture coordinates. The color value of the corresponding pixel in the video frame is sampled using the corrected texture coordinates (s_corr, t_corr), and this color value is used as the texture color at the hit point P_hit on the surface of the 3D model. The above operation is performed on all W × H pixels of the current frame to generate a dynamic projected texture covering the visible surface of the 3D model.

[0043] Simultaneously, the system acquires the detection results output by the flame or smoke detection model. Let the detection model output a flame detection box B_box in the current frame, with the pixel coordinates of its four corner points in the image coordinate system being (u1, v1), (u2, v1), (u1, v2), and (u2, v2), respectively. Here, u1 and u2 are the abscissas of the left and right boundaries of the detection box, and v1 and v2 are the ordinates of the top and bottom boundaries of the detection box, respectively. The detection confidence is P_conf. For each corner pixel coordinate of the detection box, the system calculates its corresponding ray direction vector through the inverse mapping F_inverse, and then performs a ray projection and intersection operation with the 3D model surface to obtain the 3D hit points of the four corner points, denoted as P_hit_1, P_hit_2, P_hit_3, and P_hit_4, respectively. These four 3D hit points form a closed bounding box B_3D. The six faces of the enclosing bounding box B_3D are parallel to the coordinate axes of the world coordinate system. Their spatial coordinate range is determined by the following parameters: minimum X-coordinate x_min = min(X_hit_1, X_hit_2, X_hit_3, X_hit_4), maximum X-coordinate x_max = max(X_hit_1, X_hit_2, X_hit_3, X_hit_4), minimum Y-coordinate y_min = min(Y_hit_1, Y_hit_4). The Y-coordinate values ​​are defined as follows: Y_hit_2, Y_hit_3, Y_hit_4. The maximum Y-coordinate value is y_max = max(Y_hit_1, Y_hit_2, Y_hit_3, Y_hit_4). The minimum Z-coordinate value is z_min = min(Z_hit_1, Z_hit_2, Z_hit_3, Z_hit_4). The maximum Z-coordinate value is z_max = max(Z_hit_1, Z_hit_2, Z_hit_3, Z_hit_4). The center point coordinates of the bounding box B_3D are P_center = ((x_min + x_max) / 2, (y_min + y_max) / 2, (z_min + z_max) / 2). This center point represents the physical spatial position of the flame in the current frame. The length, width, and height dimensions of the closed bounding box B_3D are L_box=x_max-x_min, W_box=y_max-y_min, and H_box=z_max-z_min, respectively.

[0044] The system calculates the repair confidence C_conf for the closed bounding box. The raycasting hit rate r_hit is defined as the ratio of the number of corner points that successfully intersect the 3D model surface to the total number of corner points (4). The original confidence C_raw = P_conf × r_hit, where P_conf is the detection confidence output by the detection model, and r_hit is the raycasting hit rate. When r_hit is lower than a preset threshold τ_hit, the frame is marked as a low-confidence frame, and the closed bounding box of this frame is not included in the subsequent calculation of reprojection error accumulation. When r_hit is not lower than τ_hit, the repair confidence C_conf = C_raw is output, and the closed bounding box B_3D of this frame is stored as the fire physical baseline.

[0045] After the dynamic projection texture and the fire physical baseline are generated, their spatial positions are compared, and the projection deviation is calculated. The system projects the four corner points of the closed bounding box B_3D onto the image plane through the forward mapping F_forward, obtaining four projection points p_proj_j=F_forward(P_hit_j), where j takes values ​​from 1 to 4. The geometric center of the projected quadrilateral region enclosed by the four projection points is calculated as p_center_proj=(p_proj_1+p_proj_2+p_proj_3+p_proj_4) / 4. Simultaneously, in the dynamic projection texture, the system extracts the pixel set corresponding to the detection box B_box and calculates the color centroid position p_color_centroid of that region. The formula for calculating the color centroid is: p_color_centroid=(Σ_{(u,v)∈B_box}I(u,v)×u / ΣI(u,v), Σ_{(u,v)∈B_box}I(u,v)×v / ΣI(u,v)), where I(u,v) is the brightness value of pixel (u,v). The single-frame projection deviation δ_frame is defined as the pixel Euclidean distance between p_center_proj and p_color_centroid, i.e., δ_frame=||p_center_proj-p_color_centroid||.

[0046] The system maintains a sliding window of length N_f, storing the single-frame projection bias δ_frame_k of the most recent consecutive N_f frames and their corresponding repair confidence C_conf_k, where k is the frame index. The cumulative reprojection error RMSE_acc across multiple frames is defined as the weighted average bias: RMSE_acc = Σ(C_conf_k × δ_frame_k) / ΣC_conf_k, where the weighted sum covers all N_f frames within the window, and C_conf_k is the repair confidence of the k-th frame.

[0047] When RMSE_acc exceeds the preset tolerance ε_acc, the system initiates the calibration parameter correction process. During the correction process, the intrinsic parameter matrix K remains unchanged, and only the translation vector t and rotation matrix R in the extrinsic parameter matrix are iteratively optimized. The optimization boundaries for the translation vector t are set: with the current translation vector t_cur as the initial value, the optimization range in the X direction is [t_x - Δt_max, t_x + Δt_max], the optimization range in the Y direction is [t_y - Δt_max, t_y + Δt_max], and the optimization range in the Z direction is [t_z - Δt_max, t_z + Δt_max], where Δt_max is the preset maximum translation adjustment. The optimization boundaries for the Euler angles corresponding to the rotation matrix R are set: with the current Euler angles (α_cur, β_cur, γ_cur) as the initial value, the optimization range for each angle is [α_cur - Δθ_max, α_cur + Δθ_max], etc., where Δθ_max is the preset maximum angle adjustment. Within the optimization boundary, with the goal of minimizing RMSE_acc, the translation vector and rotation parameters in the extrinsic parameters are iteratively optimized using the Levenberg-Marquardt algorithm. After each iteration, it is determined whether the updated translation vector exceeds the optimization boundary; if so, the components in the corresponding directions are truncated to the boundary values. Similarly, it is determined whether the updated Euler angles exceed the optimization boundary; if so, they are also truncated to the boundary values. Iteration stops when the change in RMSE_acc is less than a preset threshold ε_conv or the number of iterations reaches a preset upper limit N_max. The optimized extrinsic parameter matrix [R_opt|t_opt] is output, and the mapping correction benchmark M_corr is generated.

[0048] The system sends back the mapping correction reference M_corr to update the current mapping reference M_ref_opt, i.e., sets M_ref_opt = M_corr. The aforementioned video projection, detection box transformation, and projection deviation calculation processes are then re-executed using the updated mapping reference.

[0049] The corrected spatial location and extent of the enclosed bounding box B_3D are used as the fire source input, and the spread of flames and smoke is deduced by combining the building material properties. The horizontal spread rate V_h is calculated according to the following formula: V_h=V_base×M_mat×M_vent×F_area, where V_base is the preset baseline spread rate, M_mat is the wall material correction coefficient, M_vent is the ventilation condition correction coefficient, and F_area is the area growth correction factor, where F_area=1+ΔA, and ΔA is the flame area growth rate. The system determines the fire source starting position P_source=P_center based on the spatial location of the corrected enclosed bounding box, and determines the initial fire source dimensions L_source=L_box, W_source=W_box, and H_source=H_box based on the spatial extent. The flame spread boundary for multiple future time steps is extrapolated using the horizontal spread velocity V_h: the predicted boundary for the τth time step is a three-dimensional bounding box centered on P_source, with half the length of L_source+2×V_h×τ in the X direction, half the length of W_source+2×V_h×τ in the Y direction, and half the length of H_source+V_v×τ in the Z direction, where V_v is the vertical spread velocity.

[0050] The system integrates the updated dynamic projection texture under the mapping benchmark, the corrected bounding box, and the predicted flame spread boundaries at multiple time steps in the 3D rendering engine. The dynamic projection texture serves as the real-time texture of the 3D model surface. The corrected bounding box is marked with a solid red border at the current flame location, and the predicted boundaries at each time step are marked with dashed borders of different colors and accompanied by time labels. Finally, the 3D fire situation scene is output to the display terminal.

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for constructing a three-dimensional fire situation based on visual feedback correction, characterized in that, Includes the following steps: Obtain the 3D model of the building and the monitoring video stream, and construct a mapping benchmark based on the intrinsic and extrinsic parameters of the monitoring equipment; Based on the mapping reference, the video stream is back-projected onto the surface of the three-dimensional model and the texture coordinates are corrected according to the surface differential geometry properties to generate dynamic projection texture; at the same time, the corner points of the flame or smoke detection box in the video stream are mapped to the three-dimensional model to construct a closed bounding box as the physical reference of the fire. The spatial deviation between the dynamic projection texture and the fire physical reference is compared, the reprojection error accumulated over multiple frames is calculated, the intrinsic parameters are locked when the error exceeds the preset tolerance, and the extrinsic parameters are iteratively optimized under the physical constraints of the installation pose of the monitoring equipment to generate a mapping correction reference. The mapping correction reference is returned to update the mapping reference, and the projection and positioning are re-executed. Based on the spatial position and range of the corrected enclosed box and the building material properties, the diffusion pattern of flames and smoke is obtained, and a three-dimensional fire situation scene is output.

2. The method according to claim 1, characterized in that, The process of constructing the mapping benchmark specifically includes: The installation location coordinates and orientation angle of the monitoring equipment are marked in the three-dimensional model as initial external parameters; the pixel coordinates of at least four sets of known building structure feature points in the monitoring equipment screen are obtained, and the three-dimensional spatial coordinates corresponding to each set of feature points are marked in the three-dimensional model; Using pixel coordinates and 3D spatial coordinates as input, the intrinsic and extrinsic parameter matrices of the monitoring device are obtained. The inverse of the intrinsic and extrinsic parameter matrices is used as the mapping reference from the pixel coordinate system to the world coordinate system.

3. The method according to claim 1, characterized in that, The generation process of the dynamic projection texture specifically includes: Calculate the principal curvature and Gaussian curvature of the surface where the hit point is located, and determine the surface type based on the sign of the Gaussian curvature; For convex surfaces, the ratio of arc length to chord length is calculated as the distortion compensation coefficient; for saddle surfaces, the independent arc-chord ratios along the two principal curvature directions are calculated as the distortion compensation coefficients; for planes, the distortion compensation coefficient is set to 1; the texture coordinates are reverse-corrected using the distortion compensation coefficient to generate the dynamic projection texture.

4. The method according to claim 1, characterized in that, The specific process of mapping the corner points of the flame or smoke detection boxes in the video stream to the 3D model includes: Obtain the pixel coordinates of the four corner points of the detection box. Starting from the optical center of the monitoring device, perform ray projection along the ray direction corresponding to the four corner points respectively, and find the intersection with the surface of the three-dimensional model to obtain four three-dimensional hit points. The spatial region enclosed by the four three-dimensional hit points is defined as a closed bounding box, and the six faces of the closed bounding box are parallel to the coordinate axis plane of the three-dimensional model. The spatial coordinate range of the closed bounding box in the three-dimensional model is extracted as the physical reference of the fire situation.

5. The method according to claim 1, characterized in that, The specific calculation method for the reprojection error includes: For each frame, the pixel distance between the color centroid position of the region corresponding to the detection box in the dynamic projection texture and the projection center position of the closed bounding box in the fire physical reference projected onto the image plane is taken as the single-frame projection deviation of the frame, and the weighted average of the projection deviations of consecutive single frames is taken as the cumulative reprojection error of the multiple frames.

6. The method according to claim 1, characterized in that, The specific process of iterative optimization of external parameters includes: The intrinsic parameter matrix of the fixed monitoring equipment remains unchanged, and the installation position coordinates and orientation angle of the monitoring equipment are used as variables to be optimized; the optimization boundary of the installation position coordinates is set as a spatial sphere range with a preset radius centered on the initial calibration position, and the optimization boundary of the orientation angle is set as a cone range with a preset angle centered on the initial calibration orientation. Within the optimization boundary, the extrinsic parameters are iteratively solved with the goal of minimizing the reprojection error. After each iteration, it is determined whether the optimization variable exceeds the optimization boundary. If it does, the value is truncated to the boundary value.

7. The method according to claim 1, characterized in that, After updating the mapping benchmark, the process also includes a multi-channel video time synchronization step: The system acquires video streams output by monitoring equipment, with each frame of each video stream carrying a capture timestamp. Using the latest common timestamp of all video streams as the synchronization benchmark, video frames with timestamps earlier than the synchronization benchmark are buffered and waited for, while video frames with timestamps later than the synchronization benchmark are discarded. At the synchronization benchmark time, the multiple video streams are synchronously projected onto the surface of the three-dimensional model.

8. The method according to claim 1, characterized in that, The process of acquiring the diffusion situation specifically includes: The spatial location of the enclosed box is taken as the starting position of the fire source, and the spatial range of the enclosed box is taken as the initial size of the fire source; the structural information of the area where the fire source is located is obtained from the three-dimensional model, including the wall material type and ventilation condition parameters; The first correction factor for the horizontal spread rate is determined based on the wall material type, and the second correction factor for the horizontal spread rate is determined based on the ventilation condition parameters. The actual horizontal spread rate is obtained, and the flame spread boundary at the preset time step is obtained. The flame spread boundary at the preset time step is visualized in the 3D model using 3D bounding boxes of different colors.

9. The method according to claim 1, characterized in that, After obtaining the aforementioned diffusion situation, the process also includes a rescue path planning step: The space of the 3D model is discretized into a 3D mesh; the mesh area covered by the flame spread boundary and the smoke diffusion path is used as the no-entry zone. Assign a first passage cost to the grid nodes corresponding to corridors and stair passages, and assign a second passage cost to the grid nodes corresponding to narrow passages. The second passage cost is greater than the first passage cost. Starting from the building entrance and ending at the spatial location of the enclosed box, the path with the lowest passage cost is selected as the optimal rescue path.

10. A fire-fighting three-dimensional situation construction system based on visual feedback correction, comprising a processor for performing the method as described in any one of claims 1 to 9.

Images