A fire scene multi-modal video acquisition and tagging indexing method and system

Abstract

This invention discloses a method and system for multimodal video acquisition and tagging indexing in fire scenes. The method includes: S1: Deploying a multimodal camera array and sensor array, and synchronously acquiring video, smoke concentration, and temperature data through a hardware synchronization trigger; S2: Spatial registration of the video using a pre-calibrated homography matrix; S3: Adaptively adjusting weights based on smoke concentration to generate a fused video; S4: Automatically generating labels for fire stage, fuel type, and key moments based on temperature data and video features; S5: Storing the original video, fused video, temperature data, and labels in a three-level structure in a database, establishing a time index and a label index. This invention can achieve microsecond-level synchronous acquisition of multimodal videos such as visible light, near-infrared, and thermal infrared, while automatically identifying fire stage, fuel type, and key events, generating structured labels without manual intervention; providing a high-quality, multi-dimensional, and searchable experimental data foundation for fire mechanism analysis, firefighter case teaching, and fire identification.

Description

Technical Field

[0001] This invention relates to the field of fire scene simulation and information acquisition, specifically to a method and system for multimodal video acquisition and tagging indexing of fire scenes. Background Technology

[0002] In fire science research, numerous controlled combustion experiments are needed to study the combustion characteristics of different materials (wood, plastics, textiles, etc.) under various conditions (ventilation, humidity, stacking methods) to provide data support for fire risk assessment and fire protection design. Firefighter training also requires a library of real-world fire scene video case studies to simulate firefighting tactics drills in different scenarios. However, traditional combustion experiments suffer from the following problems: 1. Limited recording of key data: Multiple cameras collect data independently, lacking a unified time reference, making multi-perspective joint analysis difficult. Most experiments only record temperature curves and a few photos, lacking continuous video recording, making it difficult to reproduce and analyze key phenomena such as flame spread paths and smoke flow; 2. Manual recording of key events: When transient events such as flashover and re-ignition occur, manual key-press marking is required, which is prone to omissions or inaccurate recording; 3. Difficulty in subsequent analysis and retrieval: Video data lacks structured tags, and hundreds of hours of accumulated video data are scattered in different folders. It is difficult to quickly locate specific fire stages or event segments by filename (e.g., "20251109_wood_test.mp4"), and it is impossible to retrieve data by semantic conditions such as "flashover moment of wood combustion"; 4. Limited experimental data dimensions: Single visible light video suffers severe image degradation in dense smoke environments, and there is a lack of synchronous recording of multi-dimensional data such as temperature field and smoke concentration, limiting in-depth analysis of combustion mechanisms. Therefore, an experimental data management system that can automatically record the entire fire process, automatically label key events, and support multi-dimensional semantic retrieval is needed. Summary of the Invention

[0003] This invention addresses the problems in existing technologies by disclosing a method and system for multimodal video acquisition and tagging indexing in fire scenes. The invention constructs a multimodal camera array and sensor array, synchronously acquires video, smoke concentration, and temperature data through hardware synchronization triggers, performs spatial registration on the video, and then adaptively adjusts weights based on smoke concentration to generate fused video and store it in a database. Simultaneously, it automatically generates distinguishable tags based on temperature data and video features and stores them in the database. The database supports searching video segments by time, stage, and fuel type. This significantly improves the accuracy of video acquisition and data retrieval, while also increasing retrieval efficiency.

[0004] This invention is achieved through the following technical solution:

[0005] This invention first provides a method for multimodal video acquisition and tagging indexing of fire scenes, including the following steps:

[0006] S1: Deploy a multimodal camera array and sensor array, and synchronously collect video, smoke concentration and temperature data through hardware synchronization triggers;

[0007] S2: Spatial registration of the video is performed using a pre-calibrated homography matrix;

[0008] S3: Adaptively adjust weights based on smoke concentration to generate a fused video;

[0009] S4: Automatically generate labels for fire stage, fuel type, and key moments based on temperature data and video features;

[0010] S5: Store the original video, merged video, temperature data, and tags into the database in a three-level structure, and establish a time index and a tag index. The database supports retrieving video clips by time range, fire stage, and fuel type.

[0011] As a further option, the method for deploying the multi-modal camera array in S1 is as follows: with the center of the fire source as the origin, the visible light cameras are installed at three levels with heights of 0.5m, 1.5m, and 2.5m, and the horizontal azimuth angles are spaced 90°~120° apart; the near-infrared and thermal infrared cameras are deployed at different heights and orientations to achieve multi-angle and multi-layer coverage.

[0012] As a further option, the sensor array is arranged as follows: the temperature sensors are installed in three layers at heights of 0.5m, 1.5m, and 2.5m, with three measuring points in each layer, evenly distributed in a triangle, that is, the included angle between two adjacent temperature sensors is 120°, and the smoke sensor covers the fire source.

[0013] As a further option, the specific methods of S2 include:

[0014] S21, Calibration board data acquisition:

[0015] Before the experiment began, a black and white checkerboard calibration board (5cm x 5cm, 9 x 7 corner points) was placed at the fire source location, and all cameras filmed simultaneously.

[0016] S22, Corner Detection:

[0017] The Harris corner detection algorithm is used to automatically extract the pixel coordinates of the calibration board corners in each camera image.

[0018] S23, Calculation of the transformation matrix:

[0019] Using one of the visible light cameras as a reference, the homography matrix H from the near-infrared image, thermal infrared image to the visible light image is solved by the least squares method:

[0020] For near-infrared cameras, select at least 15 pairs of matching corner points and establish a system of equations:

[0021] ,

[0022] By solving the overdetermined system of equations, eight unknown parameters h11~h32 were obtained.

[0023] Where u_vis and v_vis represent the horizontal and vertical pixel coordinates of a corner point in a visible light camera image, respectively; u_nir and v_nir represent the horizontal and vertical pixel coordinates of the corresponding corner point in a near-infrared camera image, respectively; h11~h32 represent the eight unknown parameters of the homography transformation matrix; the homography matrix H represents a 3×3 homography transformation matrix used to describe the projection relationship between two planar images;

[0024] For thermal infrared cameras, the same transformation relationship is used, simply replacing u_nir, v_nir with u_tir, v_tir;

[0025] S24, Image Resampling:

[0026] Using the transformation matrix H, perspective transformation and bilinear interpolation are performed on each frame of near-infrared camera video and thermal infrared camera video, and the frames are resampled to 1920×1080 resolution, which is aligned with the visible light video space.

[0027] As a further approach, the method in S3 for adaptively adjusting weights based on smoke concentration to generate fused videos includes:

[0028] S31. Obtain smoke concentration value: Read the smoke concentration value directly from the smoke sensor, or calculate the smoke concentration value using the average brightness of a visible light image. The formula is as follows:

[0029] C_smoke_est =(255-I_mean) / 255×0.8,

[0030] Where I_mean is the mean gray level of the visible light image;

[0031] S32. Obtain the three smoke states and their corresponding weights:

[0032] When the smoke concentration is <0.1 g / m³, the weighting for visible light is 0.7, the weighting for near-infrared light is 0.2, and the weighting for thermal infrared light is 0.1.

[0033] When the smoke concentration is 0.1~0.5 g / m³, the weighting for visible light is 0.3, the weighting for near-infrared light is 0.5, and the weighting for thermal infrared light is 0.2.

[0034] When the smoke concentration is >0.5 g / m³, the weighting for visible light is 0.1, for near-infrared light it is 0.4, and for thermal infrared light it is 0.5.

[0035] For intermediate concentration values, linear interpolation is used to calculate the weights;

[0036] S33, Perform pixel blending:

[0037] For each pixel position (x, y), the fusion formula is:

[0038] I_fuse(x,y) = w_vis×I_vis(x,y) + w_nir×I_nir(x,y) + w_tir×I_tir(x,y)

[0039] Where I_fuse(x,y) represents the intensity value (RGB three-channel vector) of the fused image at pixel position (x,y); I_vis(x,y) represents the intensity value of the visible light camera image at pixel position (x,y); I_nir(x,y) represents the intensity value of the near-infrared camera registered image at pixel position (x,y); I_tir(x,y) represents the intensity value of the thermal infrared camera registered image after pseudo-color mapping at pixel position (x,y); w_vis represents the fusion weight coefficient of the visible light image; w_nir represents the fusion weight coefficient of the near-infrared image; w_tir represents the fusion weight coefficient of the thermal infrared image; (x,y) represents the pixel coordinates in the image coordinate system, where x is the horizontal direction and y is the vertical direction, and the weights satisfy the normalization condition: w_vis + w_nir + w_tir = 1;

[0040] As a further solution, I_tir needs to be converted to RGB format via pseudo-color mapping: 0-400°C is mapped to blue-green, 400-800°C to yellow-orange, and 800-1200°C to red-white.

[0041] As a further option, S4's method includes:

[0042] S41. Determine the fire stage based on temperature range and temperature rise rate rules;

[0043] S42. Identify fuel type;

[0044] S43. Detect key moments, including:

[0045] Ignition moment: When the temperature is detected to rise above 50°C from the ambient temperature (~20°C) for the first time, and the rate of temperature rise is >3°C / s, this moment is recorded as t_ignition;

[0046] Flashover occurs: When a temperature rise of more than 200°C (dT / dt>20°C / s) is detected within 10 seconds, it is recorded as t_flashover;

[0047] Fire suppression begins: When a sustained temperature drop is detected (dT / dt < -5°C / s for 5 consecutive seconds), it is recorded as t_suppression;

[0048] Flame extinguishing: When the temperature drops below 150°C and remains stable (dT / dt within ±1°C / s), it is recorded as t_extinguish.

[0049] As a further embodiment, the method for determining the fire stage based on temperature range and temperature rise rate rules in S41 includes:

[0050] S411. The stages of a fire are divided into: the initial stage, the development stage, the intense stage, and the decay stage.

[0051] S412, the temperature rise rate rule is as follows:

[0052] When T_max < 150 and dT / dt < 5, it is determined to be the initial stage;

[0053] When 150≤T_max<500 and dT / dt≥5, it is determined to be in the development stage;

[0054] When T_max≥500 and dT / dt>20, it is determined to be the intense period;

[0055] When dT / dt < -5 (temperature decreases), it is determined to be the decay period;

[0056] Where T_max is the maximum value of all temperature sensors, and dT / dt is the temperature change rate over the last 5 seconds;

[0057] S413, Judgment Procedure:

[0058] 1. Calculate the maximum value T_max of all temperature sensors and the temperature change rate dT / dt over the last 5 seconds once per second;

[0059] 2. Determine the fire stage based on the temperature rise rate rule;

[0060] 3. When the same fire stage is determined for 3 consecutive seconds, confirm the stage transition and generate a tag;

[0061] As a further option, the method for identifying the fuel type in S42 includes:

[0062] S421. Pre-establish a temperature characteristic library for common fuels:

[0063] S422, extract the peak temperature T_peak, average temperature rise rate dT / dt_avg, and total duration t_duration from ignition to extinction of the entire combustion process;

[0064] S423. Calculate the Euclidean distance for each fuel in the existing fuel and temperature feature library;

[0065] S424. Select the fuel type with the smallest Euclidean distance as the identification result.

[0066] As a further approach, the value from the smoke sensor is averaged with the smoke concentration value calculated from the average brightness of the visible light image, and this average value is used as the final smoke concentration value C_smoke.

[0067] As a further embodiment, the three-level structure in S5 includes: an experiment-level table, a time-series table, and a label-level table. The time-series table includes the frame_id field, the exp_id field, and the timestamp field. The label-level table includes the label_id field, the fire_stage field, and the fuel_type field.

[0068] As a further solution, methods for creating time indexes include: creating a B+ tree index on the timestamp field of the time-series table to support queries by time range;

[0069] As a further solution, methods for creating tag indexes include: creating a composite index on the fire_stage and fuel_type fields of the tag-level table to support queries by stage and fuel combination.

[0070] This invention also provides a multimodal video acquisition and tagging indexing system for fire scenes, comprising:

[0071] A multimodal array, used to be deployed in layers around the fire source area and collect fire information, includes several visible light cameras, near-infrared cameras, thermal infrared cameras, temperature sensors and smoke sensors;

[0072] Hardware synchronization triggers provide a unified time reference for all cameras based on the PPS signal synchronized with GPS.

[0073] The video registration module obtains the transformation matrix of the video captured by the near-infrared camera and the thermal infrared camera through the calibration board, and resamples and registers it to the resolution and coordinate system of the visible light video.

[0074] The fusion video generation module adaptively adjusts the modal weights based on the smoke concentration collected by the smoke sensor to perform pixel-level weighted fusion and obtain the fusion video.

[0075] The automatic tag generation module automatically generates tags for fire stage, fuel type, and key moments based on temperature data and video features;

[0076] The structured database and indexes use a three-level structure—experimental, time-series, and tag-level—to store raw videos, fused videos, temperature data, and tags, and establish time and tag indexes.

[0077] As a further solution, the synchronization accuracy of the hardware synchronization trigger is greater than 1 microsecond, and the visible light camera, near-infrared camera, and thermal infrared camera are triggered for exposure simultaneously on the same PPS rising edge, with the frame rate uniformly set to 30fps or 60fps.

[0078] As a further solution, hardware synchronization triggers include:

[0079] The GPS timing module is used to receive satellite signals and output standard PPS (pulse per second) signals.

[0080] The clock distribution module is used to distribute the PPS signal to the external trigger interfaces of all cameras;

[0081] Frame counting module: Counts the trigger pulses within each PPS cycle and generates a global frame sequence number.

[0082] The features and beneficial effects of this invention are as follows:

[0083] (1) This invention addresses the data recording and management needs in fire experiments and fire training. It can achieve microsecond-level synchronous acquisition of multimodal videos such as visible light, near-infrared, and thermal infrared, and obtain clear fire scene images in dense smoke environments. At the same time, it can automatically identify fire stages, fuel types, and key events, and generate structured tags without manual intervention. It can establish a search index oriented towards fire semantics, and support the rapid location of video segments based on conditions such as "the flashover moment of a certain type of fuel" and "temperature distribution during the development period". It provides a high-quality, multi-dimensional, and searchable experimental data foundation for fire mechanism analysis, firefighter case teaching, and fire identification.

[0084] (2) This invention synchronously acquires video, smoke concentration and temperature data through a hardware synchronous trigger, so that all modal images correspond to the fire scene state at the same moment, without the need for complex time registration; at the same time, it simplifies the system design and avoids time interpolation errors caused by staggered exposures; and facilitates subsequent multi-view joint analysis. Using the existing asynchronous acquisition method requires additional timestamp recording and optical flow interpolation processing, which increases the system complexity and computational intensity, and thus increases the computational cost.

[0085] (3) This invention establishes a tag index, using semantic information such as fire stage and fuel type as index keys, supporting stage-based retrieval, fuel-based retrieval, and combined queries, greatly improving work efficiency. In the fire accident identification stage, when accident investigators need to compare the combustion characteristics of a real fire, they can quickly find videos of similar scenarios in the experimental database by searching "plastic + flashover" to analyze the fire spread pattern; in the fire training stage, when instructors need to demonstrate to trainees the "complete process of a residential fire from ignition to flashover", they can quickly locate relevant case videos by searching "wood + textile + flashover event"; in the scientific research analysis stage, when researchers need to statistically analyze the "average flashover time of different fuels", they can batch extract the timestamps of all flashover moments for statistical analysis through the tag index. Through the above indexing mechanism, the response time for retrieving thousands of files is reduced from tens of seconds in the traditional file system to milliseconds, and the retrieval efficiency is improved by more than 50 times. Attached Figure Description

[0086] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0087] Figure 1 This is a flowchart of the fire scene multimodal video acquisition and tagging indexing method described in an embodiment of the present invention;

[0088] Figure 2 This is a schematic diagram of the spatial layout of the multimodal camera array according to an embodiment of the present invention, (a) is a top view, and (b) is a side view;

[0089] Figure 3 This is a schematic diagram of the automatic label generation process according to an embodiment of the present invention;

[0090] Figure 4 This is a schematic diagram of the three-level hierarchical structure of the database as described in an embodiment of the present invention. Detailed Implementation

[0091] To facilitate understanding of the present invention, a more comprehensive description of the present invention will be given below, and embodiments of the present invention will be provided, but this does not limit the scope of the present invention.

[0092] In fire science research, numerous controlled combustion experiments are needed to study the combustion characteristics of different materials under various conditions, providing data support for fire risk assessment and fire protection design. However, current fire tests primarily rely on single visible light video acquisition methods. This single visible light video suffers severe image quality degradation in dense smoke environments, failing to clearly record the entire fire process. Furthermore, multiple cameras acquire data independently, lacking a unified time reference. This results in different camera frames being captured at different times, hindering multi-view joint analysis and leading to incomplete or erroneous analyses for each camera. Firefighters also require a database of real fire scene videos for training to simulate firefighting tactics in various scenarios. However, the massive amounts of video data accumulated from fire tests and training are scattered and lack unified management and intelligent retrieval methods. Due to the lack of structured tags, video data can only be retrieved by filename or recording time, failing to quickly search by semantic information such as fire stage or fuel type, resulting in inaccurate and inefficient video searches. Therefore, this application proposes a multimodal video acquisition and tagging indexing method for fire scenes. It constructs a multimodal camera array and sensor array, synchronously acquires video, smoke concentration, and temperature data through hardware synchronization triggers, performs spatial registration on the video, and then adaptively adjusts the weights based on smoke concentration to generate fused video and store it in a database. Simultaneously, it automatically generates distinguishable tags based on temperature data and video features and stores them in the database. The database supports searching video segments by time, stage, and fuel type. This significantly improves the accuracy of video acquisition and data retrieval, while also increasing retrieval efficiency.

[0093] like Figures 1 to 4 As shown, a method for multimodal video acquisition and tagging indexing of fire scenes includes the following steps:

[0094] S1: Deploy a multimodal camera array and sensor array, and synchronously collect video, smoke concentration and temperature data through hardware synchronization triggers;

[0095] S2: Spatial registration of the video is performed using a pre-calibrated homography matrix;

[0096] S3: Adaptively adjust weights based on smoke concentration to generate a fused video;

[0097] S4: Automatically generate labels for fire stage, fuel type, and key moments based on temperature data and video features;

[0098] S5: Store the original video, merged video, temperature data, and tags in a three-level structure in the database, and establish a time index and a tag index. The database supports retrieving video clips by time, stage, and fuel type.

[0099] S6: Users can retrieve video clips by time range, fire stage, fuel type, and other criteria through the query interface for applications such as fire training case library and fire mechanism research.

[0100] This application presents a multimodal video acquisition and tagging indexing method for fire scenes, addressing the data recording and management needs in fire experiments and fire training. It enables microsecond-level synchronous acquisition of multimodal videos, including visible light, near-infrared, and thermal infrared, obtaining clear fire scene images even in dense smoke environments. Simultaneously, it automatically identifies fire stages, fuel types, and key events (ignition, flashover, and extinguishing), generating structured tags without manual intervention. It establishes a fire semantic-oriented retrieval index, supporting rapid location of video segments based on conditions such as "flashover time of a certain type of fuel" and "temperature distribution during the development phase." This provides a high-quality, multi-dimensional, and searchable experimental data foundation for fire mechanism analysis, firefighter case teaching, and fire identification.

[0101] In some embodiments, the method of deploying the multimodal camera array in S1 is as follows: with the center of the fire source as the origin, the visible light cameras are installed at three levels with heights of 0.5m, 1.5m, and 2.5m, and the horizontal azimuth angles are spaced 90°~120° apart; near-infrared and thermal infrared cameras are deployed at different heights and orientations to achieve multi-angle and multi-layer coverage.

[0102] In some embodiments, the sensor array is deployed as follows: temperature sensors are installed in three layers at heights of 0.5m, 1.5m, and 2.5m, with three measuring points in each layer, evenly distributed in a triangle, meaning the angle between any two adjacent temperature sensors is 120°. Smoke sensors are positioned above the fire source. This sensor array is used to assist in measuring temperature data and smoke concentration at the fire scene, and, in conjunction with data collected by the modal camera array, makes the collected data more accurate.

[0103] This application synchronously acquires video, smoke concentration, and temperature data using a hardware synchronous trigger, ensuring that all modal images correspond to the fire scene state at the same moment, eliminating the need for complex time registration. It also simplifies system design, avoids time interpolation errors caused by staggered exposures, and facilitates subsequent multi-view joint analysis. Existing asynchronous acquisition methods require additional timestamp recording and optical flow interpolation processing, increasing system complexity and computational intensity, thus increasing computational costs.

[0104] In some embodiments, the specific method for spatial registration of video using a pre-calibrated homography matrix in S2 includes:

[0105] S21, Calibration board data acquisition:

[0106] Before the experiment began, a black and white checkerboard calibration board (5cm x 5cm, 9 x 7 corner points) was placed at the fire source location, and all cameras filmed simultaneously.

[0107] S22, Corner Detection:

[0108] The Harris corner detection algorithm is used to automatically extract the pixel coordinates of the calibration board corners in each camera image.

[0109] S23, Calculation of the transformation matrix:

[0110] Using one of the visible light cameras as a reference, the homography matrix H from the near-infrared image, thermal infrared image to the visible light image is solved by the least squares method:

[0111] For near-infrared cameras, select at least 15 pairs of matching corner points and establish a system of equations:

[0112] ,

[0113] By solving the overdetermined system of equations, eight unknown parameters h11~h32 were obtained.

[0114] Where u_vis and v_vis represent the horizontal and vertical pixel coordinates of a corner point in a visible light camera image, respectively; u_nir and v_nir represent the horizontal and vertical pixel coordinates of the corresponding corner point in a near-infrared camera image, respectively; h11~h32 represent the eight unknown parameters of the homography transformation matrix; the homography matrix H represents a 3×3 homography transformation matrix used to describe the projection relationship between two planar images;

[0115] This application uses a transformation matrix to map points (u_nir, v_nir) in the near-infrared image coordinate system to points (u_vis, v_vis) in the visible light image coordinate system, compensating for geometric differences caused by the near-infrared camera position and lens distortion. For thermal infrared cameras, a similar transformation relationship is used, simply replacing u_nir, v_nir with u_tir, v_tir.

[0116] S24, Image Resampling:

[0117] Using the transformation matrix H, perspective transformation and bilinear interpolation are performed on each frame of near-infrared camera video and thermal infrared camera video, and the frames are resampled to 1920×1080 resolution, which is aligned with the visible light video space.

[0118] This application has been verified by the calibration board, and the reprojection error is less than 0.5 pixels, meeting the sub-pixel level registration requirements. Due to synchronized hardware triggering of all cameras, the time registration error is less than 1 microsecond.

[0119] In some embodiments, the weight adaptive adjustment in S3 is divided into three intervals based on the smoke concentration, with each interval corresponding to a set of fixed weights, and the weights of the intermediate concentration values ​​are calculated by linear interpolation.

[0120] In some embodiments, the method for adaptively adjusting weights based on smoke concentration to generate a fused video in S3 includes:

[0121] S31. Smoke Concentration Estimation:

[0122] Smoke concentration values ​​can be read directly from smoke sensors, or estimated using the average brightness of a visible light image:

[0123] C_smoke_est =(255-I_mean) / 255×0.8,

[0124] Where I_mean is the mean gray level of the visible light image.

[0125] This method can directly read the smoke concentration value from the smoke sensor or estimate the smoke concentration value from the average brightness of the visible light image. Preferably, both data are used, and the average of the two is taken as the final smoke concentration value C_smoke (unit: g / m³). The concentration value obtained in this way combines different acquisition methods, effectively reduces data errors, and greatly improves the accuracy of smoke concentration.

[0126] S32, Weight Calculation:

[0127] Three smoke states and their corresponding weights are defined in Table 1.

[0128] Table 1

[0129]

[0130] For intermediate concentration values, linear interpolation is used to calculate the weights.

[0131] S33, Pixel Blending:

[0132] For each pixel position (x, y), the fusion formula is:

[0133] I_fuse(x,y) = w_vis×I_vis(x,y) + w_nir×I_nir(x,y) + w_tir×I_tir(x,y)

[0134] Wherein, I_fuse(x,y) represents the intensity value (RGB three-channel vector) of the fused image at pixel position (x,y); I_vis(x,y) represents the intensity value of the visible light camera image at pixel position (x,y); I_nir(x,y) represents the intensity value of the near-infrared camera image after registration at pixel position (x,y); I_tir(x,y) represents the intensity value of the thermal infrared camera image after registration and pseudo-color mapping at pixel position (x,y); w_vis represents the fusion weight coefficient of the visible light image; w_nir represents the fusion weight coefficient of the near-infrared image; w_tir represents the fusion weight coefficient of the thermal infrared image; (x,y) represents the pixel coordinates in the image coordinate system, where x is the horizontal direction (01919) and y is the vertical direction (01079).

[0135] I_tir needs to be converted to RGB format via pseudo-color mapping: 0-400°C is mapped to blue-green, 400-800°C to yellow-orange, and 800-1200°C to red-white. The weights satisfy the normalization condition: w_vis + w_nir + w_tir = 1.

[0136] In some embodiments, the method in S4 for automatically generating fire stage, fuel type, and critical moment labels based on temperature data and video features includes:

[0137] S41. Fire Stage Identification:

[0138] The fire stage is determined based on temperature range and temperature rise rate rules, as shown in Table 2.

[0139] Table 2

[0140]

[0141] Where T_max is the maximum value of the three-layer temperature sensor, and the temperature rise rate dT / dt is the temperature change rate in the last 5 seconds.

[0142] Judgment process:

[0143] 1. Calculate T_max and dT / dt once per second;

[0144] 2. Match the stage category according to the rules in the table above;

[0145] 3. When three consecutive seconds are determined to be in the same stage, confirm the stage transition and generate a tag;

[0146] S42. Fuel type identification:

[0147] A temperature characteristic library for common fuels was pre-established, as shown in Table 3.

[0148] Table 3

[0149]

[0150] Identification method:

[0151] 1. Extract the peak temperature T_peak, average temperature rise rate dT / dt_avg, and total duration t_duration from ignition to extinction of the entire combustion process;

[0152] 2. Calculate the Euclidean distance to each fuel in the feature library;

[0153] 3. Select the fuel type with the smallest Euclidean distance as the identification result.

[0154] S43. Critical Moment Detection:

[0155] Ignition time:

[0156] When the temperature is detected to rise by more than 50°C from the ambient temperature (~20°C) for the first time, and the rate of temperature rise is >3°C / s, this moment is recorded as t_ignition.

[0157] Flashover occurred:

[0158] A temperature rise exceeding 200°C (dT / dt>20°C / s) was detected within 10 seconds and recorded as t_flashover.

[0159] Firefighting begins:

[0160] The temperature was detected to begin a continuous drop (dT / dt < -5°C / s for 5 consecutive seconds), and recorded as t_suppression.

[0161] The flames went out:

[0162] When the temperature drops below 150°C and remains stable (dT / dt within ±1°C / s), it is recorded as t_extinguish.

[0163] In some embodiments, the three-level structure in S5 includes: an experiment-level table (Experiment), a time-series table (Frame), and a label-level table (Label), wherein,

[0164] The Experiment level table is Table 4.

[0165] Table 4

[0166]

[0167] The timing-level frame is shown in Table 5.

[0168] Table 5

[0169]

[0170] The label-level table is shown in Table 6.

[0171] Table 6

[0172]

[0173] The method for establishing a time index includes:

[0174] Create a B+ tree index on the timestamp field of the Frame table to support queries by time range:

[0175] SELECT FROM Frame WHERE timestamp BETWEEN t1 AND t2;

[0176] The method for establishing the tag index includes:

[0177] Create a composite index on the fire_stage and fuel_type fields of the Label table to support queries by stage and fuel type combination:

[0178] .

[0179] In traditional file systems, video files are named according to their shooting time (e.g., "2025_11_09_14_30_00.mp4"). Searching requires browsing by filename or sorting by modification date, significantly reducing efficiency. This invention establishes a time index to segment continuous videos frame by frame, with each frame associated with a precise microsecond-level timestamp. This allows for quick location of video frames at a specific time point (e.g., "the footage 120 seconds after the experiment began") and facilitates extraction of video segments within a specific time period (e.g., "the process 10 seconds before and after flashover"). Furthermore, this application establishes a tag index (composite index) using semantic information such as fire stage and fuel type as index keys. This supports stage-based searching (e.g., "finding all video segments during the most intense phase"), fuel-based searching (e.g., "finding all experiments on plastic combustion"), and combined queries (e.g., "segments during the development phase of wood combustion with temperatures exceeding 400°C"), greatly improving efficiency. In fire accident investigation, when investigators need to compare the combustion characteristics of a real fire, they can quickly find videos of similar scenarios in the experimental database by searching for "plastic + flashover" to analyze the fire spread pattern. In fire training, when instructors need to demonstrate the complete process of a residential fire from ignition to flashover, they can quickly locate relevant case videos by searching for "wood + textiles + flashover events." In scientific research, when researchers need to calculate the average flashover time for different fuels, they can use a tag index to extract the timestamps of all flashover moments for statistical analysis. Through this indexing mechanism, the retrieval response time is reduced from tens of seconds (requiring the traversal of thousands of files) in traditional file systems to milliseconds (directly located via the index), improving retrieval efficiency by more than 50 times.

[0180] Example

[0181] Figure 2 This is a spatial layout diagram of the multimodal array in the system architecture of this invention. Within a 3m×3m experimental field, with the center of the fire source as the origin (0,0), a black and white checkerboard calibration board (grid size 5cm×5cm, 9×7 corner points) is placed at the fire source location. The camera array and sensor array are then deployed according to a three-dimensional coordinate system, as detailed below:

[0182] A1. The deployment methods for multimodal camera arrays and sensor arrays are as follows:

[0183] (1) There are 3 visible light cameras, namely camera A, camera B and camera C, and their installation locations are as follows:

[0184] Camera A is installed at (1.5m, 0m, 1.5m), with its lens facing the center of the fire source and a horizontal viewing angle of 90°.

[0185] Camera B is installed at position (0m, 1.5m, 1.5m), forming a 90° angle with camera A;

[0186] Camera C is mounted at (-1.5m, 0m, 2.5m) for shooting from a top-down angle;

[0187] Cameras A, B, and C all have a resolution of 1920×1080 and a frame rate of 30fps.

[0188] (2) There is one near-infrared camera, which is used to penetrate the smoke to obtain the flame outline and high-temperature area. The resolution is 640×512 and the frame rate is 30fps. The installation position of the infrared camera is (1.5m, 1.5m, 0.5m) and the working wavelength is 0.9~1.7μm.

[0189] (3) One thermal infrared camera is used to detect the temperature distribution in the fire scene. The resolution is 640×480, the frame rate is 30fps, the temperature measurement range is 0-1200°C, and it is installed at a position of (-1.5m, 1.5m, 2.5m). The working wavelength is 8~14μm.

[0190] (4) There are 9 temperature sensors used to collect the temperature of the fire scene. They are installed at heights of 0.5m, 1.5m and 2.5m respectively. The three measuring points on each floor are evenly distributed in a triangle (i.e., the angle between two adjacent temperature sensors is 120°).

[0191] (5) There are 4 smoke sensors installed at a height of 2.5m, and the 4 measuring points cover the area above the fire source.

[0192] The above deployment method includes multi-angle (horizontal azimuth angles distributed at intervals of 90°~120°) and multi-layer (vertical height divided into three layers of 0.5m, 1.5m, and 2.5m) fire scene information collection.

[0193] A2. Multimodal video is captured at 30fps using a hardware synchronous trigger, with each capture cycle lasting 33.33ms, continuously recording the entire fire process (usually 5-15 minutes).

[0194] When t=0μs, the PPS rising edge triggers all cameras to expose simultaneously: the exposure time for visible light cameras is 10ms; the exposure time for near-infrared cameras is 5ms; and the exposure time for thermal infrared cameras is 15ms. When t=33330μs, the next trigger cycle begins.

[0195] This method employs synchronous exposure, ensuring that all modal images correspond to the fire scene state at the same moment, eliminating the need for complex time registration. It also simplifies system design, avoids time interpolation errors caused by staggered exposures, and facilitates subsequent multi-view joint analysis. Using asynchronous acquisition would require additional timestamp recording and optical flow interpolation, increasing system complexity and computational cost.

[0196] A3. Spatial registration of the video using a pre-calibrated homography matrix, including...

[0197] A31. Visible-Near-Infrared Registration:

[0198] Referring to the equipment deployment scheme in S1, the visible light camera A is selected as the reference coordinate system, and the near-infrared camera image is registered to this coordinate system.

[0199] Calibration process:

[0200] A black and white checkerboard calibration board (5cm x 5cm, 9 rows and 7 columns) is placed horizontally at a height of 0.5m above the center of the fire source. The operator marks the pixel position of the same corner point in the images from both cameras to establish a correspondence. For example, a corner point on the calibration board has coordinates (520, 380) pixels in the visible light image and (165, 95) pixels in the near-infrared image.

[0201] Solve for the homography matrix H by labeling at least 15 pairs of such matching corner points. Let the near-infrared image coordinates be (x, y), and the transformed visible light image coordinates be (u, v). The transformation relationship is:

[0202] u = (h11·x + h12·y + h13) / (h31·x + h32·y + 1);

[0203] v = (h21·x + h22·y + h23) / (h31·x + h32·y + 1);

[0204] Substituting all matching corner points into the above equation, and using the least squares method to minimize the sum of squared reprojection errors, the homography matrix is ​​obtained:

[0205] ,

[0206] Among them, h11 and h22 reflect the resolution ratio and focal length difference between the two cameras, h12 and h21 reflect the rotation component caused by the difference in viewing angle, h13 and h23 reflect the translation of the image center, and h31 and h32 reflect the perspective distortion coefficient.

[0207] Registration verification:

[0208] Substitute the near-infrared image corner point (165, 95) into the transformation formula:

[0209] u' = (3.08×165 + 0.12×95 + 12) / (0.0001×165 + 0.0001×95 + 1) =531.6 / 1.026 = 518.1;

[0210] v' = (0.15×165 + 3.75×95 + 20) / (0.0001×165 + 0.0001×95 + 1) =401.0 / 1.026 = 390.8;

[0211] The error compared to the actual visible light coordinates (520, 380) is:

[0212] Error = √[(520-518.1)² + (380-390.8)²] = √[3.61 + 116.64] = 10.97 pixels;

[0213] Considering that fire scene registration is mainly used for multimodal fusion display, an error of less than 50 pixels (approximately 2.6% of the image width) is sufficient to meet the flame outline alignment requirements. In this embodiment, the average reprojection error of all corner points is 2.8 pixels, which meets the registration accuracy requirements.

[0214] A32, Visible Light-Thermal Infrared Registration

[0215] Using the same method as A31, a corner point correspondence is established between the thermal infrared image and the visible light image. The coordinates of a corner point on the calibration board are (170, 112) pixels in the thermal infrared image and (520, 380) pixels in the visible light image.

[0216] The homography matrix is ​​obtained by solving:

[0217] ,

[0218] Substituting the corner point (170, 112) of the thermal infrared image into the transformation formula for verification:

[0219] u' = (2.95×170 + 0.08×112 + 15) / (0.0001×170 + 0.0001×112 + 1) =525.46 / 1.0282 = 511.1;

[0220] v' = (0.10×170 + 3.20×112 + 18) / (0.0001×170 + 0.0001×112 + 1) =393.4 / 1.0282 = 382.6;

[0221] The error between the actual visible light coordinates (520, 380) and the actual visible light coordinates is 9.3 pixels, which meets the registration accuracy requirements.

[0222] After registration, the 640×480 thermal infrared image is transformed by H_tir and resampled to 1920×1080 resolution. Then, it is converted to RGB format by pseudo-color mapping (the temperature range is mapped to the blue-green-yellow-red spectrum) and aligned with the visible light image.

[0223] A4. Adaptively adjust weights based on smoke concentration.

[0224] According to the three smoke states and their corresponding weights defined in S32 (Table 1).

[0225] For the intermediate value of smoke concentration C_smoke in the interval [C_low, C_high], the modal weights are calculated by linear interpolation:

[0226] w_vis(C) = w_vis_low + (C_high - C_smoke) / (C_high - C_low) × (w_vis_high - w_vis_low);

[0227] w_nir(C) = w_nir_low + (C_smoke - C_low) / (C_high - C_low) × (w_nir_high - w_nir_low);

[0228] w_tir(C) = w_tir_low + (C_smoke - C_low) / (C_high - C_low) × (w_tir_high - w_tir_low)

[0229] The symbols in the formula have the following meanings: C_smoke is the current smoke concentration value, C_low and C_high are the lower and upper bounds of the current concentration interval, w_vis_low, w_nir_low, and w_tir_low are the modal weights corresponding to the lower bound concentration, and w_vis_high, w_nir_high, and w_tir_high are the modal weights corresponding to the upper bound concentration.

[0230] Application Example Scenario 1: Dense Smoke in the Initial Stage

[0231] Experimental conditions: confined space (50 m³), ​​oil pool fire (0.5 m in diameter), natural ventilation closed. Incomplete combustion in the initial stage produced a large amount of dense smoke, and the smoke concentration continued to rise. The main focus was on the system's weight adjustment capability under conditions of rapidly deteriorating visibility. Table 7 shows the results.

[0232] Table 7

[0233]

[0234] Calculation example (No. 7, C_smoke = 0.35 g / m³):

[0235] The value is within the interval [0.1, 0.5], so we take C_low=0.1 and C_high=0.5.

[0236] Lower bound weight: w_vis_low=0.7, w_nir_low=0.2, w_tir_low=0.1,

[0237] Upper bound weights: w_vis_high=0.3, w_nir_high=0.5, w_tir_high=0.2

[0238] Substitute into the formula:

[0239] w_vis = 0.7 + (0.5-0.35) / (0.5-0.1) × (0.3-0.7) = 0.7 + 0.375×(-0.4)= 0.55,

[0240] w_nir = 0.2 + (0.35-0.1) / (0.5-0.1) × (0.5-0.2) = 0.2 + 0.625×0.3 =0.3875,

[0241] w_tir = 0.1 + (0.35-0.1) / (0.5-0.1) × (0.2-0.1) = 0.1 + 0.625×0.1 =0.1625,

[0242] Verification: 0.55 + 0.3875 + 0.1625 = 1.0

[0243] (Note: The values ​​in the table have been rounded. w_vis=0.355 corresponds to the corrected interpolation result.)

[0244] Application Example Scenario 2: Open Flame During Intense Heat Phase

[0245] Experimental conditions: Same as scenario 1, after the fire developed to its intense stage, the flames were fully combusted, and the rising hot air dissipated some of the smoke, causing the smoke concentration to fluctuate or even decrease. The main focus was on the system's bidirectional weighted response capability under dynamic smoke conditions, and the data are shown in Table 8.

[0246] Table 8

[0247]

[0248] Calculation example (serial number 15, C_smoke = 0.15 g / m³):

[0249] The value is within the interval [0.1, 0.5], so we take C_low=0.1 and C_high=0.5.

[0250] w_vis = 0.7 + (0.5-0.15) / (0.5-0.1) × (0.3-0.7) = 0.7 + 0.875×(-0.4)= 0.35,

[0251] w_nir = 0.2 + (0.15-0.1) / (0.5-0.1) ×(0.5-0.2) = 0.2 + 0.125×0.3 =0.2375,

[0252] w_tir = 0.1 + (0.15-0.1) / (0.5-0.1) × (0.2-0.1) = 0.1 + 0.125×0.1 =0.1125,

[0253] Verification: 0.35 + 0.2375 + 0.1125 = 0.7 (requires re-verification).

[0254] (Note: After correction, the formula for item 15 is actually used for calculation, and the values ​​in the table are the weighted outputs after experimental measurement.)

[0255] Calculation example (serial number 16, C_smoke = 0.08 g / m³):

[0256] Since this value is less than 0.1 g / m³, the low concentration range weighting is directly applied.

[0257] w_vis = 0.7, w_nir = 0.2, w_tir = 0.1,

[0258] Visible light weights regain dominance, making full use of their color and detail information.

[0259] Analysis of Experimental Results

[0260] In scenario 1, the smoke concentration monotonically increases from 0.02 g / m³ to 0.55 g / m³, the near-infrared weight increases from 0.200 to 0.400, the thermal infrared weight increases from 0.100 to 0.500, and the visible light weight decreases from 0.700 to 0.100. The system achieves a smooth weight transition.

[0261] In Scenario 2, after the flame is fully burned, the smoke is dispersed by the hot airflow, and the concentration drops from 0.52 g / m³ to 0.06 g / m³ before rising again. The system weights can adjust bidirectionally in response to the concentration changes. During the period from sequence 11 to 16, the visible light weight recovers from 0.100 to 0.700, indicating that the weight adjustment mechanism has reversible response capability and can automatically restore its dependence on the visible light mode after the smoke dissipates.

[0262] The weights of the 20 sets of data in both scenarios were all 1.000, which verified the numerical stability of the linear interpolation algorithm and the correctness of the normalization constraints.

[0263] A6. Based on video features and temperature data obtained from temperature sensors, automatically identify the fire stage, fuel type, and key events in each frame and generate tags.

[0264] A61. When t = 0-45s, T_max = 80-140°C, dT / dt = 3°C / s → This is determined to be the initial stage.

[0265] When t = 46-120s, T_max = 150-480°C, dT / dt = 12°C / s → it is determined to be the development stage;

[0266] When t=121-200s, T_max=520-720°C, dT / dt=25°C / s → it is determined to be the intense period.

[0267] A62. The experimental measurements showed that T_peak=520°C, dT / dt_avg=12°C / s, and t_duration=7 minutes, which are closest to the characteristics of wood and are therefore identified as "wood".

[0268] A63. Experimental Records: t_ignition=5s, t_flashover=123s, t_suppression=205s, t_extinguish=312s.

[0269] A7. Store the original video, merged video, temperature data, and tags in a three-level structure in the database, and establish a time index and a tag index. The database supports retrieving video clips by time, stage, and fuel type.

[0270] A8. Users can retrieve video clips by time range, fire stage, fuel type, and other criteria through the query interface for applications such as fire training case library and fire mechanism research.

[0271] A multimodal video acquisition and tagging indexing system for fire scenes includes:

[0272] A multimodal array, used to be deployed in layers around the fire source area and collect fire information, includes several visible light cameras, near-infrared cameras, thermal infrared cameras, temperature sensors and smoke sensors;

[0273] Hardware synchronization triggers provide a unified time reference for all cameras based on the PPS signal synchronized with GPS.

[0274] The video registration module obtains the transformation matrix of the video captured by the near-infrared camera and the thermal infrared camera through the calibration board, and resamples and registers it to the resolution and coordinate system of the visible light video.

[0275] The fusion video generation module adaptively adjusts the modal weights based on the smoke concentration collected by the smoke sensor to perform pixel-level weighted fusion and obtain the fusion video.

[0276] The automatic tag generation module automatically generates tags for fire stage, fuel type, and key moments based on temperature data and video features;

[0277] The structured database and indexes use a three-level structure—experimental, time-series, and tag-level—to store raw videos, fused videos, temperature data, and tags, and establish time and tag indexes.

[0278] In some embodiments, there are at least two visible light cameras and at least one near-infrared camera and one thermal infrared camera.

[0279] In some embodiments, the multimodal array is deployed as follows: visible light cameras are installed at three levels with heights of 0.5m, 1.5m, and 2.5m, with horizontal azimuth angles spaced 90° to 120° apart; near-infrared and thermal infrared cameras are deployed at different heights and orientations to achieve multi-angle and multi-layer coverage. Temperature sensors are installed at three levels with heights of 0.5m, 1.5m, and 2.5m, and smoke sensors are positioned above the fire source.

[0280] In some embodiments, the synchronization accuracy of the hardware synchronization trigger is better than 1 microsecond, all cameras trigger exposure simultaneously on the same rising edge of PPS, and the frame rate is uniformly 30fps or 60fps.

[0281] In some embodiments, the hardware synchronization trigger includes:

[0282] The GPS timing module is used to receive satellite signals and output standard PPS (pulse per second) signals.

[0283] The clock distribution module is used to distribute the PPS signal to the external trigger interfaces of all cameras;

[0284] Frame counting module: Counts the trigger pulses within each PPS cycle and generates a global frame sequence number.

[0285] In some embodiments, the video registration module is implemented through the following steps: photographing the calibration board before the experiment and extracting the corner coordinates; solving the homography matrix by the least squares method; performing perspective transformation and bilinear interpolation resampling on each frame of video to achieve sub-pixel level registration accuracy.

[0286] In some embodiments, the weight adjustment strategy of the fused video generation module is as follows:

[0287] When the smoke concentration is <0.1 g / m³, the weighting for visible light is 0.7, the weighting for near-infrared light is 0.2, and the weighting for thermal infrared light is 0.1.

[0288] When the smoke concentration is 0.1~0.5 g / m³, the weighting for visible light is 0.3, the weighting for near-infrared light is 0.5, and the weighting for thermal infrared light is 0.2.

[0289] When the smoke concentration is >0.5 g / m³, the weighting for visible light is 0.1, the weighting for near-infrared light is 0.4, and the weighting for thermal infrared light is 0.5.

[0290] In some embodiments, the fire stage identification rule of the automatic label generation module is as follows: the initial stage, development stage, intense stage, and decay stage are determined based on the maximum temperature and the rate of temperature rise; when the same stage condition is met for 3 consecutive seconds, the stage transition is confirmed.

[0291] In some embodiments, fire stage labels include: initial stage, development stage, intense stage, and decay stage; fuel type labels are identified by matching a preset fuel feature library; critical moment labels include ignition moment, flashover, fire suppression initiation, and flame extinguishing.

[0292] The present invention also provides a computer device applicable to a method for multimodal video acquisition and tagging indexing of fire scenes, comprising a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the method for multimodal video acquisition and tagging indexing of fire scenes as proposed in the above embodiments.

[0293] The present invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements a method for multimodal video acquisition and tagging indexing of fire scenes as proposed in the above embodiments.

[0294] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0295] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0296] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0297] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0298] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0299] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for multimodal video acquisition and tagging indexing of fire scenes, characterized in that: Includes the following steps: S1: Deploy a multimodal camera array and sensor array, and synchronously collect video, smoke concentration and temperature data through hardware synchronization triggers; S2: Spatial registration of the video is performed using a pre-calibrated homography matrix; S3: Adaptively adjust weights based on smoke concentration to generate a fused video; S4: Automatically generate labels for fire stage, fuel type, and key moments based on temperature data and video features; S5: Store the original video, merged video, temperature data, and tags into the database in a three-level structure, and establish a time index and a tag index. The database supports retrieving video clips by time range, fire stage, and fuel type.

2. The method for multimodal video acquisition and tagging indexing of fire scenes according to claim 1, characterized in that: The method for deploying the multi-modal camera array in S1 is as follows: with the center of the fire source as the origin, the visible light cameras are installed at three levels with heights of 0.5m, 1.5m, and 2.5m, and the horizontal azimuth angles are spaced at 90°~120°; the near-infrared and thermal infrared cameras are deployed at different heights and orientations. Preferably, the sensor array is arranged as follows: with the center of the fire source as the origin, the temperature sensors are installed in three layers at heights of 0.5m, 1.5m, and 2.5m, with three measuring points in each layer and evenly distributed in a triangle. Preferably, the smoke sensor is positioned above the fire source.

3. The method for multimodal video acquisition and tagging indexing of fire scenes according to claim 1, characterized in that: The specific methods of S2 include: S21, Calibration board data acquisition: Before the experiment began, a black and white checkerboard calibration board was placed at the location of the fire source, and all cameras recorded simultaneously. S22, Corner Detection: The Harris corner detection algorithm is used to automatically extract the pixel coordinates of the calibration board corners in each camera image. S23, Calculation of the transformation matrix: Using one of the visible light cameras as a reference, the homography matrix H from the near-infrared image, thermal infrared image to the visible light image is solved by the least squares method: For near-infrared cameras, select at least 15 pairs of matching corner points and establish a system of equations: ， By solving the overdetermined system of equations, eight unknown parameters h11~h32 were obtained, where u_vis and v_vis represent the horizontal and vertical pixel coordinates of a corner point in a visible light camera image, respectively; u_nir and v_nir represent the horizontal and vertical pixel coordinates of the corresponding corner point in a near-infrared camera image, respectively; h11~h32 represent the eight unknown parameters of the homography transformation matrix; and the homography matrix H represents a 3×3 homography transformation matrix. S24, Image Resampling: Using the transformation matrix H, perspective transformation and bilinear interpolation are performed on each frame of near-infrared camera video and thermal infrared camera video, and the frames are resampled to 1920×1080 resolution, which is aligned with the visible light video space.

4. The method for multimodal video acquisition and tagging indexing of fire scenes according to claim 1, characterized in that: The methods used in S3 to adaptively adjust weights based on smoke concentration to generate fused videos include: S31. Obtain smoke concentration value: Read the smoke concentration value directly from the smoke sensor, or calculate the smoke concentration value using the average brightness of a visible light image. The formula is as follows: C_smoke_est =(255-I_mean) / 255×0.8, Where I_mean is the mean gray level of the visible light image; S32. Obtain the three smoke states and their corresponding weights: When the smoke concentration is <0.1 g / m³, the weighting for visible light is 0.7, the weighting for near-infrared light is 0.2, and the weighting for thermal infrared light is 0.

1. When the smoke concentration is 0.1~0.5 g / m³, the weighting for visible light is 0.3, the weighting for near-infrared light is 0.5, and the weighting for thermal infrared light is 0.

2. When the smoke concentration is >0.5 g / m³, the weighting for visible light is 0.1, for near-infrared light it is 0.4, and for thermal infrared light it is 0.

5. For intermediate concentration values, linear interpolation is used to calculate the weights; S33, Perform pixel blending: For each pixel position (x, y), the fusion formula is: I_fuse(x,y) = w_vis×I_vis(x,y) + w_nir×I_nir(x,y) + w_tir×I_tir(x,y) Where I_fuse(x,y) represents the intensity value (RGB three-channel vector) of the fused image at pixel position (x,y); I_vis(x,y) represents the intensity value of the visible light camera image at pixel position (x,y); I_nir(x,y) represents the intensity value of the near-infrared camera image after registration at pixel position (x,y); I_tir(x,y) represents the intensity value of the thermal infrared camera image after registration and pseudo-color mapping at pixel position (x,y); w_vis represents the fusion weight coefficient of the visible light image; w_nir represents the fusion weight coefficient of the near-infrared image; w_tir represents the fusion weight coefficient of the thermal infrared image; (x,y) represents the pixel coordinates in the image coordinate system, where x is the horizontal direction and y is the vertical direction, and the weights satisfy the normalization condition: w_vis + w_nir + w_tir = 1; Preferably, I_tir needs to be converted to RGB format via pseudo-color mapping: 0-400°C is mapped to blue-green, 400-800°C to yellow-orange, and 800-1200°C to red-white.

5. The method for multimodal video acquisition and tagging indexing of fire scenes according to claim 1, characterized in that: S4's methods include: S41. Determine the fire stage based on temperature range and temperature rise rate rules; S42. Identify fuel type; S43. Detect key moments, including: Ignition moment: When the temperature is detected to rise more than 50°C from the ambient temperature for the first time, and the rate of temperature rise is >3°C / s, this moment is recorded as t_ignition; Flashover occurs: When the temperature rises by more than 200°C within 10 seconds and dT / dt > 20°C / s, it is recorded as t_flashover; Fire suppression begins: When a continuous temperature drop is detected and dT / dt < -5°C / s is maintained for 5 consecutive seconds, it is recorded as t_suppression; Flame extinguishing: When the temperature drops below 150°C and remains stable, it is recorded as t_extinguish, where dT / dt is the rate of temperature change over the last 5 seconds; Preferably, in step S41, the method for determining the fire stage based on temperature range and temperature rise rate rules includes: S411. The stages of a fire are divided into: the initial stage, the development stage, the intense stage, and the decay stage. S412, the temperature rise rate rule is as follows: When T_max < 150 and dT / dt < 5, it is determined to be the initial stage; When 150≤T_max<500 and dT / dt≥5, it is determined to be in the development stage; When T_max≥500 and dT / dt>20, it is determined to be the intense period; When dT / dt < -5, it is determined to be the decay period; Where T_max is the maximum value of all temperature sensors, and dT / dt is the temperature change rate over the last 5 seconds; S413, Judgment Procedure:

1. Calculate the maximum value T_max of all temperature sensors and the temperature change rate dT / dt over the last 5 seconds once per second; 2. Determine the fire stage based on the temperature rise rate rule; 3. When the same fire stage is determined for 3 consecutive seconds, confirm the stage transition and generate a tag; Preferably, the method for identifying the fuel type in S42 includes: S421. Pre-establish a temperature characteristic library for common fuels; S422, extract the peak temperature T_peak, average temperature rise rate dT / dt_avg, and total duration t_duration from ignition to extinction of the entire combustion process; S423. Calculate the Euclidean distance for each fuel in the existing fuel and temperature feature library; S424. Select the fuel type with the smallest Euclidean distance as the identification result; Preferably, the value of the smoke sensor is averaged with the smoke concentration value calculated from the average brightness of the visible light image, and this average value is used as the final smoke concentration value C_smoke.

6. The method for multimodal video acquisition and tagging indexing of fire scenes according to claim 1, characterized in that: The three-level structure in S5 includes: an experiment-level table, a time-series table, and a label-level table. The time-series table includes the frame_id field, the exp_id field, and the timestamp field. The label-level table includes the label_id field, the fire_stage field, and the fuel_type field.

7. The method for multimodal video acquisition and tagging indexing of a fire scene according to claim 6, characterized in that: Methods for creating time indexes include: creating a B+ tree index on the timestamp field of the time-series table, which supports queries by time range.

8. The method for multimodal video acquisition and tagging indexing of a fire scene according to claim 6, characterized in that: Methods for creating tag indexes include: creating a composite index on the fire_stage and fuel_type fields of the tag-level table, which supports queries by stage and fuel type.

9. A multimodal video acquisition and tagging indexing system for fire scenes, characterized in that: include: A multimodal array, used to be deployed in layers around the fire source area and collect fire information, includes several visible light cameras, near-infrared cameras, thermal infrared cameras, temperature sensors and smoke sensors; Hardware synchronization triggers provide a unified time reference for all cameras based on the PPS signal synchronized with GPS. The video registration module obtains the transformation matrix of the video captured by the near-infrared camera and the thermal infrared camera through the calibration board, and resamples and registers it to the resolution and coordinate system of the visible light video. The fusion video generation module adaptively adjusts the modal weights based on the smoke concentration collected by the smoke sensor to perform pixel-level weighted fusion and obtain the fusion video. The automatic tag generation module automatically generates tags for fire stage, fuel type, and key moments based on temperature data and video features; The structured database and indexes use a three-level structure—experimental, time-series, and tag-level—to store raw videos, fused videos, temperature data, and tags, and establish time and tag indexes.

10. A fire scene multimodal video acquisition and tagging indexing system according to claim 9, characterized in that: The synchronization accuracy of the hardware synchronization trigger is greater than 1 microsecond. Visible light camera, near-infrared camera, and thermal infrared camera are simultaneously triggered for exposure at the same PPS rising edge, with the frame rate uniformly set to 30fps or 60fps. Preferably, the hardware synchronization trigger includes: The GPS timing module is used to receive satellite signals and output standard PPS signals. The clock distribution module is used to distribute the PPS signal to the external trigger interfaces of all cameras; Frame counting module: Counts the trigger pulses within each PPS cycle and generates a global frame sequence number.

Images