AR emergency fire fighting helmet system integrating multi-sensor image fusion

Abstract

The invention relates to the technical field of photoelectric imaging detection and augmented reality display, and discloses an AR emergency fire fighting helmet system integrated with multi-sensor image fusion, and the system comprises an active and passive multi-dimensional imaging module which synchronously collects active gating depth and passive polarization heat radiation data to construct a light field source; the time sequence synchronization and scanning control module is used for driving depth scanning and synchronously generating a zoom signal; the environment medium characteristic inversion module is used for analyzing the polarization characteristics to invert a dynamic extinction coefficient; the target physical parameter resolving module is used for executing transmission compensation by utilizing the extinction coefficient so as to resolve the real recovery radiance; and the fusion display and man-machine interaction module is used for projecting a fusion image in response to the zoom signal. Through multi-dimensional complementary imaging and physical inversion logic, perspective detection and real temperature field reduction in a dense smoke environment are realized, and in combination with object image conjugate zoom control, visual convergence conflicts can be eliminated and the sensing precision of fire scene operation can be improved.

Description

Technical Field

[0001] This invention relates to the field of optoelectronic imaging detection and augmented reality display technology, specifically to an AR emergency fire helmet system integrating multi-sensor image fusion. Background Technology

[0002] The AR emergency fire helmet system is a digital firefighting equipment for individual combat that integrates multi-band photoelectric detection sensors and near-eye display components. It is mainly deployed at fire rescue sites with extremely low visibility and complex environments to assist frontline firefighters in performing search and rescue and firefighting tasks. By superimposing real-time perceived environmental physical data, heat distribution status, and navigation guidance information onto the wearer's line of sight, it achieves penetrating perception of the fire scene environment and improves rescue response efficiency and personnel survival ability in high-risk operation scenarios.

[0003] Existing fire-fighting visual assistance technologies mainly rely on uncooled long-wave infrared thermal imaging modules for passive environmental perception. The mainstream technical solutions usually integrate thermal imagers directly into handheld observation devices or the outside of helmets, using the radiative temperature difference between objects of different materials and heat sources to generate pseudo-color thermal images. Some advanced solutions attempt to combine visible light cameras for dual-channel image fusion display, trying to retain thermal radiation temperature information while introducing some edge contours and texture details of the scene to help observers judge the distribution of surrounding obstacles and the location of trapped personnel in smoke-filled corridors or rooms.

[0004] However, existing passive thermal imaging technologies have limitations when facing high-concentration aerosol environments deep within fire zones. The scattering and absorption of infrared radiation by smoke particles causes nonlinear attenuation of the target signal along the transmission path. Simultaneously, the path radiation emitted by the high-temperature smoke medium itself superimposes onto the imaging detector, creating high-intensity background thermal noise. This results in images that are often blurry with extremely low contrast, unable to effectively penetrate smoke barriers to resolve the geometric contours or texture details of obscured targets. Furthermore, the lack of real-time inversion and compensation methods for the optical properties of the transmission medium means that the radiance data received by the detection equipment cannot reflect the true surface temperature of the target, easily leading to misjudgments by rescue personnel. The distribution of the thermal field can lead to serious misjudgments. In addition, traditional head-mounted display devices generally use fixed focal length optical projection schemes, which means that the imaging plane of the virtual image cannot dynamically adjust with changes in the observer's gaze depth. This depth mismatch between virtual information and physical scene in spatial position forces the human eye to frequently adjust between different focal lengths. The resulting visual convergence-accommodation conflict not only accelerates visual fatigue and induces dizziness, but also causes severe dynamic misalignment and delay between the virtual heat map and real objects during rapid movement or search due to the lack of image stabilization and gaze compensation mechanisms coupled with the wearer's movement state. This restricts the ability to accurately search and rescue and protect oneself in complex environments. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an AR emergency fire helmet system integrating multi-sensor image fusion, aiming to solve the problems of the lack of see-through imaging and radiation temperature measurement capabilities in traditional equipment, as well as the problems of visual convergence adjustment conflict and virtual-real dynamic alignment deviation in augmented reality displays.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] This invention provides an AR emergency fire helmet system that integrates multi-sensor image fusion. The system mainly solves the technical problem that traditional visual devices are unable to simultaneously achieve penetrating imaging, real temperature field restoration, and accurate alignment of virtual information with physical space in the context of dense smoke in a fire scene.

[0008] The system mainly includes an active and passive multidimensional imaging module, a time synchronization and scanning control module, an environmental medium property inversion module, a target physical parameter calculation module, and a fusion display and human-computer interaction module.

[0009] At the imaging acquisition level, the active and passive multidimensional imaging modules employ a dual-channel synchronous detection architecture. The active detection channel is configured as an active short-wave infrared gating imaging unit. This unit utilizes the penetrating characteristics of short-wave infrared light through smoke particles, combined with time gating technology, to precisely control the time delay between laser pulse emission and electronic shutter opening, receiving only backscattered echoes from a specific spatial depth, thereby physically filtering out background smoke noise at the front of the field of view. The passive detection channel is configured as a passive focal plane polarization thermal imaging unit, utilizing a micro-polarization array integrated on the detector's focal plane to simultaneously capture the scene's thermal radiation intensity and linear polarization state signals at multiple angles.

[0010] For polarization feature analysis, the system executes linear polarization degree calculation logic. This logic analyzes the original thermal radiation signal, separates the Stokes vector components, and quantifies the linear polarization degree of each pixel by calculating the ratio of the composite vector magnitude of the polarization component differences to the total radiation intensity. This polarization degree information reflects the surface roughness and material properties of the object, and is used to enhance the edge contour contrast of man-made objects with natural backgrounds or smoky media in thermal radiation images.

[0011] At the temporal control and spatial scanning level, the temporal synchronization and scanning control module executes the depth tomography scanning logic. During the search phase, the module linearly increases the gating delay time by a preset step size, driving the imaging slices to move layer by layer from the near field to the far field to search for the target of interest;

[0012] During the locking phase, the delay parameters are fixed in response to the system's optimal observation layer command. Simultaneously, the module generates a zoom drive signal based on object-image conjugate logic, calculates the drive voltage required for the liquid lens according to the current imaging detection distance, and adjusts the optical path diopter to ensure that the virtual image plane of the augmented reality display system is consistent with the actual imaging detection plane in terms of physical distance, eliminating the visual convergence-accommodation conflict for the wearer when observing virtual information and real scenes.

[0013] At the environmental medium property inversion level, the environmental medium property inversion module first calls the calibration parameters to perform spatial registration of multi-source images, ensuring strict alignment of active and passive images in the pixel coordinate system. Subsequently, the module executes the medium parameter inversion logic based on dual-depth slicing.

[0014] This logic selects two gated image slices at different detection distances, and calculates the average extinction coefficient of the light wave on the current smoke medium transmission path based on the difference in the ratio of the smoke echo intensity received at the two distances, combined with the inverse square law of light wave attenuation in spatial transmission. This allows for the dynamic quantification of the visibility and attenuation characteristics of the fire environment.

[0015] At the target physical parameter calculation and image enhancement level, the target physical parameter calculation module uses the aforementioned dynamic extinction coefficient to perform medium transmission model compensation on the original data source. For thermal radiation data, the system constructs a transmittance distribution map, subtracts the path radiation component and absorption loss component caused by the smoke medium from the total radiance received by the sensor, and inverts to obtain the true restored radiance of the target surface, thereby realizing the restoration of the true temperature field of the obscured target.

[0016] At the image fusion and display level, the system adopts a channel replacement fusion strategy based on color space. The multi-dimensional feature fusion unit separates the image data into luminance and chrominance components. The luminance component is weighted and synthesized from enhanced short-wave infrared texture details and linear polarization edge features to preserve the geometric structure of the scene; the chrominance component is generated by mapping the temperature data obtained from the conversion of the true restored radiance, and the color gradient is used to intuitively represent the temperature distribution of the fire scene.

[0017] Furthermore, the integrated display and human-computer interaction module incorporates an inertial measurement unit to collect the helmet's angular velocity vector in real time. The system uses the mapping relationship between pixel coordinates and spatial angles to calculate the inter-frame gaze offset, performs reverse displacement compensation on the displayed virtual image coordinates, achieves electronic image stabilization, and ensures that the virtual thermal map remains anchored to the surface of the real object throughout the wearer's movement.

[0018] Through the aforementioned system architecture, this invention achieves active tomographic detection and passive polarization resolution of complex lighting environments in fire scenes. It utilizes physical model inversion to eliminate the blurring and attenuation effects of smoke on imaging. Combined with spatial alignment and image stabilization technologies, it provides augmented reality visual assistance with fog-penetrating clarity, realistic temperature perception, and accurate spatial positioning within the firefighters' field of vision.

[0019] This invention provides an AR emergency fire helmet system integrating multi-sensor image fusion. It has the following beneficial effects:

[0020] 1. This invention utilizes the collaborative operation of active and passive multidimensional imaging modules. The system employs the time gating mechanism of active shortwave infrared to physically cut off the backscattering noise of smoke at the front of the field of view. Simultaneously, it combines the polarization resolution capability of passive thermal imaging to extract the unique edge and texture features of man-made objects from the chaotic thermal radiation signal. This dual-channel complementary detection method enables firefighters to obtain perspective images with clear outlines and a sense of depth in smoke-filled environments with zero visibility, which helps improve the efficiency of identifying obstacles and trapped targets.

[0021] 2. This invention, through the linkage of the environmental medium characteristic inversion and target physical parameter calculation modules, uses dual-depth slice data to calculate the dynamic extinction coefficient of the current smoke medium in real time, and constructs a transmission compensation model accordingly. The path radiation and absorption loss are deducted from the attenuation signal received by the sensor. This mechanism helps to correct the temperature measurement deviation caused by the interference of dense smoke medium, presenting the wearer with enhanced visual data with high fidelity and real temperature warning significance, and avoiding the situation of misjudging the core strength of the fire source due to environmental attenuation.

[0022] 3. This invention combines time-series synchronous control with inertial sensing technology. The system generates a zoom drive signal that follows the object-image conjugate relationship based on real-time depth detection, so that the focal plane of the virtual image always follows the physical position change of the actual observed target, which helps to eliminate the dizziness caused by visual convergence-accommodation conflict. At the same time, with the electronic image stabilization logic based on angular velocity monitoring, it cancels the eye movement caused by the wearer during vigorous movement, ensuring the stability of the virtual thermal map and the real fire scene objects in spatial position. Attached Figure Description

[0023] Figure 1 This is a system framework diagram of the present invention;

[0024] Figure 2 This is a schematic diagram of the method flow of the present invention.

[0025] The module includes: 100, Active and Passive Multidimensional Imaging Module; 110, Active Shortwave Infrared Gating Imaging Unit; 120, Passive Focusing Plane Polarization Thermal Imaging Unit; 200, Timing Synchronization and Scanning Control Module; 300, Environmental Medium Characteristics Inversion Module; 310, Data Preprocessing and Registration Unit; 320, Medium Parameter Inversion Unit; 400, Target Physical Parameter Calculation Module; 410, Thermal Radiation Inversion and Temperature Correction Unit; 420, Multidimensional Feature Fusion and Target Recognition Unit; and 500, Fusion Display and Human-Computer Interaction Module. Detailed Implementation

[0026] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] See attached document Figure 1 This invention provides an AR emergency fire helmet system integrating multi-sensor image fusion. This system can be deployed in individual soldier wearable equipment that includes optical sensing components, embedded computing units and near-eye display components, and is used to achieve penetrating perception, physical parameter inversion and augmented reality display in complex fire scene environments.

[0028] An AR emergency firefighting helmet system integrating multi-sensor image fusion, comprising:

[0029] The system includes an active and passive multidimensional imaging module 100, a timing synchronization and scanning control module 200, an environmental medium property inversion module 300, a target physical parameter calculation module 400, and a fusion display and human-computer interaction module 500.

[0030] The active and passive multidimensional imaging module 100 is equipped with an active laser illumination device and a passive thermal radiation detection device to acquire multidimensional light field signals of the fire environment. The active and passive multidimensional imaging module 100 transmits the acquired active shortwave infrared gating image data and passive focusing plane polarization thermal radiation image data to the subsequent processing module through a high-speed data interface.

[0031] The timing synchronization and scanning control module 200 is electrically connected to the active / passive multidimensional imaging module 100 and the fusion display and human-machine interaction module 500, respectively, and is used to generate a system-level reference clock signal. The timing synchronization and scanning control module 200 is equipped with a programmable gate array, which is used to send a precisely delayed gating trigger signal to the active / passive multidimensional imaging module 100 to perform depth tomography scanning, and simultaneously send a zoom drive signal to the fusion display and human-machine interaction module 500 to ensure that the physical positions of the imaging acquisition plane and the display virtual image plane are synchronized.

[0032] The environmental medium property inversion module 300, connected to the active-passive multidimensional imaging module 100, is used to receive passive polarized thermal radiation images. The environmental medium property inversion module 300 is equipped with dark channel polarization analysis logic and a medium mapping lookup table to identify background scattering regions in the image and invert the dynamic extinction coefficient of the current environment for the shortwave infrared band based on the average polarization degree of the background region.

[0033] The target physical parameter calculation module 400 is connected to the timing synchronization and scanning control module 200 and the environmental medium characteristic inversion module 300, respectively, and is used to perform the calculation of core physical parameters. The target physical parameter calculation module 400 uses a heterogeneous gradient cross-correlation algorithm to calculate the spatial correlation between active and passive images, thereby determining the optimal observation distance and feeding it back to the timing synchronization and scanning control module 200 for locking; at the same time, this module uses the dynamic extinction coefficient to perform atmospheric transmission compensation on the active echo data of the optimal observation layer, and combines polarization information to calculate the target's true reflectivity and true temperature field.

[0034] The fusion display and human-computer interaction module 500, connected to the target physical parameter calculation module 400, is used to construct an augmented reality view. The fusion display and human-computer interaction module 500 is equipped with an image fusion engine and an electric zoom optical component. It performs weighted fusion of the geometric contours, texture details, and true-temperature colors output by the target physical parameter calculation module 400, and projects the generated image frame onto a virtual image plane that matches the optimal viewing distance.

[0035] In one embodiment of the present invention, the aforementioned AR emergency fire helmet system integrating multi-sensor image fusion operates on an embedded computing device. This computing device includes a processor, a memory, a system bus, and peripheral interfaces. The processor is connected to the memory and peripheral interfaces via the system bus. The memory stores computer program instructions, which, when executed by the processor, implement the functions of the active / passive multi-dimensional imaging module 100, the timing synchronization and scanning control module 200, the environmental medium characteristic inversion module 300, the target physical parameter calculation module 400, and the fusion display and human-computer interaction module 500.

[0036] The processor can include heterogeneous computing units of FPGA (Field Programmable Gate Array) and SoC (System-on-a-Chip) for performing nanosecond-level timing control and parallel image matrix operations. The memory includes non-volatile storage media and high-speed random access memory for storing media-mapped lookup tables, intermediate image frame sequences, and the final generated enhanced image.

[0037] The system workflow is as follows:

[0038] The active and passive multidimensional imaging module 100 first acquires ambient light field data; the environmental medium characteristic inversion module 300 calculates the environmental extinction coefficient based on the passive polarization data; the timing synchronization and scanning control module 200 drives the system to perform depth scanning, while the target physical parameter calculation module 400 locks the optimal observation layer by calculating the correlation of heterogeneous images; in the locked state, the target physical parameter calculation module 400 calculates the target's true reflectivity and temperature in combination with the extinction coefficient; finally, the fusion display and human-computer interaction module 500 generates a multidimensional fused image, which is then projected to the wearer's eyes through the zoomed display component.

[0039] The active and passive multidimensional imaging module 100 adopts a dual-optical-path parallel calibration structure. In terms of hardware configuration, it includes an active short-wave infrared gating imaging unit 110 and a passive focusing plane polarization thermal imaging unit 120. The active short-wave infrared gating imaging unit 110 and the passive focusing plane polarization thermal imaging unit 120 are fixed by a precision mechanical bracket, and their optical field of view (FOV) is kept overlapping and matched to ensure that the spatial geometric reference of the output image is consistent.

[0040] The active shortwave infrared gating imaging unit 110 is used to acquire target echo images after removing atmospheric backscattering interference. The active shortwave infrared gating imaging unit 110 includes a pulsed laser illuminator with a center wavelength of 1550nm and a shortwave infrared gating camera that responds to the 1550nm band.

[0041] The pulsed laser illuminator is configured to emit laser pulses with a full width at half maximum (FWHM) of 10 ns to 200 ns. The short-wave infrared gating camera uses an indium gallium arsenide focal plane array sensor and integrates a high-speed electronic shutter circuit. The signal input terminal of the active short-wave infrared gating imaging unit 110 is connected to the timing synchronization and scanning control module 200 via a high-speed differential signal interface (LVDS). The active short-wave infrared gating imaging unit 110 receives a trigger signal to control the time difference between the laser emission time and the shutter opening time.

[0042] The active shortwave infrared gating imaging unit 110 achieves slice observation of space at a specific depth through a gating imaging mechanism. The working principle of the active shortwave infrared gating imaging unit 110 is as follows:

[0043] After the pulsed laser illuminator emits a pulse, the shutter of the short-wave infrared gating camera remains closed to block backscattered light from nearby smoke from entering the sensor. After a preset delay, the shutter of the short-wave infrared gating camera opens and remains gated for a preset time. The images acquired during the gating imaging process correspond to a specific spatial depth range, calculated using the following formula:

[0044]

[0045] In the formula, Drange t represents the imaging gating depth range; c represents the speed of light in the atmosphere; t represents the speed of light in the atmosphere. delay The time delay between laser pulse emission and shutter opening; t gate The shutter stays open for a specified gate time.

[0046] The passive focusing plane polarization thermal imaging unit 120 is used to simultaneously acquire the thermal radiation intensity and polarization state information of the scene. The passive focusing plane polarization thermal imaging unit 120 includes an uncooled long-wave infrared focal plane detector and a micro-polarization array integrated on the photosensitive surface of the uncooled long-wave infrared focal plane detector.

[0047] The response band of the uncooled long-wave infrared focal plane detector is 8μm to 14μm. The micro-polarization array is composed of several 2×2 arranged superpixels periodically spliced ​​together. Each superpixel contains four adjacent detector pixels, corresponding to the four linear polarization transmission directions of 0°, 45°, 90° and 135° respectively.

[0048] The passive focusing plane polarization thermal imaging unit 120 is equipped with embedded image processing logic, which is used to analyze the raw polarization mosaic image output by the uncooled long-wave infrared focal plane detector and calculate the Stokes vector and the degree of linear polarization.

[0049] The steps of the data parsing logic of the passive focusing plane polarization thermal imaging unit 120 are as follows:

[0050] The passive focusing plane polarization thermal imaging unit 120 extracts the grayscale response values ​​of four polarization channels from the raw image output by the uncooled long-wave infrared focal plane detector, denoted as I0(x,y), I... 45 (x,y), I 90 (x,y) and I 135 (x,y), where (x,y) represents the coordinate position of the pixel in the focal plane array.

[0051] The passive focusing plane polarization thermal imaging unit 120 calculates the S0, S1, and S2 components in the Stokes vector using the grayscale response values ​​of the four polarization channels. The calculation formula is as follows:

[0052]

[0053] In the formula, S0(x,y) is the total radiation intensity component, which characterizes the thermal radiation energy distribution of the scene; S1(x,y) is the difference between the horizontal and vertical linear polarization components; and S2(x,y) is the difference between the 45° and 135° linear polarization components.

[0054] The passive focusing plane polarization thermal imaging unit 120 calculates the degree of linear polarization of each pixel based on the calculated Stokes vector components. The calculation formula is as follows:

[0055]

[0056] In the formula, DoLP(x,y) is the degree of linear polarization, which ranges from 0 to 1 and is used to characterize the polarization characteristics of the radiation light radiated from the target surface; S0(x,y) is the total radiation intensity component, which characterizes the thermal radiation energy distribution of the scene; S1(x,y) is the difference between the horizontal and vertical linear polarization components; and S2(x,y) is the difference between the 45° and 135° linear polarization components.

[0057] Through the above hardware configuration and data processing logic, the active and passive multidimensional imaging module 100 can output active echo intensity images, full-field passive thermal radiation intensity images and passive linear polarization degree images corresponding to specific depth layers in real time, and transmit the above image data to subsequent modules through a high-speed parallel data bus for environmental parameter inversion and physical property calculation.

[0058] The timing synchronization and scanning control module 200 serves as the system's time-domain reference generator and logic control hub. Its hardware composition mainly includes a field-programmable gate array (FPGA) main control unit, a high-precision temperature-compensated crystal oscillator (TCXO), a digital-to-analog converter (DAC), and a multi-channel high-speed digital isolation interface.

[0059] The timing synchronization and scanning control module 200 achieves precise modulation of nanosecond-level time slices through hardware logic programming, and establishes bidirectional communication links with the active and passive multidimensional imaging module 100, the target physical parameter calculation module 400, and the fusion display and human-computer interaction module 500, respectively.

[0060] The core function of the timing synchronization and scan control module 200 is to generate and distribute strictly synchronized clock signals and trigger pulses. The timing synchronization and scan control module 200 utilizes a high-precision temperature-compensated crystal oscillator to generate a reference clock source and generates the system master clock through a phase-locked loop (PLL) frequency multiplication circuit within the field-programmable gate array. The timing synchronization and scan control module 200 is equipped with a global state machine, which divides the system operating modes into a deep scan search mode and a deep lock-on observation mode.

[0061] In the depth scan search mode, the timing synchronization and scan control module 200 drives the active and passive multidimensional imaging module 100 to perform distance tomography scanning along the line of sight. The timing synchronization and scan control module 200 generates periodic laser trigger signals and shutter gating signals.

[0062] In order to cover the preset detection depth range, the timing synchronization and scanning control module 200 linearly increases the delay between the laser emission time and the shutter opening time according to a preset time step within each imaging frame period or between adjacent frame periods.

[0063] The timing synchronization and scan control module 200 calculates the pulse delay time of the nth scan level using the following formula:

[0064] t delay (n)=t start +n·Δt;

[0065] In the formula, t delay (n) represents the delay time corresponding to the nth scan slice; t start Δt represents the initial delay time corresponding to the starting detection distance; n is the index sequence number of the scanning layer, which takes the value of an integer; Δt is the single time step increment corresponding to the depth resolution.

[0066] The timing synchronization and scan control module 200 will generate t delay (n) is converted into a count value corresponding to the clock cycle, and the laser trigger signal and shutter gating signal are sent to the active-passive multidimensional imaging module 100 through the high-speed differential signal interface (LVDS). The timing synchronization and scanning control module 200 control the active-passive multidimensional imaging module 100 to move the gated imaging range from the near field to the far field in a step-by-step manner in physical space, thereby acquiring a series of active echo images of slices at different distances.

[0067] The timing synchronization and scan control module 200 simultaneously receives feedback control signals from the target physical parameter calculation module 400 via a high-speed serial bus (SPI). When the target physical parameter calculation module 400 identifies a distance slice containing a high-confidence target (i.e., determined to be the optimal observation layer), the target physical parameter calculation module 400 sends a locking command and the corresponding optimal delay time parameter to the timing synchronization and scan control module 200.

[0068] In response to the lock command, the timing synchronization and scan control module 200 switches from the depth scan search mode to the depth lock observation mode. In the depth lock observation mode, the timing synchronization and scan control module 200 stops accumulating the scan index n and fixes the pulse delay time at the optimal feedback delay time. The timing synchronization and scan control module 200 continuously drives the active and passive multidimensional imaging modules 100 at the fixed optimal delay time, ensuring that the system continuously acquires clear echo images of the target plane, providing a stable data stream for subsequent physical parameter calculations.

[0069] The timing synchronization and scanning control module 200 is also responsible for synchronously driving the motorized zoom optical components in the fusion display and human-computer interaction module 500. The timing synchronization and scanning control module 200 operates based on the current imaging distance (in scanning mode, the current D...). n In locked mode, it is D optThe system calculates the digital focus control quantity and uses a digital-to-analog converter (DAC) to convert this digital quantity into an analog drive voltage. The timing synchronization and scan control module 200 ensures that the distance to the virtual image plane of the AR display is consistent with the current imaging detection distance to eliminate the visual convergence-accommodation conflict that occurs when the human eye observes.

[0070] The timing synchronization and scan control module 200 calculates the conversion logic of the zoom control signal as follows:

[0071]

[0072] In the formula, V focus D is the drive voltage value output to the motorized zoom lens. target The current imaging detection distance; k is the scaling factor stored in the register related to the refractive power characteristics of the zoom lens; V offset This is the lens reference bias voltage stored in the register.

[0073] Through the aforementioned hardware logic, the timing synchronization and scanning control module 200 constructs a unified spatiotemporal coordinate system for the AR emergency fire helmet system integrating multi-sensor image fusion. This ensures that in complex fire scene environments, the system can accurately project the penetrating visual information collected by the active and passive multi-dimensional imaging module 100 onto a virtual image plane that matches the wearer's line of sight depth, thus realizing the fusion of augmented reality display content and real fire scene in physical space.

[0074] The environmental medium characteristic inversion module 300 consists of a data preprocessing and registration unit 310 and a medium parameter inversion unit 320 in terms of system architecture. The environmental medium characteristic inversion module 300 adopts an embedded heterogeneous computing architecture and realizes data interaction between units through an internal high-speed bus.

[0075] The data preprocessing and registration unit 310 is used to perform noise reduction, cleaning, and spatial geometric correction on the raw images output by the active and passive multidimensional imaging module 100, generating image data with strict spatial alignment. For active shortwave infrared gated images, the data preprocessing and registration unit 310 uses an improved adaptive bilateral filtering algorithm to perform coherent speckle noise suppression, smoothing grain noise while preserving high-frequency edge information.

[0076] Subsequently, the data preprocessing and registration unit 310 uses a homography matrix pre-calibrated and stored in the onboard non-volatile Flash memory to map the active shortwave infrared image onto the coordinate system of the passive thermal imaging. The calculation logic for geometric correction performed by the data preprocessing and registration unit 310 is as follows:

[0077]

[0078] In the formula, (x,y) are the original pixel coordinates in the active shortwave infrared image; (x ′ ,y ′ ) represents the pixel coordinates corresponding to the passive thermal imaging coordinate system after transformation; H is the pre-calibrated 3×3 homography transformation matrix; h 11 to h 33 These are elements of the homography transformation matrix, representing scaling, rotation, translation, and perspective projection parameters.

[0079] The medium parameter inversion unit 320 is connected to the output of the data preprocessing and registration unit 310. It is used to calculate the optical attenuation characteristics of the fire smoke medium based on the aligned temporal gating data. The medium parameter inversion unit 320 uses the depth scan index information provided by the temporal synchronization and scan control module 200 to select two gating slices with different depths z1 and z2 (z2>z1) and calculate the path average extinction coefficient corresponding to each pixel in the field of view.

[0080] The calculation formula for the extinction coefficient performed by the medium parameter inversion unit 320 is as follows:

[0081]

[0082] In the formula, μ ext (x,y) represents the average extinction coefficient of the medium at coordinates (x,y), in m. -1 z1 and z2 are the center detection distances of the proximal and distal gating slices, respectively; I smoke (x,y,z1) represents the intensity of the smoke backscattered echo at depth z1; I smoke (x,y,z2) represents the intensity of the smoke backscattered echo at depth z2; z 2 This term is used to correct for spherical wave attenuation caused by geometric distance.

[0083] The medium parameter inversion unit 320 further constructs a long-wave infrared transmittance spectrum to correct the passive thermal radiation intensity based on the calculated extinction coefficient. The formula for calculating the medium transmittance at the target depth by the medium parameter inversion unit 320 is as follows:

[0084] τ LWIR (x,y,D target )=exp(-k ratio μ ext (x,y)·D target );

[0085] In the formula, τ LWIR D represents the atmospheric transmittance at target distance in the long-wave infrared band. targetThe target physical distance is obtained by the target physical parameter calculation module 400 based on the image fusion result feedback, or provided by the current locking depth of the timing synchronization and scan control module 200; μ ext (x,y) represents the measured extinction coefficient in the short-wave infrared band; k ratio These are preset band conversion coefficients stored in the register, used to characterize the ratio of extinction capability between the 1550nm band and the 8-14μm band.

[0086] Through the cascaded processing of data preprocessing and registration unit 310 and medium parameter inversion unit 320, environmental medium characteristic inversion module 300 outputs a spatially aligned and physically quantified environmental parameter field, providing accurate correction factors for subsequent target recognition and enhanced display of the system.

[0087] The target physical parameter calculation module 400 consists of a thermal radiation inversion and temperature correction unit 410 and a multi-dimensional feature fusion and target recognition unit 420 in terms of system architecture. The target physical parameter calculation module 400 is equipped with an embedded neural network acceleration processor (NPU) and a multi-channel video output interface in terms of hardware, which are used to perform high-concurrency pixel-level calculations.

[0088] The thermal radiation inversion and temperature correction unit 410 is used to retrieve the medium transmittance spectrum τ output by the environmental medium characteristics inversion module 300. LWIR Atmospheric compensation is performed on the original thermal radiation image acquired by the passive focusing plane polarization thermal imaging unit 120 to invert the true surface temperature of the target. Since the dense smoke in the fire field absorbs the infrared radiation emitted by the target and superimposes its own path radiation, the directly observed thermal image cannot accurately reflect the target temperature.

[0089] The thermal radiation inversion and temperature correction unit 410 first establishes a physical mapping model between the target's true radiance and the observed signal based on the inverse operation logic of Planck's radiation law. The thermal radiation inversion and temperature correction unit 410 then calculates the target radiance after atmospheric correction, using the following formula:

[0090]

[0091] In the formula, L target (x,y) represents the true restored radiance of the target at position (x,y); L sensor (x,y) represents the actual radiance value received by the thermal imaging sensor, which is obtained by converting the original grayscale value through a pre-stored detector response curve; τ LWIR (x,y) represents the long-wave infrared transmittance of the corresponding pixel point, which is input in real time by the environmental medium characteristic inversion module 300; L path The path radiance of the smoke medium is estimated from the average radiance value taken from the dark pixel region of the image; ∈obj The surface emissivity of the target is set to the average emissivity constant of typical materials in a fire scene (e.g., 0.95) preset by the system.

[0092] Subsequently, the thermal radiation inversion and temperature correction unit 410 calculates L target (x,y) converted to Celsius temperature value T real (x,y) and generate a pseudo-color temperature layer based on a preset temperature alarm threshold (e.g., 300℃).

[0093] The multi-dimensional feature fusion and target recognition unit 420 is used to fuse an active short-wave infrared gated image (providing texture details), a corrected thermal imaging image (providing temperature information), and a polarization image (providing contour features) into an augmented reality video stream suitable for human visual observation;

[0094] In order to highlight high-temperature danger areas while preserving scene details, the multi-dimensional feature fusion and target recognition unit 420 adopts channel replacement fusion logic based on the YUV color space.

[0095] The computational logic for the multi-dimensional feature fusion and target recognition unit 420 to generate the final fused image is as follows:

[0096]

[0097] In the formula, Y fused To fuse the luminance component of the image, it mainly carries the texture and contour information of the scene; I SWIREN The enhanced shortwave infrared image grayscale value output by the data preprocessing and registration unit 310; I Pol α represents the grayscale value of the linearly polarized image, used to enhance the edge features of man-made objects (such as door frames and pipes); α is a weighting coefficient stored in a register (range 0.5-0.8), used to adjust the blending ratio of texture and contour; T real For the corrected temperature, U fused The chromaticity component is determined by the corrected temperature T. real Generated using a preset heatmap color mapping function.

[0098] Through the above logic, the target physical parameter calculation module 400 outputs a fused video stream containing high-definition scene textures and accurate temperature color coding. This video stream not only eliminates smoke obscuration but also intuitively marks the high-temperature hazard source. Finally, it is transmitted to the fused display and human-computer interaction module 500 for augmented reality projection through a high-speed video interface.

[0099] The integrated display and human-computer interaction module 500 serves as the system's information output and command input terminal. In terms of hardware composition, it mainly includes a 0.7-inch high-brightness Micro-OLED display microscreen, an arrayed waveguide optical module, a 6-axis MEMS inertial measurement unit (IMU), and a low-power offline voice recognition chip.

[0100] The fusion display and human-computer interaction module 500 receives the fused video stream from the target physical parameter calculation module 400 through a high-speed video interface (such as MIPI-DSI) and projects it into the wearer's field of view.

[0101] The fusion display and human-computer interaction module 500 first performs optical projection of the augmented reality (AR) image. In order to ensure image contrast in high-brightness environments and alleviate visual fatigue caused by near-eye display, the fusion display and human-computer interaction module 500 adopts exit pupil expansion technology based on geometric waveguides.

[0102] Specifically, the fusion display and human-computer interaction module 500 drives the Micro-OLED display microscreen to display the received YUV format fused image, and the emitted light is coupled into the arrayed optical waveguide substrate. The light propagates through total internal reflection inside the waveguide and is finally diffracted through the two-dimensional pupil-expanding grating structure, forming an equivalent virtual imaging plane in front of the wearer's eyes. The focusing depth of this virtual imaging plane is determined by the physical optical path design of the optical module and is fixed at a distance of 2.5 meters from the human eye to match the observation habits of operators in fire search and rescue operations when observing targets at medium and close distances.

[0103] The Fusion Display and Human-Computer Interaction Module 500 is also responsible for performing posture-aware electronic image stabilization. Because firefighters frequently turn their heads in a fire, uncompensated superimposed images can cause visual delay and jitter. The Fusion Display and Human-Computer Interaction Module 500 uses an onboard IMU to acquire the helmet's angular velocity vector in real time. Calculate the change in head pose in the current frame relative to the previous frame.

[0104] The integrated display and human-computer interaction module 500 performs pixel-level displacement compensation on the displayed image based on posture changes. Its compensation logic is as follows:

[0105]

[0106] In the formula, (u comp ,v comp (u) represents the pixel coordinates displayed on the microscreen after compensation; raw ,v raw ) represents the original image coordinates; K is a pre-stored pixel-angle mapping matrix, which is obtained through factory calibration and used to convert physical angle changes into screen pixel displacements; The three-axis angular velocity data output by the IMU; Δt frame This refers to the frame refresh cycle of the display system.

[0107] Through this compensation logic, the integrated display and human-computer interaction module 500 ensures that the virtual heat map and the real fire scene objects maintain relative spatial positioning when the wearer turns their head quickly.

[0108] In addition, the integrated display and human-computer interaction module 500 integrates an interactive control unit for receiving control commands from firefighters in scenarios where manual operation is not possible. This unit includes an offline voice recognition circuit and an eye-tracking sensor.

[0109] The offline voice recognition circuit continuously monitors the ambient audio. When it matches a preset command term, such as activating the fog-penetrating mode or switching the temperature measurement point, it generates a corresponding digital control signal and sends it to the timing synchronization and scanning control module 200 via the I2C or UART control bus to trigger the system's imaging mode switching.

[0110] The eye-tracking sensor uses an inner infrared camera to capture the wearer's pupil position and calculates the gaze coordinates using the Purkinjet spot reflection principle. When the gaze coordinates fall within the valid area of ​​the virtual menu icon and the continuous dwell time exceeds a confirmation threshold (e.g., 1.2 seconds), the interaction control unit determines it as a valid click and sends the corresponding menu execution command to the system.

[0111] Through the aforementioned hardware and logic, the integrated display and human-computer interaction module 500 enables the intuitive and stable overlay of physically enhanced multi-dimensional fire scene information onto the real scene, and provides a non-contact control method adapted to extreme operating environments.

[0112] See attached document Figure 2 Based on the aforementioned AR emergency fire helmet system integrating multi-sensor image fusion, this invention also provides a method for fire scene environment perception and enhanced display control. This method, through an embedded main control unit, schedules the collaborative work of various functional modules, specifically including the following steps:

[0113] Step S100: Multidimensional Time Series Data Acquisition

[0114] The timing synchronization and scanning control module 200 generates a high-precision timing trigger signal. First, the timing synchronization and scanning control module 200 sends a laser emission trigger pulse to the active and passive multidimensional imaging module 100 to control the laser illumination unit to emit detection pulses;

[0115] Subsequently, the timing synchronization and scanning control module 200, according to a preset distance gating strategy, opens the electronic shutter of the gating imaging unit after a preset delay time to acquire an active short-wave infrared gating slice at a specific depth. Simultaneously, the timing synchronization and scanning control module 200 sends a synchronization frame trigger signal to drive the focal plane polarization thermal imaging unit to acquire a passive long-wave infrared thermal image and a linear polarization image within the same field of view.

[0116] Step S200: Image preprocessing and spatial registration

[0117] The data preprocessing and registration unit 310 in the environmental medium property inversion module 300 receives the multidimensional raw image data output from step S100. First, the data preprocessing and registration unit 310 uses an improved adaptive bilateral filtering algorithm to perform coherent speckle noise suppression on the active shortwave infrared gating slice.

[0118] Subsequently, based on the denoised image data, the data preprocessing and registration unit 310 calls the pre-stored homography transformation matrix and, according to the geometric correction calculation logic described in the aforementioned embodiment, maps the active image to the coordinate system of the passive thermal imaging to generate image data with strictly aligned spatial dimensions.

[0119] Step S300: Environmental medium parameter inversion

[0120] The medium parameter inversion unit 320 in the environmental medium property inversion module 300, based on the aligned image data output in step S200, selects two gated slices of different depths and calculates the average extinction coefficient of the fire smoke according to the extinction coefficient calculation formula described in the aforementioned embodiment. The medium parameter inversion unit 320, combining the current target physical distance and pre-stored band conversion coefficients, constructs a long-wave infrared transmittance spectrum for correcting thermal radiation intensity according to the aforementioned transmittance calculation logic.

[0121] Step S400: Target physical parameter calculation and image fusion

[0122] The target physical parameter calculation module 400 uses the long-wave infrared transmittance spectrum generated in step S300 and, according to the radiance inversion formula described in the aforementioned embodiment, performs atmospheric transmission correction on the original thermal radiation image that has been aligned in step S200, eliminates the absorption and path radiation interference of the smoke medium, and inverts to obtain the true surface temperature of the target.

[0123] Subsequently, the target physical parameter calculation module 400 uses color space channel replacement logic to map short-wave infrared texture information into luminance components and real surface temperature information into chromaticity components, generating a fused video stream containing high-frequency details and accurate temperature field information.

[0124] Step S500: Augmented Reality Display and Interactive Control

[0125] The fusion display and human-computer interaction module 500 receives the fused video stream output in step S400 and projects it onto a virtual imaging plane in front of the wearer through an arrayed waveguide optical module. During the display process, the fusion display and human-computer interaction module 500 reads the angular velocity data output by the inertial measurement unit in real time, and performs electronic image stabilization compensation on the image according to the pixel-level displacement compensation formula described in the aforementioned embodiment, combined with the pre-stored pixel-angle mapping matrix and the frame refresh cycle of the display system.

[0126] Meanwhile, the integrated display and human-computer interaction module 500 continuously performs interactive monitoring. When it recognizes a preset voice command or confirms a valid eye-tracking gaze operation, it generates a corresponding digital control signal and feeds it back to the timing synchronization and scanning control module 200, thereby triggering the adjustment of the system working mode or scanning parameters in a closed loop.

Claims

1. An AR emergency firefighting helmet system integrated with multi-sensor image fusion, characterized in that, The application relates to a multi-dimensional imaging system for fire scene observation. The system comprises: a multi-dimensional active-passive imaging module, which synchronously collects active gated depth image data and passive polarized thermal radiation image data of a fire scene environment, and constructs a multi-dimensional light field original data source containing spatial depth information, multi-dimensional polarization characteristics and thermal radiation characteristics; a time sequence synchronization and scanning control module, which drives the multi-dimensional active-passive imaging module to perform depth scanning, and synchronously generates a zoom driving signal according to depth information fed back by scanning, so as to establish an alignment reference of an imaging collection plane and a display virtual image plane in a physical space; an environmental medium characteristic inversion module, which analyzes polarization scattering characteristics in the multi-dimensional light field original data source, and inversely obtains a dynamic extinction coefficient representing fire scene medium attenuation characteristics; a target physical parameter solving module, which uses the dynamic extinction coefficient to perform medium transmission compensation on the multi-dimensional light field original data source, solves real surface reflectivity and real recovered radiation brightness of a target, and generates enhanced target data; 2. The integrated multi-sensor image fusion AR emergency firefighting helmet system of claim 1, wherein, a fusion display and human-computer interaction module, which adjusts a projection focal length in response to the zoom driving signal, and projects a fusion image of the enhanced target data to an optimal observation plane matched with the depth scanning. The multi-dimensional active-passive imaging module specifically comprises: an active short-wave infrared gated imaging unit and a passive split-focus plane polarized thermal imaging unit, which have parallel visual axes; the active short-wave infrared gated imaging unit intercepts smoke backscattering echoes corresponding to a spatial depth of a gated delay time sequence by controlling a delay time sequence between laser pulses and an electronic shutter; 3. The integrated multi-sensor image fusion AR firefighting helmet system of claim 2, wherein, the passive split-focus plane polarized thermal imaging unit synchronously captures multi-angle linear polarization thermal radiation signals by using a micro-polarization array at a pixel level.

4. The integrated multi-sensor image fusion AR emergency firefighting helmet system of claim 1, wherein, The passive split-focus plane polarized thermal imaging unit is configured with linear polarization degree solving logic, which is used for separating Stokes vector components from original thermal radiation signals, and quantifying linear polarization degrees of each pixel point by using a ratio of an arithmetic square root of a square sum of polarization component differences in the Stokes vector components to a total radiation intensity component. The time sequence synchronization and scanning control module is configured to perform depth search and locking logic: in a search stage, the delay time between laser emission time and shutter opening time is linearly increased by a preset time step between adjacent frame periods, and an imaging field of view is driven to move from a near field to a far field layer by layer; 5. The integrated multi-sensor image fusion AR firefighting helmet system of claim 1, wherein, in a locking stage, the delay time is fixed at an optimal delay time parameter in response to a received optimal observation layer instruction, and is used for maintaining continuous observation on a target plane.

6. The integrated multi-sensor image fusion AR firefighting helmet system of claim 1, wherein, The time sequence synchronization and scanning control module is also configured to perform object-image conjugate zooming logic, and calculates a corresponding liquid lens driving voltage according to a current imaging detection distance, so that a refractive power change caused by the driving voltage and a change of the imaging detection distance maintain an optical conjugate relationship, and ensure that a display virtual image is always focused on an imaging detection surface. The environmental medium characteristic inversion module is configured to perform data registration logic, and calls a previously calibrated homography transformation matrix to map pixel coordinates of the active gated depth image data to a coordinate system of the passive polarized thermal radiation image data, and generates image sequences with spatial dimensions aligned.

7. The integrated multi-sensor image fusion AR emergency firefighting helmet system of claim 1, wherein, The environmental medium characteristic inversion module is configured to perform a double-slice-based medium parameter inversion logic, select two gating slices of different detection distances, calculate an average extinction coefficient of light waves on a medium transmission path according to a ratio of smoke echo intensities at the two detection distances and a difference between the two detection distances, and combine a distance square inverse ratio attenuation law.

8. The integrated multi-sensor image fusion AR emergency firefighting helmet system of claim 1, wherein, The target physical parameter solving module is configured to perform a thermal radiation transmission recovery logic, construct a long-wave infrared transmittance spectrum by using the dynamic extinction coefficient, and deduct path radiation and absorption loss caused by the medium from total radiation brightness received by the sensor, to restore real recovered radiation brightness of the target surface.

9. The integrated multi-sensor image fusion AR emergency firefighting helmet system of claim 1, wherein, The target physical parameter solving module is configured to perform a channel replacement fusion logic, perform weighted summation on enhanced short-wave infrared texture information and linear polarization degree feature information to construct a luminance component of an image, and construct a chrominance component of the image by mapping thermal force map colors of temperature data converted from the real recovered radiation brightness.

10. The integrated multi-sensor image fusion AR emergency firefighting helmet system of claim 1, wherein, The fusion display and human-computer interaction module includes an inertial measurement unit; The fusion display and human-computer interaction module is configured to collect an angular velocity vector of the helmet device in real time by using the inertial measurement unit, calculate an interframe line-of-sight offset by using a pre-stored pixel and angle mapping relationship, and perform reverse displacement compensation on image coordinates displayed.

Images