Fire extinguishing unmanned aerial vehicle fire feature image recognition method and system

Abstract

The application provides a fire extinguishing unmanned aerial vehicle fire feature image recognition method and system, the method comprising: emitting a specific light signal to the surface of the optical component of the unmanned aerial vehicle camera and the front fire scene environment; receiving the light signal change after the specific light signal penetrates the surface attachments of the optical component and the front fire scene environment; analyzing and generating degradation characteristic information containing the surface attachment characteristics of the optical component and the optical characteristics of the front fire scene environment according to the light signal change; and performing reverse correction on the original image data collected by the image sensor according to the degradation characteristic information to restore the definition, contrast and detail information of the image. The application can actively emit and receive specific light signals to obtain the degradation characteristic information of the fire scene environment and the surface attachments of the optical component, and accordingly perform reverse correction on the original image, thereby effectively solving the problem of serious image degradation and difficult recognition in a complex fire scene environment.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and more specifically, to a method and system for recognizing fire feature images from firefighting drones. Background Technology

[0002] In modern emergency rescue, firefighting drones play a crucial role in fire reconnaissance and firefighting assistance. However, in complex fire environments, especially when fires occur in specialized industrial facilities and are accompanied by ground firefighting operations, structural collapses, and severe airflow disturbances, images acquired by drone cameras often suffer severe degradation, becoming blurry and unclear. This leads to high misjudgment rates and response delays in image recognition systems, seriously impacting rescue efficiency and the safety of firefighters. For example, industrial fires involving large quantities of plastic products and chemical packaging materials produce extremely concentrated, densely packed black smoke that effectively blocks visible light and infrared radiation. Furthermore, high-pressure water mist and foam extinguishing agents generated by ground firefighting operations, along with the drone's own movement through airflow, cause a moist mixture carrying smoke and dust to adhere to the surface of the drone's camera's optical components, forming an irregular layer of water droplets, foam spots, and smoke particles. After the moisture evaporates at high temperatures, these deposits leave behind a layer of stains with specific light scattering and absorption characteristics, resulting in localized image blurring, severely reduced contrast, and irregular light spots or streaks, further complicating the image degradation problem. Summary of the Invention

[0003] This application discloses a method and system for fire feature image recognition from firefighting drones, aiming to solve the problem that in complex fire scene environments, the images acquired by drone cameras are severely degraded and the information is blurry, resulting in a high misjudgment rate and response delay in the recognition system, which seriously affects rescue efficiency and the safety of firefighters.

[0004] In a first aspect, this application discloses a method for recognizing fire feature images from firefighting drones, comprising the following steps: Specific light signals are emitted onto the surface of the optical components of the drone's camera and the fire scene environment in front of it; Receive the changes in the light signal after a specific light signal penetrates the surface of the optical component and the fire scene environment ahead; Based on changes in optical signals, analyze and generate degradation characteristic information that includes the characteristics of surface deposits on optical components and the optical characteristics of the fire scene environment ahead; Based on the degradation characteristics information, the raw image data acquired by the image sensor is reverse-corrected to restore the image's sharpness, contrast, and detail.

[0005] Secondly, this application also discloses a fire feature image recognition system for firefighting drones, the system comprising: The optical signal transmitting module is used to transmit specific optical signals to the surface of the optical components of the drone camera and to the fire scene environment in front of it. The optical signal receiving module is used to receive the changes in optical signals after they penetrate the surface of the optical components and the fire scene environment in front of them. The degradation characteristic information generation module is used to analyze and generate degradation characteristic information, including the characteristics of the surface deposits of optical components and the optical characteristics of the fire scene environment, based on changes in the optical signal. The image correction module is used to reverse-correct the original image data acquired by the image sensor based on the degradation characteristic information, in order to restore the image's sharpness, contrast, and detail.

[0006] Compared with the prior art, this application has at least the following beneficial effects: This application can obtain information on the degradation characteristics of the fire environment and the surface of optical components by actively transmitting and receiving specific light signals, and then reverse correct the original image accordingly, thereby effectively solving the problem of severe image degradation and difficulty in recognition under complex fire environment, significantly improving the clarity, contrast and detail information of the image, and providing more reliable visual information for fire rescue. Attached Figure Description

[0007] Figure 1 A flowchart illustrating a fire feature image recognition method for a fire-fighting drone provided in this application.

[0008] Figure 2 This is a schematic diagram of the structure of a fire feature image recognition system for firefighting drones provided in this application. Detailed Implementation

[0009] The technical solutions in this application will now be clearly and completely described in conjunction with the accompanying drawings.

[0010] This application proposes a method for fire feature image recognition using firefighting drones, such as... Figure 1 As shown, it includes the following steps: Specific light signals are emitted onto the surface of the optical components of the drone's camera and the fire scene environment in front of it; Receive the changes in the light signal after a specific light signal penetrates the surface of the optical component and the fire scene environment ahead; Based on changes in optical signals, analyze and generate degradation characteristic information that includes the characteristics of surface deposits on optical components and the optical characteristics of the fire scene environment ahead; Based on the degradation characteristics information, the raw image data acquired by the image sensor is reverse-corrected to restore the image's sharpness, contrast, and detail.

[0011] This application, by emitting specific light signals and analyzing their changes, can accurately identify the degradation effects of surface deposits on optical components and the fire environment on images, and accordingly perform reverse correction on the original image data, thereby effectively restoring the image's clarity, contrast, and detail information, significantly improving the accuracy and reliability of UAV image recognition in complex fire environments.

[0012] In order to better understand the technical solutions proposed in this application, it is necessary to explain some key terms involved therein.

[0013] A specific optical signal refers to a light wave with a specific wavelength, frequency, polarization state, or modulation mode. Its selection is aimed at maximizing penetration of smoke, water mist, foam, and adhering substances on the surface of optical components in a fire environment, and enabling effective detection to obtain information on degradation. For example, lasers, LED beams, or broadband light sources in specific wavelength bands can be selected.

[0014] Optical component surface contaminants refer to various substances adhering to the surface of drone camera optical components (such as lenses and protective covers), including but not limited to water droplets, ice crystals, dust, smoke particles, foam residue, and oil stains. These contaminants can alter the propagation path of light, causing image distortion, blurring, or obstruction.

[0015] The fire scene environment refers to the space in front of the drone camera, between the camera and the target fire features. This area may contain smoke, water vapor, flames, hot air currents, particulate matter, etc., all of which can affect the propagation of light signals.

[0016] Optical signal variation refers to the changes in the physical properties of a specific optical signal, such as intensity, phase, polarization state, and spectral composition, after it penetrates the surface of an optical component and the fire environment ahead. These changes carry information about the properties of the medium.

[0017] Degradation characteristics information refers to a set of parameters obtained by analyzing changes in optical signals, describing the impact of surface deposits on optical components and the surrounding fire environment on image quality. This information may include the geometry, optical density, refractive index, and scattering coefficient of the deposits, as well as the smoke concentration, particle size distribution, and turbulence intensity of the fire environment.

[0018] Reverse correction refers to the reverse processing of the original image data acquired by the image sensor based on the degradation characteristics information, in order to eliminate or reduce the impact of degradation factors on image quality, thereby restoring the original sharpness, contrast and detail information of the image.

[0019] The core of the fire feature image recognition method for fire-fighting drones proposed in this application lies in overcoming the impact of complex fire scene environments on image quality through active detection and intelligent correction.

[0020] First, regarding the characteristic of emitting specific light signals onto the surface of the drone's camera's optical components and the fire scene environment ahead.

[0021] As one implementation method, a laser diode array can be used to emit one or more laser beams of specific wavelengths onto the surface of an optical component. These laser beams can illuminate the surface of the optical component in a scanning or fixed mode to detect the presence and characteristics of deposits. For example, a near-infrared laser with a wavelength of 980 nanometers can be emitted, which has good penetrability to water droplets and certain dust particles.

[0022] As another implementation, a broadband LED light source can be used to emit a modulated broadband light signal towards the fire environment. This light signal can cover the visible and near-infrared bands to obtain more comprehensive information on the environmental optical characteristics. For example, a broadband light signal that varies continuously in the range of 400 nanometers to 1000 nanometers can be emitted, and smoke concentration and particulate matter distribution can be assessed by analyzing the attenuation of light at different wavelengths.

[0023] As another implementation, a polarized light emitter can be used to emit light signals with a specific polarization state onto the surface of the optical component and into the fire environment ahead. By analyzing the changes in the polarization state of the reflected or transmitted light, microstructural information about the adhering material and the surrounding medium can be obtained. For example, linearly polarized light can be emitted, and its ellipticity change after penetrating the medium can be analyzed.

[0024] Secondly, regarding the characteristic of the change in light signal after receiving a specific light signal through the surface of the optical component and the fire scene environment ahead.

[0025] In one implementation, a photodiode array can be used to receive specific light signals reflected from the surface of an optical component. By measuring the intensity of the reflected light, it can be determined whether there are any deposits on the surface of the optical component, and the extent of their coverage. For example, when the surface of the optical component is covered with water droplets, the intensity of the reflected light will change significantly.

[0026] As another implementation method, a spectrometer can be used to receive specific light signals after they penetrate the fire environment. The spectrometer can analyze the changes in the spectral composition of the received light signals, thereby obtaining the absorption and scattering characteristics of media such as smoke and water vapor in the fire environment. For example, by analyzing the intensity of specific absorption peaks, the concentration of a certain chemical substance in the smoke can be estimated.

[0027] As another implementation method, a polarization-sensitive detector can be used to receive polarized light signals after they have penetrated the medium. By analyzing the changes in the polarization state of the received light signal, the scattering characteristics and microstructure information of the medium can be obtained. For example, when a light signal penetrates smoke composed of irregular particles, its degree of polarization will change.

[0028] Secondly, regarding the feature of analyzing and generating degradation characteristic information based on changes in optical signals, including the characteristics of surface deposits on optical components and the optical characteristics of the fire scene environment ahead.

[0029] As one implementation method, the coverage and optical density of the surface deposits on the optical component can be estimated using a preset lookup table or empirical model based on changes in the received reflected light intensity. For example, a 20% decrease in reflected light intensity may correspond to a 50% surface coverage.

[0030] As another implementation method, parameters such as the particulate size distribution, optical density, and water vapor concentration of smoke in the fire scene can be calculated using Mie scattering theory or Beer-Lambert's law, based on the spectral absorption and scattering curves obtained from spectrometer analysis. For example, the visibility of smoke can be estimated by analyzing the attenuation in the visible light band.

[0031] As another implementation method, multi-source data such as reflected light intensity, spectral changes, and polarization state changes can be combined and fused using machine learning algorithms (such as support vector machines or neural networks) to generate more comprehensive and accurate information on degradation characteristics. For example, a model can be trained to directly output the type of adhering material (water droplets, soot) and the visibility level of the fire environment based on multi-source light signal variation data.

[0032] Finally, regarding the feature of reverse correction of the raw image data acquired by the image sensor based on the degradation characteristics information in order to restore the image's sharpness, contrast, and detail information.

[0033] As one implementation, when degradation characteristics indicate the presence of water droplets or stains on the surface of optical components, an optical distortion model can be constructed based on the geometry and optical properties of the deposits. This model is then used to perform geometric correction and deblurring on the original image data. For example, if the water droplets create a convex lens effect, corresponding inverse distortion correction is performed on the image.

[0034] As another implementation, when degradation characteristics indicate the presence of dense smoke in the fire area ahead, the original image data can be dehazed using an atmospheric scattering model (such as the dark channel prior model by He et al.) based on the optical density and scattering characteristics of the smoke, in order to restore the image's contrast and color information. For example, pixel-level brightness compensation can be performed on the image by estimating the smoke transmittance map.

[0035] As another implementation method, a multi-scale image fusion algorithm can be used, combining information on the degrading characteristics of deposits and the environment. For example, local adaptive filtering can be used to process local blurring caused by deposits, while a global contrast enhancement algorithm can be used to address the overall contrast reduction caused by smoke. Finally, the different processing results are fused to achieve comprehensive image restoration.

[0036] The fire feature image recognition method for firefighting drones proposed in this application works by systematically solving the image degradation problem in complex fire scene environments through active detection and intelligent correction. First, the drone emits specific light signals to the surface of its camera's optical components and the fire scene environment ahead. These light signals undergo physical changes as they penetrate the surface of the optical components (e.g., water droplets, stains) and the fire scene environment (e.g., dense smoke, water mist). The drone then receives these changed light signals. Through in-depth analysis of the received light signal changes, degradation characteristic information can be accurately generated, including the characteristics of the surface deposits on the optical components (e.g., geometry, optical density) and the optical characteristics of the fire scene environment (e.g., smoke concentration, particulate matter distribution). This degradation characteristic information forms the basis for quantifying the degree and type of image degradation. Finally, based on this detailed degradation characteristic information, targeted reverse correction is performed on the raw image data acquired by the image sensor. For example, if the degradation information indicates geometric distortion caused by water droplets, geometric remapping is performed; if it indicates a decrease in contrast caused by dense smoke, defogging and contrast enhancement are performed. Through this series of collaborative steps, this method can effectively restore the sharpness, contrast, and detail of images, thereby providing high-quality image data for the accurate identification of fire features.

[0037] This application offers significant advantages over existing technologies: Current image recognition methods for firefighting drones primarily rely on passively receiving ambient light and then using image processing algorithms to correct degraded images. However, this approach often struggles to accurately identify the cause of degradation in complex and variable fire environments, especially when there are deposits on the surfaces of optical components, leading to poor correction results and potentially introducing new artifacts. For example, when image blurring is caused by water droplets on the lens surface, traditional dehazing algorithms may not effectively address the geometric distortion problem.

[0038] The core of this application lies in the introduction of an active detection mechanism. By emitting specific light signals onto the surface of optical components and the fire environment ahead, and analyzing the changes after penetration, information on the physical characteristics that cause image degradation can be directly and quantitatively obtained. For example, by analyzing the changes in reflection or transmission of specific light signals after penetrating the surface of the optical components containing deposits, the type, thickness, and geometry of the deposits can be directly determined without relying solely on guesswork based on the blurriness of the image itself. This active detection method makes the acquisition of degradation characteristic information more accurate and comprehensive.

[0039] Furthermore, this application performs reverse correction on the original image data based on these precise degradation characteristic information. This correction is not blind image enhancement, but a targeted processing based on a deep understanding of the causes of degradation. For example, if the degradation characteristic information indicates the presence of high-concentration smoke in the fire scene, a defogging algorithm based on a smoke optical model can be used; if it indicates the presence of irregular water droplets on the surface of optical components, a distortion correction algorithm based on a droplet geometric model can be used. This targeted correction method significantly improves the accuracy and efficiency of image restoration, avoiding the problems of over-correction or under-correction that may occur in traditional methods.

[0040] Through the above-described solution, this application effectively overcomes the challenge of image degradation in complex fire environments, significantly improving the image recognition capabilities and decision support level of firefighting drones in fire reconnaissance and auxiliary firefighting missions. For example, in industrial fires, even if the drone's camera is covered by a mixture of water mist and smoke, this method can still recover clear images of the fire source and structural details through active detection and intelligent correction, thereby providing firefighters with crucial visual information and ensuring the smooth progress of rescue operations.

[0041] In real-world fire environments, highly reflective particles, such as metal dust, incompletely burned carbon particles, or special chemical substances, may be present on the surfaces of optical components or in the smoke. These particles produce strong specular reflection or complex scattering, making it difficult for traditional methods to accurately assess their degrading impact on image quality, resulting in inaccurate generated degradation characteristic information. If these problems are not addressed, subsequent image remediation may not fully restore the image's sharpness, contrast, and detail, affecting the accurate identification of fire features. To address this, this application proposes a more refined method for generating degradation characteristic information. By detecting the absorption intensity of a specific infrared band and analyzing the scattering pattern of the light signal, the contribution of highly reflective particles is identified and quantified, thereby generating corrected degradation characteristic information.

[0042] In this regard, this application further proposes the following steps for analyzing and generating degradation characteristic information, including the characteristics of surface deposits on optical components and the optical characteristics of the fire scene environment, based on changes in the optical signal: Detecting the absorption intensity in specific infrared bands to determine the presence of chemicals associated with highly reflective particles; Analyze the scattering patterns of optical signals in the visible and near-infrared bands to identify anomalous scattering phenomena; When the absorption intensity of a specific infrared band exceeds a preset threshold and abnormal scattering is detected, the presence of highly reflective particles is confirmed. Based on the absorption intensity of a specific infrared band, the optical density of smoke, particle size distribution, and the contribution of highly reflective particles in the surface of optical components obtained from the analysis of changes in optical signals are corrected to generate corrected degradation characteristic information.

[0043] Specifically, detecting the absorption intensity of a specific infrared band refers to using spectral analysis techniques to detect the degree to which infrared light within a specific wavelength range is absorbed. Certain highly reflective particles, such as specific metal oxides or carbon-based particles, possess unique absorption peaks in specific bands of the infrared spectrum. By monitoring the intensity of these absorption peaks, the presence and concentration of these particles can be determined. The aim is to provide evidence at the chemical composition level to aid in the identification of highly reflective particles. Analyzing the scattering patterns of light signals in the visible and near-infrared bands can be understood as analyzing the changes in the propagation direction of incident light after penetrating the medium. Highly reflective particles typically produce scattering patterns different from ordinary smoke or water mist, such as stronger forward or backscattering, or unusual scattering peaks at specific angles. Through detailed analysis of these scattering patterns, anomalous scattering phenomena caused by highly reflective particles can be identified. The aim is to provide evidence at the physical optics level to aid in the identification of highly reflective particles. In practical applications, when the absorption intensity in a specific infrared band exceeds a preset threshold and anomalous scattering phenomena are identified, the presence of highly reflective particles can be confirmed. The preset threshold, set based on experimental data or experience, distinguishes between normal environmental scattering and significant absorption or scattering caused by highly reflective particles. Only when both independent pieces of evidence (chemical absorption and physical scattering anomalies) simultaneously meet the conditions can the presence of highly reflective particles be reliably confirmed, avoiding misjudgment. Furthermore, based on the absorption intensity of a specific infrared band, the contribution of highly reflective particles in smoke optical density, particulate size distribution, and surface deposits of optical components obtained from optical signal variation analysis is corrected to generate corrected degradation characteristic information. This means that once the presence of highly reflective particles is confirmed, their absorption and scattering effects on the light signal will be quantified and separated from the overall degradation characteristic information or modeled separately. For example, the contribution of highly reflective particles can be calculated separately and then weighted and fused or independently corrected with traditional degradation parameters such as smoke optical density and particulate size distribution, thereby enabling the final generated degradation characteristic information to more accurately reflect the true degradation situation of the fire environment and the surface of optical components.

[0044] This application's solution addresses the problem of inaccurate generation of degradation characteristic information in complex fire environments using traditional methods by introducing a specialized identification and quantification mechanism for highly reflective particles. Specifically, firstly, by detecting the absorption intensity in a specific infrared band, substances associated with highly reflective particles can be identified at the chemical composition level, providing direct evidence for particle type determination. Secondly, by analyzing the scattering patterns of light signals in the visible and near-infrared bands, the unique scattering characteristics caused by highly reflective particles can be captured at the physical optics level, which differs significantly from the scattering behavior of ordinary smoke or water mist. It is precisely because this approach combines two independent detection methods—chemical absorption and physical scattering—that it can more reliably and accurately confirm the presence of highly reflective particles, avoiding potential misjudgments from single detection methods. Furthermore, once the presence of highly reflective particles is confirmed, their unique influence on the light signal (absorption and scattering) is quantified and used to correct the overall degradation characteristic information. This correction mechanism ensures that traditional degradation parameters such as smoke optical density and particulate size distribution are not confused by the effects of highly reflective particles, thus enabling the final generated degradation characteristic information to more accurately reflect the true optical characteristics of the fire environment and the surface of optical components, providing a more reliable input for subsequent image reverse correction.

[0045] Through the above technical solution, this application effectively solves the problem of inaccurate generation of degradation characteristic information in the presence of highly reflective particles using traditional methods. Specifically, by combining absorption intensity detection in a specific infrared band with visible / near-infrared scattering mode analysis, this solution can identify and confirm the presence of highly reflective particles in the fire environment or on the surface of optical components with higher accuracy and reliability. This dual verification mechanism significantly reduces the false positive rate and ensures accurate identification of complex degradation sources. More importantly, once highly reflective particles are identified, their unique contribution to the optical signal is quantified and used to correct the overall degradation characteristic information, thereby enabling the generated degradation characteristic information to more comprehensively and accurately reflect the actual degradation situation. Compared with the basic solution, this application has the advantage of being able to more finely characterize the optical environment of the fire scene, providing a more accurate basis for subsequent image reverse correction, and thus significantly improving the recognition accuracy and reliability of fire-fighting drones for fire feature images in complex fire environments.

[0046] In real-world fire environments, the optical components of drone cameras may be affected by a combination of factors, including high temperature, high humidity, smoke particles, and fire extinguishing agent spraying, leading to the formation of complex composite droplet deposits. The geometry, internal structure, and refractive index distribution of these composite droplets dynamically change, causing geometric distortion and alterations in local optical properties of the image. This makes it difficult to accurately correct for image degradation based solely on analysis of changes in light signals, resulting in incomplete recovery of image sharpness, contrast, and detail.

[0047] In response, this application further proposes the following steps for reverse correction of the original image data acquired by the image sensor based on degradation characteristic information to restore the image's sharpness, contrast, and detail information: Collect local temperature data on the surface of optical components; Collect local humidity data on the surface of optical components; Collect micro-airflow velocity data on the surface of optical components; Collect micro-airflow direction data on the surface of optical components; Collect high-frequency micro-vibration data on the surface of optical components; Collect chemical composition indication signals near optical components; Based on the collected local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and chemical component indication signals, the geometry, internal structure, and refractive index distribution of the composite droplet are predicted. Based on the predicted geometry, internal structure, and refractive index distribution of the composite droplet, predictive geometric distortion information is generated. Based on predictive geometric distortion information, the original image data is geometrically remapped to restore the image's sharpness, contrast, and detail. Based on the predicted internal structure and refractive index distribution of the composite droplets, the local optical properties of the image are adjusted to restore the image's sharpness, contrast, and detail.

[0048] Specifically, to accurately capture the dynamic characteristics of composite droplets on the surface of optical components, multi-dimensional environmental parameters need to be collected. Local temperature data refers to the real-time temperature value of a specific area on the surface of the optical component; its changes affect the evaporation and condensation rates of droplets, as well as surface tension. Local humidity data reflects the water vapor content in the air near the surface of the optical component, directly related to droplet formation and growth. Micro-airflow velocity and direction data describe the intensity and direction of airflow over the surface of the optical component, which is crucial for predicting droplet deformation, movement, and adhesion stability. High-frequency micro-vibration data refers to the high-frequency micro-vibrations on the surface of the optical component caused by drone flight, wind, or internal vibrations; these vibrations may lead to droplet breakage, merging, or detachment. Chemical composition indicator signals are used to detect the presence of specific chemical substances near the optical component, such as fire extinguishing agent residues or combustion products; these substances may alter the surface tension, viscosity, or refractive index of droplets, forming composite droplets.

[0049] Based on the collected local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and chemical component indication signals, a droplet evolution model can be constructed to predict the geometry, internal structure, and refractive index distribution of composite droplets. Geometry refers to the two-dimensional or three-dimensional contour of the droplet on the surface of the optical component, such as a spherical, flat, or irregular shape. Internal structure may involve the distribution of different components within the droplet, such as smoke particles or fire extinguishing agent particles encapsulated within a water droplet. Refractive index distribution describes the refractive ability of different regions within the droplet, directly affecting the path deflection of light after it penetrates the droplet.

[0050] In practical applications, once the geometry, internal structure, and refractive index distribution of the composite droplet are predicted, predictive geometric distortion information can be generated. This information quantifies the light path deflection and image pixel position shift caused by the presence of the droplet. For example, the convex or concave lens effect of the droplet can cause local magnification, reduction, or distortion of the image. Based on this predictive geometric distortion information, the original image data can be geometrically remapped, i.e., the distorted pixels can be repositioned to their true locations through an inverse transformation, thereby restoring the image's sharpness, contrast, and detail. Simultaneously, based on the predicted internal structure and refractive index distribution of the composite droplet, the local optical properties of the image can be adjusted. This includes compensating for the brightness, color saturation, and contrast of local image regions to counteract the effects of the droplet on light absorption, scattering, and reflection, further restoring the image's sharpness, contrast, and detail.

[0051] This application addresses the limitations of traditional methods in handling image degradation caused by complex composite droplets by introducing real-time sensing of the microenvironment parameters on the surface of optical components. Specifically, in a fire environment, composite droplets composed of water, smoke, fire extinguishing agents, etc., easily form on the surface of the optical components of drone cameras. The physical and optical properties of these droplets are dynamically changing, and their degrading effect on the image far exceeds simple occlusion or uniform scattering. Traditional methods rely solely on changes in the overall light signal to generate degrading characteristic information, making it difficult to capture this local, dynamic, and complex degrading mechanism.

[0052] This application, by collecting local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and chemical composition indicator signals, can comprehensively and in real-time acquire key environmental factors affecting the formation, evolution, and optical properties of composite droplets. This multi-dimensional data is used to accurately predict the geometry, internal structure, and refractive index distribution of composite droplets. For example, local temperature and humidity data can predict the droplet's evaporation-condensation equilibrium; micro-airflow velocity and direction data can predict the droplet's deformation and adhesion state; high-frequency micro-vibration data can reveal the droplet's stability; and chemical composition indicator signals can identify the droplet's chemical composition.

[0053] It is precisely because of the ability to accurately predict these dynamic characteristics of composite droplets that subsequent image retouching steps become highly targeted. By generating predictive geometric distortion information, the deflection effect of the droplets on the light path can be accurately reversed, allowing for geometric remapping of the original image data and correcting image distortion and deformation caused by the droplets. Simultaneously, by adjusting the local optical properties of the image, local brightness, contrast, and color distortions caused by the droplets' absorption, scattering, and reflection of light can be compensated. This refined retouching based on physical models and real-time environmental awareness is far more accurate and effective than general retouching based on overall degradation information, thus enabling a more thorough restoration of image sharpness, contrast, and detail.

[0054] Through the above technical solution, this application can significantly improve the accuracy and reliability of image recognition for firefighting drones in complex fire environments. Compared with methods that rely solely on changes in light signals for general inverse correction, this application uses real-time sensing of the micro-environmental parameters on the surface of optical components to accurately predict the geometry, internal structure, and refractive index distribution of composite droplets. This generates more refined and accurate predictive geometric distortion information and allows for targeted adjustments to the local optical properties of the image. This refined correction mechanism effectively solves the limitations of traditional methods in handling the degradation of complex, dynamic images caused by composite droplets, avoiding incomplete correction due to dynamic changes in droplets. Consequently, the image clarity, contrast, and detail information can be more thoroughly restored, enabling the drone to more clearly capture fire features, such as flame spread paths, types of burning materials, and the location of trapped personnel, greatly enhancing fire situational awareness and firefighting efficiency.

[0055] In real-world fire environments, the aforementioned environmental parameters can change rapidly and drastically, causing abrupt changes in the state of the composite droplets. Relying solely on conventional frequency environmental parameter acquisition and prediction models may fail to capture these abrupt changes accurately and in a timely manner, thus affecting the accuracy of predicting the composite droplet characteristics and reducing the effectiveness of image backcorrection. To address this, this application proposes a more dynamic and responsive prediction method. This method aims to monitor the instantaneous changes in environmental parameters in real time and activate a high-frequency measurement and model correction mechanism upon detecting potential abrupt changes, ensuring accurate prediction of the composite droplet characteristics even under complex and variable fire conditions.

[0056] The steps described above for predicting the geometry, internal structure, and refractive index distribution of the composite droplet based on collected local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and chemical component indicator signals include: When collecting local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and chemical component indication signals, monitor the instantaneous rate of change of each environmental parameter. When the instantaneous rate of change of any environmental parameter exceeds the preset dynamic threshold, or when the combined change pattern of multiple environmental parameters does not conform to the preset synergistic effect pattern, a sudden change warning of the droplet state is triggered. After triggering the mutation warning, the high-frequency image sequence acquisition mode is started, and the auxiliary photodetector array is activated at the same time to measure the instantaneous light refraction and reflection patterns of the composite droplet in multiple local areas. Based on the rapid deformation trajectory of the droplet edge in the high-frequency image sequence and the instantaneous light refraction and reflection modes measured by the auxiliary photodetector array, the key parameters in the droplet evolution model are corrected. Based on the revised droplet evolution model, the geometry, internal structure, and refractive index distribution of the composite droplet are re-predicted in the very short time to come.

[0057] Specifically, monitoring the instantaneous rate of change of each environmental parameter refers to continuously or frequently sampling environmental parameters such as local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and chemical component indicator signals, and calculating the ratio of the change in a very short time interval to the time interval. This instantaneous rate of change reflects the dynamic characteristics and trends of the environmental parameter. The preset dynamic threshold can be understood as a critical rate of change value set for different environmental parameters, which can be adjusted according to actual working conditions or historical data. When the rate of change of any parameter exceeds this threshold, it is considered abnormal or drastic. A discrepancy between the combined change pattern of multiple environmental parameters and the preset synergistic effect pattern refers to identifying synergistic change behaviors inconsistent with normal or expected patterns by analyzing the interrelationships or coupling relationships between different environmental parameters. For example, under specific fire conditions, temperature increases are usually accompanied by humidity decreases. If a situation occurs where temperature increases but humidity increases instead of decreasing, it may be identified as a discrepancy in the synergistic effect pattern. Triggering a sudden change warning of droplet state refers to the system issuing a signal when the above conditions are met, indicating that the state of the composite droplet may be undergoing a rapid and nonlinear change, requiring more refined observation and prediction measures.

[0058] Furthermore, after triggering the mutation warning, the high-frequency image sequence acquisition mode is activated. This refers to the drone camera or specialized imaging equipment continuously capturing images at a frame rate far exceeding the conventional rate to obtain the continuous deformation process of the droplet within a very short time. Simultaneously, the auxiliary photodetector array is activated, which involves activating a miniature photodetector array deployed near the surface of the optical components. These sensors can directly measure the instantaneous refraction and reflection patterns of light passing through or reflecting from the composite droplet, thus providing direct evidence of the droplet's internal structure and optical properties. Based on the rapid deformation trajectory of the droplet edge in the high-frequency image sequence, image processing algorithms are used to accurately track the dynamic changes of the droplet boundary in consecutive image frames, quantifying its deformation speed and direction. The instantaneous light refraction and reflection patterns measured by the auxiliary photodetector array provide physical quantitative data on the droplet's influence on light. This data is used to correct key parameters in the droplet evolution model, such as the surface tension coefficient, evaporation rate, viscosity, or refractive index coefficient as a function of temperature, enabling the model to better fit the observed droplet behavior. Therefore, based on the modified droplet evolution model, the geometry, internal structure, and refractive index distribution of the composite droplet are re-predicted within a very short time in the future (note: this very short time can be preset as needed, for example, 0.1 seconds, i.e., a time threshold can be set), ensuring the real-time performance and accuracy of the prediction results.

[0059] This application's solution, by introducing monitoring of the instantaneous rate of change of environmental parameters, enables real-time perception of the dynamics of the fire environment. When a drastic change in a single environmental parameter or an abnormal coordinated change in multiple environmental parameters is detected, the system immediately triggers a sudden change warning for the droplet state. This indicates that the droplet state may undergo a rapid, nonlinear transition, and conventional prediction models may fail. To address this sudden change, the system no longer relies solely on indirect environmental parameters but instead initiates a high-frequency image sequence acquisition mode and activates an auxiliary photodetector array. The high-frequency image sequence directly captures the macroscopic deformation of the droplet, while the auxiliary photodetector array directly measures the microscopic optical response of the droplet. These direct, high-frequency measurement data are used to correct key parameters in the droplet evolution model, enabling the model to adaptively adjust based on real-time observations, thereby more accurately reflecting the current physical state and optical properties of the droplet. Finally, based on the corrected droplet evolution model, the geometry, internal structure, and refractive index distribution of the composite droplet are re-predicted within a very short timeframe in the future, ensuring high-precision prediction results even when the droplet state changes rapidly, providing a reliable foundation for subsequent image back-correction.

[0060] Through the above technical solution, this application can significantly improve the accuracy and real-time performance of predicting the characteristics of composite droplets on the surface of UAV camera optical components in dynamic and complex fire environments. Traditional prediction methods may lag or produce errors due to rapid changes in environmental parameters, resulting in poor image correction effects. However, this application, by introducing monitoring of the instantaneous rate of change of environmental parameters and identification of synergistic interaction modes, can promptly warn of abrupt changes in droplet state. Furthermore, after the abrupt change warning is triggered, real-time, high-precision data on droplet deformation and optical properties are obtained by initiating high-frequency image sequence acquisition and direct measurement using an auxiliary photodetector array. This data is used to dynamically correct key parameters in the droplet evolution model, enabling the prediction model to adaptively adjust to match the actual droplet state. Thus, this application ensures accurate prediction of the geometry, internal structure, and refractive index distribution of composite droplets even under extreme dynamic conditions, thereby providing a more accurate basis for reverse correction of the raw image data acquired by the image sensor, ultimately significantly restoring the image's clarity, contrast, and detail information, and greatly improving the fire feature image recognition capability of firefighting UAVs in harsh environments.

[0061] In some embodiments, the step of monitoring the instantaneous rate of change of each environmental parameter when collecting local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and chemical component indication signals further includes: Noise suppression and outlier removal are performed on the raw data collected, including local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and chemical component indicator signals.

[0062] Through the above technical solution, this application can significantly improve the accuracy and robustness of monitoring the instantaneous change rate of surface environmental parameters of UAV camera optical components in harsh fire environments. Specifically, by performing refined noise suppression and outlier removal on the raw collected data, misjudgments caused by data contamination are effectively avoided, ensuring the purity of the input data.

[0063] In real-world fire environments, abrupt changes in the geometry, internal structure, and refractive index distribution of composite droplets are often the result of complex synergistic effects of multiple environmental parameters, rather than simple changes in a single parameter or deviations from a preset pattern. Relying solely on preset fixed patterns or single thresholds for judgment may fail to accurately capture these complex and dynamic synergistic effects, leading to insufficient sensitivity or accuracy in abrupt change warnings and affecting the timeliness and effectiveness of subsequent image correction.

[0064] In response, this application further proposes the following steps for triggering a sudden change warning of the droplet state when the instantaneous change rate of any environmental parameter exceeds a preset dynamic threshold, or when the combined change pattern of multiple environmental parameters does not conform to a preset synergistic effect pattern: Acquire local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and instantaneous change rate of chemical component indicator signals; Calculate the degree of correlation between instantaneous rates of change at different time scales; Based on the degree of correlation, identify the strength of synergistic effects between environmental parameters; The strength of the synergistic effect is compared with the preset synergistic effect mode; When the strength of the synergistic effect deviates from the preset synergistic effect mode, and the deviation exceeds the preset allowable range, a sudden change warning of the droplet state is triggered.

[0065] Specifically, acquiring the instantaneous change rate of local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and chemical component indicator signals refers to calculating the rate of change of environmental parameter data over continuous time points after real-time acquisition by sensors and preprocessing (e.g., noise suppression and outlier removal). The instantaneous change rate can be calculated using methods such as differential, moving average, or Kalman filtering to reflect the real-time dynamics of the environmental parameters.

[0066] Furthermore, calculating the correlation between instantaneous change rates at different time scales involves cross-analysis of the instantaneous change rates of the multiple environmental parameters obtained above to quantify their mutual influence and synchronicity. For example, cross-correlation functions, Granger causality analysis, dynamic time warping (DTW), or deep learning-based time series correlation models can be used to assess the correlation or causal relationship of different parameter change trends at short, medium, or long time scales. The aim is to reveal the potential, nonlinear, dynamic coupling relationships between environmental parameters.

[0067] In this context, identifying the synergistic effect strength among environmental parameters, based on the degree of correlation, can be understood as comprehensively evaluating the strength of the combined effect of these parameters on the droplet state based on the calculated correlation between instantaneous change rates using specific algorithms or models (e.g., principal component analysis, factor analysis, support vector machine, or neural network models). The synergistic effect strength is a comprehensive indicator that reflects the potential energy or trend of multiple environmental factors jointly driving abrupt changes in the droplet state.

[0068] In practical applications, comparing the synergistic effect strength with a preset synergistic effect pattern refers to comparing the currently calculated synergistic effect strength with a pre-established synergistic effect pattern representing environmental parameters under normal or stable droplet states. The preset synergistic effect pattern can be obtained through training on historical data, such as typical correlation patterns between the change rates of various environmental parameters under stable fire conditions or non-mutational states. The comparison can be achieved by calculating the distance between the two (e.g., Euclidean distance, Mahalanobis distance) or their similarity (e.g., cosine similarity).

[0069] Therefore, when the intensity of the synergistic effect deviates from the preset synergistic effect mode, and the deviation exceeds the preset allowable range, a sudden change warning for the droplet state is triggered. This means that when the intensity of the synergistic effect of environmental parameters deviates significantly from the normal mode, and this deviation exceeds the acceptable fluctuation range, the system will determine that a sudden change in the droplet state is about to occur or is already occurring, and thus issue a timely warning. The preset allowable range can be dynamically adjusted according to the actual application scenario and the requirements for warning sensitivity.

[0070] This application's solution acquires the instantaneous change rates of multiple environmental parameters and further calculates the correlation between these change rates at different time scales, enabling a more comprehensive and in-depth understanding of the dynamic interactions of various factors in a fire environment. Traditional single-parameter threshold judgments or fixed-pattern comparisons often fail to capture complex and ever-changing synergistic effects. By quantifying this correlation, this application can identify a more refined synergistic intensity between environmental parameters, thus overcoming the limitations of relying solely on preset combination change patterns. When this synergistic intensity deviates significantly from the preset stable pattern, it indicates that the droplet state may be under the combined influence of multiple factors and is about to undergo a sudden change. This judgment mechanism based on synergistic intensity allows the system to more accurately identify the true trend of droplet state changes, avoiding false alarms or missed alarms caused by single-parameter fluctuations or simple pattern discrepancies, thereby providing more reliable early warning information for subsequent image correction.

[0071] Through the aforementioned technical solution, this application can more accurately capture the precursors of abrupt changes in the state of complex droplets in a fire environment. By deeply analyzing the correlation and synergistic effect between the instantaneous change rates of environmental parameters, the system can identify complex dynamic changes that are difficult to detect using traditional methods, significantly improving the accuracy and sensitivity of droplet state change warnings. This not only helps reduce false alarms and missed alarms, but also ensures that high-frequency image sequence acquisition and auxiliary photodetector arrays can be activated in a timely manner at critical moments, thus providing more timely and accurate input data for the correction of droplet evolution models. Ultimately, this improved warning mechanism enables more accurate reverse correction of the raw image data acquired by the image sensor, effectively restoring the image's clarity, contrast, and detail information, greatly enhancing the ability of fire-fighting drones to identify fire feature images in complex fire environments.

[0072] In actual fire environments, the spread of fire, changes in combustibles, and the urgency of the mission are all dynamic. If a fixed allowable range is used, the early warning system may become too sensitive and generate false alarms, or it may fail to issue timely warnings at critical moments due to an excessively large allowable range, thereby affecting the accuracy and timeliness of fire-fighting drones in recognizing fire feature images.

[0073] In response, this application further proposes the following steps for triggering a sudden change warning of the droplet state when the synergistic effect strength deviates from the preset synergistic effect mode and the deviation exceeds the preset allowable range: Obtain overall stability information of the fire environment, including the spread rate of the fire source, the type of combustible material, and the intensity of heat radiation at the fire site; Obtain historical droplet mutation event frequency data, which includes statistical patterns of droplet mutation occurrence under different fire types; Obtain information on the urgency of the current drone mission, including the criticality of the reconnaissance area, the priority of target identification, and the urgency of the rescue operation. Based on the overall stability information of the fire scene environment, the preset allowable range benchmark value is adjusted to obtain the adjusted benchmark value. Based on the frequency data of historical droplet mutation events, the dynamic adjustment range of the adjusted baseline value is corrected to obtain the corrected dynamic adjustment range. Based on the urgency information of the drone mission, the final value of the dynamic adjustment range after weighted adjustment and correction is used to generate a dynamic allowable range that adapts to the current fire scene environment and mission requirements. When the strength of the synergistic effect deviates from the preset synergistic effect mode, and the deviation exceeds the dynamic tolerance range, a sudden change warning of the droplet state is triggered.

[0074] Specifically, acquiring overall stability information about the fire environment aims to assess the current macroscopic situation of the fire. The spread rate of the fire source reflects the intensity and development trend of the fire; the type of combustible material affects the characteristics of smoke and heat generation, thus influencing droplet formation and evaporation; and the intensity of thermal radiation from the fire directly relates to the rapid increase in surface temperature of optical components, potentially leading to rapid droplet evaporation or condensation. This information is used to provide an initial baseline value based on the current fire conditions for a predetermined permissible range.

[0075] Obtaining historical droplet mutation event frequency data can be understood as summarizing and utilizing past experience. This data contains statistical patterns of droplet mutation occurrence under different fire types, such as the probability and characteristics of droplet mutation under specific combustible materials or fire conditions. These statistical patterns are used to correct the dynamic adjustment range of the aforementioned benchmark values, making the adjustment of the allowable range more consistent with actual physical laws and historical experience.

[0076] In practical applications, acquiring information on the urgency of the current UAV mission specifically refers to considering the strategic importance of the mission. Factors such as the criticality of the reconnaissance area, the priority of target identification, and the urgency of the rescue operation determine the sensitivity requirements of the early warning system's response to droplet abrupt changes. For example, during high-priority rescue missions, even minute changes in droplet state may require immediate warning to ensure accurate image recognition. Therefore, this information is used to weight and adjust the dynamic adjustment range to generate a final dynamic tolerance range adapted to the current fire scene environment and mission requirements. Thus, a droplet state abrupt change warning is only triggered when the intensity of the synergistic effect deviates from the preset synergistic effect pattern, and this deviation exceeds the aforementioned dynamic tolerance range. This means that the triggering of the warning no longer depends on a fixed threshold but is intelligently adjusted based on the actual fire scene conditions, historical data, and mission requirements.

[0077] This application's solution achieves dynamic adjustment of a preset allowable range by incorporating overall stability information of the fire environment, frequency data of historical droplet mutation events, and urgency information of the UAV mission. Specifically, the overall stability information of the fire environment is used to set a baseline value for the allowable range, ensuring that the early warning system can make a preliminary judgment based on the current macroscopic situation of the fire. Furthermore, the frequency data of historical droplet mutation events is used to correct the dynamic adjustment range of this baseline value, making the adjustment of the allowable range more refined and able to reflect the statistical regularity of droplet mutation occurrences under different fire types. On this basis, the urgency information of the UAV mission is used to weight the corrected dynamic adjustment range, thereby generating a dynamic allowable range that is highly adaptable to the current fire environment and mission requirements. It is precisely because of the dynamic nature of the allowable range that the early warning system can flexibly adjust its sensitivity according to the actual situation, avoiding false alarms or missed alarms that may be caused by fixed thresholds, thereby improving the accuracy and reliability of droplet state mutation early warning.

[0078] Through the aforementioned technical solution, this application can intelligently adjust the permissible range of droplet state change warning based on real-time changes in the fire scene environment, historical experience, and the urgency of the mission. This frees the warning system from the limitations of static thresholds, enabling it to more accurately identify real changes in droplet state, effectively reducing false alarm rates, and ensuring timely warnings at critical moments. Consequently, the accuracy, reliability, and timeliness of fire feature image recognition for firefighting drones in complex and ever-changing fire scene environments are significantly improved, providing more reliable visual information support for fire rescue decision-making.

[0079] Obtaining accurate and comprehensive overall stability information of the fire environment in real-world fire scenarios presents numerous challenges. For example, dense smoke, intense heat, and complex air convection can lead to partial data loss, interference, or even inconsistencies between different sensors. If these issues are not addressed, the acquired overall stability information may be biased, affecting the accuracy of dynamic tolerance adjustments and potentially causing misjudgments or missed warnings of sudden droplet state changes, thereby reducing the reliability and safety of firefighting drones. To address this, this application proposes a more robust and accurate method for acquiring overall stability information of the fire environment. Through multi-source data fusion and intelligent data processing, it ensures high-quality stability information even under complex fire conditions.

[0080] The steps for obtaining overall stability information of the fire environment, including the fire spread rate, the type of combustible material, and the intensity of heat radiation from the fire, include: The drone is equipped with a multispectral sensor array to collect multispectral images and video streams of the fire scene; Multispectral images and video streams are processed to analyze changes in the color, brightness, and shape of flames, as well as the diffusion patterns and speed of smoke, thereby estimating the spread rate of the fire source. Infrared thermal imagers were used to measure the temperature distribution on the surface of a fire. Based on the physical laws of flame spread, predict the future trajectory of the fire; When there are local missing or interference in multispectral images or infrared thermal imaging data, analyze the stable data in adjacent time frames and interpolate or replace the missing data. When inconsistencies exist between multispectral images and infrared thermal imaging data, sensor data with a higher signal-to-noise ratio and lower error should be accepted.

[0081] Specifically, the drones are configured to carry a multispectral sensor array capable of simultaneously acquiring images and video streams of the fire scene across multiple spectral bands (e.g., visible light, near-infrared, mid-infrared, etc.). Multispectral data allows for a more comprehensive identification of fire source characteristics, such as the spectral fingerprints of different combustibles. Processing the multispectral images and video streams involves using image processing algorithms and pattern recognition technology to perform in-depth analysis of the acquired data. For example, analyzing flame color changes can determine combustion temperature and stage; changes in flame brightness and shape can assess fire intensity and spread trends; and smoke diffusion patterns and speeds can estimate the fire spread rate. Comprehensive analysis of these parameters helps in forming an accurate estimate of the fire spread rate.

[0082] Furthermore, measuring the temperature distribution on the surface of a fire using infrared thermal imagers aims to obtain information on the intensity of thermal radiation within the fire. Infrared thermal imagers can penetrate some smoke and directly sense the heat radiation from object surfaces, thus providing information on the internal temperature distribution of the fire. This is crucial for assessing the overall intensity of the fire and potential hazardous areas. By combining this with the physical laws of flame spread, such as heat conduction, convection, and radiation models, the future trajectory of the fire can be predicted, providing forward-looking information for drone-based firefighting strategies and early warning of abrupt changes in droplet states.

[0083] In practical applications, when multispectral images or infrared thermal imaging data exhibit localized gaps or interference, such as due to dense smoke obstruction, sensor malfunction, or strong light interference, it is necessary to analyze stable data from adjacent time frames to interpolate or replace the missing data. This ensures the continuity and integrity of the overall stability information of the fire scene environment. For example, time series analysis methods can be used to predictively fill in the missing data based on valid data from the previous or next time step. Furthermore, when inconsistencies exist between multispectral images and infrared thermal imaging data, such as due to differences in sensor calibration errors or environmental influences, it is necessary to adopt sensor data with higher signal-to-noise ratios and lower errors. This is typically achieved through real-time quality assessment of sensor data, such as calculating the signal-to-noise ratio, evaluating data volatility, or comparing it with known physical models, thereby selecting the most reliable data source for information fusion to improve the accuracy of the overall stability information.

[0084] This application's solution, by incorporating a multispectral sensor array and an infrared thermal imager, achieves comprehensive acquisition of multi-dimensional and multi-layered information about the fire environment, overcoming the limitations of single sensors in acquiring information under complex fire conditions. Multispectral data provides detailed characteristics of flames and smoke, aiding in the accurate estimation of fire spread rates and combustible material types; infrared thermal imaging data directly reflects the heat radiation intensity and temperature distribution of the fire scene, compensating for the shortcomings of visible light data in smoke penetration. By fusing these multi-source heterogeneous data, a more complete and accurate overall stability information of the fire environment can be constructed.

[0085] Furthermore, the solution proposed in this application effectively addresses the data quality degradation problem caused by the complex environment of a fire scene by introducing an interpolation substitution mechanism for missing or interfered data, and a reliability acceptance strategy for inconsistent data. When sensor data is partially missing or interfered with due to environmental factors (such as dense smoke or high temperature), intelligent filling is performed by analyzing stable data from adjacent time frames, ensuring the continuity of information. When there are contradictions between different sensor data, a more reliable data source is selected by evaluating the signal-to-noise ratio and error, avoiding misjudgments caused by data conflicts. This ensures that the overall stability information of the fire scene environment input to the subsequent dynamic tolerance adjustment module is of high quality and high reliability, thus providing a solid data foundation for the accuracy of droplet state change early warning.

[0086] Through the above technical solutions, this application can significantly improve the accuracy and reliability of acquiring overall fire environment stability information. The combination of multispectral and infrared thermal imaging makes the assessment of fire source spread rate, type of combustibles, and thermal radiation intensity more comprehensive and accurate, especially in environments with dense smoke and low visibility, effectively penetrating obstacles to acquire key data. In addition, the intelligent processing mechanism for data missing, interference, and inconsistencies greatly enhances the system's data robustness in harsh fire environment conditions, avoiding misjudgments caused by data quality issues. As a result, the generated overall fire environment stability information more realistically reflects the fire situation, making subsequent adjustments to the dynamic tolerance range of droplet state change warnings more accurate and timely, effectively reducing the risk of false alarms and missed alarms, and significantly improving the decision-making accuracy and safety of firefighting drones performing missions in complex fire scenes.

[0087] The following is a specific example to illustrate this.

[0088] Suppose a firefighting drone is conducting a fire scene reconnaissance mission; the surface of its camera's optical components may be covered with complex droplets. To accurately predict abrupt changes in the droplet state, precise information on the overall stability of the fire scene environment is needed.

[0089] Specifically, the drone carries a multispectral sensor array that continuously acquires visible light and near-infrared images of the fire site, while an infrared thermal imager simultaneously measures the temperature distribution on the fire surface. When the drone flies over a dense smoke area, the visible light portion of the multispectral image may appear partially blurred or missing, but the near-infrared image can still provide some information on the flame outline and smoke diffusion. At this point, the system uses the clear temperature distribution data provided by the infrared thermal imager, combined with the smoke diffusion pattern in the near-infrared image, to make a preliminary estimate of the fire spread rate.

[0090] Furthermore, if the infrared thermal imager data exhibits a momentary anomaly due to localized high-temperature interference, the system will identify the anomaly and perform interpolation based on the infrared data from the previous stable time frame. Simultaneously, it will combine this with the flame brightness variation trend provided by the multispectral sensor array to correct the thermal radiation intensity of the fire scene. For example, if the previous frame's infrared data indicated a temperature of 800℃ in a certain area, while the current frame abnormally displays 1200℃, and the multispectral data does not show a dramatic increase in flame intensity, the system will either accept the stable data from the previous frame or perform smooth interpolation.

[0091] Furthermore, if there is a slight discrepancy between the multispectral sensor array's identification of the type of combustible material and the infrared thermal imager's assessment of fire intensity (for example, multispectral data shows wood burning, but infrared data shows a slightly lower thermal radiation intensity than typical wood burning), the system will evaluate the signal-to-noise ratio and historical error rate of the two sensor data. If the infrared thermal imager has a higher signal-to-noise ratio and lower error rate in the current environment, the fire intensity indicated by the infrared data will be prioritized, and the overall assessment of the fire spread rate and thermal radiation intensity will be adjusted accordingly.

[0092] Through the aforementioned multi-source data acquisition, data processing, and intelligent fusion mechanism, even in situations where the fire environment is complex and changeable and the quality of sensor data is unstable, the system can continuously output high-precision and high-reliability information on the overall stability of the fire environment, thereby providing a solid basis for the dynamic adjustment of the allowable range for subsequent droplet state change warnings.

[0093] Specifically, the steps described above, when multispectral images or infrared thermal imaging data show local missing data or interference, to analyze stable data from adjacent time frames and interpolate or replace the missing data, include: Identify the boundaries and extent of missing or interfering areas; Based on the boundaries and extent of the missing or interfering regions, extract the local texture features and color distribution around the missing or interfering regions from the valid data of the current time frame; Select a stable time frame that is closest in time to the current time frame and has high data quality; In a stable time frame, regions with similar features to the missing or interfering regions are searched based on the boundaries and extent of the missing or interfering regions, as well as local texture features and color distribution. Based on the image content of similar regions and combined with the dynamic change trend of the region in a stable time frame, content is filled into the missing or interfering regions. Smooth the filled area.

[0094] Identifying the boundaries and extent of missing or interfering regions can be achieved through image segmentation algorithms or methods based on pixel intensity, color, and texture anomaly detection. For example, edge detection, region growing, or machine learning models can be used to accurately define damaged areas. Furthermore, local texture features and color distributions around the missing or interfering regions are extracted from the valid data of the current time frame to provide contextual information for subsequent image content infilling. Specifically, techniques such as Local Binary Patterns (LBP), Gabor filters, or color histograms can be used to quantify these features.

[0095] Selecting a stable timeframe that is temporally closest to the current timeframe and has high data quality ensures the highest timeliness and reliability of the data used for interpolation or replacement. Data quality can be evaluated based on metrics such as signal-to-noise ratio, sharpness, and completeness. Within the stable timeframe, regions with similar characteristics to the missing or interfering regions are searched based on their boundaries and extent, as well as local texture features and color distribution. The aim is to find an optimal reference source for content filling. This can be achieved using feature matching algorithms such as SIFT, SURF, or deep learning-based feature extraction and matching methods. Content filling of the missing or interfering regions is performed based on the image content of similar regions and the dynamic change trend of these regions within the stable timeframe. This involves mapping the pixel or structural information of the found similar regions to the missing regions. Combining this with dynamic change trends ensures that the filled content not only matches the static appearance but also maintains reasonable consistency in the temporal sequence. Finally, smoothing the filled region eliminates potential visual discontinuities or artifacts that may occur during the filling process, making the repaired image visually more natural and consistent. This can be achieved through techniques such as Gaussian filtering, mean filtering, or Poisson image editing.

[0096] The solution presented in this application, through the detailed steps described above, systematically addresses the problem of localized missing or interfered areas in multispectral images or infrared thermal imaging data during acquisition. First, it accurately identifies the damaged areas, laying the foundation for subsequent restoration. Second, it extracts local features from surrounding valid data, providing contextual clues for restoration. Next, it identifies regions with similar characteristics from stable time frames that are closest in time and have high data quality, ensuring the accuracy and timeliness of the filled content. By combining the image content of similar regions and their dynamic change trends for filling, the visual information and dynamic characteristics of the missing areas can be effectively restored. Finally, through smoothing processing, it ensures that the restored image is visually seamless.

[0097] The above technical solution effectively interpolates or replaces missing or interfering areas in multispectral images or infrared thermal imaging data with high quality, even in complex fire environments where data acquisition is susceptible to interference. This significantly improves the completeness, accuracy, and reliability of overall fire environment stability information, avoiding errors in estimating fire spread rate, combustible material type, and fire thermal radiation intensity due to missing or distorted data. This provides a more solid data foundation for the decision-making and actions of firefighting drones.

[0098] Based on the same inventive concept, this application also discloses a fire feature image recognition system for fire-fighting drones, such as... Figure 2 As shown, the system includes: The optical signal transmitting module 1 is used to transmit specific optical signals to the surface of the optical components of the drone camera and to the fire scene environment in front of it; Optical signal receiving module 2 is used to receive the changes in optical signal after a specific optical signal penetrates the surface of the optical component and the fire scene environment in front of it. The degradation characteristic information generation module 3 is used to analyze and generate degradation characteristic information, including the characteristics of the surface deposits of optical components and the optical characteristics of the fire scene environment, based on changes in the optical signal. Image correction module 4 is used to reverse correct the original image data acquired by the image sensor based on the degradation characteristic information, so as to restore the image's clarity, contrast and detail information.

[0099] The system provided in this application can actively transmit and receive specific light signals to obtain information on the degradation characteristics of the fire environment and the surface deposits of optical components, and then reverse-correct the original image accordingly. This effectively solves the problem of severe image degradation and difficulty in recognition under complex fire environment conditions, significantly improves the clarity, contrast and detail of the image, and provides more reliable visual information for fire rescue.

[0100] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application.

Claims

1. A method for recognizing fire feature images from firefighting drones, characterized in that, include: Specific light signals are emitted onto the surface of the optical components of the drone's camera and the fire scene environment in front of it; Receive the change in the light signal after the specific light signal penetrates the surface of the optical component and the fire scene environment in front; Based on the changes in the optical signal, analyze and generate degradation characteristic information including the characteristics of the surface deposits of the optical components and the optical characteristics of the fire scene environment in front; Based on the degradation characteristics information, the original image data acquired by the image sensor is reverse-corrected to restore the image's clarity, contrast, and detail.

2. The method for fire feature image recognition of a fire-fighting drone according to claim 1, characterized in that, The step of analyzing and generating degradation characteristic information, including the characteristics of the surface deposits of the optical component and the optical characteristics of the fire scene environment ahead, based on the changes in the optical signal includes: Detecting the absorption intensity in specific infrared bands to determine the presence of chemicals associated with highly reflective particles; Analyze the scattering patterns of the optical signal in the visible and near-infrared bands to identify anomalous scattering phenomena; When the absorption intensity of the specific infrared band is detected to exceed a preset threshold and the abnormal scattering phenomenon is identified, the presence of the highly reflective particles is confirmed. Based on the absorption intensity of the specific infrared band, the smoke optical density, particulate size distribution, and the contribution of highly reflective particles in the surface of the optical components obtained from the optical signal change analysis are corrected to generate corrected degradation characteristic information.

3. The method for fire feature image recognition of a fire-fighting drone according to claim 1, characterized in that, The step of reverse-correcting the original image data acquired by the image sensor based on the degradation characteristic information to restore the image's sharpness, contrast, and detail includes: Collect local temperature data on the surface of the optical component; Collect local humidity data on the surface of the optical component; Collect micro-airflow velocity data on the surface of the optical component; Collect micro-airflow direction data on the surface of the optical component; Collect high-frequency micro-vibration data on the surface of the optical component; Acquire chemical composition indication signals near the optical components; Based on the collected local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and chemical component indication signals, the geometry, internal structure, and refractive index distribution of the composite droplet are predicted. Based on the predicted geometry of the composite droplet, its internal structure, and its refractive index distribution, predictive geometric distortion information is generated. Based on the predictive geometric distortion information, the original image data is geometrically remapped to restore the image's sharpness, contrast, and detail. Based on the predicted internal structure of the composite droplet and the refractive index distribution, the local optical properties of the image are adjusted to restore the image's sharpness, contrast, and detail.

4. The method for fire feature image recognition of a fire-fighting drone according to claim 3, characterized in that, The step of predicting the geometry, internal structure, and refractive index distribution of the composite droplet based on the collected local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and chemical component indication signal includes: When collecting the local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and chemical component indication signal, the instantaneous rate of change of each environmental parameter is monitored. When the instantaneous rate of change of any environmental parameter exceeds the preset dynamic threshold, or when the combined change pattern of multiple environmental parameters does not conform to the preset synergistic effect pattern, a sudden change warning of the droplet state is triggered. After triggering the mutation warning, a high-frequency image sequence acquisition mode is started, and an auxiliary photodetector array is activated at the same time to measure the instantaneous light refraction and reflection patterns of the composite droplet in multiple local areas. Based on the rapid deformation trajectory of the droplet edge in the high-frequency image sequence and the instantaneous light refraction and reflection modes measured by the auxiliary photodetector array, the key parameters in the droplet evolution model are corrected. Based on the revised droplet evolution model, the geometry, internal structure, and refractive index distribution of the composite droplet are re-predicted in the very short time to come.

5. The method for fire feature image recognition of a fire-fighting drone according to claim 4, characterized in that, The step of monitoring the instantaneous rate of change of each environmental parameter when collecting the local temperature data, the local humidity data, the micro-airflow velocity data, the micro-airflow direction data, the high-frequency micro-vibration data, and the chemical component indication signal further includes: Noise suppression and outlier removal are performed on the original acquired data of the local temperature data, local humidity data, micro-airflow velocity data, micro-airflow direction data, high-frequency micro-vibration data, and chemical component indication signals.

6. The method for fire feature image recognition of a fire-fighting drone according to claim 4, characterized in that, The step of triggering a sudden change warning for the droplet state when the instantaneous change rate of any environmental parameter exceeds a preset dynamic threshold, or when the combined change pattern of multiple environmental parameters does not conform to a preset synergistic effect pattern, includes: Acquire the local temperature data, the local humidity data, the micro-airflow velocity data, the micro-airflow direction data, the high-frequency micro-vibration data, and the instantaneous change rate of the chemical component indicator signal; Calculate the degree of correlation between the instantaneous rates of change at different time scales; Based on the degree of correlation, the strength of the synergistic effect between the environmental parameters is identified; The strength of the synergistic effect is compared with a preset synergistic effect mode; When the intensity of the synergistic effect deviates from the preset synergistic effect mode, and the deviation exceeds the preset allowable range, a sudden change warning for the droplet state is triggered.

7. The method for fire feature image recognition of a fire-fighting drone according to claim 6, characterized in that, The step of triggering a sudden change warning for the droplet state when the synergistic effect strength deviates from the preset synergistic effect mode and the deviation exceeds a preset allowable range includes: Obtain overall stability information of the fire scene environment; Obtain historical droplet mutation event frequency data, which includes statistical patterns of droplet mutation occurrence under different fire types; Obtain the urgency information of the current drone mission, which includes the criticality of the reconnaissance area, the priority of target identification, and the urgency of the rescue operation; Based on the overall stability information of the fire scene environment, the baseline value of the preset allowable range is adjusted to obtain the adjusted baseline value; Based on the frequency data of the occurrence of the historical droplet mutation events, the dynamic adjustment range of the adjusted baseline value is corrected to obtain the corrected dynamic adjustment range; Based on the urgency information of the drone mission, the final value of the corrected dynamic adjustment range is adjusted by weighting to generate a dynamic allowable range that adapts to the current fire scene environment and mission requirements. When the intensity of the synergistic effect deviates from the preset synergistic effect mode, and the deviation exceeds the dynamic tolerance range, a sudden change warning for the droplet state is triggered.

8. The method for fire feature image recognition of a fire-fighting drone according to claim 7, characterized in that, The steps for obtaining overall stability information of the fire scene environment include: The drone is equipped with a multispectral sensor array to collect multispectral images and video streams of the fire scene; The multispectral images and video streams are processed to analyze the color, brightness, and shape changes of the flames, as well as the diffusion patterns and speed of the smoke, thereby estimating the spread rate of the fire source. Infrared thermal imagers were used to measure the temperature distribution on the surface of a fire. Based on the physical laws of flame spread, predict the future trajectory of the fire; When the multispectral image or the infrared thermal imaging data has local missing or interference, analyze the stable data of adjacent time frames and interpolate or replace the missing data. When there is an inconsistency between the multispectral image and the infrared thermal imaging data, the sensor data with a higher signal-to-noise ratio and lower error is accepted.

9. A method for fire feature image recognition of a fire-fighting drone according to claim 8, characterized in that, When the multispectral image or the infrared thermal imaging data has local missing data or interference, the step of analyzing stable data in adjacent time frames and interpolating or replacing the missing data includes: Identify the boundaries and extent of the missing or interfering regions; Based on the boundary and extent of the missing or interfering region, extract the local texture features and color distribution around the missing or interfering region from the valid data of the current time frame; Select a stable time frame that is closest in time to the current time frame and has high data quality; In the stable time frame, based on the boundary and range of the missing or interfering region, as well as the local texture features and color distribution, a region with similar features to the missing or interfering region is searched; Based on the image content of the similar regions and combined with the dynamic change trend of the regions in the stable time frames, the missing or interfering regions are filled with content. Smooth the filled area.

10. A fire feature image recognition system for firefighting drones, characterized in that, The system includes: The optical signal transmitting module is used to transmit specific optical signals to the surface of the optical components of the drone camera and to the fire scene environment in front of it; An optical signal receiving module is used to receive the changes in the optical signal after the specific optical signal penetrates the surface of the optical component and the fire scene environment in front. The degradation characteristic information generation module is used to analyze and generate degradation characteristic information including the characteristics of the surface deposits of the optical component and the optical characteristics of the fire scene environment in front, based on the changes in the optical signal. The image correction module is used to reverse correct the original image data acquired by the image sensor based on the degradation characteristic information, so as to restore the image's clarity, contrast, and detail information.

Images