A smoke identification and classification early warning method, system, device and storage medium

Abstract

The application relates to a smoke identification and classification early warning method, which comprises the following steps: carrying an infrared camera, an infrared spectrum sensor and a temperature and humidity sensor on a UAV to carry out smoke inspection activities on a set target area; collecting infrared visual images, infrared spectrum data and environmental parameters of the target area by using the UAV, and pre-processing the collected data; the infrared spectrum data comprises the concentrations of CO2 and CO, and the environmental parameters comprise temperature, humidity and wind speed; performing multi-modal data fusion on the pre-processed data to construct a fusion feature vector, and identifying a smoke type according to the fusion feature vector by using a trained deep learning classification model; determining an early warning level according to the identified smoke type, and dynamically realizing early warning triggering based on the early warning level and a set classification response rule. The application can accurately distinguish and make early warning decisions on smoke in a complex environment of urban comprehensive management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of urban comprehensive management, and in particular to a smoke identification and classification early warning method, system, device and storage medium. BACKGROUND

[0002] In comprehensive management, early identification of smoke is particularly crucial for fire warning and pollution control. However, the urban environment is complex and variable, and smoke sources are diverse - there are fire smoke, industrial emissions, and it may also come from steam, garbage incineration or construction dust, plus interference factors such as high-rise buildings blocking, weather changes (such as smog), etc. Traditional smoke detection systems often cause high false alarm rates due to misjudgment. This not only wastes emergency resources of fire, environmental protection and other departments, but also may cause unnecessary public panic.

[0003] Smoke is often a "signal light" of problems such as fire and industrial pollution. It is of great significance to protect urban safety and avoid resource waste to quickly identify it and issue an alarm. In the past, we mainly relied on fixed-position sensors or manual patrols to discover smoke. These methods have small coverage, slow response, and are particularly prone to false positives.

[0004] In recent years, unmanned aerial vehicle technology has begun to play a role in urban management due to its flexibility and wide field of view. Some unmanned aerial vehicle smoke detection systems now can determine whether there is smoke by carrying a high-definition camera combined with image recognition technology. However, the problem is that these systems can only "see" smoke, but cannot distinguish the type of smoke - is it dangerous fire smoke? Or normal emissions from a factory? Is it water vapor? Is it garbage incineration? Or construction dust? Without distinguishing the type, the accuracy of the warning is insufficient, and there is a problem of resource waste caused by misjudgment. Therefore, how to achieve intelligent differentiation of smoke types through technical means has become a technical bottleneck that needs to be broken through in urban comprehensive management. SUMMARY

[0005] The present application provides a smoke identification and classification early warning method, system, device and storage medium, which is suitable for accurate differentiation and early warning decision of multiple smoke sources in complex urban environment.

[0006] In a first aspect, the present application provides a smoke identification and classification early warning method, comprising:

[0007] An infrared camera, an infrared spectrum sensor and a temperature and humidity sensor are mounted on an unmanned aerial vehicle to carry out smoke inspection activities in a set target area;

[0008] The unmanned aerial vehicle collects infrared vision images, infrared spectrum data and environmental parameters of the target area, and pre-processes the collected data; the infrared spectrum data includes the concentrations of CO2 and CO, and the environmental parameters include temperature, humidity and wind speed;

[0009] Multimodal data fusion is performed on the preprocessed data to construct a fusion feature vector, and a trained deep learning classification model is used to identify the smoke type based on the fusion feature vector;

[0010] The warning level is determined based on the identified smoke type, and the warning is dynamically triggered based on the warning level and the set graded response rules.

[0011] Furthermore, the preprocessing of the collected data includes:

[0012] An adaptive filtering algorithm is used to suppress stripe noise and random noise in the collected data, and a Gaussian mixture model is combined to separate smoke from the background.

[0013] Furthermore, the step of constructing a fusion feature vector by performing multimodal data fusion on the preprocessed data includes:

[0014] First-order difference and wavelet transform were performed on the concentration sequences of CO2 and CO to extract the transient mutation index and construct a 6-dimensional spectral feature vector.

[0015] Temperature, humidity, and wind speed are adaptively normalized to construct a 3D environmental feature vector. The original visual features are adaptively reduced in dimensionality and activated to construct a 256-dimensional visual feature vector.

[0016] The 6-dimensional spectral feature vector, the 3-dimensional environmental feature vector, and the 256-dimensional visual feature vector are weighted and fused under time-series consistency constraints to obtain a fused feature vector.

[0017] Furthermore, the concentration sequences of CO2 and CO are subjected to first-order difference and wavelet transform to extract transient mutation indexes and construct a 6-dimensional spectral feature vector, including:

[0018] Let the CO2 concentration sequence be... C ( t ), t = 1,2,...,T, then the first-order difference Δ C ( t )= C ( t )- C ( t- 1) Wavelet high-frequency coefficient energy ,in d j For the db4 wavelet decomposition of the th j Layer detail factor, J This represents the total number of layers in the db4 wavelet decomposition.

[0019] Calculate the transient mutation index Then the 6-dimensional spectral eigenvector is obtained as follows: ;in The mean value of the CO2 spectral characteristics. The standard deviation of the CO2 spectral characteristics. The slope of the CO2 spectral characteristics. The mean value of the CO spectral characteristics. denoted as the standard deviation of the CO spectral characteristics.

[0020] Furthermore, the process of adaptively normalizing temperature, humidity, and wind speed to construct a 3D environmental feature vector, and adaptively reducing and activating the original visual features to construct a 256-dimensional visual feature vector includes:

[0021] A predefined set of scene labels {sunny, rainy, foggy, night, strong wind} is used to maintain sliding statistics for each scene, which are updated online. ;in, This represents the scene average at the current time step t. For the previous time step t The average value of scenario 1 For smoothing coefficients, The observation value at the current time step t;

[0022] Normalizing the environmental features yields a 3D environmental feature vector:

[0023] ;

[0024] Where T, H, and W represent temperature, humidity, and wind speed, respectively, in the environmental characteristics. , , These are the average values ​​of temperature, humidity, and wind speed in the current scenario. , , These represent the standard deviations of temperature, humidity, and wind speed in the current scenario;

[0025] Calculate channel importance scores for 512-dimensional raw visual features. ,reserve The first 256 channels from high to low are selected, and the rest are set to zero. After global average pooling, a 256-dimensional visual feature vector is obtained. Where h and w represent the height and width of the feature map, respectively, and i represents the channel of the feature map. This represents the value of the feature map.

[0026] Further, the step of performing time-consistency-constrained weighted fusion of the 6-dimensional spectral feature vector, the 3-dimensional environmental feature vector, and the 256-dimensional visual feature vector to obtain a fused feature vector includes:

[0027] Maintain the expected value of the fused features from the previous frame Calculate the temporal consistency score for each modality in the current frame:

[0028]

[0029] in For the fusion features at time step t-1, For the characteristics of mode m at the current time step t, Let m be the characteristic transformation function of mode m. These represent visual features, spectral features, and environmental features, respectively.

[0030] Calculate the fusion weights for each modality The fused feature vector is obtained as follows:

[0031] ;

[0032] in For smoothing parameters, The exponential consistency score for mode m. For the number of modes, The sum of the indexed consistency scores for all modalities. Visual feature vectors The fusion weight, Spectral eigenvectors The fusion weight, Environmental feature vector The fusion weight.

[0033] Furthermore, the step of determining the warning level based on the identified smoke type, and dynamically triggering the warning based on the warning level and the set graded response rules, includes:

[0034] When the smoke type is fire smoke, it is determined as a Level 1 warning, triggering fire linkage, pushing location information to the emergency command center, and starting drone tracking;

[0035] When the smoke type is industrial illegal emission smoke, it is determined as a Level 2 warning, which is pushed to the environmental protection regulatory department and recorded in the database;

[0036] When the smoke type is residential smoke, it is determined to be a Level 3 warning, and community management personnel are notified to verify.

[0037] Secondly, this application provides a smoke recognition and classification early warning system, including:

[0038] The equipment is equipped with modules for mounting infrared cameras, infrared spectral sensors, and temperature and humidity sensors on drones to conduct smoke inspections of designated target areas.

[0039] The data acquisition module is used to collect infrared visual images, infrared spectral data, and environmental parameters of the target area using a drone, and to preprocess the collected data; the infrared spectral data includes the concentrations of CO2 and CO, and the environmental parameters include temperature, humidity, and wind speed;

[0040] The smoke recognition module is used to perform multimodal data fusion on the preprocessed data to construct a fusion feature vector, and to use a trained deep learning classification model to identify the smoke type based on the fusion feature vector;

[0041] The classification and early warning module is used to determine the early warning level based on the identified smoke type, and dynamically trigger the early warning based on the early warning level and the set graded response rules.

[0042] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the smoke recognition and classification warning method as described above.

[0043] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the smoke recognition and classification warning method described above.

[0044] The above-mentioned technical solution of this application has the following advantages:

[0045] The smoke recognition and classification early warning method provided in the first aspect of this application conducts smoke inspection activities in a designated target area by equipping a drone with an infrared camera, infrared spectral sensor, and temperature and humidity sensor. The drone collects infrared visual images, infrared spectral data, and environmental parameters of the target area, and preprocesses the collected data. The infrared spectral data includes CO2 and CO concentrations, and the environmental parameters include temperature, humidity, and wind speed. Multimodal data fusion is performed on the preprocessed data to construct a fusion feature vector, and a trained deep learning classification model is used to identify the smoke type based on the fusion feature vector. An early warning level is determined based on the identified smoke type, and early warning triggering is dynamically implemented based on the early warning level and set hierarchical response rules. This method can accurately distinguish and make early warning decisions for diverse smoke sources in complex urban environments, reducing the ecological and environmental hazards of fires. It better adapts to complex urban environments and improves the timeliness of drone response in fire prevention and disaster relief in urban comprehensive management.

[0046] It is understood that the beneficial effects of the second, third and fourth aspects mentioned above can be found in the relevant descriptions in the first aspect above, and will not be repeated here. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 A flowchart of the smoke recognition and classification early warning method provided in this application;

[0049] Figure 2 This is a structural diagram of the smoke recognition and classification early warning system provided in this application;

[0050] Figure 3 A structural diagram of the electronic device provided in this application. Detailed Implementation

[0051] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0052] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0053] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0054] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. "A plurality" means "two or more."

[0055] This application presents a method for automatic smoke identification, detection, and early warning in urban comprehensive management based on UAV vision, belonging to the field of fire identification and early warning technology. By utilizing UAVs, infrared high-definition cameras, environmental sensors, and intelligent recognition technologies, it can accurately distinguish and make early warning decisions regarding diverse smoke sources in the complex environment of urban comprehensive management. This reduces the ecological and environmental hazards of fires, better adapts to complex urban environments, and improves the timeliness of UAV response in fire prevention and disaster relief within urban comprehensive management.

[0056] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0057] This application provides a smoke recognition and classification warning method, such as... Figure 1 As shown, the specific steps include: Equipping a drone with an infrared camera, infrared spectral sensor, and temperature and humidity sensor to conduct smoke inspection activities in a designated target area; using the drone to collect infrared visual images, infrared spectral data, and environmental parameters of the target area, and preprocessing the collected data; the infrared spectral data includes the concentrations of CO2 and CO, and the environmental parameters include temperature, humidity, and wind speed; performing multimodal data fusion on the preprocessed data to construct a fusion feature vector, and using a trained deep learning classification model to identify the smoke type based on the fusion feature vector; determining the warning level based on the identified smoke type, and dynamically triggering the warning based on the warning level and the set graded response rules.

[0058] Multiple inspection drones are used to conduct inspections of the planned comprehensive management area, transmitting images and data collected by airborne sensors in real time. The collected data is then analyzed and preprocessed. Multimodal features related to smoke are extracted from the preprocessed data. A deep learning classification model is built to identify smoke from real-time images or video data. After automatically identifying smoke, a positioning algorithm is used to confirm the location information of the smoke point and determine its regional location. Based on accurate smoke type identification and hierarchical response rules, early warning triggering is dynamically implemented.

[0059] Multimodal data fusion integrates different types of data (such as infrared visual images, infrared spectral data, and environmental parameters) to construct a more comprehensive feature representation. In this embodiment, combining infrared visual images with infrared spectral data and environmental parameters improves the robustness of smoke type identification. Deep learning classification models are based on deep neural networks (such as CNN and Transformer), learning high-level features from the data through training to achieve classification tasks. In this embodiment, the deep learning classification model is used for accurate classification of smoke types, replacing the preset rule models of existing technologies. The warning level association mechanism dynamically adjusts the warning level and triggers corresponding response measures based on smoke type, severity, and environmental parameters, avoiding resource waste.

[0060] In some embodiments, the preprocessing of the collected data includes:

[0061] An adaptive filtering algorithm is used to suppress stripe noise and random noise in the collected data, and a Gaussian mixture model is combined to separate smoke from the background.

[0062] In some embodiments, the step of constructing a fusion feature vector by performing multimodal data fusion on the preprocessed data includes:

[0063] First-order difference and wavelet transform were performed on the concentration sequences of CO2 and CO to extract the transient mutation index and construct a 6-dimensional spectral feature vector. Scene-adaptive normalization was performed on temperature, humidity, and wind speed to construct a 3-dimensional environmental feature vector. Adaptive dimensionality reduction and activation were performed on the original visual features to construct a 256-dimensional visual feature vector. The 6-dimensional spectral feature vector, the 3-dimensional environmental feature vector, and the 256-dimensional visual feature vector were weighted and fused with temporal consistency constraints to obtain a fused feature vector.

[0064] In some embodiments, performing first-order difference and wavelet transform on the concentration sequences of CO2 and CO to extract the transient mutation index and construct a 6-dimensional spectral feature vector includes:

[0065] Let the CO2 concentration sequence be... C ( t ), t = 1,2,...,T, then the first-order difference Δ C ( t )= C ( t )- C ( t- 1) Wavelet high-frequency coefficient energy ,in d j For the db4 wavelet decomposition of the th j Layer detail factor, J This represents the total number of layers in the db4 wavelet decomposition.

[0066] Calculate the transient mutation index Then the 6-dimensional spectral eigenvector is obtained as follows: ;in The mean value of the CO2 spectral characteristics. The standard deviation of the CO2 spectral characteristics. The slope of the CO2 spectral characteristics. The mean value of the CO spectral characteristics. denoted as the standard deviation of the CO spectral characteristics.

[0067] In some embodiments, the step of scene-adaptive normalization of temperature, humidity, and wind speed to construct a 3D environmental feature vector, and adaptive dimensionality reduction and activation of the original visual features to construct a 256-dimensional visual feature vector includes:

[0068] A predefined set of scene labels {sunny, rainy, foggy, night, strong wind} is used to maintain sliding statistics for each scene, which are updated online. ;in, This represents the scene average at the current time step t. For the previous time step t The average value of scenario 1 For smoothing coefficients, The observation value at the current time step t;

[0069] Normalizing the environmental features yields a 3D environmental feature vector:

[0070] ;

[0071] Where T, H, and W represent temperature, humidity, and wind speed, respectively, in the environmental characteristics. , , These are the average values ​​of temperature, humidity, and wind speed in the current scenario. , , These represent the standard deviations of temperature, humidity, and wind speed in the current scenario;

[0072] Calculate channel importance scores for 512-dimensional raw visual features. ,reserve The first 256 channels from high to low are selected, and the rest are set to zero. After global average pooling, a 256-dimensional visual feature vector is obtained. Where h and w represent the height and width of the feature map, respectively, and i represents the channel of the feature map. This represents the value of the feature map.

[0073] In some embodiments, the step of performing time-consistency-constrained weighted fusion of the 6-dimensional spectral feature vector, the 3-dimensional environmental feature vector, and the 256-dimensional visual feature vector to obtain a fused feature vector includes:

[0074] Maintain the expected value of the fused features from the previous frame Calculate the temporal consistency score for each modality in the current frame:

[0075]

[0076] in For the fusion features at time step t-1, For modality m The characteristics at the current time step t, Let m be the characteristic transformation function of mode m. These represent visual features, spectral features, and environmental features, respectively.

[0077] Calculate the fusion weights for each modality The fused feature vector is obtained as follows:

[0078] ;

[0079] in For smoothing parameters, The exponential consistency score for mode m. For the number of modes, The sum of the indexed consistency scores for all modalities. Visual feature vectors The fusion weight, Spectral eigenvectors The fusion weight, Environmental feature vector The fusion weight.

[0080] In some embodiments, determining the warning level based on the identified smoke type and dynamically triggering the warning based on the warning level and the set graded response rules includes: when the smoke type is fire smoke, it is determined as a Level 1 warning, triggering fire linkage, pushing location information to the emergency command center, and starting drone tracking; when the smoke type is industrial illegal emission smoke, it is determined as a Level 2 warning, pushing to the environmental protection regulatory department and recording in the database; when the smoke type is residential smoke, it is determined as a Level 3 warning, notifying community management personnel to verify.

[0081] To address the shortcomings of existing technologies, this application aims to construct a method for intelligent identification, classification, and early warning of urban smoke based on UAV vision. Leveraging the advantages of UAVs' more comprehensive and intelligent "seeing," it creates an early warning system capable of accurately identifying and classifying urban smoke, thus resolving the pain points of existing solutions.

[0082] 1. Core objective: Intelligent brain, to build a precise identification system for multi-source smoke in cities, so that the system can truly "recognize" the various types of smoke in the city.

[0083] Multispectral fusion perception: Introducing infrared cameras and infrared spectral sensors to simultaneously acquire the target's thermal radiation distribution (temperature field) and visual features (texture, color). Through coupled analysis of thermal and visual features, it can reliably distinguish between high-temperature smoke (>150℃) and normal-temperature steam / dust (<50℃), reducing false alarm rates at the source.

[0084] Fine-grained classification network: Design an attention-based multi-label classifier that can simultaneously identify 7 typical urban smoke sources: ① fire smoke, ② industrial emissions, ③ restaurant fumes, ④ vehicle exhaust, ⑤ garbage burning, ⑥ construction dust, and ⑦ natural water mist.

[0085] 2. Algorithm goal: To bring data to life, understand the dynamics of smoke, and achieve multimodal data fusion of smoke fluid.

[0086] Spatiotemporal joint modeling: A lightweight CNNTransformer fusion architecture is designed to construct a multimodal dataset covering scenarios such as fire smoke, industrial emissions, steam, and fog, labeling smoke types and severity. FocalLoss is employed to mitigate class imbalance and improve small target detection capabilities. A joint feature vector is constructed by combining infrared spectral data (such as CO2 and CO concentration) with visual features, and a graph neural network (GNN) is used to model the spatiotemporal dynamics of smoke diffusion.

[0087] 3. Environmental Adaptability Goals: To enable the system to "understand" the weather and "know" the location, and to establish a multi-source meteorological and geographic data fusion mechanism.

[0088] Dynamic threshold adjustment: Integrates real-time meteorological data access interfaces (wind speed, wind direction, humidity), and uses algorithms to analyze and quantify the degree of deviation between smoke movement and the wind field. When the deviation index exceeds the threshold, model recalibration is automatically triggered to ensure detection accuracy under strong wind conditions.

[0089] Urban geographic knowledge is embedded: heat maps of urban functional areas are constructed, and differentiated judgment thresholds are applied to different areas such as industrial zones, restaurant streets, and transportation hubs. For example, higher alarm thresholds for oil fume concentration are allowed in restaurant areas, while sensitive fire warning thresholds are used in forest park areas.

[0090] 4. Hardware Configuration Goals: Upgrade the hardware, adding a high-definition infrared camera and a multi-functional environmental sensor.

[0091] (1) Unmanned aerial vehicle platform: equipped with a high-definition infrared camera (resolution ≥1080p, frame rate ≥30fps) to replace the existing visible light camera and improve imaging capabilities in low light and complex environments.

[0092] (2) Auxiliary sensors: Integrating infrared spectral sensors, temperature and humidity sensors and GPS modules to collect multi-dimensional environmental data.

[0093] 5. Early warning mechanism objectives: Intelligent early warning, establishment of a tiered response strategy, accurate alarm, saving time and effort.

[0094] Multi-level alarm system: Level 1 alarm (fire and smoke): triggers fire-fighting linkage and pushes location information to the emergency command center; Level 2 alarm (industrial illegal emissions): pushes to the environmental protection regulatory department; Level 3 alarm (residential smoke): notifies community management personnel to verify.

[0095] Resource optimization results: Through accurate smoke type identification and graded response, the target is to reduce the number of ineffective fire dispatches by more than 80%, significantly reducing social governance costs.

[0096] The following is a description through specific embodiments.

[0097] Example

[0098] This embodiment proposes an intelligent urban smoke recognition, classification, and early warning method based on UAV vision. Through hardware upgrades, algorithm optimization, and innovative early warning mechanisms, it achieves accurate classification of smoke types and low false alarm rates. The technical solution is based on over 500 field experiments and simulation tests, covering three scenarios: urban areas, forests, and industrial zones. The core process is as follows: UAV patrol → infrared data acquisition → preprocessing and noise reduction → multimodal feature extraction → deep learning classification → dynamic early warning triggering.

[0099] I. Hardware Configuration

[0100] 1. Drone Platform: The vehicle uses a six-axis multi-rotor drone (such as DJIM300RTK) as its carrier, which has a flight time of 55 minutes to cover the cruising needs of complex urban environments. It has a maximum payload of 2.7kg and is suitable for cruising in complex urban environments.

[0101] 2. Core Sensors:

[0102] High-definition infrared camera: Uses an uncooled long-wave infrared (LWIR) camera (such as FLIRBoson), with a resolution of 640×480, a frame rate of 30Hz, a wavelength range of 814μm, and automatic gain control (AGC) and digital detail enhancement (DDE) functions.

[0103] 3. Auxiliary sensors:

[0104] Infrared spectral sensors (such as Tellurex NDIR sensors): measure CO2 and CO concentrations, with a range of 0-5000 ppm and an accuracy of ±3%FS.

[0105] Temperature and humidity sensor (e.g., SHT31): Measurement range 40~125℃, humidity 0-100%RH, accuracy ±2%RH. GPS / IMU module (e.g., ublox ZEDF9P): Positioning accuracy ±1.5m, attitude measurement accuracy ±0.3°.

[0106] 4. Data Acquisition Unit:

[0107] Synchronization triggering mechanism: Camera and sensor data are synchronized via PPS (pulses per second) signal to ensure that the image is time-aligned with the spectrum and environmental parameters (error ≤ 10ms).

[0108] Storage and transmission: The video stream is compressed using H.265 encoding and transmitted to the ground station in real time via 4G / 5G network, while the original data (format: TIFF / RAW) is stored locally.

[0109] II. Data Acquisition and Preprocessing

[0110] 1. Daily Patrol Route Planning: Based on a city grid map, plan daily patrol routes for drones, covering key areas (such as industrial parks, utility tunnels, restaurant streets, transportation hubs, forests, etc.). Dynamic Route Adjustment: Based on real-time wind speed and temperature data, optimize patrol routes through AI algorithms. When wind speed ≥15m / s, automatically detour around areas with strong winds to optimize patrol efficiency.

[0111] 2. Data Acquisition: Real-time acquisition of smoke images, videos, and infrared spectral data, synchronously recording time, location, and environmental parameters (temperature, humidity, wind speed, CO2 concentration).

[0112] 3. Image Preprocessing:

[0113] Radiation calibration: using the formula T=(V_adc) V_offset) / G converts the raw data from the infrared camera into the actual temperature value, where V_adc is the ADC output voltage, V_offset is the bias voltage, and G is the gain coefficient.

[0114] 4. Noise reduction: An adaptive filtering algorithm is used to suppress stripe noise and random noise, and the signal-to-noise ratio (SNR) is improved to ≥40dB.

[0115] 5. Dynamic background modeling: Combine Gaussian mixture model (GMM) to separate the foreground (smoke) from the background.

[0116] Gaussian Mixture Model (GMM): Models the background of 100 consecutive frames of images, with an update rate of 1 frame per second, to adapt to changes in lighting (such as sunrise / sunset).

[0117] Foreground extraction: The smoke region is separated by background subtraction, with the threshold set to grayscale difference ≥ 20 (8-bit image).

[0118] Dynamic background modeling improves the accuracy of target detection by separating the foreground (moving target) from the background in a video using algorithms. In this embodiment, it can suppress environmental interference and highlight smoke features.

[0119] III. Deep Learning Classification Model Design

[0120] 1. Model Architecture:

[0121] CNN branch: Based on the improved ResNet50 architecture, the original fully connected layers are replaced with global average pooling (GAP) modules, outputting 512-dimensional feature vectors.

[0122] Transformer branch: Input 256×256 image patches of 16 consecutive frames (time dimension), capture temporal dynamics through multi-head self-attention (MHSA), and output 256-dimensional temporal feature vector.

[0123] Feature fusion: Spatial and temporal feature vectors are concatenated and then input into a fully connected layer (size 1024→256→N), where N is the number of smoke types (such as fire, industrial emissions, steam, etc.).

[0124] 2. Training data: Construct a multi-scenario smoke dataset (fire smoke, industrial emissions, steam, fog, etc.) and label the smoke type and severity.

[0125] (1) Construction of multi-scenario datasets: Add "adversarial scenarios". Based on the original "fire, industry, steam, fog", artificial adversarial samples that are difficult to distinguish are constructed, including: white steam + high CO2: simulate the confusion between "industrial leakage" and "fire"; nighttime car headlight reflection + fog: simulate "fake smoke"; barbecue stall oil smoke + high humidity: simulate "life interference".

[0126] Data collection method: A smoke generator and an environmental control chamber were used to precisely adjust the CO2 concentration (500~5000 ppm) and temperature and humidity (20~35°C, 60~95% RH); visible light and infrared video and spectrometer data were recorded simultaneously, and the "ambiguity level" was marked (1=clear, 5=extremely difficult to distinguish).

[0127] A "smoke adversarial dataset" was constructed, containing 12,000 difficult examples, filling a gap in publicly available data.

[0128] (2) Data quality assessment: Introducing "multimodal consistency score"

[0129] To filter out sensor malfunctions or incorrectly labeled samples, a multimodal consistency score is proposed:

[0130] In the initial training phase, single-modal models are used to predict the output of the visual model. P v Spectral model output P s Consistency of calculation: After removing samples with CMCS < 0.3 (approximately 4.7%), the model convergence speed improved by 31%.

[0131] Among them, the visual model output P v This represents the output vector of the visual model. This vector is typically a feature representation or probability distribution extracted from visual data (such as images). For example, in image classification tasks, P v It can be a feature vector containing visual information such as the color, texture, and shape of an image; or a probability vector representing the confidence level of an image belonging to each category.

[0132] Spectral model output P s This represents the output vector of the spectral model. This vector is typically a feature representation or probability distribution extracted from spectral data (such as infrared spectra or multispectral images). For example, in remote sensing or chemical analysis... P s This indicates the spectral characteristics of a substance, such as the values ​​of reflectance or absorptivity in different wavelength bands.

[0133] 3. Data Augmentation:

[0134] (1) Enhanced physical consistency

[0135] Traditional enhancements (rotation, noise) can compromise multimodal consistency. This embodiment proposes a physical consistency preservation enhancement:

[0136] During visual enhancement, the spectrum and environmental labels are adjusted simultaneously: if the image brightness is increased by 20% (simulating strong light), the temperature sensor reading increases by 2°C (direct sunlight effect), and the wind speed decreases by 0.1 m / s (thermal stability); if the image contrast decreases by 15% (simulating smog), the CO2 concentration increases by 50 ppm, and the humidity increases by 5%. The enhancement rules are derived from real-world environmental calibration experiments, establishing a "visual-physical mapping table".

[0137] Training on augmented data improved field test accuracy by 4.8%, while traditional augmentation resulted in a 2.1% decrease.

[0138] (2) Modal dropout enhancement

[0139] To prevent the model from over-relying on a particular modality, a training-period modal dropout is proposed: each batch randomly "shuts down" (sets) a particular modality to zero with a probability p=0.15, forcing the model to learn the modal redundancy mechanism; during testing, all modalities are enabled, significantly improving robustness (in the "spectrometer failure" simulation experiment, the accuracy only decreased by 5.2%, while without this strategy it decreased by 18.9%).

[0140] 4. Loss Function and Optimization:

[0141] Focal loss is a loss function used to address class imbalance. By suppressing the contribution of easily classified samples, it improves the model's ability to learn from difficult-to-classify samples. In this embodiment, it can improve the ability to identify low-frequency smoke types (such as chemical leaks).

[0142] To address class imbalance, a "spectral reweighting Focal Loss" is introduced. The original Focal Loss only considers prediction confidence. This embodiment proposes a spectral reweighting mechanism that incorporates class "difficulty in distinguishing" factors.

[0143] Define the visual similarity matrix between categories The visual similarity between categories is quantified through EfficientNet feature calculation:

[0144]

[0145] in Let C be the visual similarity matrix between categories, where C is the total number of categories. This is a symmetric matrix that stores the visual similarity between all pairs of categories. For the first i Class prototype characteristics (class center); For the first j Class prototype characteristics (class center); τ The scaling parameter controls the sensitivity of similarity to changes in distance. τ When the value is large, the similarity changes relatively smoothly and is not sensitive to distance. τ When the value is small, the similarity changes steeply and is sensitive to distance.

[0146] Calculate the category confusion penalty weight: ,in α i For the first i The penalty weight for the class; this value is used to adjust the penalty weight of the first class in the loss function. i Class weights are designed to increase the weight of easily confused classes, making the model pay more attention to these classes; λThis is a hyperparameter used to control the strength of the obfuscation penalty, which is set to 0.5 here. It adjusts the degree of influence of the obfuscation item on the overall weight. S ij For the first i Class and the j Visual similarity between classes; the higher the value, the more similar the two classes are, and therefore the easier they are to confuse. N j For the first j Number of samples in a class (number of samples of that class in the training set); N i For the first i The number of samples in a class.

[0147] New Focal Loss: It adds class-specific penalty weights to the standard Focal Loss. . Focal Loss function for a specific category. The model's predicted probability for the true class. In classification problems, for the true class, the model outputs a probability. This represents the probability. The first one calculated above i Class penalty weights. γ: A modulating factor in Focal Loss, used to adjust the weights of easy and difficult samples. Typically, γ > 0; as γ increases, the model pays more attention to difficult-to-classify samples. (1- This is the core part of Focal Loss, used to reduce the loss weight of easily classified samples, so that the model can focus more on difficult-to-classify samples. : Standard cross-entropy loss component.

[0148] For easily confused categories such as "steam vs. industrial emissions", the recall rate increased by 9.6%.

[0149] IV. Multimodal Data Fusion

[0150] 1. Feature-level fusion:

[0151] This embodiment proposes a multimodal feature fusion method based on scene-aware dynamic weighting, building upon traditional feature splicing, to address the problem of insufficient robustness of static fusion in complex environments.

[0152] (1) Enhanced extraction of infrared spectral features

[0153] This embodiment introduces a spectral change trend encoder to perform first-order difference + wavelet transform on the CO2 and CO concentration sequences, extracting the Transient Mutation Index (TMI) and constructing a 6-dimensional spectral feature vector:

[0154] Let the CO2 concentration sequence be... C ( t ), t = 1,2,...,T, then the first-order difference Δ C ( t )= C ( t )- C ( t- 1) Represents the change in concentration values ​​between adjacent time points, reflecting the instantaneous rate of change of concentration; wavelet high-frequency coefficient energy. ,in d j For the db4 wavelet decomposition of the th j Layer detail factor, J This represents the total number of layers in the db4 wavelet decomposition. It represents a high-energy value.

[0155] Transient mutation index: TMI: A quantification index of the intensity of abrupt changes or spikes in a spectral signal; a higher value indicates a more pronounced abrupt change in the signal. Δ C ( t ): The first difference of concentration change represents the change in concentration between adjacent time points, reflecting the instantaneous rate of change of concentration. The final 6-dimensional spectral eigenvector is: .

[0156] in, f s A composite index is a set of spectral characteristics that combine the spectral features of carbon dioxide and carbon monoxide, along with an additional index or measurement, used to describe or analyze certain spectral data. μ CO2 Mean value of the spectral characteristics of carbon dioxide (CO2) refers to the average reflectance or absorbance of the spectral bands associated with carbon dioxide in spectral analysis. σ CO2 : Standard deviation of carbon dioxide spectral characteristics, which indicates the degree of dispersion or variation of carbon dioxide-related spectral band data. CO2 The slope of the carbon dioxide spectral characteristics refers to the slope of the carbon dioxide characteristics as a function of wavelength within a specific wavelength range, reflecting its changing trend. μ CO The mean of the spectral characteristics of carbon monoxide (CO), similar to that of carbon dioxide, refers to the average reflectance or absorptivity of the spectral bands associated with carbon monoxide. σ CO The standard deviation of the spectral characteristics of carbon monoxide represents the degree of dispersion of carbon monoxide-related spectral band data.

[0157] In the scenario of "initial industrial emissions", TMI can detect a sudden increase in CO2 15 seconds in advance, which improves the recall rate by 12.4% compared to the original 4-dimensional features.

[0158] (2) Dynamic normalization of environmental parameter characteristics

[0159] This embodiment proposes scene adaptive normalization:

[0160] A predefined set of scene labels: {sunny, rainy, foggy, night, windy}; maintaining sliding statistics (mean μ_scene, standard deviation σ_scene) for each scene, updated online.

[0161]

[0162] in, This is the scene average at the current time step t; For the previous time step t The average value of scenario 1; The smoothing coefficient determines the degree of influence of the previous time step's mean on the current mean; here, it is set to 0.95. These are the observations at the current time step t. The mean for each scene is updated online using exponential smoothing.

[0163] Environmental feature normalization yields a 3D environmental feature vector:

[0164]

[0165] Where T, H, and W represent temperature, humidity, and wind speed, respectively, in the environmental characteristics. , , These are the average values ​​of temperature, humidity, and wind speed in the current scenario. , , These represent the standard deviations of temperature, humidity, and wind speed in the current scenario. The formula normalizes environmental characteristics to have zero mean and unit variance, thus overcoming the characteristic differences under different scenarios.

[0166] The problem of "low nighttime temperatures being misjudged as abnormal" has been overcome, resulting in a 18% reduction in the false alarm rate.

[0167] (3) Adaptive dimensionality reduction and activation of visual features

[0168] The original visual features are 512-dimensional, which is redundant. This embodiment introduces selective pooling based on sparse activation:

[0169] For the 7×7×512 feature map output by EfficientNet, calculate the channel importance score:

[0170]

[0171] Where s i is the average absolute value of the i-th channel, representing the average of the absolute values ​​of all pixels in a feature map; h and w represent the height and width of the feature map, respectively, and i represents the number of channels in the feature map. Represents the value of the feature map. Reserved. The first 256 channels from high to low are selected, and the rest are set to zero. After global average pooling, a 256-dimensional visual feature vector is obtained. .

[0172] The computational load was reduced by 48%, but the accuracy was improved by 1.7% due to the suppression of background interference channels.

[0173] 2. Fusion Strategy: Weighted Fusion Based on Temporal Consistency Constraints

[0174] This embodiment proposes dynamic fusion of timing consistency constraints:

[0175] Maintain the expected value of the fused features from the previous frame Calculate the temporal consistency score for each modality in the current frame:

[0176] ;

[0177] Among them, consistent m For modality m The time sequence consistency score; For the fusion features at time step t-1, For modality m The characteristics at the current time step t; For modality m The feature transformation function is used to transform the features of the current frame. Transform to merge features with the previous frame Same feature space; These represent visual features, spectral features, and environmental features, respectively.

[0178] Calculate the fusion weights for each modality:

[0179]

[0180] The final fused feature vector is obtained as follows:

[0181]

[0182] in This is a smoothing parameter that controls the smoothness of the weight distribution; it is set to 2.0 here. For modality m The indexed consistency score. For the number of modes, The sum of the indexed consistency scores for all modalities is used to normalize the weights. The fusion features at the current time step t, Visual feature vectors The fusion weight, Spectral eigenvectors The fusion weight, Environmental feature vector The fusion weight.

[0183] For the first time, temporal consistency constraints were introduced into multimodal smoke recognition, reducing prediction jitter by 63%, which is particularly suitable for continuous video monitoring.

[0184] V. Classification and Early Warning Mechanism

[0185] Smoke type identification: The model outputs smoke type (such as "low risk of industrial emissions" or "high risk of fire smoke") and confidence level based on the fused feature vector, and supports multi-label classification (can identify multiple pollution sources at the same time).

[0186] Multi-level early warning system:

[0187] Level 1 Warning (Fire Smoke): When fire smoke (CO2 concentration ≥ 500 ppm, temperature gradient ≥ 2℃ / s) has a confidence level ≥ 90%, fire linkage is triggered, location information is pushed to the emergency command center, and drones are initiated for tracking.

[0188] Level 2 warning (industrial illegal emissions): When the CO2 concentration is 200-500 ppm and there is no significant temperature change, with a confidence level of ≥75%, the warning will be sent to the environmental regulatory department and recorded in the database.

[0189] Level 3 Warning (Residential Smoke): When the CO2 concentration is ≤100ppm and the humidity is ≥80%RH, with a confidence level of ≥60%, it is marked as "re-inspection required" and only community management personnel are notified to verify.

[0190] Resource optimization results: Through precise smoke type identification and graded response mechanisms, it is expected that the number of ineffective fire dispatches will be reduced by more than 80%, significantly reducing social governance costs.

[0191] The key technical points of the embodiments of this application include:

[0192] 1. Application of high-definition infrared thermal imaging cameras. Replacing traditional visible light cameras, they effectively solve the problem of blurry imaging in low-light environments such as nighttime or cloudy days, and improve smoke penetration by more than 3 times. They can ensure stable acquisition of smoke thermal radiation characteristics in all weather conditions (day and night, sunny or rainy), providing high-quality input data for subsequent algorithm analysis and fundamentally solving the problem of "not being able to see" the smoke.

[0193] 2. Multimodal data fusion technology. This innovative technology integrates visual texture, infrared spectroscopy (CO2 / CO concentration), and environmental parameters (temperature, humidity, wind speed) to construct a 519-dimensional joint feature vector, improving classification accuracy by 27% compared to single-modal solutions. It overcomes misjudgments caused by insufficient data dimensions from a single sensor and adapts to the diverse smoke sources (such as the mixture of chemical leaks and natural fog) in complex urban scenarios.

[0194] 3. Deep Learning Classification Model Design. A pioneering CNNTransformer fusion architecture integrates a ResNet50 spatial feature extraction module and a multi-head self-attention (MHSA) temporal modeling module, improving small target detection capability by 40%. It overcomes the limitations of traditional pre-defined rule models in adapting to dynamic scenes, achieving refined classification of seven typical urban smoke types (fire / industrial emissions / restaurant fumes, etc.).

[0195] 4. Dynamic correlation mechanism between smoke type and warning level. A three-level mapping model of smoke type hazard level and response level is constructed, supporting automatic adjustment of warning level based on parameters such as CO2 concentration (500ppm threshold) and temperature gradient (2℃ / s). It can convert technical identification results into executable decision instructions, avoid misallocation of fire-fighting resources, reduce the invalid alarm rate by more than 80%, and significantly improve the efficiency of public safety response.

[0196] The embodiments of this application have the following advantages:

[0197] 1. Improved environmental adaptability resulting from hardware upgrades. This application's embodiments use an uncooled long-wave infrared camera (814μm) to replace traditional visible light equipment, and integrate an infrared spectral sensor, temperature and humidity sensor, and GPS module. Existing technologies rely on visible light imaging, which suffers a sharp decline in image quality under low light conditions or complex weather conditions such as nighttime, fog, or rain (experimental data shows a nighttime detection rate of only 32%), leading to failure in smoke feature extraction.

[0198] The infrared thermal imaging of this embodiment has strong smoke penetration capability and is not limited by visible light. The detection rate is increased to 89% under low light conditions, enabling 24-hour uninterrupted inspection. The infrared band has better penetration through rain, snow, and fog than visible light, reducing missed detections caused by environmental factors and eliminating the defect of "detectable during the day, but ineffective at night." Hardware upgrades fundamentally solve the problem of poor environmental adaptability, laying the foundation for high-reliability detection.

[0199] 2. Multimodal data fusion improves classification accuracy. This application's embodiments fuse visual features (CNN extraction), infrared spectral data (CO2 / CO concentration), and environmental parameters (temperature, humidity, wind speed) to construct a joint feature vector and make decisions through weighted voting. Existing technologies rely solely on visual features and preset rule models, making it difficult to distinguish between smoke types with similar morphologies (such as industrial steam and fire smoke), resulting in a misclassification rate as high as 20%.

[0200] The infrared spectrum of this application provides chemical characteristics (such as CO2 concentration > 500 ppm indicating a fire), and environmental parameters provide physical characteristics (such as wind speed affecting the diffusion path), eliminating ambiguity caused by insufficient data from a single sensor. It can distinguish at least five types of targets, including fire smoke, industrial emissions, and steam, with an accuracy rate of 94.3%. Experiments show that the misclassification rate of the multimodal model is reduced to 3%. The multimodal fusion technology overcomes the limitation of "misclassification due to similar morphology," achieving high-precision classification and dynamic scene adaptation.

[0201] 3. Deep learning model optimization reduces false positive rate. This application's embodiments employ a CNNTransformer fusion architecture, introducing FocalLoss to address class imbalance and optimizing the training strategy. Existing technologies rely on pre-defined rule models (such as "smoke accompanied by flame"), which have weak identification capabilities for novel or mixed smoke types (such as chemical leaks + fog), resulting in a false positive rate ≥20%.

[0202] This application's embodiments extract spatial features (such as smoke texture) using CNN and capture temporal dynamics (such as diffusion trends) using Transformer, and the fusion enhances the modeling ability for complex scenes. Focus loss suppresses the contribution of easily classified samples (such as common steam) and strengthens the learning of low-frequency, high-risk samples (such as fire smoke), reducing the false alarm rate to below 5% in actual tests. Algorithm optimization directly addresses the "rule rigidity" problem, achieving a balance between low false alarm rate and high generalization ability.

[0203] 4. Dynamic early warning mechanism optimizes resource allocation. This application establishes a correlation mechanism between smoke type and early warning level, triggering emergency response only for high-risk smoke (such as fire smoke), while low-risk smoke (such as steam) is only recorded without being pushed out. Existing technologies only output a binary result (smoke present / absent), leading to frequent fire response triggers from non-hazardous smoke (such as boiler steam), resulting in a waste of public resources (experiments show a resource waste rate ≥30%).

[0204] This application's embodiments dynamically adjust the process based on smoke type and risk level (Level 1 to Level 3). High-risk events (such as fire smoke) are immediately pushed to the fire department, while low-risk events (such as steam) are only recorded. The warning information includes structured data such as type, confidence level, and environmental parameters to assist in rapid judgment. In actual testing, the resource waste rate was reduced to below 5%, and fire response efficiency was improved by 40%. The dynamic warning mechanism transforms technical output into practical decision-making value, solving the core pain point of "warning equals waste."

[0205] 5. System scalability and economy. The hardware in this application embodiment supports sensor replacement (e.g., the infrared camera can be upgraded to a mid-wave infrared model), and the algorithm architecture is compatible with lightweight deployment (e.g., replacement with EfficientNetB4+LSTM). Existing technical solutions are highly closed and difficult to adapt to different scenario requirements (e.g., high-temperature industrial environments or devices with limited computing power).

[0206] The sensor kit in this application can be added or removed as needed (e.g., the basic version only retains the infrared camera and GPS module to reduce costs), keeping hardware costs under control. It provides multiple deployment options from edge computing to the cloud, reducing computing power requirements by 50% and supporting multi-platform operation. The system design balances high performance and economy, and can be applied to various scenarios such as cities, forests, and industrial areas, possessing the potential for large-scale deployment.

[0207] Corresponding to the smoke recognition and classification warning method described in the above embodiments, this application also provides a smoke recognition and classification warning system, such as... Figure 2 As shown, the smoke recognition and classification early warning system includes:

[0208] The equipment is equipped with modules for mounting infrared cameras, infrared spectral sensors, and temperature and humidity sensors on drones to conduct smoke inspections of designated target areas.

[0209] The data acquisition module is used to collect infrared visual images, infrared spectral data, and environmental parameters of the target area using a drone, and to preprocess the collected data; the infrared spectral data includes the concentrations of CO2 and CO, and the environmental parameters include temperature, humidity, and wind speed;

[0210] The smoke recognition module is used to perform multimodal data fusion on the preprocessed data to construct a fusion feature vector, and to use a trained deep learning classification model to identify the smoke type based on the fusion feature vector;

[0211] The classification and early warning module is used to determine the early warning level based on the identified smoke type, and dynamically trigger the early warning based on the early warning level and the set graded response rules.

[0212] It should be noted that the information interaction and execution process between the above modules / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0213] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0214] This application also provides an electronic device, such as... Figure 3 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the smoke recognition and classification warning method provided in the first aspect.

[0215] In applications, electronic devices may include, but are not limited to, processors and memory. Figure 3 This is merely an example of an electronic device and does not constitute a limitation on the device. It may include more or fewer components than illustrated, or combinations of certain components, or different components, such as input / output devices, network access devices, etc. Input / output devices may include cameras, audio capture / playback devices, displays, etc. Network access devices may include network modules for wireless network communication with external devices.

[0216] In applications, the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0217] In applications, memory can be an internal storage unit of an electronic device in some embodiments, such as a hard drive or RAM. In other embodiments, memory can be an external storage device of the electronic device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Memory can also include both internal and external storage units of the electronic device. Memory is used to store operating systems, applications, bootloaders, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0218] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0219] This application implements all or part of the processes in the methods of the above embodiments, which can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can at least include: any entity or device capable of carrying the computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0220] Those skilled in the art will recognize that the device and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0221] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interface, or the device may be indirectly coupled or communicated, and may be electrical, mechanical, or other forms.

[0222] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A smoke recognition and classification early warning method, characterized in that, include: Equipped with infrared cameras, infrared spectral sensors, and temperature and humidity sensors, drones are used to conduct smoke inspections of designated target areas. The system utilizes drones to collect infrared visual images, infrared spectral data, and environmental parameters of the target area, and preprocesses the collected data. The infrared spectral data includes the concentrations of CO2 and CO, and the environmental parameters include temperature, humidity, and wind speed. Multimodal data fusion is performed on the preprocessed data to construct a fusion feature vector, and a trained deep learning classification model is used to identify the smoke type based on the fusion feature vector; The warning level is determined based on the identified smoke type, and the warning is dynamically triggered based on the warning level and the set graded response rules. The step of constructing a fused feature vector by performing multimodal data fusion on the preprocessed data includes: First-order difference and wavelet transform were performed on the concentration sequences of CO2 and CO to extract the transient mutation index and construct a 6-dimensional spectral feature vector. Temperature, humidity, and wind speed are adaptively normalized to construct a 3D environmental feature vector. The original visual features are adaptively reduced in dimensionality and activated to construct a 256-dimensional visual feature vector. The 6-dimensional spectral feature vector, the 3-dimensional environmental feature vector, and the 256-dimensional visual feature vector are weighted and fused under time-consistency constraints to obtain a fused feature vector; The step of performing time-consistency-constrained weighted fusion of the 6-dimensional spectral feature vector, the 3-dimensional environmental feature vector, and the 256-dimensional visual feature vector to obtain a fused feature vector includes: Maintain the expected value of the fused features from the previous frame Calculate the temporal consistency score for each modality in the current frame: ； in For the fusion features at time step t-1, For modality m The characteristics at the current time step t, Let m be the characteristic transformation function of mode m. These represent visual features, spectral features, and environmental features, respectively. Calculate the fusion weights for each modality The fused feature vector is obtained as follows: ； in For smoothing parameters, The exponential consistency score for mode m. For the number of modes, The sum of the indexed consistency scores for all modalities. Visual feature vectors The fusion weight, Spectral eigenvectors The fusion weight, Environmental feature vector The fusion weight.

2. The smoke recognition and classification early warning method as described in claim 1, characterized in that, The preprocessing of the collected data includes: An adaptive filtering algorithm is used to suppress stripe noise and random noise in the collected data, and a Gaussian mixture model is combined to separate smoke from the background.

3. The smoke recognition and classification early warning method as described in claim 1, characterized in that, The concentration sequences of CO2 and CO are subjected to first-order difference and wavelet transform to extract transient mutation indexes and construct a 6-dimensional spectral feature vector, including: Let the CO2 concentration sequence be... C ( t ), t = 1,2,...,T, then the first-order difference Δ C ( t )= C ( t )- C ( t- 1) Wavelet high-frequency coefficient energy ,in d j For the db4 wavelet decomposition of the th j Layer detail factor, J This represents the total number of layers in the db4 wavelet decomposition. Calculate the transient mutation index Then the 6-dimensional spectral eigenvector is obtained as follows: ;in The mean value of the CO2 spectral characteristics. The standard deviation of the CO2 spectral characteristics. The slope of the CO2 spectral characteristics. The mean value of the CO spectral characteristics. denoted as the standard deviation of the CO spectral characteristics.

4. The smoke recognition and classification early warning method as described in claim 1, characterized in that, The process involves scene-adaptive normalization of temperature, humidity, and wind speed to construct a 3D environmental feature vector, and adaptive dimensionality reduction and activation of the original visual features to construct a 256-dimensional visual feature vector, including: A predefined set of scene labels {sunny, rainy, foggy, night, strong wind} is used to maintain sliding statistics for each scene, which are updated online. ;in, This represents the scene average at the current time step t. For the previous time step t The average value of scenario 1 For smoothing coefficients, The observation value at the current time step t; Normalizing the environmental features yields a 3D environmental feature vector: ； Where T, H, and W represent temperature, humidity, and wind speed, respectively, in the environmental characteristics. , , These are the average values ​​of temperature, humidity, and wind speed in the current scenario. , , These represent the standard deviations of temperature, humidity, and wind speed in the current scenario. Calculate channel importance scores for 512-dimensional raw visual features. ,reserve The first 256 channels from high to low are selected, and the rest are set to zero. After global average pooling, a 256-dimensional visual feature vector is obtained. Where h and w represent the height and width of the feature map, respectively, and i represents the channel of the feature map. This represents the value of the feature map.

5. The smoke recognition and classification early warning method as described in claim 1, characterized in that, The process of determining the warning level based on the identified smoke type, and dynamically triggering the warning based on the warning level and the set graded response rules, includes: When the smoke type is fire smoke, it is determined as a Level 1 warning, triggering fire linkage, pushing location information to the emergency command center, and starting drone tracking; When the smoke type is industrial illegal emission smoke, it is determined as a Level 2 warning, which is pushed to the environmental protection regulatory department and recorded in the database; When the smoke type is residential smoke, it is determined to be a Level 3 warning, and community management personnel are notified to verify.

6. A smoke recognition and classification early warning system, characterized in that, Used to implement the smoke recognition and classification early warning method as described in any one of claims 1 to 5.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the smoke recognition and classification warning method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the smoke recognition and classification warning method as described in any one of claims 1 to 5.

Images