Forest fire monitoring method and device based on deep learning and feature fusion

Abstract

The invention relates to a forest fire monitoring method and device based on deep learning and feature fusion, and particularly relates to the technical field of computer vision. Comprising the steps of obtaining forest video image data; performing preliminary identification on the preprocessed forest region video image data based on a lightweight deep learning model to obtain suspected forest fire information and target features; wherein the lightweight deep learning model is obtained by compressing and optimizing a convolutional neural network model based on an attention mechanism; and in combination with the suspected forest fire information, carrying out secondary identification on the fused target features so as to detect a fire event and the geographic coordinates of the corresponding ignition point. According to the method, the attention mechanism is introduced, key features of the forest fire flames and the smoke are more concerned, the forest fire, especially the early-stage soft fire or the smoke, can be more accurately recognized, and the forest fire monitoring precision is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a wildfire monitoring method and device based on deep learning and feature fusion. Background Art

[0002] Wildfires are one of the serious natural disasters globally. In recent years, due to frequent extreme droughts, power transmission lines pose a huge threat due to wildfire disasters. Traditional wildfire monitoring methods, such as manual observation from watchtowers, satellite remote sensing, etc., have problems such as poor real-time performance, low accuracy, and being easily restricted by weather conditions. In recent years, wildfire monitoring technology based on ground cameras has developed rapidly, but there are still the following deficiencies: ① High false alarm rate: Traditional image processing methods have poor adaptability to complex environments and are prone to misjudging clouds, light changes, etc. as fires; ② Weak early fire recognition ability: For early small fires or smoke, the recognition rate of traditional methods is relatively low; ③ Difficult model deployment: High-precision deep learning models usually have large computational requirements and are difficult to deploy on resource-constrained edge devices; ④ Limitations of single-feature recognition: Relying solely on a single color or texture feature for recognition is easily interfered with and has poor robustness.

[0003] Therefore, it is of great significance to develop an intelligent ground camera wildfire monitoring system and method with high precision, high efficiency, and low false alarm rate. Summary of the Invention

[0004] To solve the above technical problems or at least partially solve the above technical problems, this application provides a wildfire monitoring method and device based on deep learning and feature fusion, which can more accurately identify wildfires, especially early small fires or smoke, and improve the accuracy of wildfire monitoring.

[0005] To achieve the above object, the technical solutions provided by the embodiments of this application are as follows:

[0006] In a first aspect, this application provides a wildfire monitoring method based on deep learning and feature fusion, including: obtaining forest area video image data; performing preliminary recognition on the preprocessed forest area video image data based on a lightweight deep learning model to obtain suspected wildfire information and target features; wherein, the lightweight deep learning model is obtained after compression and optimization of a convolutional neural network model based on an attention mechanism; combining the suspected wildfire information, performing secondary recognition on the fused target features to detect fire events and the corresponding geographical coordinates of the ignition points.

[0007] As an optional implementation manner provided in the embodiments of this application, the convolutional neural network model based on an attention mechanism is the MobileNetV3 model pre-trained on ImageNet; the inverted residual module of the MobileNetV3 model includes a convolutional block attention module.

[0008] As an alternative implementation provided in the embodiments of the present application, the training process of the lightweight deep learning model includes the following steps: fine-tuning a convolutional neural network model based on an attention mechanism using a wildfire image dataset to optimize the model parameters so that the model adapts to the fire recognition task; compressing the fine-tuned convolutional neural network model based on the attention mechanism to obtain an intermediate model; evaluating the performance metrics of the intermediate model on a test set and adjusting the compression parameters until the performance metrics meet the preset performance metrics to obtain a lightweight deep learning model.

[0009] As an alternative implementation provided in the embodiments of the present application, the target features include color features, texture features, motion features, and deep learning features; among them, the color features include statistical information in the RGB color space and statistical information in the HSV color space; the texture features include local binary pattern texture descriptors and gray-level co-occurrence matrix texture descriptors; the motion features include optical flow features and motion pattern features; the deep learning features include semantic level features and spatio-temporal correlation features.

[0010] In a second aspect, the present application provides a wildfire monitoring device based on deep learning and feature fusion. The device includes:

[0011] An image acquisition module for acquiring forest area video image data;

[0012] A preliminary recognition module for preliminarily recognizing the preprocessed forest area video image data based on a lightweight deep learning model to obtain suspected wildfire information and target features; among them, the lightweight deep learning model is obtained after compressing and optimizing a convolutional neural network model based on an attention mechanism.

[0013] A secondary recognition module for secondarily recognizing the fused target features in combination with the suspected wildfire information to detect fire events and the corresponding geographical coordinates of the ignition points.

[0014] As an alternative implementation provided in the embodiments of the present application, the device further includes a training module for training the lightweight deep learning model, including:

[0015] A fine-tuning unit for fine-tuning a convolutional neural network model based on an attention mechanism using a wildfire image dataset to optimize the model parameters so that the model adapts to the fire recognition task;

[0016] A compression unit for compressing the fine-tuned convolutional neural network model based on the attention mechanism to obtain an intermediate model;

[0017] An adjustment unit for evaluating the performance metrics of the intermediate model on a test set and adjusting the compression parameters until the performance metrics meet the preset performance metrics to obtain a lightweight deep learning model.

[0018] In a third aspect, the present application provides a wildfire monitoring system based on deep learning and feature fusion, and the system includes:

[0019] A front-end camera for collecting forest area video image data;

[0020] And a wildfire monitoring device based on deep learning and feature fusion as described in the first aspect or any one of its optional implementation manners, so as to implement the wildfire monitoring method based on deep learning and feature fusion as described in the first aspect or any one of its optional implementation manners.

[0021] In a fourth aspect, the present application provides an electronic device, including: a processor, a memory, and a computer program stored on the memory and executable on the processor, and when the computer program is executed by the processor, it implements the wildfire monitoring method based on deep learning and feature fusion as described in the first aspect or any one of its optional implementation manners.

[0022] In a fifth aspect, the present application provides a computer-readable storage medium, including: a computer program stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the wildfire monitoring method based on deep learning and feature fusion as described in the first aspect or any one of its optional implementation manners.

[0023] In a sixth aspect, the present application provides a computer program product, including: the computer program product includes a computer program, and when the computer program runs on a computer, it causes the computer to implement the wildfire monitoring method based on deep learning and feature fusion as described in the first aspect or any one of its optional implementation manners.

[0024] The technical solutions provided by the embodiments of the present application have the following advantages compared with the prior art:

[0025] The embodiments of the present disclosure provide a wildfire monitoring method and device based on deep learning and feature fusion. The method first obtains forest area video image data; then performs preliminary identification on the preprocessed forest area video image data based on a lightweight deep learning model to obtain suspected wildfire information and target features; wherein, the lightweight deep learning model is obtained after compression and optimization of a convolutional neural network model based on an attention mechanism; further, in combination with the suspected wildfire information, secondary identification is performed on the fused target features to detect a fire event and the corresponding geographical coordinates of the ignition point. In this way, the present application introduces an attention mechanism, pays more attention to the key features of wildfire flames and smoke, and thus can more accurately identify wildfires, especially early small fires or smoke, improving the accuracy of wildfire monitoring. Description of the Drawings

[0026] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings required for use in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0028] Figure 1 It is a schematic flowchart of a wildfire monitoring method based on deep learning and feature fusion provided by an embodiment of this application;

[0029] Figure 2 It is a schematic structural diagram of a wildfire monitoring device based on deep learning and feature fusion provided by an embodiment of this application;

[0030] Figure 3 It is a schematic structural diagram of an electronic device described in an embodiment of this application. Detailed implementation manners

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the technical terms required for use in the description of the embodiments or the prior art will be briefly introduced below.

[0032] The RGB color model is an additive color model that represents various colors by mixing three primary colors, red, green, and blue, in different proportions. In this space, the color of each pixel is determined by the values of these three color channels, and the value range is usually from 0 to 255.

[0033] The HSV color model represents colors as three parameters: hue, saturation, and value. Hue represents the type of color, and the value range is usually from 0 to 360 degrees; saturation represents the vividness of the color, and the value range is from 0 to 1; value represents the brightness of the color, and the value range is also from 0 to 1. This color space is more in line with the human perception of colors and is convenient for extracting and analyzing color features.

[0034] Local Binary Pattern (LBP) is a texture description method based on local regions. Its basic idea is to generate a binary code by comparing the gray values of the central pixel and the neighboring pixels to describe local texture features.

[0035] The Gray Level Co-occurrence Matrix (GLCM) is a method for describing texture features by statistically counting the frequencies of gray-level combinations of two pixels with specific spatial relationships in an image. It takes into account the gray values of the pixels in the image and their relative positional relationships.

[0036] To more clearly understand the above objects, features, and advantages of the present application, the solutions of the present application will be further described below. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.

[0037] In the following description, many specific details are set forth in order to fully understand the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present application, rather than all of the embodiments.

[0038] To solve some or all of the technical problems existing in the related art, an embodiment of the present application provides a wildfire monitoring method and device based on deep learning and feature fusion. The method first obtains forest area video image data; then, based on a lightweight deep learning model, it performs preliminary identification on the preprocessed forest area video image data to obtain suspected wildfire information and target features; among them, the lightweight deep learning model is obtained after compression and optimization of a convolutional neural network model based on the attention mechanism; further, in combination with the suspected wildfire information, it performs secondary identification on the fused target features to detect fire events and the corresponding geographical coordinates of the ignition points. In this way, the present application introduces the attention mechanism, pays more attention to the key features of wildfire flames and smoke, and thus can more accurately identify wildfires, especially early small fires or smoke, improving the accuracy of wildfire monitoring.

[0039] The wildfire monitoring method based on deep learning and feature fusion provided in the embodiment of the present application can be implemented by a wildfire monitoring device or electronic device based on deep learning and feature fusion. The electronic device includes, but is not limited to, a personal computer, a laptop computer, a tablet computer, a smart phone, etc. The operating system of the electronic device may include Android, iOS developed by Apple Inc., Windows developed by Microsoft Corporation in the United States, etc., and the embodiments of the present application do not limit this. The electronic device can run independently to implement the present application, or can be connected to the network and implement the present application through interactive operations with other computer devices in the network. Among them, the network where the electronic device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a Virtual Private Network (VPN) network, etc.

[0040] It should be noted that the protection scope of the wildfire monitoring method based on deep learning and feature fusion described in the embodiments of the present application is not limited to the execution order of the steps listed in this embodiment. Any solution achieved by adding or reducing steps of the prior art and replacing steps according to the principle of the present application is included in the protection scope of the present application.

[0041] As Figure 1 shown, Figure 1 is a schematic flowchart of a wildfire monitoring method based on deep learning and feature fusion according to an embodiment of the present application. This method can be executed by a wildfire monitoring device based on deep learning and feature fusion, where the device can be implemented by software and / or hardware and is generally integrated in an electronic device. This method mainly includes the following steps S101 to S103:

[0042] S101. Obtain forest area video image data.

[0043] In some embodiments, the forest area video image data is collected in real time by multiple high-definition cameras deployed at key positions in the forest area (such as the entrance of the forest area, flammable areas, high-risk areas, etc.).

[0044] After performing step S101, preprocessing such as image enhancement, denoising, image cropping and normalization is performed on the forest area video image data. Optionally, means such as histogram equalization are used to enhance the contrast of the image, making details such as vegetation and terrain in the forest area clearer and facilitating the identification of possible wildfire signs. Filtering methods are adopted, such as median filtering to remove salt-and-pepper noise and Gaussian filtering to reduce Gaussian noise, to improve the image quality. The image is cropped to remove unnecessary edge parts, and at the same time, normalization processing is performed on image pixels, etc., to unify the image size and data format.

[0045] S102. Based on a lightweight deep learning model, perform preliminary identification on the preprocessed forest area video image data to obtain suspected wildfire information and target features.

[0046] Among them, the lightweight deep learning model is obtained after compression and optimization of a convolutional neural network model based on an attention mechanism. The lightweight deep learning model can run on an edge computing device, and the edge computing device is composed of multiple embedded computing devices and is connected to the aforementioned high-definition cameras. The application of the edge computing layer realizes real-time fire warning, which wins precious time for fire fighting and rescue. The lightweight model reduces the requirements for hardware devices and reduces the system deployment and maintenance costs.

[0047] In some embodiments, the convolutional neural network model based on the attention mechanism can be the MobileNetV3 model pre-trained on ImageNet. MobileNetV3 is a lightweight convolutional neural network proposed by Google and belongs to the MobileNet series. The MobileNetV3 model can decompose the standard convolution into depthwise convolution and pointwise convolution, reducing the computational amount.

[0048] The depthwise convolution is shown in formula (1):

[0049]

[0050] In formula (1), O i,j,k is the value of the output feature map at position (i, j) and channel k. I i+m-1,j+m-1,k is the value of the input feature map at position (i + m - 1, j + m - 1) and channel k, and K m,m,k is the value of the convolution kernel in m rows and m columns and channel k. M is the size of the convolution kernel, and * represents the convolution operation.

[0051] The pointwise convolution is shown in formula (2):

[0052]

[0053] In formula (2), O i,j,l is the value of the output feature map at position (i, j) and channel l. I i,j,k is the value of the input feature map at position (i, j) and channel k, and W 1,1,k,l is the value of the convolution kernel in k rows and l columns. K is the number of channels of the input feature map. The pointwise convolution is usually used to implement the linear combination of feature maps, which is equivalent to performing a linear transformation at each pixel point.

[0054] In some embodiments, the inverted residual module of the MobileNetV3 model includes a Convolutional Block Attention Module (CBAM). In this application, the CBAM module is used to enhance the attention of the MobileNetV3 model to the features of flames and smoke.

[0055] The channel attention of the CBAM module learns the channel weights through global average pooling, max pooling, and two multi-layer perceptrons.

[0056] M c = σ(MLP(AvgPool(F)) + MLP(MaxPool(F))) (3)

[0057] In formula (3), M cis the output, F is the input feature map, AvgPool is the average pooling layer, MaxPool is the maximum pooling layer, MLP is the multi-layer perceptron, and σ is the sigmoid activation function. The insertion position and number of CBAM modules can be determined according to experimental results, and this application does not specifically limit it.

[0058] In some embodiments, the training process of the lightweight deep learning model includes the following steps: using a wildfire image dataset to fine-tune the convolutional neural network model based on the attention mechanism, and optimizing the model parameters so that the model can adapt to the fire identification task; compressing the fine-tuned convolutional neural network model based on the attention mechanism to obtain an intermediate model; evaluating the performance indicators of the intermediate model on the test set, adjusting the compression parameters until the performance indicators meet the preset performance indicators, and obtaining a lightweight deep learning model.

[0059] In some embodiments, the wildfire image dataset is labeled training data, including known wildfire images and normal forest area images.

[0060] In some embodiments, pruning, quantization and other techniques are used to compress the fine-tuned attention-based convolutional neural network model.

[0061] Pruning is based on the sparsity principle of neural networks. It is believed that some connections or parameters in the network contribute little to the performance of the model. By removing these unimportant connections or parameters, the model is compressed. An optional implementation provided in the embodiment of the present application is to remove unimportant connections or parameters according to the set pruning strategy and threshold to obtain a pruned model. For example, the convolution kernel connection that contributes little to the classification is pruned.

[0062] Quantization is to convert the original high-precision data types (such as 32-bit floating point numbers) in the model into low-precision data types (such as 8-bit integers) for storage and calculation, reducing the storage and calculation costs by reducing the representation accuracy of the data. An optional implementation provided in the embodiment of the present application is to quantize the pruned model and convert the parameters and calculations in the model from high-precision data types to low-precision data types.

[0063] Optionally, after pruning and quantizing the intermediate model, use a small amount of data set to fine-tune the intermediate model to adapt the model to low-precision data representation and restore and improve the performance of the model. Evaluate the performance indicators of the intermediate model on the test set to check whether the model's accuracy, recall rate, F1 value and other indicators meet the requirements. If not, adjust the pruning and quantization parameters and repeat the above steps until satisfactory performance is achieved.

[0064] In some embodiments, feature extraction methods, model parameters, etc. can be continuously optimized and adjusted according to the recognition accuracy, misjudgment rate, etc. in actual applications to continuously improve the ability to identify suspected wildfire information.

[0065] In the embodiments of this application, a converged lightweight deep learning model is applied, which can process forest area video image data, identify suspected wildfire information, and extract target features, including color features, texture features, and motion features. When the color features, texture features, motion features, and deep learning features simultaneously meet certain wildfire feature conditions, and the probability of suspected wildfire output by the model exceeds a certain threshold, it is determined that suspected wildfire information is found.

[0066] Optionally, a feature-level fusion strategy is adopted, and weighted fusion is used to fuse the target features. Different weights are assigned to them according to the importance of color features, texture features, motion features, and deep learning features in wildfire recognition. For example, through experimental analysis, it is found that color features are relatively important in wildfire recognition and can be assigned a higher weight, such as 0.4; texture features are the second, with a weight of 0.3; the weight of motion features is 0.15, and the weight of deep learning features is 0.15. Then, each feature is multiplied by its corresponding weight and added together to obtain the fused feature. This application fuses color, texture, motion, and deep learning features, improving the accuracy and robustness of fire recognition.

[0067] Among them, color features include statistical information in RGB and HSV color spaces, texture features include LBP and GLCM texture descriptors, motion features include optical flow features and motion pattern features; deep learning features include semantic level features and spatio-temporal correlation features.

[0068] (1) Color features

[0069] The statistical information in the RGB color space includes pixel mean, pixel variance, color value frequency, etc. Optionally, analyze the statistical information of channels related to the colors of flames and smoke, such as red, orange, etc. in the image, such as calculating the pixel proportion, mean, etc. of a specific color area. Flames usually present warm colors such as red and orange, and smoke may be grayish-white.

[0070] The statistical information in the HSV color space includes hue mean, hue variance, saturation mean, saturation variance, etc. Optionally, focus on the hue feature because flames and smoke have their unique manifestations in hue, and relevant data of saturation and lightness can also be statistically analyzed. For example, the saturation of flames is relatively high and the lightness is brighter. Through these features, it is judged whether there is suspected wildfire.

[0071] (2) Texture features

[0072] For each pixel in the video image data of the forest area, the lightweight deep learning model compares the gray value of its neighboring pixels with the gray value of the central pixel. If the gray value of the neighboring pixel is greater than or equal to the gray value of the central pixel, it is recorded as 1; otherwise, it is recorded as 0. These comparison results are combined into a binary number in a certain order and then converted into a decimal number, which is used as the LBP value of the central pixel. The texture of the smoke area generated by wildfires often differs from that of normal forest vegetation. The LBP feature can assist in distinguishing them. For example, the LBP texture of the smoke area may be more blurred and uniform.

[0073] For a given image, specified directions (such as horizontal, vertical, diagonal, etc.) and distances, the lightweight deep learning model counts the number of times pixels with gray value i and pixels with gray value j appear simultaneously in that direction and distance. These statistical results are combined into a matrix, which is the GLCM. By constructing the gray-level co-occurrence matrix and analyzing the frequency of pixel gray combinations in different directions and distances, for areas related to wildfires, the characteristics such as the periodicity and directionality of the texture will change, and thus the texture manifestations suspected of wildfires can be identified.

[0074] (3) Motion features

[0075] Motion features are key information for analyzing the motion state of objects in dynamic images such as videos and can be the motion speed and direction of pixel points. Optical flow refers to the motion speed and direction of pixel points of an object in an image between consecutive frames, reflecting the motion of the object on the image plane. The optical flow method is a method for obtaining motion features by calculating the optical flow of pixel points in an image. The optical flow method includes, but is not limited to, sparse optical flow method and dense optical flow method. By using the optical flow method to calculate the motion speed and direction of pixel points in an image, the smoke generated by wildfires usually drifts with the wind, and the flame also has dynamic changes such as swaying. By detecting these areas with specific motion features, it can assist in judging whether there is a wildfire situation.

[0076] Motion features can also include motion pattern features. The flame and smoke of wildfires have specific motion patterns, such as the flickering, jumping and spreading patterns of the flame, and the floating and spreading patterns of the smoke. Analyzing these motion pattern features helps to distinguish wildfires from other objects or phenomena with similar motions.

[0077] (4) Deep learning features

[0078] Deep learning features include semantic level features and spatiotemporal correlation features. Semantic level features include object category features, such as flame and smoke features, and can also identify flammable objects in forest areas, such as dry trees, fallen leaves and other targets. The presence of these flammable objects may increase the risk of wildfires. Semantic level features also include scene semantic features. For example, in clear and dry weather, the probability of wildfires is relatively high, and weather conditions can be analyzed through images. Topographic features can also be extracted, such as mountains, valleys and other terrains, which may affect the spread direction and speed of wildfires.

[0079] The moment-to-moment correlation features include time series features, such as the drifting direction and speed of smoke or flames, the flickering and spreading patterns of flames, etc. They also include spatial context features, such as the positional relationship between flames, smoke, and surrounding vegetation and the sky, which helps to more accurately identify and locate abnormal situations.

[0080] The above embodiment introduces an attention mechanism, which enables the model to pay more attention to the key features of flames and smoke, and can more accurately identify fires, especially small fires and smoke in the early stages, thereby reducing the waste of human and material resources.

[0081] S103. Combined with the suspected wildfire information, secondary recognition is performed on the fused target features to detect the fire event and the corresponding geographic coordinates of the fire point.

[0082] In some embodiments, a high-performance service cluster can perform secondary identification based on suspected wildfire information uploaded by edge computing devices and fusion features. Combined with the geographic location information and calibration parameters of the camera, the geometric positioning algorithm is used to calculate the geographic coordinates of the fire point.

[0083] In some embodiments, fire warning information is released, and the system sends the confirmed fire events and the corresponding geographical coordinates of the fire points to forest area managers, fire departments and other relevant personnel through SMS, APP push, system messages, sound and light alarms, etc. The location of suspected wildfires can be accurately located on the map, making it convenient to take timely response measures.

[0084] In summary, the present application provides a fusion of color, texture, motion and deep learning features, which improves the accuracy and robustness of fire identification. At the same time, the attention mechanism is introduced to make the model pay more attention to the key features of flames and smoke, and can more accurately identify fires, especially small fires and smoke in the early stages, reducing the waste of manpower and material resources. The application of the edge computing layer realizes real-time fire warning, which buys precious time for fire fighting and rescue. The lightweight model reduces the requirements for hardware equipment and reduces the system deployment and maintenance costs.

[0085] like Figure 2 As shown, Figure 2The structural schematic diagram of a wildfire monitoring device based on deep learning and feature fusion provided by an embodiment of this application. The device includes:

[0086] An image acquisition module 201, which acquires forest area video image data;

[0087] A preliminary recognition module 202, which is used to perform preliminary recognition on the preprocessed forest area video image data based on a lightweight deep learning model to obtain suspected wildfire information and target features; among them, the lightweight deep learning model is obtained after compression and optimization of a convolutional neural network model based on an attention mechanism;

[0088] A secondary recognition module 203, which is used to perform secondary recognition on the fused target features in combination with the suspected wildfire information to detect fire events and the corresponding geographical coordinates of the ignition points.

[0089] As an optional implementation manner provided in the embodiment of this application, the convolutional neural network model based on an attention mechanism is the MobileNetV3 model pre-trained on ImageNet; the inverted residual module of the MobileNetV3 model includes a convolutional block attention module.

[0090] As an optional implementation manner provided in the embodiment of this application, the device further includes a training module, which is used to train the lightweight deep learning model, including:

[0091] A fine-tuning unit, which is used to fine-tune the convolutional neural network model based on an attention mechanism using a wildfire image data set, optimize the model parameters, so that the model adapts to the fire recognition task;

[0092] A compression unit, which is used to compress the fine-tuned convolutional neural network model based on an attention mechanism to obtain an intermediate model;

[0093] An adjustment unit, which is used to evaluate the performance metrics of the intermediate model on a test set, and adjust the compression parameters until the performance metrics meet the preset performance metrics to obtain the lightweight deep learning model.

[0094] As an optional implementation manner provided in the embodiment of this application, the target features include color features, texture features, motion features, and deep learning features; among them, the color features include statistical information in the RGB color space and statistical information in the HSV color space; the texture features include local binary pattern texture descriptors and gray-level co-occurrence matrix texture descriptors; the motion features include optical flow features and motion pattern features; the deep learning features include semantic hierarchical features and spatio-temporal association features.

[0095] For the specific limitations of the wildfire monitoring device based on deep learning and feature fusion, reference can be made to the limitations of the wildfire monitoring method based on deep learning and feature fusion in the above text, which will not be elaborated here. Each module in the above wildfire monitoring device based on deep learning and feature fusion can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or be stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0096] In some embodiments, the present application provides a wildfire monitoring system based on deep learning and feature fusion. The system includes: a front-end camera for collecting forest area video image data; and the wildfire monitoring device based on deep learning and feature fusion mentioned above to implement the steps in the wildfire monitoring method based on deep learning and feature fusion in various embodiments described in this specification.

[0097] In the above embodiments, the image acquisition module obtains the forest area video image data from the front-end camera, and subsequent preliminary recognition and secondary recognition are sequentially completed within the same device. This system is suitable for use in relatively centralized computing centers or small monitoring scenarios. Since the functions are concentrated in the wildfire monitoring device, the maintenance and upgrade of the system are relatively simple, mainly focusing on the update and maintenance of the software modules and algorithms inside the monitoring device, and it is easier to deploy and test by making adjustments in a single software system.

[0098] In other embodiments, the wildfire monitoring system based on deep learning and feature fusion may include:

[0099] A front-end camera for collecting forest area video image data;

[0100] An edge computing device for obtaining the forest area video image data and running a lightweight deep learning model to perform preliminary recognition on the preprocessed forest area video image data to obtain suspected wildfire information and target features; wherein, the lightweight deep learning model is obtained after compression and optimization of a convolutional neural network model based on the attention mechanism;

[0101] A cloud server for performing secondary recognition on the fused target features in combination with the suspected wildfire information to detect fire events and the corresponding geographical coordinates of the ignition points.

[0102] The above embodiments provide a distributed system, which consists of a front-end camera, an edge computing device, and a cloud server. Data collection, preliminary processing, and final in-depth processing are carried out on different devices respectively, realizing the close combination of functions and devices. During the processing, the low latency and fast response capabilities of the edge device and the powerful computing capabilities of the cloud server are utilized. The distributed system is suitable for large-scale forest area monitoring scenarios, has stronger scalability, and can conveniently add more front-end cameras and edge computing devices in different forest areas, connect them to the cloud server for centralized management and in-depth analysis, can better meet the monitoring needs of different geographical regions, and at the same time can adapt to different business requirements and data volumes by adjusting the configurations of the edge computing device and the cloud server. Utilize the advantages of different devices to achieve a more efficient and scalable forest fire monitoring function.

[0103] In practical applications, any of the above systems can be selected according to factors such as specific monitoring requirements, budget, computing resources, and maintenance capabilities.

[0104] The logical structure of the forest fire monitoring system based on deep learning and feature fusion provided by this application includes:

[0105] The front-end perception layer is used to collect forest area video image data and perform preprocessing.

[0106] The edge computing layer is used to run a lightweight deep learning model to perform preliminary identification on the preprocessed forest area video image data and obtain suspected forest fire information and target features; among them, the lightweight deep learning model is obtained after compression and optimization of a convolutional neural network model based on the attention mechanism.

[0107] The cloud processing layer is used to perform secondary identification on the fused target features in combination with the suspected forest fire information to detect fire events and the corresponding geographical coordinates of the ignition points.

[0108] In some embodiments, the logical structure of the system further includes a data transmission layer, which is used to connect the front-end perception layer, the edge computing layer, and the cloud processing layer, and is responsible for the efficient and reliable transmission of data, and can use wired or wireless networks.

[0109] In some embodiments, the logical structure of the system further includes a user interaction layer, which is used to provide a user interface for real-time monitoring, historical data query, system configuration, etc.

[0110] Optionally, the cloud processing layer uses a high-performance server cluster.

[0111] It should be noted that in the logical architecture of the wildfire monitoring system based on deep learning and feature fusion provided in the above embodiments, each logical layer can be implemented in actual software and hardware devices. For example, the front-end perception layer can be implemented in a front-end camera, the edge computing layer can be implemented in edge computing devices, the cloud processing layer can be implemented in cloud servers, and the data transmission layer can be implemented in data transmission devices, etc., so as to provide clearer guidance for the implementation and deployment of the system and implement the steps in the wildfire monitoring methods based on deep learning and feature fusion in various embodiments of the present application described in this specification.

[0112] In one embodiment, the present application provides an electronic device, which can be a terminal, and its internal structure diagram can be as Figure 3 shown. The electronic device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a carrier network, near-field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a stutter detection method. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball, or a touchpad provided on the housing of the electronic device, or an external keyboard, a touchpad, or a mouse, etc.

[0113] Those skilled in the art can understand that Figure 3 the structure shown in

[0114] In one embodiment, the wildfire monitoring device based on deep learning and feature fusion provided by the present application can be implemented in the form of a computer program, and the computer program can run on an electronic device such as Figure 3 shown. Each program module constituting the wildfire monitoring device based on deep learning and feature fusion can be stored in the memory of the electronic device. For example, Figure 2The image acquisition module 201, the preliminary recognition module 202 and the secondary recognition module 203 are shown. The computer program composed of various program modules enables the processor to execute the steps of the wildfire monitoring method based on deep learning and feature fusion in various embodiments of the present application described in this specification.

[0115] For example, Figure 3 The electronic device shown can be Figure 2 The image acquisition module 201 in the wildfire monitoring device based on deep learning and feature fusion shown is used to acquire forest video image data; the electronic device can use the preliminary recognition module 202 to perform preliminary recognition of the preprocessed forest video image data based on the lightweight deep learning model to obtain suspected wildfire information and target features; wherein the lightweight deep learning model is a convolutional neural network model based on the attention mechanism that is compressed and optimized; the electronic device can use the secondary recognition module 203 to perform secondary recognition of the fused target features in combination with the suspected wildfire information to detect fire events and the corresponding geographical coordinates of the ignition point.

[0116] In one embodiment, the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0117] Acquire forest video image data; perform preliminary recognition on the preprocessed forest video image data based on a lightweight deep learning model to obtain suspected wildfire information and target features; the lightweight deep learning model is a convolutional neural network model based on the attention mechanism that is compressed and optimized; combined with the suspected wildfire information, perform secondary recognition on the fused target features to detect fire events and the corresponding geographic coordinates of the ignition point.

[0118] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the convolutional neural network model based on the attention mechanism is a MobileNetV3 model pre-trained on ImageNet; the inverted residual module of the MobileNetV3 model includes a convolutional block attention module.

[0119] In one embodiment, the processor also implements the following steps when executing the computer program: the training process of the lightweight deep learning model includes the following steps: using the wildfire image data set to fine-tune the convolutional neural network model based on the attention mechanism, and optimize the model parameters to make the model adapt to the fire identification task; compressing the fine-tuned convolutional neural network model based on the attention mechanism to obtain an intermediate model; evaluating the performance indicators of the intermediate model on the test set, adjusting the compression parameters until the performance indicators meet the preset performance indicators, and obtaining a lightweight deep learning model.

[0120] In one embodiment, when the processor executes the computer program, the following steps are further implemented: The target features include color features, texture features, motion features, and deep learning features; among them, the color features include the statistical information in the RGB color space and the statistical information in the HSV color space; the texture features include the local binary pattern texture descriptor and the gray-level co-occurrence matrix texture descriptor; the motion features include the optical flow features and the motion pattern features; the deep learning features include the semantic hierarchical features and the spatio-temporal correlation features.

[0121] When the processor in the electronic device provided by this application executes the computer program, it first obtains the forest area video image data; then, based on the lightweight deep learning model, it performs a preliminary identification on the preprocessed forest area video image data to obtain the suspected wildfire information and target features; among them, the lightweight deep learning model is obtained after the convolutional neural network model based on the attention mechanism is compressed and optimized; further, in combination with the suspected wildfire information, a secondary identification is performed on the fused target features to detect the fire event and the corresponding geographical coordinates of the ignition point. In this way, this application introduces the attention mechanism, pays more attention to the key features of wildfire flames and smoke, and thus can identify wildfires more accurately, especially small fires or smoke in the early stage, improving the accuracy of wildfire monitoring.

[0122] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a computer, the following steps are implemented:

[0123] Obtain the forest area video image data; based on the lightweight deep learning model, perform a preliminary identification on the preprocessed forest area video image data to obtain the suspected wildfire information and target features; among them, the lightweight deep learning model is obtained after the convolutional neural network model based on the attention mechanism is compressed and optimized; in combination with the suspected wildfire information, perform a secondary identification on the fused target features to detect the fire event and the corresponding geographical coordinates of the ignition point.

[0124] In one embodiment, when the computer program is executed by a computer, the following steps are further implemented: The convolutional neural network model based on the attention mechanism is the MobileNetV3 model pre-trained on ImageNet; the inverted residual module of the MobileNetV3 model includes a convolutional block attention module.

[0125] In one embodiment, when the computer program is executed, the following steps are further implemented: The training process of the lightweight deep learning model includes the following steps: Using the wildfire image dataset to fine-tune the convolutional neural network model based on the attention mechanism, and optimizing the model parameters to make the model adapt to the fire recognition task; Compressing the fine-tuned convolutional neural network model based on the attention mechanism to obtain an intermediate model; Evaluating the performance metrics of the intermediate model on the test set, and adjusting the compression parameters until the performance metrics meet the preset performance metrics to obtain the lightweight deep learning model.

[0126] In one embodiment, when the computer program is executed, the following steps are further implemented: The target features include color features, texture features, motion features, and deep learning features; Among them, the color features include the statistical information in the RGB color space and the statistical information in the HSV color space; The texture features include local binary pattern texture descriptors and gray-level co-occurrence matrix texture descriptors; The motion features include optical flow features and motion pattern features; The deep learning features include semantic hierarchical features and spatio-temporal correlation features.

[0127] When the computer program in the computer-readable storage medium provided by this application is executed, first obtain the forest area video image data; Then, based on the lightweight deep learning model, perform preliminary identification on the preprocessed forest area video image data to obtain suspected wildfire information and target features; Among them, the lightweight deep learning model is obtained after compression and optimization of the convolutional neural network model based on the attention mechanism; Further, in combination with the suspected wildfire information, perform secondary identification on the fused target features to detect the fire event and the corresponding geographical coordinates of the ignition point. In this way, this application introduces the attention mechanism, pays more attention to the key features of wildfire flames and smoke, and thus can more accurately identify wildfires, especially early small fires or smoke, improving the accuracy of wildfire monitoring.

[0128] Those skilled in the art should understand that the embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0129] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0130] In the present application, the processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0131] In the present application, the memory may include non-permanent memory in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0132] In this application, computer-readable media include both permanent and non-permanent, removable and non-removable storage media. The storage media can implement information storage by any method or technology, and the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transitory media such as modulated data signals and carrier waves.

[0133] It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0134] The above are only specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application will not be limited to these embodiments herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A wildfire monitoring method based on deep learning and feature fusion, characterized in that: include: Obtain forest area video image data; Based on a lightweight deep learning model, the preprocessed forest area video image data is preliminarily identified to obtain suspected wildfire information and target features; wherein the lightweight deep learning model is a convolutional neural network model based on an attention mechanism that is compressed and optimized; Combined with the suspected wildfire information, the fused target features are re-identified to detect the fire event and the corresponding geographic coordinates of the fire point.

2. The method according to claim 1, characterized in that: The convolutional neural network model based on the attention mechanism is a MobileNetV3 model pre-trained on ImageNet; the inverted residual module of the MobileNetV3 model includes a convolutional block attention module.

3. The method according to claim 1, characterized in that The training process of the lightweight deep learning model includes the following steps: Use the wildfire image dataset to fine-tune the convolutional neural network model based on the attention mechanism and optimize the model parameters to make the model suitable for fire identification tasks; Compress the fine-tuned attention-based convolutional neural network model to obtain an intermediate model; The performance index of the intermediate model is evaluated on the test set, and the compression parameters are adjusted until the performance index meets the preset performance index to obtain the lightweight deep learning model.

4. The method according to claim 1, characterized in that: The target features include color features, texture features, motion features and deep learning features; Wherein, the color features include statistical information of RGB color space and statistical information of HSV color space; The texture features include local binary pattern texture descriptors and gray level co-occurrence matrix texture descriptors; The motion features include optical flow features and motion pattern features; The deep learning features include semantic level features and spatiotemporal correlation features.

5. A forest fire monitoring device based on deep learning and feature fusion, characterized in that: include: An image acquisition module is used to acquire forest area video image data; A preliminary recognition module is used to perform preliminary recognition on the preprocessed forest area video image data based on a lightweight deep learning model to obtain suspected wildfire information and target features; wherein the lightweight deep learning model is a convolutional neural network model based on an attention mechanism that is compressed and optimized; The secondary recognition module is used to combine the suspected wildfire information and perform secondary recognition on the fused target features to detect the fire event and the corresponding geographical coordinates of the fire point.

6. The device according to claim 5, characterized in that The device also includes a training module for training a lightweight deep learning model, including: A fine-tuning unit is used to fine-tune the convolutional neural network model based on the attention mechanism using the wildfire image dataset and optimize the model parameters to make the model suitable for the fire identification task; The compression unit is used to compress the fine-tuned attention-based convolutional neural network model to obtain an intermediate model; An adjustment unit is used to evaluate the performance index of the intermediate model on the test set, adjust the compression parameters until the performance index meets the preset performance index, and obtain the lightweight deep learning model.

7. A forest fire monitoring system based on deep learning and feature fusion, characterized in that: include: Front-end camera, used to collect video image data of forest areas; And, the wildfire monitoring device based on deep learning and feature fusion as described in claim 5 or 6, to implement the wildfire monitoring method based on deep learning and feature fusion as described in any one of claims 1 to 4.

8. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the wildfire monitoring method based on deep learning and feature fusion as described in any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the wildfire monitoring method based on deep learning and feature fusion as described in any one of claims 1 to 4 is implemented.

10. A computer program product, characterized in that include: The computer program product includes a computer program, and when the computer program is run on a computer, the computer is enabled to implement the wildfire monitoring method based on deep learning and feature fusion as described in any one of claims 1 to 4.

Images