A forest lightning fire monitoring method and system based on artificial intelligence

Abstract

The present application provides a forest lightning fire monitoring method and system based on artificial intelligence, which relates to the field of computer data intelligent analysis technology. The method includes: obtaining a sensor monitoring data group for a forest area; extracting the atmospheric electric field time series characteristics corresponding to the atmospheric electric field instrument data based on a long short-term memory network module; constructing a lightning graph structure, updating the node characteristics of each node in the graph structure based on a graph neural network to construct a lightning feature matrix; processing regional monitoring images based on a convolutional neural network to extract an image feature matrix; inputting the atmospheric electric field time series characteristics, the lightning feature matrix, and the image feature matrix into a lightning fire recognition model to determine whether there is a lightning fire risk in the forest area, and the lightning fire recognition model adopts a multimodal fusion deep learning model. As a result, the comprehensive analysis of lightning events and the dynamic monitoring of fire conditions are enhanced, and the efficiency of fire risk identification and early warning and the speed of emergency response are greatly improved.

Description

Technical Field

[0001] The present application relates to the technical field of computer data intelligent analysis, and in particular to a forest lightning fire monitoring method and system based on artificial intelligence. Background Art

[0002] With the intensification of global climate change and the frequent occurrence of natural disasters, forest fires have become one of the important factors threatening the ecological environment and human safety. Forest lightning fires, as a natural disaster mainly caused by lightning, are characterized by strong suddenness, rapid spread, and great difficulty in extinguishing. In recent years, with the advancement of my country's meteorological observation level and lightning positioning technology, relevant domestic institutions have carried out some beneficial experiments and research on the prevention of forest lightning fires.

[0003] Lightning protection facilities are installed at appropriate locations along the main path of lightning activity to actively attract and efficiently intercept lightning currents. Through the effective discharge of these facilities (such as lightning arresters), the energy of lightning strikes can be reduced, thereby achieving the purpose of regional lightning protection. However, such physical protection measures require the deployment of lightning protection facilities in the lightning strike area in advance, but since lightning strikes are often sudden and random, it is difficult to achieve comprehensive risk coverage of forest areas, resulting in frequent forest lightning fires, which seriously threaten the safety of people's lives and property.

[0004] In response to the above problems, the industry has not yet proposed a better technical solution. Summary of the invention

[0005] The present application provides a forest lightning fire monitoring method, system, storage medium, computer program product and electronic device based on artificial intelligence, which are used to at least solve the problems of limited coverage and insufficient early warning capability of lightning physical protection measures in current related technologies.

[0006] In a first aspect, an embodiment of the present application provides a forest lightning fire monitoring method based on artificial intelligence, comprising: obtaining a sensor monitoring data group for a forest area, the sensor monitoring data group comprising atmospheric electric field instrument data, lightning location data and a regional monitoring image; extracting the atmospheric electric field time series characteristics corresponding to the atmospheric electric field instrument data based on a long short-term memory network module; the atmospheric electric field time series characteristics comprising the electric field intensity change rate and the electric field spectrum characteristics; parsing at least one lightning event corresponding to the lightning location data, and constructing a lightning graph structure according to each of the lightning events, and updating the node features of each node in the graph structure based on a graph neural network to construct a lightning feature matrix; the lightning event comprises a lightning timestamp, a lightning location, a lightning intensity, a lightning type and a lightning duration; the lightning graph structure comprises a plurality of nodes and edges, each node is defined by a corresponding lightning event, and each edge is defined by the association information between the lightning events of the two connected nodes; processing the regional monitoring image based on a convolutional neural network to extract the image feature matrix; the image feature matrix includes image edge features, image texture features and image operation features; the atmospheric electric field time series features, the lightning feature matrix and the image feature matrix are input into the lightning fire recognition model to determine whether there is a lightning fire risk in the forest area; the lightning fire recognition model adopts a multimodal fusion deep learning model, which includes an input module, a feature fusion module, a Transformer encoder and a risk prediction module; the input module is used to receive the atmospheric electric field time series features, the lightning feature matrix and the image feature matrix; the feature fusion module is used to fuse the atmospheric electric field time series features, the lightning feature matrix and the image feature matrix through an attention mechanism to determine the corresponding multimodal fusion feature matrix; the multimodal fusion features are processed based on the Transformer encoder to obtain the corresponding multimodal encoding feature matrix; the multimodal encoding feature matrix is ​​processed based on the risk prediction module to determine the corresponding lightning fire risk prediction result.

[0007] In a second aspect, an embodiment of the present application provides an artificial intelligence-based forest lightning fire monitoring system, comprising: an acquisition unit, used to acquire a sensor monitoring data group for a forest area, the sensor monitoring data group comprising atmospheric electric field instrument data, lightning location data and regional monitoring images; an electric field feature extraction unit, used to extract the atmospheric electric field time series features corresponding to the atmospheric electric field instrument data based on a long short-term memory network module; the atmospheric electric field time series features comprise an electric field intensity change rate and an electric field spectrum feature; a lightning feature extraction unit, used to parse at least one lightning event corresponding to the lightning location data, and construct a lightning graph structure according to each of the lightning events, and update the node features of each node in the graph structure based on a graph neural network to construct a lightning feature matrix; the lightning event comprises a lightning timestamp, a lightning location, a lightning intensity, a lightning type and a lightning duration; the lightning graph structure comprises a plurality of nodes and edges, each node is defined by a corresponding lightning event, and each edge is defined by the association information between the lightning events of the two connected nodes; an image feature extraction unit, used to process the lightning event based on a convolutional neural network; The regional monitoring image is processed to extract an image feature matrix; the image feature matrix includes image edge features, image texture features and image operation features; a lightning fire prediction unit is used to input the atmospheric electric field time series features, the lightning feature matrix and the image feature matrix into a lightning fire recognition model to determine whether there is a lightning fire risk in the forest area; the lightning fire recognition model adopts a multimodal fusion deep learning model, which includes an input module, a feature fusion module, a Transformer encoder and a risk prediction module; the input module is used to receive the atmospheric electric field time series features, the lightning feature matrix and the image feature matrix; the feature fusion module is used to fuse the atmospheric electric field time series features, the lightning feature matrix and the image feature matrix through an attention mechanism to determine a corresponding multimodal fusion feature matrix; the multimodal fusion features are processed based on the Transformer encoder to obtain a corresponding multimodal encoding feature matrix; the multimodal encoding feature matrix is ​​processed based on the risk prediction module to determine a corresponding lightning fire risk prediction result.

[0008] In a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps of the artificial intelligence-based forest lightning fire monitoring method of any embodiment of the present application.

[0009] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of the artificial intelligence-based forest lightning fire monitoring method of any embodiment of the present application are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the artificial intelligence-based forest lightning fire monitoring method of any embodiment of the present application.

[0011] The forest lightning fire monitoring method based on artificial intelligence provided by this application can produce at least the following technical effects:

[0012] (1) The time series features of the atmospheric electric field extracted based on the long short-term memory network (LSTM), including the rate of change of the electric field intensity and the electric field spectrum features, can reflect the dynamic changes of the atmospheric electric field in real time. The lightning graph structure constructed by the lightning location data is updated with node features through the graph neural network (GNN) to form a high-dimensional lightning feature matrix. This feature matrix not only contains information such as the time, location, and intensity of the lightning, but also integrates the correlation between lightning events, so that the lightning fire identification model can more accurately judge the potential lightning fire risk. The feature matrix of the regional monitoring image processed by the convolutional neural network (CNN) can monitor the environmental changes in the forest area in real time and quickly identify possible signs of fire.

[0013] (2) The atmospheric electric field time series features, lightning feature matrix and image feature matrix are multimodally fused through the attention mechanism to form a multimodal fusion feature matrix, which can capture the correlation between different data sources, enhance the comprehensiveness and accuracy of feature expression, and improve the accuracy of fire identification. Then, the multimodal fusion feature matrix is ​​processed based on the Transformer encoder to generate a multimodal encoding feature matrix. The self-attention mechanism is used to effectively process and integrate different modal data, so that the model can maintain a high recognition accuracy when facing complex data. Therefore, by analyzing and processing the multimodal encoding feature matrix and applying the multimodal fusion deep learning model, real-time monitoring and early warning of forest lightning fires can be achieved, which greatly improves the identification and response speed of fire risks and effectively reduces the threat of lightning fires to the ecological environment and human safety.

[0014] Through this technical solution, multiple sensor data and advanced deep learning technologies are integrated, and multi-source data can be processed and analyzed in real time. It is possible to quickly analyze and predict fire risks in the early stages of lightning strikes, realize real-time monitoring and early warning of forest lightning fires, enhance the comprehensive analysis of lightning strikes and dynamic monitoring of fire conditions, greatly improve the efficiency of fire risk identification and early warning and emergency response speed, take measures in advance, and effectively reduce the threat of lightning fires to the ecological environment and human safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A flowchart showing an example of a forest lightning fire monitoring method based on artificial intelligence according to an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram of structural connection of an example of a lightning fire identification model according to an embodiment of the present application is shown;

[0018] Figure 3 An operation flow chart of an example of constructing a lightning feature matrix through a deep graph convolutional network according to an embodiment of the present application is shown;

[0019] Figure 4 A schematic diagram showing an exemplary structural connection of a convolutional neural network according to an embodiment of the present application is shown;

[0020] Figure 5 A structural block diagram of an example of an artificial intelligence-based forest lightning fire monitoring system according to an embodiment of the present application is shown;

[0021] Figure 6 It is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0024] Figure 1 A flowchart of an example of an artificial intelligence-based forest lightning fire monitoring method according to an embodiment of the present application is shown.

[0025] Regarding the executor of the method of the embodiment of the present application, it can be any controller or processor with computing or processing capabilities. By integrating multiple sensor data and advanced deep learning technology, the warning accuracy and real-time performance of lightning fires are significantly improved, and the comprehensive analysis of lightning events and dynamic monitoring of fire conditions are enhanced.

[0026] In some examples, it can be a lightning fire monitoring platform, and can be integrated and configured in an electronic device or terminal through software, hardware, or a combination of software and hardware, and the type of terminal or electronic device can be diverse, such as a mobile phone, tablet computer, or desktop computer, etc.

[0027] like Figure 1 As shown, in step S110, a sensor monitoring data group for the forest area is obtained, and the sensor monitoring data group includes atmospheric electric field instrument data, lightning location data and regional monitoring images.

[0028] In some embodiments, various types of sensors are deployed in the forest area, including atmospheric electric field meters, lightning location equipment, and high-resolution cameras, etc., so as to collect various types of sensor monitoring data.

[0029] More specifically, atmospheric electric field instruments are used to monitor changes in the atmospheric electric field in real time, capture precursor information of lightning activity, and a high frequency of data collection should be guaranteed to ensure the real-time and continuity of data. Advanced lightning location networks are used to obtain accurate location and intensity information of lightning in real time through multi-point collaborative positioning, thereby improving the accuracy and precision of data. Through surveillance cameras deployed throughout the forest area, image data is collected in real time to ensure the comprehensiveness and continuity of data collection. Therefore, by fully deploying various sensors in the forest area, the latest environmental information can be obtained in real time, providing a reliable data basis for subsequent processing.

[0030] In step S120, a long short-term memory network module is used to extract the atmospheric electric field time series characteristics corresponding to the atmospheric electric field instrument data, and the atmospheric electric field time series characteristics include the electric field intensity change rate and the electric field spectrum characteristics.

[0031] In some embodiments, by preprocessing the atmospheric electric field data, noise and outliers can be removed to ensure data quality, and a long short-term memory network (LSTM) model can be designed and trained to extract the temporal characteristics of the atmospheric electric field data. Thus, by inputting the continuously monitored atmospheric electric field instrument data into the LSTM model, the electric field intensity change rate and electric field spectrum characteristics are obtained.

[0032] Here, the rate of change of the electric field strength is calculated by the rate of change of the electric field strength, and can be calculated based on the first-order derivative to capture the rapid changes of the electric field before lightning activity. By extracting the spectrum features using the fast Fourier transform (FFT), the electric field spectrum features are obtained, thereby identifying the changes in specific frequency components and capturing the frequency characteristics before lightning activity. Here, the LSTM model can capture the time series dependency of the atmospheric electric field data, dynamically monitor the changes in the electric field by extracting the time series features, and improve the accuracy of feature extraction.

[0033] In step S130, at least one lightning event corresponding to the lightning location data is parsed, and a lightning graph structure is constructed according to each lightning event. The node features of each node in the graph structure are updated based on the graph neural network to construct a lightning feature matrix.

[0034] Here, the lightning event includes lightning timestamp, lightning location, lightning intensity, lightning type and lightning duration. The lightning graph structure includes multiple nodes and edges, each node is defined by a corresponding lightning event, and each edge is defined by the association information between the lightning events of the two connected nodes.

[0035] In some embodiments, the edge feature of each edge can be determined based on the association information corresponding to the connected nodes. For example, the edge feature can be used to express the geographical distance, time difference, intensity difference and type similarity of the lightning events corresponding to two nodes. Therefore, by constructing a lightning graph structure, the relationship between multiple lightning events can be comprehensively analyzed, and high-risk lightning event combinations can be identified. GNN is used to capture the complex associations between different lightning events, which helps to improve the accuracy of lightning fire risk prediction.

[0036] In step S140, the regional monitoring image is processed based on a convolutional neural network to extract an image feature matrix, where the image feature matrix includes image edge features, image texture features, and image operation features.

[0037] More specifically, image edge features represent edge features in an image, which can identify the boundary area where a fire occurs. Image texture features represent texture features of an image, which can identify abnormal texture changes in the early stages of a fire. Image motion features represent motion features in an image sequence, which can identify dynamic changes in the occurrence of a fire.

[0038] In this way, important information in the image can be efficiently extracted, high-dimensional image features can be provided, the accuracy and robustness of feature extraction can be improved, and the visual characteristics of fire occurrence can be identified.

[0039] In step S150, the atmospheric electric field time series characteristics, the lightning feature matrix and the image feature matrix are input into a lightning fire identification model to determine whether there is a lightning fire risk in the forest area.

[0040] In some examples of the embodiments of the present application, the lightning fire identification model adopts a multimodal fusion deep learning model. Here, through multimodal data fusion, it is possible to comprehensively analyze information from different data sources and improve the accuracy of fire risk prediction. By using deep learning models, especially the powerful modeling capabilities of Transformer encoders, it is possible to better capture and understand the relationship between complex features and provide accurate lightning fire risk prediction.

[0041] Figure 2 A schematic diagram of the structural connection of an example of a lightning fire identification model according to an embodiment of the present application is shown.

[0042] like Figure 2 As shown, the lightning fire identification model 200 includes an input module 210 , a feature fusion module 220 , a Transformer encoder 230 and a risk prediction module 240 .

[0043] The input module 210 is used to receive the atmospheric electric field time series characteristics, the lightning characteristic matrix and the image characteristic matrix.

[0044] Here, by combining the temporal characteristics of the atmospheric electric field, the lightning characteristic matrix and the image features, the model can utilize information in more dimensions, comprehensively capture the relevant factors of lightning activity and fire risk, and significantly improve the prediction accuracy.

[0045] The feature fusion module 220 is used to fuse the atmospheric electric field time series features, the lightning feature matrix and the image feature matrix through the attention mechanism to determine the corresponding multimodal fusion feature matrix.

[0046] In some implementations, a cross-modal attention mechanism can be used to fuse features of different modalities, dynamically adjust the importance of features of each modality, better capture the relationship between features, and fuse them into multi-modal fusion features.

[0047] The multimodal fusion features are processed based on the Transformer encoder 230 to obtain a corresponding multimodal encoding feature matrix.

[0048] The Transformer encoder has powerful feature expression capabilities, can handle complex feature relationships, extract high-level feature representations, and improve the generalization ability and robustness of the model.

[0049] The risk prediction module 240 processes the multimodal coding feature matrix to determine the corresponding lightning fire risk prediction result.

[0050] Here, on the one hand, the lightning risk prediction result can be a prediction result used to indicate whether there is a lightning fire risk. On the other hand, the lightning risk prediction result can also be a multi-dimensional prediction result, such as the probability of occurrence of lightning activity and the fire risk level, to achieve multi-task identification output.

[0051] Through this embodiment, advanced neural network technologies such as cross-modal attention mechanism and Transformer encoder are adopted to efficiently integrate multi-source data, fully utilize the advantages of different data, and improve the performance of the overall forest lightning fire monitoring system.

[0052] In the process of practicing this application, further improvements were made to the details of the lightning fire identification model to enhance the performance of the lightning fire monitoring system.

[0053] More specifically, the feature fusion module 220 is used to fuse the atmospheric electric field time series features, the lightning feature matrix and the image feature matrix by adopting a multi-head attention mechanism to determine the corresponding multimodal fusion feature matrix:

[0054] Q=W Q X, formula (1)

[0055] K=W K X, formula (2)

[0056] V=W V X, formula (3)

[0057]

[0058]

[0059] In the formula, X represents the input feature, which is X atm , X ltn or X img ;X atm represents the atmospheric electric field time series characteristic matrix, X img represents the image feature matrix; W Q ,W K ,W V Denote the trainable weight matrices of query, key and value respectively, Q, K, V denote the query matrix, key matrix and value matrix respectively; A is the attention weight matrix, d k is the scaling factor; F attention is the attention feature matrix, which represents the feature matrix after weighting by the attention mechanism, F att_atm ,Fatt_ltn ,F att_img They represent the atmospheric electric field time series feature matrix, lightning feature matrix and image feature matrix after weighting by the attention mechanism, respectively. fusion Represents the multimodal fusion feature matrix.

[0060] The Transformer encoder 230 is used to process the multimodal fusion features to generate a multimodal encoding feature matrix:

[0061] F ffn =σ(W1F fusion +b1), formula (7)

[0062]

[0063] Where W1, W2 are the weight matrices of the feedforward neural network, b1, b2 are bias terms, and F enc represents the multimodal encoding feature matrix, Represents the output feature matrix of the feedforward neural network;

[0064] The risk prediction module 240 uses a multi-task learning layer to determine the corresponding lightning fire risk prediction result by processing the multi-modal encoding feature matrix, and the lightning fire risk prediction result includes the probability of occurrence of lightning activities and the fire risk level:

[0065] P ltn =sigmoid(W ltn F enc +b ltn ), formula (10)

[0066] P fr =softmax(W fr F enc +b fr ), formula (11)

[0067] Where W ltn ,W fr represents the weight matrix of the multi-task learning layer, b ltn ,b fr is the bias term of the multi-task learning layer; P ltn represents the probability of lightning activity, P fr represents the predicted probability distribution of fire risk level; sigmoid represents the sigmoid function, and softmax represents the softmax function.

[0068] In this embodiment, the probability of lightning activity and the fire risk level are predicted simultaneously through the multi-task learning module, which can not only effectively identify the risk probability, but also obtain the corresponding risk level, providing strong support for real-time monitoring and early warning, such as giving priority to deploying defenses in advance for locations with high risk probability and high risk level. Therefore, by integrating multimodal features and multi-task learning, the robustness and stability of the model are enhanced, and the performance of the model in different environments and data conditions is improved.

[0069] In some examples of the embodiments of the present application, the lightning fire recognition model can adopt a comprehensive loss function that integrates dynamic target-weighted loss and adversarial training loss. Through dynamic target weighted loss, the weight of each sample is dynamically adjusted so that the model can pay more attention to important targets in the current training stage. Using adversarial training loss, the robustness of the model is improved by generating adversarial samples and calculating the loss of adversarial samples.

[0070] Based on this, the comprehensive loss function L of the lightning fire identification model is total for:

[0071] L total =L dm +λL ad , formula (12)

[0072]

[0073] Where, L dm is the dynamic target weighted loss, L ad is the adversarial training loss; λ is the loss weight hyperparameter, N is the total number of samples in the data sample set; α u is the dynamic weight factor of the u-th sample, L ltn,u is the lightning activity prediction loss of the u-th sample, L fr,u is the fire risk prediction loss of the u-th sample; represents the lightning activity prediction loss of the vth adversarial example, represents the fire risk prediction loss of the vth adversarial example.

[0074] Here, adversarial samples refer to samples generated by making small, almost imperceptible perturbations to the original input data, which can cause the model to make incorrect predictions. Adversarial samples are usually used to test and improve the robustness of the model to ensure that the model can maintain high performance in the face of malicious attacks or changes in data distribution. For example, common adversarial sample generation methods, such as the Fast Gradient Sign Method (FGSM) or the Projected Gradient Descent (PGD), can be used to generate adversarial samples. Through adversarial training, adversarial attacks can be effectively defended to ensure the security and reliability of the model in practical applications.

[0075] Through the dynamic target weighted loss of the embodiment of the present application, the model can adaptively focus on important samples in the current training stage, and optimize the learning process of the model by dynamically adjusting the sample weights, thereby improving the prediction accuracy. By generating adversarial samples and calculating the loss of adversarial samples, the performance of the model in the face of adversarial samples is improved, thereby enhancing the robustness and stability of the model. By dynamically adjusting sample weights and adversarial training, the model can better learn the characteristics of different samples and improve the generalization ability under different data distributions. As a result, the dynamic target weighted loss and balance weight parameters can dynamically allocate attention between different tasks. By optimizing multi-task learning, the model can better handle multiple prediction tasks in complex scenarios, so that the model performs better in multi-task learning and improves overall performance.

[0076] For the above-mentioned long short-term memory network module, in some examples of the embodiments of the present application, it can use a bidirectional long short-term memory network (Bidirectional Long Short-Term Memory, Bi-LSTM) to process the atmospheric electric field instrument data, which can significantly improve the performance of the lightning fire identification system. Through the bidirectional long short-term memory network, the forward and backward information of the sequence data can be processed simultaneously, and a more comprehensive and rich feature representation can be generated, so that the model can not only understand the impact of the past on the present, but also use future information for more accurate prediction. Thus, Bi-LSTM can capture the subtle changes in the rate of change of electric field intensity and the characteristics of the electric field spectrum. Through the combination of forward and backward LSTM, more representative time series features are extracted, which significantly improves the prediction accuracy of the model. The enhanced time series feature representation can help the model to more accurately identify the early signs of lightning activity and fire risk, and improve the risk prediction ability of the lightning fire identification model.

[0077] In some examples of the embodiments of the present application, the graph neural network can adopt a deep graph convolutional network. Figure 3An operational flowchart of an example of constructing a lightning feature matrix through a deep graph convolutional network according to an embodiment of the present application is shown.

[0078] like Figure 3 As shown, in step S310, feature information of each node and edge in the graph structure is received through the input layer.

[0079] In step S320, an adjacency matrix is ​​constructed.

[0080] More specifically, the adjacency matrix A is constructed by the following formula:

[0081]

[0082] It should be noted that in order to calculate the association information indicated by the edge features between two lightning events, the geographical distance, time difference, intensity difference and type similarity can be comprehensively considered. The following is a detailed calculation formula:

[0083] d ij =w1·dist ij +w2·Δt ij +w3·ΔI ij +w4·(1-sim ij ), formula (16)

[0084]

[0085] Δt ij =|t i -t j |, formula (18)

[0086] ΔI ij =|I i -I j |, formula (19)

[0087]

[0088] In the formula, the element A in A ij represents the connection weight between node i and node j; I unit is the identity matrix; represents the adjacency matrix after adding the self-loop; d ij Represents the degree of association between node i and node j, which is calculated by the association information corresponding to the edge; lat i and lon i denote the latitude and longitude of lightning event i, lat j and lon j denote the latitude and longitude of lightning event j respectively; t i and t jI represents the timestamp of lightning events i and j respectively; i and I j Respectively represent the current intensity of lightning events i and j; type i and type j denote the types of lightning events i and j respectively.

[0089] Here, by calculating the distance between two lightning events, reflecting the spatial position relationship of lightning events, the spatial distribution of lightning activities and fire risks can be effectively predicted. By calculating the time difference between two lightning events, the model can capture the temporal dynamic changes of lightning activities. By calculating the difference in current intensity between two lightning events, the relationship between high-intensity and low-intensity lightning can be identified, thereby more accurately assessing the fire risk. By judging the similarity of the two lightning event types, the model can distinguish different types of lightning events (such as cloud-to-ground lightning and cloud-to-cloud lightning), thereby more accurately capturing the characteristics and relationships of different types of lightning events. Therefore, by comprehensively considering geographical distance, time difference, intensity difference and type similarity, and through the fusion of multi-dimensional features, the model's ability to express node relationships is improved, and the accuracy and comprehensiveness of feature representation are enhanced.

[0090] In step S330, a normalized adjacency matrix is ​​used to maintain the balance of feature propagation.

[0091] In some examples of the embodiments of the present application, the normalized adjacency matrix Calculated by the following formula:

[0092]

[0093] In the formula, yes The degree matrix of represents the degree of node i, Represents the adjacency matrix after adding the self-loop The connection weight between node i and node j; Degree matrix The inverse matrix of the square root of .

[0094] Here, the normalized adjacency matrix balances the influence of different nodes during feature propagation, ensuring that the model can adaptively fuse the features of each node and improve the overall prediction performance.

[0095] In step S340, the node features are updated by aggregating the information of neighbor nodes through a graph convolution operation.

[0096] More specifically, each graph convolutional layer is used to perform the following operations:

[0097]

[0098] X ltn =H (L) , formula (25)

[0099] In the formula, H (l) is the node feature matrix of the lth layer; is the normalized adjacency matrix, W (l) is the trainable weight matrix of the lth graph convolutional layer, L is the number of graph convolutional layers, σ represents the activation function; X ltn Represents the lightning feature matrix.

[0100] In step S350, node features are extracted and fused layer by layer through multi-layer graph convolution, and finally a lightning feature matrix is ​​output, which contains a high-level feature representation of each node.

[0101] In this embodiment, the multi-layer graph convolution structure enables the model to extract features layer by layer from low-level to high-level, generate more expressive node feature representations, and better capture the subtle differences of lightning events through deep feature extraction, enhancing the expressiveness of the model. In addition, based on the multi-layer graph convolution network, noise can be effectively suppressed when aggregating neighbor node information, improving the accuracy and stability of feature extraction, and making the model more robust in the face of noise and abnormal data.

[0102] In some examples of the embodiments of the present application, the convolutional neural network adopts a multi-scale attention fusion convolutional neural network (Multi-scale Attention Fusion Convolutional Neural Network, MAF-CNN).

[0103] Figure 4 A schematic diagram of an example structural connection of a convolutional neural network according to an embodiment of the present application is shown.

[0104] like Figure 4 As shown, the convolutional neural network 400 includes a multi-scale feature extraction layer 410, an attention layer 420 and a multi-level feature fusion layer 430.

[0105] The multi-scale feature extraction layer 410 is used to perform a convolution operation on the regional monitoring image using multiple convolution kernels to extract corresponding multi-scale image features, and each convolution kernel has a corresponding scale.

[0106] In this embodiment, convolution operations are performed using convolution kernels of different sizes to extract multi-scale image features, and features of different scales are combined to improve the ability to capture objects of different sizes in the image.

[0107] The attention layer 420 includes a channel attention module 421 and a spatial attention module 423. The multi-scale image features are processed by the attention layer to obtain corresponding attention features. The channel attention module 421 is used to weight the features of different channels through the channel attention mechanism to highlight important features. The features of different spatial positions are weighted through the spatial attention module 423 to enhance the feature representation of important areas.

[0108] In this embodiment, based on the channel attention mechanism and the spatial attention mechanism, it is possible to highlight important features and important areas and enhance the accuracy of feature representation.

[0109] The multi-level feature fusion layer 430 processes the attention features through a feature pyramid attention network to determine the corresponding image feature matrix.

[0110] Here, the Feature Pyramid Attention Network (FPAN) is used to combine features at different levels to improve the detection ability of objects in images and generate more expressive image features, which can significantly improve the prediction accuracy of lightning activity and fire risk.

[0111] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of actions combined, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0112] Figure 5 A structural block diagram of an example of an artificial intelligence-based forest lightning fire monitoring system according to an embodiment of the present application is shown.

[0113] like Figure 5 As shown, the artificial intelligence-based forest lightning fire monitoring system 500 includes an acquisition unit 510, an electric field feature extraction unit 520, a lightning feature extraction unit 530, an image feature extraction unit 540 and a lightning fire prediction unit 550.

[0114] The acquisition unit 510 is used to acquire a sensor monitoring data group for a forest area, wherein the sensor monitoring data group includes atmospheric electric field instrument data, lightning location data, and regional monitoring images.

[0115] The electric field feature extraction unit 520 is used to extract the atmospheric electric field time series features corresponding to the atmospheric electric field instrument data based on the long short-term memory network module; the atmospheric electric field time series features include the electric field intensity change rate and the electric field spectrum features.

[0116] The lightning feature extraction unit 530 is used to parse at least one lightning event corresponding to the lightning location data, and construct a lightning graph structure according to each of the lightning events, and update the node features of each node in the graph structure based on the graph neural network to construct a lightning feature matrix; the lightning event includes a lightning timestamp, a lightning location, a lightning intensity, a lightning type, and a lightning duration; the lightning graph structure includes multiple nodes and edges, each node is defined by a corresponding lightning event, and each edge is defined by the association information between the lightning events of two connected nodes.

[0117] The image feature extraction unit 540 is used to process the regional monitoring image based on a convolutional neural network to extract an image feature matrix; the image feature matrix includes image edge features, image texture features and image operation features.

[0118] The lightning fire prediction unit 550 is used to input the atmospheric electric field time series characteristics, the lightning feature matrix and the image feature matrix into a lightning fire identification model to determine whether there is a lightning fire risk in the forest area.

[0119] The lightning fire identification model adopts a multimodal fusion deep learning model, which includes an input module, a feature fusion module, a Transformer encoder and a risk prediction module;

[0120] The input module is used to receive the atmospheric electric field time series characteristics, the lightning characteristic matrix and the image characteristic matrix;

[0121] The feature fusion module is used to fuse the atmospheric electric field time series features, the lightning feature matrix and the image feature matrix through an attention mechanism to determine a corresponding multimodal fusion feature matrix;

[0122] Processing the multimodal fusion features based on the Transformer encoder to obtain a corresponding multimodal encoding feature matrix;

[0123] The multimodal coding feature matrix is ​​processed based on the risk prediction module to determine a corresponding lightning fire risk prediction result.

[0124] In some embodiments, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the steps of any of the above-mentioned artificial intelligence-based forest lightning fire monitoring methods of the present application.

[0125] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any step of the above-mentioned artificial intelligence-based forest lightning fire monitoring method.

[0126] In some embodiments, the embodiments of the present application also provide an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the artificial intelligence-based forest lightning fire monitoring method.

[0127] Figure 6 is a schematic diagram of the hardware structure of an electronic device for executing an artificial intelligence-based forest lightning fire monitoring method provided by another embodiment of the present application, such as Figure 6 As shown, the device includes:

[0128] One or more processors 610 and memory 620, Figure 6 A processor 610 is taken as an example.

[0129] The device for executing the forest lightning fire monitoring method based on artificial intelligence may further include: an input device 630 and an output device 640 .

[0130] The processor 610, the memory 620, the input device 630 and the output device 640 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0131] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the forest lightning fire monitoring method based on artificial intelligence in the embodiment of the present application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, realizing the forest lightning fire monitoring method based on artificial intelligence in the above method embodiment.

[0132] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The input device 630 can receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 640 can include a display device such as a display screen.

[0134] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, the artificial intelligence-based forest lightning fire monitoring method in any of the above method embodiments is executed.

[0135] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0136] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0137] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0138] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0139] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0140] (4) Other onboard electronic devices with data interaction functions, such as on-board devices installed in vehicles.

[0141] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A forest lightning fire monitoring method based on artificial intelligence, comprising: Acquire a sensor monitoring data set for a forest area, wherein the sensor monitoring data set includes atmospheric electric field instrument data, lightning location data, and regional monitoring images; A long short-term memory network module is used to extract the atmospheric electric field time series characteristics corresponding to the atmospheric electric field instrument data; the atmospheric electric field time series characteristics include the electric field intensity change rate and the electric field spectrum characteristics; Parsing at least one lightning event corresponding to the lightning location data, and constructing a lightning graph structure according to each of the lightning events, and updating the node features of each node in the graph structure based on a graph neural network to construct a lightning feature matrix; the lightning event includes a lightning timestamp, a lightning location, a lightning intensity, a lightning type, and a lightning duration; The lightning graph structure includes multiple nodes and edges, each node is defined by a corresponding lightning event, and each edge is defined by association information between lightning events of two connected nodes; Processing the regional monitoring image based on a convolutional neural network to extract an image feature matrix; the image feature matrix includes image edge features, image texture features, and image operation features; Inputting the atmospheric electric field time series characteristics, the lightning characteristic matrix and the image characteristic matrix into a lightning fire identification model to determine whether there is a lightning fire risk in the forest area; The lightning fire identification model adopts a multimodal fusion deep learning model, which includes an input module, a feature fusion module, a Transformer encoder and a risk prediction module; The input module is used to receive the atmospheric electric field time series characteristics, the lightning characteristic matrix and the image characteristic matrix; The feature fusion module is used to fuse the atmospheric electric field time series features, the lightning feature matrix and the image feature matrix through an attention mechanism to determine a corresponding multimodal fusion feature matrix; Processing the multimodal fusion features based on the Transformer encoder to obtain a corresponding multimodal encoding feature matrix; The multimodal coding feature matrix is ​​processed based on the risk prediction module to determine a corresponding lightning fire risk prediction result.

2. The method according to claim 1, wherein: The graph neural network adopts a deep graph convolutional network and is used to perform the following operations: Construct the adjacency matrix A: d ij =w1·dist ij +w2·Δt ij +w3·ΔI ij +w4·(1-sim ij ) Δt ij =|t i -t j | ΔI ij =|I i -I j | In the formula, the element A in A ij represents the connection weight between node i and node j; I unit is the identity matrix; represents the adjacency matrix after adding the self-loop; d ij Represents the degree of association between node i and node j, which is calculated by the association information corresponding to the edge; lat i and lon i denote the latitude and longitude of lightning event i, lat j and lon j denote the latitude and longitude of lightning event j respectively; t i and t j denote the timestamps of lightning events i and j respectively; I i and I j denote the current intensity of lightning events i and j respectively; type i and type j denote the types of lightning events i and j respectively; Normalized adjacency matrix Calculated by the following formula: In the formula, yes The degree matrix of represents the degree of node i, Represents the adjacency matrix after adding the self-loop The connection weight between node i and node j; Degree matrix The inverse matrix of the square root of ; Each graph convolutional layer is used to perform the following operations: X ltn =H (L) In the formula, H (l) is the node feature matrix of the lth layer; is the normalized adjacency matrix, W (l) is the trainable weight matrix of the lth graph convolutional layer, L is the number of graph convolutional layers, σ represents the activation function; X ltn Represents the lightning feature matrix.

3. The method according to claim 2, wherein: The convolutional neural network adopts a multi-scale attention fusion convolutional neural network, which includes a multi-scale feature extraction layer, an attention layer and a multi-level feature fusion layer; The multi-scale feature extraction layer is used to perform a convolution operation on the regional monitoring image using multiple convolution kernels to extract corresponding multi-scale image features; each of the convolution kernels has a corresponding scale; The attention layer includes a channel attention module and a spatial attention module, which are used to process the multi-scale image features to obtain corresponding attention features; the channel attention module is used to weight the features of different channels through the channel attention mechanism to highlight important features; the spatial attention module is used to weight the features of different spatial positions to enhance the feature representation of important areas; The multi-level feature fusion layer processes the attention features through a feature pyramid attention network to determine a corresponding image feature matrix.

4. The method according to claim 3, wherein: The long short-term memory network module is a bidirectional long short-term memory network.

5. The method according to claim 4, wherein: The feature fusion module is used to fuse the atmospheric electric field time series features, the lightning feature matrix and the image feature matrix by adopting a multi-head attention mechanism to determine a corresponding multimodal fusion feature matrix: Q =W Q X K=W K X V=W V X F attention =OFF F fusion =Concat[F att_atm ,F att_ltn ,F att_img ] In the formula, X represents the input feature, which is X atm , X ltn or X img ;X atm represents the atmospheric electric field time series characteristic matrix, X img represents the image feature matrix; W Q ,W K ,W V Denote the trainable weight matrices of query, key and value respectively, Q, K, V denote the query matrix, key matrix and value matrix respectively; A is the attention weight matrix, d k is the scaling factor; F attention is the attention feature matrix, which represents the feature matrix after weighting by the attention mechanism, F att_atm ,F att_ltn ,F att_img They represent the atmospheric electric field time series feature matrix, lightning feature matrix and image feature matrix after weighting by the attention mechanism, respectively. fusion Represents the multimodal fusion feature matrix; The Transformer encoder is used to process the multimodal fusion features to generate a multimodal encoding feature matrix: F ffn =σ(W1F fusion +b1) Where W1, W2 are the weight matrices of the feedforward neural network, b1, b2 are bias terms, and F enc represents the multimodal encoding feature matrix, Represents the output feature matrix of the feedforward neural network; The risk prediction module adopts a multi-task learning layer to determine the corresponding lightning fire risk prediction result by processing the multimodal encoding feature matrix, and the lightning fire risk prediction result includes the probability of occurrence of lightning activity and the fire risk level: P ltn =sigmoid(W ltn F enc +b ltn ) P fr =softmax(W fr F enc +b fr ) Where W ltn ,W fr represents the weight matrix of the multi-task learning layer, b ltn ,b fr is the bias term of the multi-task learning layer; P ltn represents the probability of lightning activity, P fr represents the predicted probability distribution of fire risk level; sigmoid represents the sigmoid function, and softmax represents the softmax function.

6. The method according to claim 5, wherein: The comprehensive loss function L of the lightning fire identification model is total for: THE total =L dm +λL ad Where, L dm is the dynamic target weighted loss, L ad is the adversarial training loss; λ is the loss weight hyperparameter, N is the total number of samples in the data sample set; α u is the dynamic weight factor of the u-th sample, L ltn,u is the lightning activity prediction loss of the u-th sample, L fr,u is the fire risk prediction loss of the u-th sample; represents the lightning activity prediction loss of the vth adversarial example, represents the fire risk prediction loss of the vth adversarial example.

7. A forest lightning fire monitoring system based on artificial intelligence, comprising: An acquisition unit, used to acquire a sensor monitoring data group for a forest area, wherein the sensor monitoring data group includes atmospheric electric field instrument data, lightning location data, and a regional monitoring image; An electric field feature extraction unit, used for extracting the atmospheric electric field time series features corresponding to the atmospheric electric field instrument data based on a long short-term memory network module; the atmospheric electric field time series features include electric field intensity change rate and electric field spectrum features; A lightning feature extraction unit, configured to analyze at least one lightning event corresponding to the lightning location data, construct a lightning graph structure according to each of the lightning events, and update the node features of each node in the graph structure based on a graph neural network to construct a lightning feature matrix; the lightning event includes a lightning timestamp, a lightning location, a lightning intensity, a lightning type, and a lightning duration; The lightning graph structure includes multiple nodes and edges, each node is defined by a corresponding lightning event, and each edge is defined by association information between lightning events of two connected nodes; An image feature extraction unit, used for processing the regional monitoring image based on a convolutional neural network to extract an image feature matrix; the image feature matrix includes image edge features, image texture features and image operation features; A lightning fire prediction unit, used for inputting the atmospheric electric field time series characteristics, the lightning feature matrix and the image feature matrix into a lightning fire recognition model to determine whether there is a lightning fire risk in the forest area; The lightning fire identification model adopts a multimodal fusion deep learning model, which includes an input module, a feature fusion module, a Transformer encoder and a risk prediction module; The input module is used to receive the atmospheric electric field time series characteristics, the lightning characteristic matrix and the image characteristic matrix; The feature fusion module is used to fuse the atmospheric electric field time series features, the lightning feature matrix and the image feature matrix through an attention mechanism to determine a corresponding multimodal fusion feature matrix; Processing the multimodal fusion features based on the Transformer encoder to obtain a corresponding multimodal encoding feature matrix; The multimodal coding feature matrix is ​​processed based on the risk prediction module to determine a corresponding lightning fire risk prediction result.

Images