Multi-modal information analysis-based lightning fire identification method

Abstract

The invention relates to the technical field of forest lightning fire identification and prediction, and discloses a multi-modal information analysis-based lightning fire identification method, which comprises the steps of collecting monitoring data of M sub-regions of a to-be-monitored region according to a preset time interval t in a first preset time period T, and converting the monitoring data of each sub-region into a first feature sequence for representation; preprocessing the monitoring data of the M sub-regions of the to-be-monitored region; constructing graph structure data according to the monitoring data of the M sub-regions of the to-be-monitored region; constructing and training a forest lightning fire prediction model; and generating an early warning measure according to the probability value of the forest lightning fire occurring in the M sub-regions within the second preset time period G.

Description

Technical Field

[0001] The invention relates to the field of forest lightning fire identification and prediction, and more specifically, to a lightning fire identification method based on multimodal information analysis. Background Art

[0002] In recent years, affected by global climate change and the increase in extreme weather events, lightning fires in my country have become increasingly serious, causing losses to my country's ecological environment and the safety of people's lives and property. At present, the identification and monitoring methods for lightning fires mainly include: 1. Meteorological data monitoring method, which predicts the location and probability of lightning fires by combining observed meteorological data with historical data; 2. Remote sensing technology identification method, which identifies forest lightning fires by combining remote sensing and surface environment; 3. UAV monitoring method, which obtains forest image information by carrying a computer module on the drone, and analyzes and identifies forest lightning fires.

[0003] However, the meteorological data monitoring method can predict the location and probability of lightning fires by combining observed meteorological data with historical data, but cannot accurately identify and monitor forest lightning fires; the remote sensing technology identification method identifies forest lightning fires by combining remote sensing and surface environment. In complex terrain or remote areas, the accuracy and coverage of the global lightning positioning network data required by the method will be limited, affecting the accuracy of the lightning location; the drone monitoring method obtains forest image information by carrying a computer module, and can identify forest lightning fires, but will be affected by environmental factors such as light, smoke, and leaf obstruction, resulting in a decrease in recognition rate. Summary of the invention

[0004] The present invention provides a lightning fire identification method based on multimodal information analysis, which solves the technical problems of low lightning fire identification probability and inaccurate position positioning in the related art.

[0005] The present invention provides a lightning fire identification method based on multimodal information analysis, comprising:

[0006] Step S101, within a first preset time period T, monitoring data of M sub-areas of the area to be monitored are collected at preset time intervals t, and the monitoring data of each sub-area is converted into a first feature sequence representation;

[0007] The nth sequence unit of the first characteristic sequence of the mth sub-region represents the monitoring data of the mth sub-region at the nth time point, where 1≤m≤M, 1≤n≤N, and N=T / t;

[0008] Monitoring data include: meteorological data and remote sensing images;

[0009] Meteorological data include: temperature, humidity, air pressure, wind speed, wind direction, rainfall, CO gas concentration, NO2 gas concentration, O3 gas concentration, PM1.0 concentration, PM2.5 concentration, PM10 concentration;

[0010] Step S102, preprocessing the monitoring data of the M sub-areas of the monitored area, and converting the first feature sequence of each sub-area into a second feature sequence representation;

[0011] The nth sequence unit of the second feature sequence of the mth sub-region represents the pre-processed monitoring data of the mth sub-region at the nth time point;

[0012] Step S103, constructing graph structure data according to the monitoring data of the M sub-areas of the area to be monitored;

[0013] Graph structure data includes: nodes, initial features of nodes, and edges between nodes;

[0014] Establish a mapping relationship between nodes and sub-areas;

[0015] The initial feature of the node is represented by the second feature sequence of the sub-region with which the mapping relationship is established;

[0016] Step S104, constructing and training a forest lightning fire prediction model;

[0017] The input of the forest lightning fire prediction model is graph structure data, and the output value represents the probability value of forest lightning fire occurring in M ​​sub-areas within the second preset time period G;

[0018] Step S105 , generating early warning measures according to the probability values ​​of forest lightning fires occurring in the M sub-areas within the second preset time period G.

[0019] Furthermore, T, t, M, m, n, N, and G are all custom parameters.

[0020] Furthermore, the pre-processing step includes:

[0021] Step S201, for missing values ​​in the meteorological data, filling processing is performed by taking the average value of the value of the previous sequence unit and the value of the next sequence unit;

[0022] Step S202, normalizing the meteorological data by using the maximum and minimum normalization method;

[0023] Step S203, performing atmospheric correction on the remote sensing image through an atmospheric correction model;

[0024] Step S204, the normalized difference vegetation index value of each sub-area in the remote sensing image of the monitored area is calculated by the normalized difference vegetation index, and is spliced ​​after the monitoring data. The calculation formula of the normalized difference vegetation index is as follows:

[0025]

[0026] NIR represents the reflectivity in the near-infrared band, and Red represents the reflectivity in the red light band.

[0027] Furthermore, the ways to construct edges include:

[0028] Build edges between adjacent sub-regions;

[0029] It is determined that the angle between the line connecting the center points of each sub-area within the first range and the wind direction is less than or equal to the angle threshold, and then edges are constructed between the sub-areas;

[0030] The cosine similarity of the second feature sequence of each sub-region within the second range is calculated. If the cosine similarity is greater than or equal to the similarity threshold, an edge is constructed between the sub-regions.

[0031] Further, the forest lightning fire prediction model includes: a first hidden layer, a second hidden layer and a first classifier;

[0032] The first hidden layer includes: N first units, a splicer, a second unit and an extractor;

[0033] N first units share weight parameters and bias parameters;

[0034] The nth first unit inputs the remote sensing image in the nth sequence unit of the second feature sequence of each node, outputs the remote sensing image feature, and converts it into a vector representation;

[0035] The nth splicer is used to splice the vector representation of the remote sensing image feature output by the nth first unit to the meteorological data of the nth sequence unit of the second feature sequence to obtain a combined vector;

[0036] The nth second unit inputs the nth sequence unit of the second feature sequence of each node and outputs the first updated feature;

[0037] The extractor is used to use the first updated feature output by the Nth second unit as the initial feature of the node;

[0038] The second hidden layer inputs graph structure data and outputs a graph update matrix, where each row vector of the graph update matrix corresponds to a second updated feature of a node;

[0039] The second updated feature of each node is input into the first classifier, and the classification space of the first classifier represents the probability value of the occurrence of forest lightning fire in the sub-area of ​​the mapping relationship established by the nodes within the second preset time period G.

[0040] Furthermore, the first unit is constructed based on a convolutional neural network.

[0041] Furthermore, the second unit is constructed based on a long short-term memory network unit.

[0042] Furthermore, the calculation formula of the second hidden layer includes:

[0043]

[0044] Where S represents the graph update matrix output by the second hidden layer, represents the second updated feature of the mth node, H m represents the initial feature of the mth node, W m represents the weight parameter corresponding to the mth node, represents the second adjacency matrix, A represents the adjacency matrix of the second hidden layer input graph structure data, the size of the adjacency matrix is ​​M×M, the elements of the adjacency matrix represent the connection relationship between the nodes, and the element value is represented by 0 or 1, 0 means that there is no connection relationship between the nodes, otherwise there is a connection relationship, I represents the unit matrix, the size of the unit matrix is ​​M×M, the element values ​​on the diagonal of the unit matrix are assigned to 1, and the remaining element values ​​are assigned to 0, represents the normalized degree matrix, The value of the mth element on the diagonal of The size of the degree matrix is ​​M×M. The elements of the degree matrix represent the number of edge connections with the node, and ReLU represents the ReLU activation function.

[0045] Furthermore, sample labels of training samples used to train the forest lightning fire prediction model are obtained through manual labeling.

[0046] Furthermore, the probability values ​​of forest lightning fires occurring in the sub-areas are divided into three levels. The probability value between 0 and 0.2 is a low-risk level, and no special measures are taken at the low-risk level; the probability value between 0.2 and 0.5 is a medium-risk level, and the early warning measures for the medium-risk level are to increase the frequency of data collection, pre-arrange fire-fighting resources, and issue warnings through the media to remind local residents to pay attention to fire prevention; the probability value between 0.5 and 1 is a high-risk level, and the early warning measures for the high-risk level are to immediately issue emergency warnings to relevant departments and the public, immediately mobilize fire brigades to the vicinity of high-risk areas, and make preparations for fire fighting.

[0047] The beneficial effects of the present invention are as follows: the present invention uses a forest lightning fire prediction model to comprehensively consider the influence of vegetation conditions in adjacent areas and changes in meteorological data in different areas in the time dimension to identify and predict forest lightning fires, and generates early warning measures based on probability values, thereby achieving accurate identification, monitoring and timely early warning of forest lightning fires. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the lightning fire identification method based on multimodal information analysis of the present invention;

[0049] Figure 2 It is a flow chart of preprocessing monitoring data of the present invention;

[0050] Figure 3 It is a schematic diagram of three ways of constructing edges of graph structure data of the present invention. DETAILED DESCRIPTION

[0051] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the contents of this specification. Each example may omit, replace or add various processes or components as needed. In addition, the features described relative to some examples may also be combined in other examples.

[0052] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in one or more embodiments of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0053] like Figure 1 to Figure 3 As shown, the lightning fire identification method based on multimodal information analysis includes:

[0054] Step S101, within a first preset time period T, monitoring data of M sub-areas of the area to be monitored are collected at preset time intervals t, and the monitoring data of each sub-area is converted into a first feature sequence representation;

[0055] The nth sequence unit of the first characteristic sequence of the mth sub-region represents the monitoring data of the mth sub-region at the nth time point, where 1≤m≤M, 1≤n≤N, and N=T / t;

[0056] Monitoring data include: meteorological data and remote sensing images;

[0057] Meteorological data include: temperature, humidity, air pressure, wind speed, wind direction, rainfall, CO gas concentration, NO2 gas concentration, O3 gas concentration, PM1.0 concentration, PM2.5 concentration, PM10 concentration;

[0058] Step S102, preprocessing the monitoring data of the M sub-areas of the monitored area, and converting the first feature sequence of each sub-area into a second feature sequence representation;

[0059] The nth sequence unit of the second feature sequence of the mth sub-region represents the pre-processed monitoring data of the mth sub-region at the nth time point;

[0060] Step S103, constructing graph structure data according to the monitoring data of the M sub-areas of the area to be monitored;

[0061] Graph structure data includes: nodes, initial features of nodes, and edges between nodes;

[0062] Establish a mapping relationship between nodes and sub-areas;

[0063] The initial feature of the node is represented by the second feature sequence of the sub-region with which the mapping relationship is established;

[0064] Step S104, constructing and training a forest lightning fire prediction model;

[0065] The input of the forest lightning fire prediction model is graph structure data, and the output value represents the probability value of forest lightning fire occurring in M ​​sub-areas within the second preset time period G;

[0066] Step S105 , generating early warning measures according to the probability values ​​of forest lightning fires occurring in the M sub-areas within the second preset time period G.

[0067] In one embodiment of the present invention, T, t, M, m, n, N, and G are all custom parameters. Preferably, T is set to 30 minutes, t is set to 2 minutes, N is 15, M is set according to the actual situation of the monitored area, and G is set to 30 minutes.

[0068] In one embodiment of the present invention, PM1 refers to particles in the air with a diameter less than or equal to 1 micron, PM2.5 refers to particles in the air with a diameter less than or equal to 2.5 microns, and PM10 refers to particles in the air with a diameter less than or equal to 10 microns.

[0069] In one embodiment of the present invention, a drone is used to carry gas survey equipment and meteorological sensors to collect meteorological data of the area to be monitored.

[0070] In one embodiment of the present invention, Figure 2 As shown, the preprocessing steps include:

[0071] Step S201, for missing values ​​in the meteorological data, filling processing is performed by taking the average value of the value of the previous sequence unit and the value of the next sequence unit;

[0072] Step S202, normalizing the meteorological data by using the maximum and minimum normalization method;

[0073] Step S203, performing atmospheric correction on the remote sensing image through an atmospheric correction model;

[0074] The atmospheric correction model can be ATCOR model, MODTRAN model, etc.

[0075] Step S204, the normalized difference vegetation index value of each sub-area in the remote sensing image of the monitored area is calculated by the normalized difference vegetation index, and is spliced ​​after the monitoring data. The calculation formula of the normalized difference vegetation index is as follows:

[0076]

[0077] NIR represents the reflectivity in the near-infrared band, and Red represents the reflectivity in the red light band.

[0078] In one embodiment of the present invention, Figure 3 As shown, the ways to construct edges include:

[0079] 1. Build edges between adjacent sub-regions;

[0080] 2. If the angle between the line connecting the center points of each sub-area within the first range and the wind direction is less than or equal to the angle threshold, then an edge is constructed between the sub-areas. Preferably, the angle threshold is set to 30 degrees;

[0081] 3. Calculate the cosine similarity of the second feature sequence of each sub-region within the second range. If the cosine similarity is greater than or equal to the similarity threshold, an edge is constructed between the sub-regions. Preferably, the similarity threshold is set to 0.7.

[0082] In one embodiment of the present invention, the forest lightning fire prediction model includes: a first hidden layer, a second hidden layer and a first classifier;

[0083] The first hidden layer includes: N first units, a splicer, a second unit and an extractor;

[0084] N first units share weight parameters and bias parameters;

[0085] The nth first unit inputs the remote sensing image in the nth sequence unit of the second feature sequence of each node, outputs the remote sensing image feature, and converts it into a vector representation;

[0086] The nth splicer is used to splice the vector representation of the remote sensing image feature output by the nth first unit to the meteorological data of the nth sequence unit of the second feature sequence to obtain a combined vector;

[0087] The nth second unit inputs the nth sequence unit of the second feature sequence of each node and outputs the first updated feature;

[0088] The extractor is used to use the first updated feature output by the Nth second unit as the initial feature of the node;

[0089] The second hidden layer inputs graph structure data and outputs a graph update matrix, where each row vector of the graph update matrix corresponds to a second updated feature of a node;

[0090] The second updated feature of each node is input into the first classifier, and the classification space of the first classifier represents the probability value of the occurrence of forest lightning fire in the sub-area of ​​the mapping relationship established by the nodes within the second preset time period G.

[0091] In one embodiment of the present invention, the first unit is constructed based on a convolutional neural network.

[0092] In one embodiment of the present invention, the second unit is constructed based on a long short-term memory network unit. The second unit can also be constructed based on a gated recurrent unit or a recurrent neural unit.

[0093] In one embodiment of the present invention, the calculation formula of the second hidden layer includes:

[0094]

[0095] Where S represents the graph update matrix output by the second hidden layer, represents the second updated feature of the mth node, H m represents the initial feature of the mth node, W m represents the weight parameter corresponding to the mth node, represents the second adjacency matrix, A represents the adjacency matrix of the second hidden layer input graph structure data, the size of the adjacency matrix is ​​M×M, the elements of the adjacency matrix represent the connection relationship between the nodes, and the element value is represented by 0 or 1, 0 means that there is no connection relationship between the nodes, otherwise there is a connection relationship, I represents the unit matrix, the size of the unit matrix is ​​M×M, the element values ​​on the diagonal of the unit matrix are assigned to 1, and the remaining element values ​​are assigned to 0, represents the normalized degree matrix, The value of the mth element on the diagonal of The size of the degree matrix is ​​M×M. The elements of the degree matrix represent the number of edge connections with the node, and ReLU represents the ReLU activation function.

[0096] In one embodiment of the present invention, sample labels of training samples for training a forest lightning fire prediction model are obtained by manual labeling.

[0097] In one embodiment of the present invention, the first unit is pre-trained before training the forest lightning fire prediction model. During the pre-training process, the feature map output by the first unit is expanded into a vector representation and then input into the second classifier. The classification space of the second classifier represents the type of vegetation, and the classification label of the second classifier is manually labeled.

[0098] In one embodiment of the present invention, the loss function of the forest lightning fire prediction model adopts the cross entropy loss function.

[0099] In one embodiment of the present invention, when using a drone to carry a gas detector to collect CO gas concentration, NO2 gas concentration, and O3 gas concentration, the system will monitor the gas concentration in real time. Once a gas concentration is detected to exceed a preset threshold, the system will immediately send an early warning to the relevant department via SMS and email. During this period, the drone will continue to perform the collection task along the predetermined route. Preferably, the CO gas concentration threshold, NO2 gas concentration threshold, and O3 gas concentration threshold are set to 35ppm, 100ppb, and 100ppb, respectively.

[0100] In one embodiment of the present invention, the probability value of forest lightning fire occurring in a sub-area is divided into three levels. The probability value between 0 and 0.2 is a low risk level, and no special measures are taken for the low risk level; the probability value between 0.2 and 0.5 is a medium risk level, and the early warning measures for the medium risk level are to increase the frequency of data collection to ensure that abnormalities can be discovered in time, pre-arrange fire-fighting resources to ensure that they can be mobilized in a short time, and issue warnings through the media to remind local residents to pay attention to fire prevention; the probability value between 0.5 and 1 is a high risk level, and the early warning measures for the high risk level are to immediately issue an emergency warning to relevant departments and the public, immediately mobilize the fire brigade to the vicinity of the high risk area, and make preparations for fire fighting.

[0101] The above describes an embodiment of the present embodiment, but the present embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present embodiment, ordinary technicians in this field can also make many forms, all of which are within the protection of the present embodiment.

Claims

1. A lightning fire identification method based on multimodal information analysis, characterized in that: include: Step S101, within a first preset time period T, monitoring data of M sub-areas of the area to be monitored are collected at preset time intervals t, and the monitoring data of each sub-area is converted into a first feature sequence representation; The nth sequence unit of the first characteristic sequence of the mth sub-region represents the monitoring data of the mth sub-region at the nth time point, where 1≤m≤M, 1≤n≤N, and N=T / t; Monitoring data include: meteorological data and remote sensing images; Meteorological data include: temperature, humidity, air pressure, wind speed, wind direction, rainfall, CO gas concentration, NO2 gas concentration, O3 gas concentration, PM1.0 concentration, PM2.5 concentration, PM10 concentration; Step S102, preprocessing the monitoring data of the M sub-areas of the monitored area, and converting the first feature sequence of each sub-area into a second feature sequence representation; The nth sequence unit of the second feature sequence of the mth sub-region represents the pre-processed monitoring data of the mth sub-region at the nth time point; Step S103, constructing graph structure data according to the monitoring data of the M sub-areas of the area to be monitored; Graph structure data includes: nodes, initial features of nodes, and edges between nodes; Establish a mapping relationship between nodes and sub-areas; The initial feature of the node is represented by the second feature sequence of the sub-region with which the mapping relationship is established; Step S104, constructing and training a forest lightning fire prediction model; The input of the forest lightning fire prediction model is graph structure data, and the output value represents the probability value of forest lightning fire occurring in M ​​sub-areas within the second preset time period G; Step S105 , generating early warning measures according to the probability values ​​of forest lightning fires occurring in the M sub-areas within the second preset time period G.

2. The lightning fire identification method based on multimodal information analysis according to claim 1 is characterized in that: T, t, M, m, n, N, and G are all custom parameters.

3. The lightning fire identification method based on multimodal information analysis according to claim 1 is characterized in that: The preprocessing steps include: Step S201, for missing values ​​in the meteorological data, filling processing is performed by taking the average value of the value of the previous sequence unit and the value of the next sequence unit; Step S202, normalizing the meteorological data by using the maximum and minimum normalization method; Step S203, performing atmospheric correction on the remote sensing image through an atmospheric correction model; Step S204, the normalized difference vegetation index of each sub-region in the remote sensing image of the monitored area is calculated by the normalized difference vegetation index, and the normalized difference vegetation index is spliced ​​after the monitoring data. The calculation formula of the normalized difference vegetation index is as follows: NIR represents the reflectivity in the near-infrared band, and Red represents the reflectivity in the red light band.

4. The lightning fire identification method based on multimodal information analysis according to claim 1 is characterized in that: Ways to construct edges include: Build edges between adjacent sub-regions; It is determined that the angle between the line connecting the center points of each sub-area within the first range and the wind direction is less than or equal to the angle threshold, and then edges are constructed between the sub-areas; The cosine similarity of the second feature sequence of each sub-region within the second range is calculated. If the cosine similarity is greater than or equal to the similarity threshold, an edge is constructed between the sub-regions.

5. The lightning fire identification method based on multimodal information analysis according to claim 1 is characterized in that: The forest lightning fire prediction model includes: a first hidden layer, a second hidden layer and a first classifier; The first hidden layer includes: N first units, a splicer, a second unit and an extractor; N first units share weight parameters and bias parameters; The nth first unit inputs the remote sensing image in the nth sequence unit of the second feature sequence of each node, outputs the remote sensing image feature, and converts it into a vector representation; The nth splicer is used to splice the vector representation of the remote sensing image feature output by the nth first unit to the meteorological data of the nth sequence unit of the second feature sequence to obtain a combined vector; The nth second unit inputs the nth sequence unit of the second feature sequence of each node and outputs the first updated feature; The extractor is used to use the first updated feature output by the Nth second unit as the initial feature of the node; The second hidden layer inputs graph structure data and outputs a graph update matrix, where each row vector of the graph update matrix corresponds to a second updated feature of a node; The second updated feature of each node is input into the first classifier, and the classification space of the first classifier represents the probability value of the occurrence of forest lightning fire in the sub-area of ​​the mapping relationship established by the nodes within the second preset time period G.

6. The lightning fire identification method based on multimodal information analysis according to claim 5 is characterized in that: The first unit is built on convolutional neural networks.

7. The method for identifying lightning fires based on multimodal information analysis according to claim 5, characterized in that: The second unit is built on the basis of long short-term memory network units.

8. The method for identifying lightning fires based on multimodal information analysis according to claim 5, characterized in that: The calculation formula for the second hidden layer includes: Where S represents the graph update matrix output by the second hidden layer, represents the second updated feature of the mth node, H m represents the initial feature of the mth node, W m represents the weight parameter corresponding to the mth node, represents the second adjacency matrix, A represents the adjacency matrix of the second hidden layer input graph structure data, the size of the adjacency matrix is ​​M×M, the elements of the adjacency matrix represent the connection relationship between the nodes, and the element value is represented by 0 or 1, 0 means that there is no connection relationship between the nodes, otherwise there is a connection relationship, I represents the unit matrix, the size of the unit matrix is ​​M×M, the element values ​​on the diagonal of the unit matrix are assigned to 1, and the remaining element values ​​are assigned to 0, represents the normalized degree matrix, The value of the mth element on the diagonal of The size of the degree matrix is ​​M×M. The elements of the degree matrix represent the number of edge connections with the node, and ReLU represents the ReLU activation function.

9. The lightning fire identification method based on multimodal information analysis according to claim 1 is characterized in that: The sample labels of the training samples used to train the forest lightning fire prediction model are obtained through manual labeling.

10. The lightning fire identification method based on multimodal information analysis according to claim 1, characterized in that: The probability values ​​of forest lightning fires occurring in the sub-areas are divided into three levels. The probability value between 0 and 0.2 is a low-risk level, and no special measures are taken at the low-risk level; the probability value between 0.2 and 0.5 is a medium-risk level, and the early warning measures for the medium-risk level are to increase the frequency of data collection, pre-arrange fire-fighting resources, issue warnings through the media, and remind local residents to pay attention to fire prevention; the probability value between 0.5 and 1 is a high-risk level, and the early warning measures for the high-risk level are to immediately issue emergency warnings to relevant departments and the public, immediately mobilize fire brigades to the vicinity of high-risk areas, and make preparations for fire fighting.

Images