Forest fire prevention monitoring method and system based on multi-modal data and intelligent collaboration

Abstract

The application relates to the technical field of forest fire prevention monitoring, in particular to a forest fire prevention monitoring method and system based on multi-modal data and intelligent cooperation, which comprises the following steps: acquiring thermal infrared images of a forest area at each moment, and temperatures and wind speeds at each moment at different monitoring points in the forest area; calculating temperature abnormality degrees of each pixel point and extracting all abnormal regions; determining distribution characteristic values, heat source evaluation values and suspected degrees of the abnormal regions and screening out suspected regions; acquiring matching points corresponding to each pixel point in each gray-scale image; calculating change confusion degrees and flicker evaluation values of each pixel point and screening out flicker points; determining flame edge values of each flicker point and extracting flame regions in each gray-scale image. The application improves the accuracy and reliability of forest fire identification, and significantly reduces the false alarm rate caused by complex environmental interference.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of forest fire prevention monitoring, in particular to a forest fire prevention monitoring method and system based on multi-modal data and intelligent collaboration. BACKGROUND

[0002] Traditional methods rely on a single sensor, such as visible light or infrared camera images, for monitoring and identification, which is easily affected by complex natural environmental phenomena such as vegetation shaking, light changes, and temperature increases, resulting in a high false positive rate in monitoring and identifying forest fires, and lacking multi-dimensional, cross-modal cross-validation capabilities for fire judgment, making it difficult to comprehensively assess forest fire risk, resulting in low identification accuracy and insufficient reliability for early forest fires. SUMMARY

[0003] To solve the above technical problems, a forest fire prevention monitoring method and system based on multi-modal data and intelligent collaboration are provided to solve the existing problems.

[0004] The technical problem of the present application is solved by providing a forest fire prevention monitoring method and system based on multi-modal data and intelligent collaboration, comprising the following steps:

[0005] In a first aspect, the present application provides a forest fire prevention monitoring method based on multi-modal data and intelligent collaboration, which comprises the following steps:

[0006] Obtain the thermal infrared image of the forest area at each time, and the temperature and wind speed at each time at different monitoring points in the forest area;

[0007] Analyze the temperature anomaly degree of each pixel point in the thermal infrared image, and extract all abnormal regions;

[0008] Based on the decay distribution of the pixel values of the pixel points in the abnormal region and the change rate of the pixel values of the pixel points in the thermal infrared image of the adjacent time, the distribution characteristic value of the abnormal region is determined;

[0009] Analyze the pixel value difference in the abnormal region in the thermal infrared image of the adjacent time, and the temperature difference and distribution characteristic value at the monitoring point, to obtain the heat source evaluation value of the abnormal region, combine the temperature anomaly degree, determine the suspicious degree of the abnormal region, and screen out the suspected region;

[0010] Collect the monitoring video of the suspected region, register each frame of gray image with the historical frame of gray image, and obtain the matching points of each pixel point in each frame of gray image;

[0011] The all matching points of each pixel point are classified, the frame sequence distribution and the difference of corresponding gray values of the matching points in different categories are analyzed, the change confusion degree of each pixel point is calculated, the flicker evaluation value of each pixel point is calculated combined with the gray value difference of each pixel point and the matching point and the fluctuation of the wind speed at the monitoring point, and the flicker points are screened out;

[0012] The flame edge value of each flicker point is determined combined with the flicker evaluation value and the matching point state of the flicker point through the gradient change and the gray deviation of the pixel points in the neighborhood of each flicker point, the flame region in each gray scale image is extracted, and an alarm is triggered.

[0013] Preferably, the temperature anomaly degree of each pixel point in the thermal infrared image is analyzed, and all abnormal regions are extracted, including:

[0014] The Z-score of the pixel value of each pixel point in the thermal infrared image is calculated, and the cumulative sum of the Z-scores of all pixel points in the neighborhood of each pixel point in the thermal infrared image is calculated.

[0015] The temperature anomaly degree is the product of the Z-score and the cumulative sum of each pixel point in the thermal infrared image.

[0016] A segmentation threshold of the temperature anomaly degree of all pixel points in the thermal infrared image is obtained, which is denoted as a first segmentation threshold; the pixel points with a temperature anomaly degree greater than or equal to the first segmentation threshold in the thermal infrared image are denoted as abnormal points.

[0017] The connected domain analysis is performed on all abnormal points in the thermal infrared image, and the connected domain is extracted as an abnormal region.

[0018] Preferably, the distribution characteristic value of the abnormal region is determined, including:

[0019] Taking the pixel point corresponding to the maximum pixel value in the abnormal region as a starting point, extending along multiple predetermined directions, taking the pixel point extending to the boundary of the abnormal region as an ending point, and taking all pixel points from the starting point to the ending point in each direction as characteristic pixel points.

[0020] The difference between each characteristic pixel point and its adjacent characteristic pixel point in each direction is calculated, which is denoted as a first difference.

[0021] The change rate of the pixel value between any characteristic pixel point in the thermal infrared image at each time and the pixel point at the corresponding position in the thermal infrared image at the previous time is calculated.

[0022] The difference between each characteristic pixel point and its adjacent characteristic pixel point in each direction is calculated, which is denoted as a second difference.

[0023] The number of characteristic pixel points with the first difference and the second difference greater than or equal to 0 in each direction and the total number of all characteristic pixel points in each direction are counted.

[0024] The distribution characteristic value is the sum of the ratios of the number of all directions in the abnormal area to the total number.

[0025] Preferably, the heat source evaluation value of the abnormal area is obtained by:

[0026] The sum of the differences between the pixel values of all pixel points in the abnormal area in the thermal infrared image at each time and the pixel values of the corresponding pixel points in the thermal infrared image at the previous time is calculated, and is denoted as a first sum value;

[0027] The temperature difference between each monitoring point included in the abnormal area at each time and at the previous time is calculated, and is denoted as a temperature difference; and the sum of the temperature differences of all monitoring points included in the abnormal area at each time is denoted as a second sum value;

[0028] The ratio of the first sum value to the second sum value is denoted as a relative ratio; and the result of negative mapping of the difference between the relative ratio and a preset value is taken as the synergy degree of the abnormal area;

[0029] The average value of the temperatures of all monitoring points included in the abnormal area at each time is calculated;

[0030] The heat source evaluation value is the product of the average value, the distribution characteristic value, and the synergy degree.

[0031] Preferably, the suspected degree of the abnormal area is determined, and a suspected area is selected by: taking the normalized result of the product between the sum of the temperature abnormal degrees of all pixel points in the abnormal area and the heat source evaluation value as the suspected degree of the abnormal area; and selecting the abnormal area with a suspected degree greater than or equal to a preset threshold value in the thermal infrared image at each time as the suspected area.

[0032] Preferably, the change confusion degree of each pixel point is calculated by:

[0033] The gray values of all matching points corresponding to each pixel point in each gray image are clustered to obtain a plurality of clustering clusters;

[0034] Each gray image corresponds to a frame number, all matching points in each clustering cluster are arranged in ascending order according to the frame numbers of the gray images to which the matching points belong, the difference between the frame number of each matching point in each clustering cluster and the frame number of the adjacent matching point thereof is taken as the time interval of each matching point, and the dispersion degree of the time intervals of all matching points in each clustering cluster is taken as the time dispersion degree of each clustering cluster;

[0035] The sum of the differences between the gray values of all matching points in each clustering cluster and the gray value of the matching point corresponding to the minimum time interval is taken as the gray difference amount of each clustering cluster; and the product of the time dispersion degree and the gray difference amount of each clustering cluster is calculated.

[0036] The degree of variation disorder is the sum of the product values ​​of all clusters corresponding to each pixel in each frame of grayscale image.

[0037] Preferably, the step of calculating the flicker evaluation value of each pixel and filtering out flickering points includes:

[0038] Count the number of pixels in each frame of a grayscale image whose grayscale value is different from the grayscale values ​​of all their matching pixels;

[0039] The mean wind speed of all monitoring points in the suspected area at each time point is calculated and recorded as the average wind speed; the dispersion of the average wind speed at the time of acquisition corresponding to each grayscale image of the suspected area and all previous times is recorded as the wind speed fluctuation.

[0040] The ratio of the quantity to the wind speed fluctuation is used as the flash intensity of each pixel.

[0041] The flicker evaluation value is the product of the degree of variation disorder of each pixel in each frame of grayscale image and the flicker intensity;

[0042] The segmentation threshold for obtaining the flicker evaluation value of all pixels in each frame of grayscale image is denoted as the second segmentation threshold; pixels with flicker evaluation values ​​greater than or equal to the second segmentation threshold are denoted as flickering points.

[0043] Preferably, determining the flame edge value of each flash point includes:

[0044] Calculate the gradient intensity of each flashing point in each frame of grayscale image; obtain the maximum gradient intensity of all pixels in the neighborhood of each flashing point.

[0045] Calculate the sum of the differences between the gray values ​​of all pixels in the neighborhood of each flashing point and the preset gray value, and record it as the relative difference.

[0046] The ratio of the maximum gradient intensity to the relative difference is used as the edge salience of each scintillation point;

[0047] Count the number of times the matching point corresponding to each flashing point does not belong to the flashing point;

[0048] The flame edge value is the product of the number of times, the flicker evaluation value, and the edge salience.

[0049] Preferably, the extraction of the flame region from each grayscale image frame includes:

[0050] Obtain the segmentation threshold of the flame edge value of all flashing points in each frame of grayscale image, and denote it as the third segmentation threshold; flashing points with flame edge values ​​greater than or equal to the third segmentation threshold are regarded as flame edge points;

[0051] The closed region formed by all flame edge points in each frame of gray-scale image is taken as a flame region.

[0052] In a second aspect, the embodiments of the present application further provide a forest fire prevention monitoring system based on multi-modal data and intelligent cooperation, which comprises a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the steps of the forest fire prevention monitoring method based on multi-modal data and intelligent cooperation as described in any one of the above aspects when executing the computer program.

[0053] The present application has at least the following beneficial effects:

[0054] The present application calculates the temperature anomaly degree of each pixel point by the temperature size represented by the pixel value of the pixel point in the thermal infrared image collected by satellite remote sensing, filters the abnormal pixel points, and extracts the abnormal area, which has the beneficial effect of locating all potential heat source points from the thermal infrared image and preliminarily identifying the area where the temperature anomaly occurs; determine the distribution characteristic value of the abnormal area, which has the beneficial effect of considering the heat attenuation distribution of the pixel value of the pixel point in the abnormal area, as well as the trend of change with time, reducing the interference caused by the phenomenon of temperature reverse increase, and preliminarily identifying the real fire area that meets the fire change characteristics; obtain the heat source evaluation value of the abnormal area, which has the beneficial effect of further reducing the interference caused by the phenomenon of temperature reverse increase by comparing the difference change of the pixel value of the pixel point in the abnormal area in the thermal infrared image with the temperature difference change collected by the intelligent sensor in the corresponding area on the ground, reflecting the possibility that the abnormal area belongs to the real heat source area; determine the suspected degree of the abnormal area, and screen out the suspected area, which has the beneficial effect of comprehensively evaluating the possibility that the abnormal area belongs to the fire area, improving the identification accuracy of the fire area; start the unmanned aerial vehicle to collect the monitoring video of the suspected area, analyze the matching condition of the pixel points in different frames of gray images, and obtain the matching points corresponding to each pixel point in each frame of gray image, which has the beneficial effect that the state of the same physical point at different time points can be accurately analyzed in time sequence, so as to evaluate the flicker characteristics of the flame in the subsequent process; classify the matching points, calculate the change confusion degree of each pixel point, which has the beneficial effect of considering the gray difference of different matching points corresponding to the pixel points in the gray image under the same category, as well as the confusion situation of the time corresponding to the matching points under the same category, so as to effectively distinguish the situation of regular flame flicker and irregular leaf shaking; calculate the flicker evaluation value of each pixel point, and screen out the flicker point, which has the beneficial effect of considering the number of times that the gray values of the pixel points and the matching points are inconsistent, and eliminating the influence of wind speed fluctuation on the frequency of flame flicker, so as to reflect the possibility that the pixel point belongs to the flame; determine the flame edge value of each flicker point, extract the flame area in each frame of gray image and trigger the alarm, which has the beneficial effect of considering the gradient change of the flicker point and the deviation of the gray value from the gray of the real flame color, as well as the number of times that the matching point does not belong to the flicker point, reflecting the spreading nature of the real flame edge, so as to accurately screen out the pixel points located on the flame edge, effectively detect the fire area, and improve the accuracy and reliability of forest fire identification by fusing multiple modal data, while significantly reducing the false positive rate caused by complex environmental interference. BRIEF DESCRIPTION OF DRAWINGS

[0055] The multi-modal data and intelligent cooperative forest fire prevention monitoring method of the present application will be further described in detail below in conjunction with the drawings.

[0056] Figure 1A step flowchart of a forest fire prevention monitoring method based on multi-modal data and intelligent cooperation provided by an embodiment of the present application is shown in FIG. 1.

[0057] Figure 2 A step flowchart of a flame area acquisition method provided by an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer and more apparent, the forest fire prevention monitoring method and system based on multi-modal data and intelligent cooperation proposed by the present application are further described in detail below in combination with the drawings and implementation examples. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0060] Please refer to Figure 1 which shows a step flowchart of a forest fire prevention monitoring method based on multi-modal data and intelligent cooperation provided by an embodiment of the present application. The method includes the following steps:

[0061] Step 1, acquiring thermal infrared images of a forest area at each time, and temperatures and wind speeds at different monitoring points in the forest area at each time.

[0062] Forest fires occur randomly and suddenly, and can cause great losses in a short time. It is extremely difficult to completely prevent forest fires. The emergence of remote monitoring technology and the continuous improvement of artificial intelligence algorithms provide an advanced early warning means for forest fire prevention. Now, forest fire monitoring is widely carried out by ground patrol, lookout tower monitoring, aerial patrol, remote video monitoring and satellite remote sensing monitoring, etc.

[0063] If a single data source is used for fire prevention monitoring, due to the influence of various environmental factors, there is a high false positive rate in identifying fire conditions, and the forest fire condition cannot be accurately judged. Therefore, by constructing an air-space-ground integrated and fully covered monitoring network, a variety of modal data are fused to monitor the forest fire condition, wherein air represents satellite remote sensing, sky represents unmanned aerial vehicle, and ground represents ground sensor. The data collected by the three ways are used for monitoring, specifically as follows:

[0064] Scanning the area where the forest is located by satellite remote sensing to acquire thermal infrared images at each time in real time;

[0065] Preselecting monitoring points in the forest area and deploying intelligent sensors at the monitoring points to collect the temperatures and wind speeds at different monitoring points in the forest area in real time;

[0066] When the temperature abnormal area is found through the thermal infrared image, the suspected area of the fire is determined through the temperature change of the intelligent sensor at the abnormal area, and the flight instruction is sent. The unmanned aerial vehicle collects the monitoring video of the suspected area by using the monitoring camera carried according to the received instruction.

[0067] In the embodiment, the acquisition time interval of the thermal infrared image is 1 hour, and the data acquisition time interval of the intelligent sensor is 30 seconds. As other implementation manners, the implementer can set it according to the actual situation.

[0068] Since the acquisition time of the thermal infrared image and the acquisition time of the intelligent sensor are inconsistent, time alignment is performed through a timestamp alignment algorithm. The timestamp alignment algorithm is a known technology and will not be described here.

[0069] All collected data are normalized, for example, the pixel values of all pixel points in the thermal infrared image are normalized, and all temperatures and wind speeds are normalized respectively. In the embodiment, the maximum and minimum normalization method is used for normalization. The maximum and minimum normalization method is a known technology and will not be described here. As other implementation manners, the implementer can set it according to the actual situation.

[0070] It should be noted that the pixel value of the pixel point in the thermal infrared image is the radiation intensity.

[0071] Thus, the thermal infrared image of the forest area at each time and the temperature and wind speed at each time at different monitoring points in the forest area are obtained.

[0072] Step 2, analyze the temperature abnormality of each pixel point in the thermal infrared image, and extract all abnormal areas; based on the attenuation distribution of the pixel value of the pixel point in the abnormal area and the change rate of the pixel value of the pixel point in the thermal infrared image at the adjacent time, the distribution characteristic value of the abnormal area is determined; analyze the pixel value difference in the abnormal area in the thermal infrared image at the adjacent time, and the temperature difference and the distribution characteristic value at the monitoring point, to obtain the heat source evaluation value of the abnormal area, and determine the suspected degree of the abnormal area combined with the temperature abnormality, and screen out the suspected area.

[0073] The traditional method usually detects the forest fire according to the flame spreading flicker characteristics and the frame difference method; in the forest area, the influence of the bad environment such as strong wind, rain, fog, snow and the like, can influence the monitoring performance of the sensor, so that the monitoring precision of the forest fire is low, and the phenomenon of false alarm of the fire is prone to appear. For example, the flicker characteristics of the flame are the rapid change and irregular brightness change of the pixel value, when the vegetation is shaken by the strong wind, the shaking of the vegetation also causes the rapid change of the pixel value in the image, and the change is similar to the flicker characteristics of the flame in some cases; the random change of the illumination intensity in the morning and evening period can cause the sudden increase of the brightness of some regions, so as to be misjudged as the fire; secondly, the temperature inverse increase is a natural meteorological phenomenon, in a certain layer of atmosphere, the air temperature increases with the increase of the height, or remains unchanged, the atmosphere layer with the characteristics of 'cold at the bottom and hot at the top' is called the inversion layer, due to the existence of the inversion layer, the heat radiation signal generated by the ground heat source is accumulated, so that the temperature of the region is abnormally high, and only through the thermal infrared image, the temperature abnormal region can be misjudged as the fire region.

[0074] Therefore, the temperature abnormality degree is calculated through the abnormal situation of the pixel value of each pixel point in the thermal infrared image, and specifically,

[0075] The Z-score of the pixel value of each pixel point in the thermal infrared image is calculated.

[0076] In the embodiment, the Z-score is calculated by using the Z-score standardization method, and the calculation of the Z-score is a known technology, which will not be described herein.

[0077] The accumulated sum of the Z-scores of all pixel points in the neighborhood of each pixel point in the thermal infrared image is calculated.

[0078] In the embodiment, the size of the neighborhood is 3*3, and as other implementation manners, the implementer can set it according to the actual situation.

[0079] The product of the Z-score and the accumulated sum of each pixel point in the thermal infrared image is taken as the temperature abnormality degree of each pixel point in the thermal infrared image.

[0080] It should be noted that the greater the Z-score, the greater the deviation of the pixel value of the pixel point from the overall distribution, the greater the accumulated sum, the greater the regional aggregation of the temperature abnormality in the neighborhood of the pixel point, the greater the obtained temperature abnormality degree, and the higher the temperature abnormality of the pixel point, the more likely it is a real heat source.

[0081] The segmentation threshold of the temperature abnormality degree of all pixel points in the thermal infrared image is obtained, and is denoted as a first segmentation threshold.

[0082] In the embodiment, the average value of the temperature abnormality degree of all pixel points in all thermal infrared images in the historical period is collected as the first segmentation threshold.

[0083] The pixel point with a temperature anomaly degree greater than or equal to the first segmentation threshold in the thermal infrared image is recorded as an abnormal point.

[0084] The connected domain analysis is performed on all the abnormal points in the thermal infrared image, and all the connected domains are extracted as abnormal regions.

[0085] In the embodiment, the Two-Pass algorithm in the connected domain marking algorithm is used to extract the connected domain, wherein the Two-Pass algorithm is a known technology, and will not be described here. As other embodiments, the implementer can use other methods of the prior art, for example, the Seed-Filling algorithm, and the embodiment does not specially limit this.

[0086] Secondly, when the temperature reverse increase phenomenon occurs, an abnormal region will also be generated in the thermal infrared image, but this heat comes from the atmospheric inversion layer, not from the ground fire source, and thus, will interfere with the real fire region in the thermal infrared image. Firstly, the distribution characteristic value is calculated through the change trend of the pixel value of the pixel point between the adjacent time thermal infrared images, and the distribution characteristic value is specifically:

[0087] The pixel point with the maximum pixel value in the abnormal region in the thermal infrared image is obtained as a center point.

[0088] It should be noted that if there are multiple pixel points with the maximum pixel value in the abnormal region, the pixel point corresponding to the minimum sum of the distances of all the pixel points on the boundary of the abnormal region is selected as the center point. Secondly, the center point is more likely to be located near the fire source, and the pixel value is the highest.

[0089] The center point in the abnormal region is taken as a starting point, and is extended along multiple predetermined directions, and the pixel point extended to the boundary of the abnormal region is taken as an end point. All the pixel points from the starting point to the end point in each direction are recorded as characteristic pixel points.

[0090] In the embodiment, the up, down, left and right four directions of the center point are selected for extension, and as other embodiments, the implementer can set it according to the actual situation.

[0091] The difference between each characteristic pixel point and its adjacent characteristic pixel point in each direction is calculated, and is recorded as a first difference.

[0092] In the embodiment, the difference between each characteristic pixel point and its next characteristic pixel point in each direction is calculated, and is recorded as a first difference.

[0093] The change rate of the pixel value between any characteristic pixel point in the thermal infrared image of each time and the pixel point at the corresponding position in the thermal infrared image of the previous time is calculated.

[0094] It should be noted that the calculation of the change rate is a known technology, and the specific process is: the absolute value of the difference value between any feature pixel point in the thermal infrared image at each time and the pixel point at the corresponding position in the thermal infrared image at the previous time is recorded as the difference amount; the time interval between the adjacent two thermal infrared images is obtained, and the ratio of the difference amount to the time interval is the change rate.

[0095] The difference between the change rate of each feature pixel point and its adjacent feature pixel point in each direction is calculated, and is recorded as the second difference.

[0096] In this embodiment, the difference between the change rate of each feature pixel point and its next feature pixel point in each direction is calculated, and is recorded as the second difference.

[0097] The number of feature pixel points in each direction whose first difference and second difference are greater than or equal to 0 is counted.

[0098] The total number of all feature pixel points in each direction is counted; the sum of the ratios of the number to the total number in all directions in the abnormal area is calculated as the distribution feature value of the abnormal area.

[0099] It should be noted that the first difference and the second difference reflect the gradient change of the pixel value between the feature pixel points. The more the abnormal area conforms to the characteristics of high temperature and fast change rate near the fire source, and the temperature gradually decreases and the change rate slows down away from the fire source in this direction, the greater the obtained distribution feature value, indicating that the temperature distribution and change rate distribution in the abnormal area are more in line with the typical characteristics of a fire, and the higher the possibility that the abnormal area is a real fire area.

[0100] Further, by synchronizing the difference change of the pixel value of the pixel point in the abnormal area in the thermal infrared image at adjacent time and the difference change of the temperature collected by the intelligent sensor on the ground corresponding to the abnormal area at adjacent time, the coordination degree is calculated, specifically:

[0101] The sum of the difference between the pixel value of all pixel points in the abnormal area in the thermal infrared image at each time and the pixel value of the pixel point at the corresponding position in the thermal infrared image at the previous time is calculated, and is recorded as the first sum value.

[0102] In this embodiment, the sum of the absolute value of the difference between the pixel value of all pixel points in the abnormal area in the thermal infrared image at each time and the pixel value of the pixel point at the corresponding position in the thermal infrared image at the previous time is calculated, and is recorded as the first sum value.

[0103] The difference between the temperature of each monitoring point in the abnormal area at each time and at the previous time is calculated, and is recorded as the temperature difference.

[0104] In the embodiment, the absolute value of the temperature difference between each time and the previous time of each monitoring point contained in the abnormal area is calculated, and the temperature difference is denoted as a temperature difference.

[0105] The sum of the temperature differences of all monitoring points contained in the abnormal area at each time is denoted as a second sum value.

[0106] The ratio of the first sum value to the second sum value is denoted as a relative ratio, and the result of the negative mapping of the difference between the relative ratio and a preset value is used as the coordination degree of the abnormal area.

[0107] In the embodiment, the preset value is 1, and in other embodiments, the implementer can set it according to the actual situation. Secondly, the absolute value of the difference between the relative ratio and the preset value is negatively mapped, and the specific negative mapping process is as follows: the negative mapping is performed by an exponential function, and the difference between the relative ratio and the preset value is denoted as , and the result of is used as the result of negative mapping, wherein is an exponential function with a natural constant as the base.

[0108] It should be noted that the greater the first sum value, the more intense the change of the pixel value of the pixel point in the abnormal area in the thermal infrared image at the adjacent time, the greater the second sum value, the greater the temperature difference collected by the intelligent sensor in the abnormal area at the adjacent time, and the closer the relative ratio to 1, the greater the coordination degree, indicating that the change of the pixel value of the pixel point in the thermal infrared image collected by the satellite and the temperature change of the monitoring point in the abnormal area have a high consistency, reflecting that the abnormal area is less likely to be caused by the temperature inverse increase phenomenon, and the possibility of being a real fire source area is greater.

[0109] In addition, the temperature of the forest is low in the morning, and gradually increases after being irradiated by sunlight, which also causes a coordinated change in the pixel value in the thermal infrared image and the temperature collected by the intelligent sensor. Therefore, the heat source evaluation value is calculated by combining the distribution characteristic value and the coordination degree and the temperature collected by the intelligent sensor in real time, and the calculation is as follows:

[0110] The average value of the temperature of all monitoring points contained in the abnormal area at each time is calculated.

[0111] It should be noted that the temperature is a normalized value, and the greater the average value, the more likely the abnormal area is a real fire source area.

[0112] The product of the average value, the distribution characteristic value, and the coordination degree is used as the heat source evaluation value of the abnormal area.

[0113] It should be noted that the greater the heat source evaluation value, the more likely the abnormal area is a real fire source area.

[0114] Further, based on the temperature anomaly degree and the heat source evaluation value, a suspicious degree is determined, and the abnormal area is screened, specifically:

[0115] The normalized result of the product of the sum of the temperature anomaly degrees of all pixel points in the abnormal area and the heat source evaluation value is taken as the suspicious degree of the abnormal area;

[0116] It should be noted that the greater the sum of the temperature anomaly degrees of all pixel points, the higher the possibility that the temperature of the abnormal area is abnormal, and the greater the obtained suspicious degree, indicating that the possibility that the abnormal area belongs to a fire area is higher.

[0117] In the present embodiment, a sigmoid function is used for normalization processing, wherein the sigmoid function is a known technology and will not be described here. As other embodiments, the implementer can use other methods of prior art, such as tanh function, and the present embodiment does not make special restrictions.

[0118] Select the abnormal area with a suspicious degree greater than or equal to a preset threshold value in each moment of the thermal infrared image, and mark it as a suspected area.

[0119] In the present embodiment, the setting process of the preset threshold value is: taking the mean of the suspicious degrees of all abnormal areas in all thermal infrared images in the historical period as the preset threshold value.

[0120] Thus, the suspected area is obtained.

[0121] Step 3, register each frame of gray image with the historical frame of gray image, and obtain the matching points of each pixel point in each frame of gray image; classify all matching points of each pixel point, analyze the frame sequence distribution and the difference of corresponding gray values in different categories, calculate the change confusion degree of each pixel point, combine the gray value difference of each pixel point and the matching point, and calculate the flicker evaluation value of each pixel point, and screen out the flicker point.

[0122] Using the monitoring camera carried by the unmanned aerial vehicle, real-time monitoring video of the suspected area is collected, and after frame processing, gray processing is performed to obtain each frame of gray image of the suspected area.

[0123] In the present embodiment, the frame rate of the frame processing is 30fps, which is consistent with the time interval of the data collection of the intelligent sensor. As other embodiments, the implementer can set it according to the actual situation, and the gray processing is performed by using the weighted average method, wherein the frame processing and the weighted average method of gray processing are known technologies and will not be described here.

[0124] All frames of gray images before each frame of gray image are grouped into an association set.

[0125] The pixel point in each frame of the gray image is matched with the pixel point in all frames of the gray image in the association set, and the pixel point in each frame of the gray image in the association set which is matched with the pixel point is recorded as a matching point;

[0126] In the embodiment, the SIFT (Scale-invariant feature transform) image matching algorithm is used for matching, wherein the SIFT image matching algorithm is a known technology, and will not be described here again. As other embodiments, the implementer can use other methods in the prior art, for example, the SURF image matching algorithm, and the embodiment does not specially limit this.

[0127] Secondly, the real flame flickers, the edge of the flame is usually high frequency, irregular, and the boundary is fuzzy, and in different frames of the gray image, the gray value of the pixel point has a large change, for example, when the flame flicker disappears, the gray value of the pixel point is the gray value of the fixed scene after the flame is blocked, when the flame flicker does not disappear, the gray value of the pixel point is the gray value corresponding to the color of the flame, and the gray value is quite different from the gray value when the flame flicker disappears. Therefore, the change of the gray value of the pixel point in the gray image has a certain regularity when the flame flickers, and therefore, the change of the gray value of all the matching points corresponding to each pixel point in each frame of the gray image is analyzed, and the change confusion degree is calculated, and specifically, the change confusion degree is calculated as follows:

[0128] The gray values of all the matching points corresponding to each pixel point in each frame of the gray image are clustered to obtain a plurality of clustering clusters.

[0129] In the embodiment, the k-means clustering algorithm is used for clustering to obtain two clustering clusters, wherein the k-means clustering algorithm is a known technology, and will not be described here again. As other embodiments, the implementer can use other methods in the prior art, for example, the hierarchical clustering algorithm, and the embodiment does not specially limit this.

[0130] It should be noted that through clustering, the pixel points in the gray image when the flame disappears and appears in the flame flicker process are divided into two categories.

[0131] Each frame of the gray image corresponds to a frame number, all the matching points in each clustering cluster are arranged in ascending order according to the frame number of the gray image to which the matching point belongs, and the difference between the frame number of each matching point in each clustering cluster and the frame number of the adjacent matching point is used as the time interval of each matching point.

[0132] In the embodiment, the difference between the frame number of each matching point in each clustering cluster and the frame number of the previous matching point is calculated as the time interval of each matching point.

[0133] a dispersion degree of time intervals of all the matching points in each cluster is calculated as a time dispersion degree of each cluster;

[0134] In the embodiment, the dispersion degree is measured by a coefficient of variation of time intervals of all the matching points in each cluster, and the coefficient of variation is a well-known technique and will not be described here. As other embodiments, the implementer can use other methods such as information entropy, and the embodiment does not specially limit this.

[0135] a sum of differences of gray values between all the matching points and the matching point corresponding to the minimum time interval in each cluster is calculated as a gray difference amount of each cluster;

[0136] In the embodiment, the sum of absolute values of the differences of gray values between all the matching points and the matching point corresponding to the minimum time interval in each cluster is calculated as the gray difference amount of each cluster.

[0137] a product value of the time dispersion degree and the gray difference amount of each cluster is calculated; and a sum value of the product values of all the cluster corresponding to each pixel point in each gray scale image is taken as a change confusion degree of each pixel point in each gray scale image.

[0138] It should be noted that the greater the time dispersion degree, the more irregular the change time of the pixel points in the cluster, the more irregular the flicker of the flame, and the greater the gray difference amount, the greater the difference of the gray values of the pixel points, and vice versa. The greater the change confusion degree, the more random the change of the gray values of the pixel points, and the smaller the possibility of the pixel points belonging to the flame features.

[0139] Secondly, the inconsistency of the gray values of each pixel point and the matching points and the fluctuation of the wind speed are analyzed, and a flicker intensity is calculated, specifically:

[0140] the number of gray values of each pixel point in each gray scale image that is different from the gray values of all the matching points is counted;

[0141] a mean value of the wind speeds of all the monitoring points contained in the suspected area at each moment is calculated and taken as an average wind speed;

[0142] each gray scale image corresponds to a collected moment;

[0143] a dispersion degree of the average wind speeds of all the moments before the moment corresponding to each gray scale image of the suspected area is taken as a wind speed fluctuation degree;

[0144] In the embodiment, the discrete degree is measured by calculating the coefficient of variation of the average wind speed of all time points before and at the corresponding time of each frame of the grayscale image of the suspected area, wherein the calculation of the coefficient of variation is a known technology and will not be described here.

[0145] The ratio of the number to the wind speed fluctuation degree is taken as the flicker intensity of each pixel point.

[0146] It should be noted that the more intense the wind speed fluctuation is, the greater the relative fluctuation degree is, and the more frequent the flicker of the flame is. Therefore, by calculating the flicker intensity, the disturbance of the wind speed fluctuation on the flame flicker is removed, and the greater the flicker intensity is, the greater the possibility of belonging to the flame edge is.

[0147] Further, based on the change chaos degree and the flicker intensity, a flicker evaluation value is determined, specifically:

[0148] The product of the change chaos degree and the flicker intensity of each pixel point in each frame of the grayscale image is taken as the flicker evaluation value of each pixel point in each frame of the grayscale image.

[0149] It should be noted that the greater the flicker evaluation value is, the more likely the pixel point is a pixel point at the flame, and vice versa, the more likely the pixel point is an interference point.

[0150] Further, based on the flicker evaluation value, the pixel points are screened, specifically:

[0151] A segmentation threshold of the flicker evaluation values of all pixel points in each frame of the grayscale image is obtained, denoted as a second segmentation threshold.

[0152] In the embodiment, the maximum inter-class variance method is used to obtain the segmentation threshold, wherein the maximum inter-class variance method is a known technology and will not be described here. As other embodiments, the implementer can use other methods of the prior art, for example, the cross-validation method, and the present embodiment does not specially limit this.

[0153] The pixel point with the flicker evaluation value greater than or equal to the second segmentation threshold is denoted as a flicker point.

[0154] Thus, each flicker point in each frame of the grayscale image corresponding to the suspected area is obtained.

[0155] Step 4, the flame edge value of each flicker point is determined by the gradient change and the grayscale deviation of the pixel points in the neighborhood of each flicker point, combined with the flicker evaluation value and the matching point state of the flicker point, the flame region in each frame of the grayscale image is extracted, and an alarm is triggered.

[0156] Further, the real flame edge spreads to the surroundings with time, the pixel points represented by the flame edge in the continuous gray image have dynamic changeability in the case of the flicker points; and the gray value of the flame edge changes sharply, usually showing high gradient value. Thus, by analyzing the case that the gray values of the pixel points in the neighborhood of the flicker points are close to the color of the flame edge and the gradient change of the flicker points, the edge saliency is calculated, specifically as follows:

[0157] The gradient intensity of each flicker point in each gray image is calculated; the maximum gradient intensity of all the pixel points in the neighborhood of each flicker point is obtained;

[0158] In the embodiment, the size of the neighborhood is 3x3, and as other implementation manners, the implementer can set it according to the actual situation; it should be noted that the calculation of the gradient intensity of the pixel points of the image is a known technology, which will not be described here.

[0159] The sum of the differences between the gray values of all the pixel points in the neighborhood of each flicker point and the preset gray value is calculated, denoted as the relative difference amount;

[0160] It should be noted that the preset gray value is the gray value corresponding to the flame edge, since the color of the flame generally presents a change interval from red to yellow, therefore, the gray value corresponding to the flame is obtained through certain prior knowledge or experiment, for example, the mean value of the gray values of all the pixel points corresponding to the flame edge is obtained as the gray value corresponding to the flame edge, in the embodiment, the gray value corresponding to the flame is 185, and as other implementation manners, the implementer can set it according to the actual situation.

[0161] The ratio of the maximum gradient intensity to the relative difference amount is taken as the edge saliency of each flicker point;

[0162] It should be noted that the greater the maximum gradient intensity, the more intense the gray change of the pixel points in the neighborhood of the flicker point, there is a sharp brightness mutation, the edge feature represented is more obvious, the possibility of being the flame edge is greater, the smaller the relative difference amount, the closer the gray of the pixel points in the neighborhood to the color feature of the flame, the greater the obtained edge saliency, which indicates that the edge feature in the neighborhood is significant and the gray value is close to the color feature of the flame, which is more likely to be the flame edge.

[0163] Secondly, the flame edge value is determined by the case that each flicker point in each gray image in the association set does not belong to the flicker point in different gray images, combined with the flicker evaluation value and the edge saliency, specifically as follows:

[0164] The number of times that the corresponding matching points of each flicker point in each gray image in the association set do not belong to the flicker points is counted;

[0165] It should be noted that the more the number is, the more the flickering point is not fixed at the same position, the less likely it is caused by the shaking of the leaves of the forest area or the interference of light, and the more the flame edge grows.

[0166] The product of the number, the flicker evaluation value and the edge prominence is taken as the flame edge value of each flickering point in each frame of gray-scale image.

[0167] It should be noted that the greater the flame edge value is, the more the flickering point conforms to the characteristics of the flame edge, and the more likely it is a pixel point on the flame edge.

[0168] A segmentation threshold of the flame edge values of all flickering points in each frame of gray-scale image is obtained, denoted as a third segmentation threshold.

[0169] In this embodiment, the process of obtaining the third segmentation threshold is that the mean value of the flame edge values of all flickering points in each frame of gray-scale image is taken as the third segmentation threshold.

[0170] The flickering point with the flame edge value greater than or equal to the third segmentation threshold is taken as a flame edge point.

[0171] The closed region formed by all flame edge points in each frame of gray-scale image is taken as a flame region.

[0172] According to the identified flame region, an alarm is triggered immediately, and the fire point position is accurately positioned on an electronic map in combination with the GPS coordinates and image recognition of the unmanned aerial vehicle, so that the fire alarm information is sent to relevant personnel.

[0173] Further, the step flowchart of the method for obtaining the flame region provided by the embodiment of the present application is shown in Figure 2 .

[0174] Based on the same inventive concept as the above method, the embodiment of the present application also provides a forest fire prevention monitoring system based on multi-modal data and intelligent cooperation, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of any one of the methods in the above forest fire prevention monitoring method based on multi-modal data and intelligent cooperation are implemented.

[0175] It should be understood that, although Figure 1 the steps in the flowchart are displayed in sequence according to the arrows, these steps are not necessarily executed in sequence according to the arrows. Unless explicitly stated herein, the execution of these steps has no strict sequence limitation, and these steps can be executed in other orders. Moreover, Figure 1At least one of the steps in the above embodiments can include a plurality of sub-steps or a plurality of stages, which are not necessarily executed at the same time, but can be executed at different times, and the order of execution of the sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or at least a part of the sub-steps or stages of other steps.

[0176] The technical features of the above embodiments can be combined in any manner. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described, but it should be understood that any combination of the technical features is within the scope of the present disclosure as long as the combination does not result in contradictions.

[0177] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the present application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, as long as it does not depart from the technical solution of the present application, is within the protection scope of the technical solution of the present application.

Claims

1.A forest fire prevention monitoring method based on multi-modal data and intelligent collaboration, characterized in that, The method comprises the following steps: Obtain the thermal infrared image of the forest area at each time, and the temperature and wind speed at each time at different monitoring points in the forest area; Analyze the temperature anomaly degree of each pixel point in the thermal infrared image, and extract all abnormal regions; Determine the distribution feature value of the abnormal region based on the attenuation distribution of the pixel value of the pixel point in the abnormal region and the change rate of the pixel value of the pixel point in the thermal infrared image at the adjacent time; Analyze the pixel value difference in the thermal infrared image at the adjacent time, the temperature difference at the monitoring point, and the distribution feature value in the abnormal region to obtain the heat source evaluation value of the abnormal region, determine the suspected degree of the abnormal region in combination with the temperature anomaly degree, and screen out the suspected region; Collect the monitoring video of the suspected region, register each gray image with the historical gray image, and obtain the matching points of each pixel point in each gray image; Classify all matching points of each pixel point, analyze the frame sequence distribution and corresponding gray value difference of the matching points in different categories, calculate the change confusion degree of each pixel point, calculate the flicker evaluation value of each pixel point in combination with the gray value difference of each pixel point and the matching point, and the fluctuation of the wind speed at the monitoring point, and screen out the flicker points; Determine the flame edge value of each flicker point by the gradient change and gray deviation of the pixel points in the neighborhood of each flicker point, in combination with the flicker evaluation value and the matching point state of the flicker point, extract the flame region in each gray image and trigger an alarm. 2.The forest fire monitoring method based on multi-modal data and intelligent cooperation according to claim 1, wherein, The method comprises the following steps: Calculate the Z-score of the pixel value of each pixel point in the thermal infrared image; calculate the cumulative sum of the Z-scores of all pixel points in the neighborhood of each pixel point in the thermal infrared image; The temperature anomaly degree is the product of the Z-score of each pixel point in the thermal infrared image and the cumulative sum; Obtain the segmentation threshold of the temperature anomaly degree of all pixel points in the thermal infrared image, denoted as the first segmentation threshold; the pixel points in the thermal infrared image with a temperature anomaly degree greater than or equal to the first segmentation threshold are denoted as abnormal points; Perform connected component analysis on all abnormal points in the thermal infrared image, and extract the connected component as an abnormal region. 3.The forest fire monitoring method based on multi-modal data and intelligent cooperation according to claim 1, wherein, The method comprises the following steps: Take the pixel point corresponding to the maximum pixel value in the abnormal region as the starting point, extend along multiple predetermined directions, take the pixel point extending to the boundary of the abnormal region as the end point, and take all pixel points from the starting point to the end point in each direction as feature pixel points; Calculate the difference between the pixel values of each feature pixel point and its adjacent feature pixel point in each direction, denoted as the first difference; Calculate the change rate of the pixel value between any feature pixel point in the thermal infrared image at each time and the pixel point at the corresponding position in the thermal infrared image at the previous time; Calculate the difference between the change rates of each feature pixel point and its adjacent feature pixel point in each direction, denoted as the second difference; Count the number of feature pixel points with the first difference and the second difference greater than or equal to 0 in each direction and the total number of all feature pixel points in each direction; The distribution feature value is the sum of the ratios of the number to the total number in all directions in the abnormal region. 4.The forest fire monitoring method based on multi-modal data and intelligent cooperation according to claim 1, wherein, The method comprises the following steps: a sum of differences between pixel values of all pixel points in the abnormal region in the thermal infrared image at each time and pixel values of corresponding positions in a thermal infrared image at a previous time of each time, is recorded as a first sum value; a temperature difference between each monitoring point in the abnormal region at each time and a previous time of each time is recorded as a temperature difference, and a sum of the temperature differences of all monitoring points in the abnormal region at each time is recorded as a second sum value; a ratio of the first sum value to the second sum value is recorded as a relative ratio, and a result of a negative mapping of a difference between the relative ratio and a preset value is used as a coordination degree of the abnormal region; an average value of temperatures of all monitoring points in the abnormal region at each time is calculated; the heat source evaluation value is a product of the average value, the distribution characteristic value, and the coordination degree. 5.The forest fire monitoring method based on multi-modal data and intelligent cooperation according to claim 1, wherein, the suspected degree of the abnormal region is determined, and a suspected region is selected, including: a normalized result of a product between a sum of temperature abnormal degrees of all pixel points in the abnormal region and the heat source evaluation value is used as the suspected degree of the abnormal region; and an abnormal region with a suspected degree greater than or equal to a preset threshold in the thermal infrared image at each time is selected as the suspected region. 6.The forest fire monitoring method based on multi-modal data and intelligent cooperation according to claim 1, wherein, the change confusion degree of each pixel point is calculated, including: gray values of all matching points corresponding to each pixel point in each gray image are clustered to obtain a plurality of clustering clusters; each gray image corresponds to a frame number, all matching points in each clustering cluster are arranged in ascending order according to the frame number of the gray image to which the matching points belong, a difference between the frame number of each matching point in each clustering cluster and the frame number of a neighboring matching point of the matching point is used as a time interval of the matching point, and a discrete degree of the time intervals of all matching points in each clustering cluster is used as a time dispersion degree of each clustering cluster; a sum of differences between gray values of all matching points in each clustering cluster and a matching point corresponding to a minimum time interval is used as a gray difference amount of each clustering cluster, and a product value of the time dispersion degree and the gray difference amount of each clustering cluster is calculated; the change confusion degree of each pixel point is calculated, including: 7.The forest fire monitoring method based on multi-modal data and intelligent cooperation according to claim 1, wherein, a number of gray values of each pixel point in each gray image that are different from gray values of all matching points of the pixel point is counted; an average value of wind speeds of all monitoring points in the suspected region at each time is recorded as an average wind speed, and a discrete degree of the average wind speed at each time before the time corresponding to each gray image of the suspected region is used as a wind speed fluctuation degree; a ratio of the number to the wind speed fluctuation degree is used as a flicker intensity of each pixel point; the flicker evaluation value of each pixel point in each gray image is the product of the change confusion degree and the flicker intensity; a segmentation threshold value of the flicker evaluation value of all pixel points in each gray image is recorded as a second segmentation threshold value, and a pixel point with a flicker evaluation value greater than or equal to the second segmentation threshold value is recorded as a flicker point. the flame edge value of each flicker point is determined, including: 8.The forest fire monitoring method based on multi-modal data and intelligent cooperation according to claim 1, wherein, a gradient intensity of each flicker point in each gray image is calculated, and a maximum gradient intensity of all pixel points in a neighborhood of each flicker point is obtained. ​ Calculate the sum of the difference between the gray value of all pixel points in the neighborhood of each flicker point and the preset gray value, denoted as the relative difference amount; Take the ratio of the maximum gradient intensity and the relative difference amount as the edge saliency of each flicker point; Count the number of times that the matching points corresponding to each flicker point do not belong to flicker points; The flame edge value is the product of the number of times, the flicker evaluation value and the edge saliency. 9.The forest fire monitoring method based on multi-modal data and intelligent cooperation according to claim 1, wherein, The extraction of the flame region in each frame of gray-scale image comprises: Obtain the segmentation threshold of the flame edge value of all flicker points in each frame of gray-scale image, denoted as the third segmentation threshold; take the flicker points with the flame edge value greater than or equal to the third segmentation threshold as the flame edge points; Take the closed region formed by all flame edge points in each frame of gray-scale image as the flame region. 10.A forest fire prevention monitoring system based on multi-modal data and intelligent cooperation, comprising a memory, a processor and a computer program stored in the memory and running on the processor, characterized in that, The processor implements the steps of the forest fire prevention monitoring method based on multi-modal data and intelligent collaboration according to any one of claims 1-9 when executing the computer program.

Images