Forest fire prevention monitoring method and system based on multi-modal data and intelligent cooperation

Abstract

The invention relates to the technical field of forest fire prevention monitoring, in particular to a forest fire prevention monitoring method and system based on multi-modal data and intelligent collaboration, and the method comprises the steps: obtaining thermal infrared images of a forest region at all moments, and the temperatures and wind speeds of different monitoring points in the forest region at all moments; calculating the temperature anomaly degree of each pixel point, and extracting all abnormal areas; determining a distribution characteristic value, a heat source evaluation value and a suspected degree of the abnormal region, and screening out a suspected region; acquiring a matching point corresponding to each pixel point in each frame of grayscale image; calculating the change confusion degree and the flicker evaluation value of each pixel point, and screening flicker points; and determining a flame edge value of each flicker point, and extracting a flame region in each frame of grayscale image. According to the method, the accuracy and reliability of forest fire identification are improved, and meanwhile, the false alarm rate caused by complex environment interference is remarkably reduced.

Description

Technical Field

[0001] This application relates to the field of forest fire prevention and monitoring technology, specifically to a forest fire prevention and monitoring method and system based on multimodal data and intelligent collaboration. Background Technology

[0003] Traditional methods often rely on images captured by single sensors, such as visible light or infrared cameras, for monitoring and identification. These methods are easily affected by complex natural environmental phenomena such as vegetation movement, changes in light intensity, and temperature inversions, resulting in a high false alarm rate in the monitoring and identification of forest fires. They also lack multi-dimensional and cross-modal cross-validation capabilities for assessing fire conditions, making it difficult to comprehensively evaluate forest fire risks. Consequently, the accuracy and reliability of early forest fire identification are low. Summary of the Invention

[0004] To address the aforementioned technical challenges, a forest fire monitoring method and system based on multimodal data and intelligent collaboration are provided to resolve existing issues.

[0005] The solution to the technical problem in this application is to provide a forest fire monitoring method and system based on multimodal data and intelligent collaboration, including the following steps: In a first aspect, embodiments of this application provide a forest fire monitoring method based on multimodal data and intelligent collaboration, the method comprising the following steps: Acquire thermal infrared images of the forest area at various times, as well as the temperature and wind speed at different monitoring points within the forest area at various times; Analyze the temperature anomaly of each pixel in the thermal infrared image and extract all abnormal regions; Based on the attenuation distribution of pixel values ​​in the abnormal region and the rate of change of pixel values ​​in the thermal infrared images at adjacent time points, the distribution characteristic values ​​of the abnormal region are determined. By analyzing the differences in pixel values ​​within abnormal regions in thermal infrared images at adjacent time points, as well as the temperature differences and distribution characteristics at monitoring points, the heat source assessment value of the abnormal region is obtained. Combined with the temperature anomaly degree, the suspicion degree of the abnormal region is determined, and suspected regions are screened out. Collect surveillance video of the suspected area, register each grayscale image frame with historical grayscale images, and obtain the matching points of each pixel in each grayscale image frame; All matching points of each pixel are classified, and the frame order distribution and gray value differences of matching points in different categories are analyzed. The degree of change disorder of each pixel is calculated. Combining the gray value difference between each pixel and the matching point, as well as the fluctuation of wind speed at the monitoring point, the flicker evaluation value of each pixel is calculated, and flickering points are screened out. By analyzing the gradient changes and grayscale deviations of pixels in the neighborhood of each flashing point, combined with the flashing evaluation value and the matching point status of the flashing point, the flame edge value of each flashing point is determined, the flame region in each frame of grayscale image is extracted, and an alarm is triggered.

[0006] Preferably, the analysis of temperature anomalies in each pixel of the thermal infrared image and the extraction of all abnormal regions include: Calculate the Z-score of the pixel value of each pixel in the thermal infrared image; calculate the sum of the Z-scores of all pixels in the neighborhood of each pixel in the thermal infrared image; The temperature anomaly is the product of the Z-scores of each pixel in the thermal infrared image and their sum. Obtain the segmentation threshold of the temperature anomaly of all pixels in the thermal infrared image, and denote it as the first segmentation threshold; denote the pixels in the thermal infrared image whose temperature anomaly is greater than or equal to the first segmentation threshold as anomaly points; Connectivity analysis was performed on all abnormal points in the thermal infrared image, and the connected components were extracted as abnormal regions.

[0007] Preferably, determining the distribution characteristic values ​​of the abnormal region includes: Starting from the pixel corresponding to the maximum pixel value in the abnormal region, the process extends along multiple predetermined directions, with the pixel on the boundary of the abnormal region as the endpoint. All pixels from the starting point to the endpoint in each direction are recorded as feature pixels. Calculate the difference in pixel value between each feature pixel and its neighboring feature pixels in each direction, and denote it as the first difference; Calculate the rate of change of pixel value between any feature pixel in the thermal infrared image at each time step and the corresponding pixel in the thermal infrared image at the previous time step. Calculate the difference in the rate of change between each feature pixel and its neighboring feature pixels in each direction, and denote it as the second difference; Count the number of feature pixels in each direction where both the first and second differences are greater than or equal to 0, and the total number of all feature pixels in each direction; The distribution characteristic value is the sum of the ratios of the number of each anomaly in all directions within the region to the total number.

[0008] Preferably, obtaining the heat source assessment value of the abnormal area includes: The sum of the differences between the pixel values ​​of all pixels in the abnormal region in the thermal infrared image at each time moment and the pixel values ​​of the corresponding pixels in the thermal infrared image at the previous time moment is denoted as the first sum value. The temperature difference between each monitoring point in the abnormal region and the previous time is calculated and denoted as temperature difference; the sum of the temperature differences of all monitoring points in the abnormal region at each time is denoted as the second sum value. The ratio of the first sum to the second sum is denoted as the relative ratio; the result of negative mapping of the difference between the relative ratio and the preset value is taken as the degree of synergy of the abnormal region. Calculate the average temperature of all monitoring points within the abnormal region at each time point; The heat source assessment value is the product of the average value, the distribution characteristic value, and the degree of synergy.

[0009] Preferably, determining the suspicion level of an abnormal region and filtering out suspected regions includes: taking the normalized result of the product between the sum of the temperature anomalies of all pixels in the abnormal region and the heat source evaluation value as the suspicion level of the abnormal region; and selecting abnormal regions in the thermal infrared images at each time point whose suspicion level is greater than or equal to a preset threshold as suspected regions.

[0010] Preferably, the calculation of the variation disorder of each pixel includes: Cluster the gray values ​​of all matching points corresponding to each pixel in each grayscale image to obtain multiple clusters; Each grayscale image corresponds to a frame number. All matching points within each cluster are sorted in ascending order according to the frame number of their respective grayscale images. The difference between the frame number of each matching point within each cluster and the frame number of its adjacent matching points is taken as the time interval of each matching point. The dispersion of the time interval of all matching points within each cluster is calculated as the time dispersion of each cluster. Calculate the sum of the differences in gray values ​​between all matching points within each cluster and the matching point corresponding to the minimum time interval, and use this as the gray value difference of each cluster; calculate the product of the time dispersion and the gray value difference of each cluster. The degree of variation disorder is the sum of the product values ​​of all clusters corresponding to each pixel in each frame of grayscale image.

[0011] Preferably, the step of calculating the flicker evaluation value of each pixel and filtering out flickering points includes: Count the number of pixels in each frame of a grayscale image whose grayscale value is different from the grayscale values ​​of all their matching pixels; The mean wind speed of all monitoring points in the suspected area at each time point is calculated and recorded as the average wind speed; the dispersion of the average wind speed at the time of acquisition corresponding to each grayscale image of the suspected area and all previous times is recorded as the wind speed fluctuation. The ratio of the quantity to the wind speed fluctuation is used as the flash intensity of each pixel. The flicker evaluation value is the product of the degree of variation disorder of each pixel in each frame of grayscale image and the flicker intensity; The segmentation threshold for obtaining the flicker evaluation value of all pixels in each frame of grayscale image is denoted as the second segmentation threshold; pixels with flicker evaluation values ​​greater than or equal to the second segmentation threshold are denoted as flickering points.

[0012] Preferably, determining the flame edge value of each flash point includes: Calculate the gradient intensity of each flashing point in each frame of grayscale image; obtain the maximum gradient intensity of all pixels in the neighborhood of each flashing point. Calculate the sum of the differences between the gray values ​​of all pixels in the neighborhood of each flashing point and the preset gray value, and record it as the relative difference. The ratio of the maximum gradient intensity to the relative difference is used as the edge salience of each scintillation point; Count the number of times the matching point corresponding to each flashing point does not belong to the flashing point; The flame edge value is the product of the number of times, the flicker evaluation value, and the edge salience.

[0013] Preferably, the extraction of the flame region from each grayscale image frame includes: Obtain the segmentation threshold of the flame edge value of all flashing points in each frame of grayscale image, and denote it as the third segmentation threshold; flashing points with flame edge values ​​greater than or equal to the third segmentation threshold are regarded as flame edge points; The closed region formed by all flame edge points in each frame of grayscale image is taken as the flame region.

[0014] Secondly, embodiments of this application also provide a forest fire prevention and monitoring system based on multimodal data and intelligent collaboration, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any of the above-described forest fire prevention and monitoring methods based on multimodal data and intelligent collaboration.

[0015] This application has at least the following beneficial effects: This application utilizes the temperature represented by the pixel values ​​of pixels in thermal infrared images acquired through satellite remote sensing to calculate the temperature anomaly degree of each pixel. Abnormal pixels are then filtered to extract anomalous regions. The advantages of this approach include locating all potential heat sources from the thermal infrared images and initially identifying areas with temperature anomalies. Furthermore, it determines the distribution characteristics of these anomalous regions, taking into account the heat attenuation distribution represented by the pixel values ​​within the anomalous regions and their trends over time, reducing interference from temperature inversion phenomena, and initially identifying real fire areas that conform to fire change characteristics. Finally, it obtains heat source assessment values ​​for the anomalous regions. The beneficial effect lies in comparing the changes in pixel values ​​over time within abnormal areas of thermal infrared images with the temperature differences collected by smart sensors in the corresponding areas on the ground. This further reduces interference caused by temperature inversion and reflects the likelihood that the abnormal area belongs to a real fire source. Determining the suspicion level of abnormal areas and filtering out suspected areas has the beneficial effect of comprehensively assessing the possibility that the abnormal area belongs to a fire area, thus improving the accuracy of fire area identification. Furthermore, launching drones to collect monitoring videos of suspected areas and analyzing the matching of pixels in different frames of grayscale images yields the matching points corresponding to each pixel in each frame of grayscale image. Its advantages include the ability to perform precise temporal analysis of the state of the same physical point at different times, facilitating subsequent evaluation of the flickering characteristics of flames; classifying matching points and calculating the degree of variation in each pixel, which takes into account the grayscale differences of different matching points within the same category in a grayscale image, as well as the disorder in the timing of the occurrence of matching points within the same category, thus effectively distinguishing between regular flame flickering and irregular leaf swaying; calculating the flicker evaluation value of each pixel and filtering out flickering points, which takes into account the number of times the grayscale values ​​of pixels and matching points are inconsistent and eliminates... The influence of wind speed fluctuations on the frequency of flame flashing is used to reflect the probability that a pixel belongs to the flame. The flame edge value of each flashing point is determined, and the flame area in each frame of grayscale image is extracted and an alarm is triggered. Its beneficial effect is that it takes into account the gradient change of the flashing point, the deviation of the grayscale value from the grayscale of the real flame color, and the number of times the matching point does not belong to the flashing point, reflecting the spread of the real flame edge. Thus, pixels located on the flame edge are accurately screened out, and fire areas are effectively detected. By fusing multimodal data, the accuracy and reliability of forest fire identification are improved, while the false alarm rate caused by complex environmental interference is significantly reduced. Attached Figure Description

[0016] The forest fire monitoring method based on multimodal data and intelligent collaboration of this application will be further described in detail below with reference to the accompanying drawings.

[0017] Figure 1A flowchart illustrating the steps of the forest fire monitoring method based on multimodal data and intelligent collaboration provided in this application embodiment; Figure 2 A flowchart illustrating the steps of the method for obtaining the flame region provided in this application embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of the forest fire monitoring method and system based on multimodal data and intelligent collaboration proposed in this application, in conjunction with the accompanying drawings and implementation examples, is provided. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0020] Please see Figure 1 The diagram illustrates a flowchart of a forest fire monitoring method based on multimodal data and intelligent collaboration according to an embodiment of this application. The method includes the following steps: Step 1: Obtain thermal infrared images of the forest area at various times, as well as the temperature and wind speed at different monitoring points within the forest area at various times.

[0021] Forest fires are characterized by their randomness and suddenness, causing immense damage in a short period. Preventing forest fires is extremely difficult. The emergence of remote monitoring technology and the continuous improvement of artificial intelligence algorithms have provided advanced early warning methods for forest fire prevention. Currently, forest fire monitoring widely employs methods such as ground patrols, lookout tower monitoring, aerial patrols, remote video monitoring, and satellite remote sensing.

[0022] If a single data source is used for fire monitoring, the identification of fires will have a high false alarm rate due to the influence of various environmental factors, making it impossible to accurately determine the status of forest fires. Therefore, an integrated, all-coverage monitoring network combining air, space, and ground is constructed to monitor forest fires by integrating data from multiple modalities. Here, air refers to satellite remote sensing, space refers to drones, and ground refers to ground sensors. Data is collected through these three methods for monitoring, specifically: By scanning the forest area with satellite remote sensing, thermal infrared images of various times can be obtained in real time. Monitoring points are pre-selected within the forest area, and smart sensors are deployed at these points to collect real-time data on temperature and wind speed at different monitoring points within the forest area. When an abnormal temperature area is detected through thermal infrared images, the temperature change data collected by the intelligent sensors at the abnormal area is used to determine the suspected fire area and send flight commands. The drone then uses its onboard surveillance camera to collect surveillance video of the suspected area based on the received commands. In this embodiment, the acquisition time interval of the thermal infrared image is 1 hour, and the data acquisition time interval of the smart sensor is 30 seconds. As other implementation methods, the implementer can set them according to the actual situation.

[0023] Since the acquisition time of the thermal infrared image is inconsistent with the acquisition time of the smart sensor, time alignment is performed using a timestamp alignment algorithm. The timestamp alignment algorithm is a well-known technology and will not be described in detail here.

[0024] All collected data are normalized. For example, for thermal infrared images, the pixel values ​​of all pixels are normalized, and the temperatures and wind speeds are normalized separately. In this embodiment, the maximum-minimum normalization method is used for normalization. The maximum-minimum normalization method is a well-known technique and will not be described in detail here. As other implementation methods, implementers can set their own methods according to the actual situation.

[0025] It should be noted that the pixel value of a pixel in a thermal infrared image is the radiation intensity.

[0026] Thus, thermal infrared images of the forest area at various times were obtained, as well as the temperature and wind speed at different monitoring points within the forest area at various times.

[0027] Step 2: Analyze the temperature anomaly of each pixel in the thermal infrared image and extract all abnormal regions; based on the attenuation distribution of pixel values ​​in the abnormal regions and the rate of change of pixel values ​​in the thermal infrared images at adjacent times, determine the distribution characteristic value of the abnormal regions; analyze the differences in pixel values ​​in the abnormal regions in the thermal infrared images at adjacent times, as well as the temperature differences and distribution characteristic values ​​at the monitoring points, to obtain the heat source assessment value of the abnormal regions; combine the temperature anomaly to determine the suspicion degree of the abnormal regions and screen out the suspected regions.

[0028] Traditional methods typically detect forest fires based on the flickering characteristics of flames and frame difference methods. However, in forested areas, adverse environmental conditions such as strong winds, rain, fog, and snow can affect the monitoring performance of sensors, resulting in lower accuracy in detecting forest fires and a higher likelihood of false alarms. For example, the flickering characteristics of flames manifest as rapid changes in pixel values ​​and irregular brightness variations. When strong winds blow and vegetation sways, this movement also causes rapid changes in pixel values ​​in the image, which in some cases resemble the flickering characteristics of flames. Random changes in light intensity during dawn and dusk can cause sudden increases in brightness in certain areas, leading to misjudgments of fires. Secondly, temperature inversion is a natural meteorological phenomenon. In a certain layer of the atmosphere, the temperature either increases with altitude or remains constant. This "cold below, warm above" atmospheric layer is called a temperature inversion layer. Due to the presence of a temperature inversion layer, thermal radiation signals from ground heat sources are accumulated, causing abnormally high temperatures in that area. Using only thermal infrared images, this area with abnormal temperatures may be misjudged as a fire area.

[0029] Therefore, the temperature anomaly degree is calculated by analyzing the anomalies in the pixel values ​​of different pixels in the thermal infrared image, specifically as follows: Calculate the Z-score of the pixel value of each pixel in the thermal infrared image; In this embodiment, the Z-score is calculated using the Z-score normalization method. The calculation of the Z-score is a well-known technique and will not be described in detail here.

[0030] Calculate the sum of the Z scores of all pixels in the neighborhood of each pixel in a thermal infrared image; In this embodiment, the size of the neighborhood is 3×3. As for other implementation methods, the implementer can set it according to the actual situation.

[0031] The product of the Z-score of each pixel in the thermal infrared image and the sum is used as the temperature anomaly of each pixel in the thermal infrared image. It should be noted that the larger the Z score, the greater the deviation of the pixel value from the overall distribution. The larger the sum, the more regional the temperature anomaly in the neighborhood of the pixel is, which is more consistent with the characteristics of a fire. The greater the temperature anomaly degree, the higher the temperature anomaly at the pixel, and the more likely it is to be a real heat source.

[0032] Obtain the segmentation threshold for the temperature anomaly of all pixels in the thermal infrared image, and denote it as the first segmentation threshold; In this embodiment, the average temperature anomaly of all pixels in all thermal infrared images over a historical period is collected as the first segmentation threshold.

[0033] Pixels in the thermal infrared image whose temperature anomaly is greater than or equal to the first segmentation threshold are recorded as anomalies. Perform connected component analysis on all abnormal points in the thermal infrared image and extract all connected components as abnormal regions. In this embodiment, the Two-Pass algorithm is used to extract connected components in the connected component labeling algorithm. The Two-Pass algorithm is a well-known technology and will not be described in detail here. As other implementation methods, implementers may use other methods of the prior art, such as the Seed-Filling algorithm, etc. This embodiment does not impose any special restrictions on this.

[0034] Secondly, when a temperature inversion occurs, it can also create abnormal areas in thermal infrared images. However, this heat originates from the atmospheric inversion layer, not from a ground-based fire source. Therefore, it can interfere with the actual fire area in the thermal infrared image. First, by analyzing the trend of pixel values ​​changing between adjacent thermal infrared images, distribution characteristic values ​​are calculated, specifically: The pixel with the maximum pixel value in the abnormal region of the thermal infrared image is obtained as the center point; It should be noted that if there are multiple pixels with the highest pixel value in the abnormal area, the pixel corresponding to the minimum sum of distances to all pixels on the boundary of the abnormal area is selected as the center point; secondly, the more likely the center point is to be located near the fire source, the higher its pixel value will be.

[0035] Starting from the center point of the abnormal region, the process extends along multiple predetermined directions, with the pixels extending to the boundary of the abnormal region as the endpoints. All pixels from the starting point to the endpoint in each direction are recorded as feature pixels. In this embodiment, the center point is extended in four directions: up, down, left, and right. As for other implementation methods, the implementer can set them according to the actual situation.

[0036] Calculate the difference in pixel value between each feature pixel and its neighboring feature pixels in each direction, and denote it as the first difference; In this embodiment, the difference between the pixel value of each feature pixel point and its next feature pixel point in each direction is calculated and denoted as the first difference.

[0037] Calculate the rate of change of pixel value between any feature pixel in the thermal infrared image at each time step and the corresponding pixel in the thermal infrared image at the previous time step. It should be noted that the calculation of the rate of change is a well-known technique. The specific process is as follows: the absolute value of the difference between the pixel value of any feature pixel in the thermal infrared image at each time moment and the corresponding pixel in the thermal infrared image at the previous time moment is recorded as the difference quantity; the time interval between the acquisition of two adjacent thermal infrared images is obtained, and the ratio of the difference quantity to the time interval is the rate of change.

[0038] Calculate the difference in the rate of change between each feature pixel and its neighboring feature pixels in each direction, and denote it as the second difference; In this embodiment, the difference in the rate of change between each feature pixel and its next feature pixel in each direction is calculated and denoted as the second difference.

[0039] Count the number of feature pixels in each direction where both the first and second differences are greater than or equal to 0. Count the total number of all feature pixels in each direction; calculate the sum of the ratios of the number of each feature pixel in each direction to the total number of feature pixels in the abnormal region, and use this sum as the distribution feature value of the abnormal region. It should be noted that the first and second differences reflect the gradient changes in pixel values ​​between feature pixels. The more the abnormal area conforms to the characteristics of high temperature and rapid change rate near the fire source, and gradually decreasing temperature and slower change rate away from the fire source, the larger the obtained distribution feature value. This indicates that the temperature distribution and change rate distribution within the abnormal area are more consistent with the typical characteristics of a fire, and the higher the probability that the abnormal area is a real fire source area.

[0040] Furthermore, by comparing the differences in pixel values ​​of pixels within abnormal regions in thermal infrared images at adjacent time points with the synchronization of temperature differences collected by intelligent sensors on the ground corresponding to the abnormal regions at adjacent time points, the degree of coordination is calculated, specifically as follows: The sum of the differences between the pixel values ​​of all pixels in the abnormal region in the thermal infrared image at each time moment and the pixel values ​​of the corresponding pixels in the thermal infrared image at the previous time moment is denoted as the first sum value. In this embodiment, the sum of the absolute values ​​of the differences between the pixel values ​​of all pixels in the abnormal region in the thermal infrared image at each time moment and the pixel values ​​of the corresponding pixels in the thermal infrared image at the previous time moment is calculated and denoted as the first sum value.

[0041] The temperature difference between each monitoring point in the abnormal area and the previous time is calculated and denoted as temperature difference. In this embodiment, the absolute value of the temperature difference between each monitoring point in the abnormal area at each time and the previous time is calculated and denoted as the temperature difference.

[0042] The sum of the temperature differences of all monitoring points included in the abnormal area at each time point is recorded as the second sum value; The ratio of the first sum to the second sum is denoted as the relative ratio; the result of negative mapping of the difference between the relative ratio and the preset value is taken as the degree of synergy of the abnormal region. In this embodiment, the preset value is 1. In other implementations, the implementer can set it according to the actual situation. Next, a negative mapping is performed on the absolute value of the difference between the relative ratio and the preset value. Specifically, the negative mapping process is as follows: a negative mapping is performed using an exponential function. Let the difference between the relative ratio and the preset value be denoted as... ,Will The result is used as the result of the negative mapping, where, It is an exponential function with the natural constant as the base.

[0043] It should be noted that the larger the first sum, the more drastic the pixel value changes of pixels in the abnormal area of ​​the thermal infrared image at adjacent time points. The larger the second sum, the greater the temperature difference collected by the smart sensor in the abnormal area at adjacent time points. The closer the relative ratio is to 1, the greater the degree of coordination, indicating that the pixel value changes of pixels in the abnormal area of ​​the thermal infrared image collected by the satellite are more consistent with the temperature changes of the monitoring points in the abnormal area. This reflects that the abnormal area is less likely to be caused by a temperature inversion phenomenon and is more likely to belong to a real fire source area.

[0044] Furthermore, the forest temperature is lower in the morning and gradually rises after being exposed to sunlight. This causes a coordinated change in pixel values ​​in the thermal infrared image and the temperature collected by the smart sensor. Therefore, by combining the distribution feature value and the degree of coordination with the real-time temperature collected by the smart sensor, a heat source assessment value is calculated, specifically: Calculate the average temperature of all monitoring points within the abnormal region at each time point; It should be noted that the temperature is a normalized value. The larger the average value, the more likely the abnormal area is to be a real fire source area.

[0045] The product of the average value, the distribution characteristic value, and the degree of synergy is used as the heat source assessment value of the abnormal area. It should be noted that the higher the heat source assessment value, the more likely the abnormal area is to be a real fire source area.

[0046] Furthermore, based on the degree of temperature anomaly and the heat source assessment value, the degree of suspicion is determined, and the abnormal areas are screened, specifically as follows: The normalized result of the product of the sum of the temperature anomalies of all pixels in the abnormal region and the heat source assessment value is used as the suspicion degree of the abnormal region. It should be noted that the larger the sum of the temperature anomalies of all pixels, the higher the probability that the temperature of the abnormal area is abnormal; the higher the probability that the abnormal area belongs to a fire area.

[0047] In this embodiment, the sigmoid function is used for normalization. The sigmoid function is a well-known technique and will not be described in detail here. As other implementation methods, implementers may use other methods of the prior art, such as the tanh function. This embodiment does not impose any special restrictions on this.

[0048] Select abnormal regions in the thermal infrared images at each time point whose suspicion level is greater than or equal to a preset threshold, and record them as suspected regions. In this embodiment, the process of setting the preset threshold is as follows: the average value of the suspicion of all abnormal areas in all thermal infrared images in the historical period is used as the preset threshold.

[0049] At this point, the suspected area has been identified.

[0050] Step 3: Register each grayscale image frame with historical grayscale images to obtain matching points for each pixel in each grayscale image frame; classify all matching points for each pixel, analyze the frame order distribution and grayscale value differences of matching points in different categories, calculate the degree of change disorder of each pixel, and calculate the flicker evaluation value of each pixel by combining the grayscale value difference between each pixel and the matching point, as well as the wind speed fluctuation at the monitoring point, and screen out the flickering points.

[0051] Using the surveillance camera mounted on the drone, real-time surveillance video of the suspected area is collected, and the video is processed in frames and then converted to grayscale to obtain each frame of grayscale image of the suspected area. In this embodiment, the frame rate of the frame processing is 30fps, which is consistent with the time interval of data acquisition by the smart sensor. As other implementation methods, the implementer can set it according to the actual situation and use the weighted average method for grayscale processing. Frame processing and weighted average method grayscale are well known technologies and will not be described in detail here.

[0052] Form an association set of all grayscale images preceding each grayscale image; For any pixel in each frame of grayscale image, image registration is performed with all pixels in all frames of grayscale images in the association set. The pixels in each frame of grayscale image in the association set that match the given pixel are recorded as matching points. In this embodiment, the SIFT (Scale-invariant feature transform) image matching algorithm is used for matching. The SIFT image registration algorithm is a well-known technology and will not be described in detail here. As other implementation methods, implementers may use other methods of the prior art, such as the SURF image registration algorithm, etc. This embodiment does not impose any special restrictions on this.

[0053] Secondly, real flames flicker, and their edges are typically high-frequency, irregular, and blurry. This manifests as significant variations in pixel grayscale values ​​across different frames of grayscale images. For example, when the flickering flame disappears, the pixel's grayscale value reflects the fixed image after the flame has disappeared. When the flickering flame persists, the pixel's grayscale value reflects the color of the flame, showing a significant difference from the value when the flickering flame has disappeared. Therefore, the changes in pixel grayscale values ​​in the grayscale image during flame flickering exhibit a certain regularity. Thus, by analyzing the changes in grayscale values ​​of all matching points corresponding to each pixel in each frame of the grayscale image, the degree of variation disorder can be calculated. Specifically: Cluster the gray values ​​of all matching points corresponding to each pixel in each grayscale image to obtain multiple clusters; In this embodiment, the k-means clustering algorithm is used to obtain two clusters. The k-means clustering algorithm is a well-known technology and will not be described in detail here. As other implementation methods, implementers may use other methods of existing technology, such as hierarchical clustering algorithms. This embodiment does not impose any special restrictions on this.

[0054] It should be noted that, through clustering, the pixels in the grayscale image during the flame flashing process are divided into two categories: the pixels where the flame disappears and the pixels where the flame appears.

[0055] Each grayscale image corresponds to a frame number. All matching points within each cluster are sorted in ascending order according to the frame number of their respective grayscale images. The difference between the frame number of each matching point within each cluster and the frame number of its adjacent matching points is taken as the time interval between each matching point. In this embodiment, the difference between the frame number of each matching point within each cluster and the frame number of its predecessor is calculated as the time interval between each matching point.

[0056] Calculate the dispersion of the time interval between all matching points within each cluster, and use it as the time dispersion of each cluster; In this embodiment, the degree of dispersion is measured by the coefficient of variation of the time interval between all matching points in each cluster. The calculation of the coefficient of variation is a well-known technique and will not be described in detail here. As other implementation methods, implementers may use other methods of the prior art, such as information entropy. This embodiment does not impose any special restrictions on this.

[0057] The sum of the differences in gray values ​​between all matching points within each cluster and the matching point corresponding to the minimum time interval is calculated as the gray value difference of each cluster. In this embodiment, the sum of the absolute values ​​of the differences in grayscale values ​​between all matching points within each cluster and the matching point corresponding to the minimum time interval is calculated as the grayscale difference of each cluster.

[0058] Calculate the product of the temporal dispersion and the grayscale difference of each cluster; use the sum of the products of all clusters corresponding to each pixel in each frame of grayscale image as the degree of variation disorder of each pixel in each frame of grayscale image. It should be noted that the greater the time dispersion, the less regular the change time of the pixels within the cluster, and the more irregular and chaotic the flame flickering. The greater the grayscale difference, the greater the difference in grayscale values ​​of the pixels. Conversely, the greater the possibility that the grayscale values ​​of the pixels represent the same background or flame. The greater the degree of change disorder, the more random the grayscale change of the pixel, and the less likely it is to belong to the flame feature.

[0059] Secondly, the inconsistencies in grayscale values ​​between each pixel and its matching point, as well as fluctuations in wind speed, are analyzed to calculate the flicker intensity, specifically: Count the number of pixels in each frame of a grayscale image whose grayscale value is different from the grayscale values ​​of all their matching pixels; Calculate the average wind speed of all monitoring points in the suspected area at each time point, and record it as the average wind speed; Each grayscale image corresponds to a single acquisition time. The degree of dispersion of the average wind speed at the corresponding time and all previous times in each grayscale image of the suspected area is used as the wind speed fluctuation. In this embodiment, the degree of dispersion is measured by calculating the coefficient of variation of the average wind speed at the corresponding time and all previous times of each grayscale image of the suspected area. The calculation of the coefficient of variation is a well-known technique and will not be described in detail here.

[0060] The ratio of the quantity to the wind speed fluctuation is used as the flash intensity of each pixel. It should be noted that the more violent the wind speed fluctuation, the greater the relative fluctuation and the more frequent the flame flicker. Therefore, by calculating the flicker intensity and eliminating the interference of wind speed fluctuation on the flame flicker, the greater the flicker intensity, the greater the possibility that it belongs to the edge of the flame.

[0061] Furthermore, based on the degree of variation disorder and flicker intensity, a flicker evaluation value is determined, specifically as follows: The product of the degree of variation disorder of each pixel in each frame of grayscale image and the flicker intensity is used as the flicker evaluation value of each pixel in each frame of grayscale image. It should be noted that the larger the flicker evaluation value, the more likely the pixel is to be a pixel at the flame location; conversely, the smaller the value, the more likely the pixel is to be an interference point.

[0062] Furthermore, based on the flicker evaluation value, pixels are filtered, specifically as follows: The segmentation threshold for obtaining the flicker evaluation value of all pixels in each frame of grayscale image is denoted as the second segmentation threshold. In this embodiment, the Otsu's method is used to obtain the segmentation threshold. Otsu's method is a well-known technique and will not be described in detail here. As other implementation methods, implementers may use other methods of the prior art, such as cross-validation. This embodiment does not impose any special restrictions on this.

[0063] Pixels with a flicker evaluation value greater than or equal to the second segmentation threshold are recorded as flickering points; At this point, we have obtained each flickering point in each frame of grayscale image corresponding to the suspected region.

[0064] Step 4: By combining the gradient changes and grayscale deviations of the pixels in the neighborhood of each flashing point with the flashing evaluation value and the matching point status of the flashing point, determine the flame edge value of each flashing point, extract the flame area in each frame of grayscale image and trigger an alarm.

[0065] Furthermore, the edge of a real flame spreads outwards over time, and the number of pixels representing the flame edge in a continuous grayscale image that are flashing points is dynamically changing; moreover, the grayscale value of the flame edge changes drastically, typically exhibiting a high gradient value. Therefore, by analyzing the similarity between the grayscale values ​​of pixels in the vicinity of the flashing point and the color of the flame edge, as well as the gradient changes of the flashing point, edge saliency is calculated, specifically as follows: Calculate the gradient intensity of each flashing point in each frame of grayscale image; obtain the maximum gradient intensity of all pixels in the neighborhood of each flashing point. In this embodiment, the size of the neighborhood is 3×3. As for other implementation methods, the implementer can set it according to the actual situation. It should be noted that the calculation of the gradient intensity of the pixel points of the image is a well-known technique, and will not be described in detail here.

[0066] Calculate the sum of the differences between the gray values ​​of all pixels in the neighborhood of each flashing point and the preset gray value, and record it as the relative difference. It should be noted that the preset grayscale value is the grayscale value corresponding to the edge of the flame. Since the color of the flame generally varies from red to yellow, the grayscale value corresponding to the flame can be obtained through certain prior knowledge or experiments. For example, an image of a real flame can be obtained, and the average grayscale value of all pixels corresponding to the edge of the flame can be obtained as the grayscale value corresponding to the edge of the flame. In this embodiment, the grayscale value corresponding to the flame is 185. In other implementation methods, the implementer can set it according to the actual situation.

[0067] The ratio of the maximum gradient intensity to the relative difference is used as the edge salience of each scintillation point; It should be noted that the larger the maximum gradient intensity, the more drastic the grayscale change of the pixels in the neighborhood of the flashing point, and the more obvious the edge features it represents, the greater the possibility that it is the edge of a flame. The smaller the relative difference, the closer the grayscale of the pixels in the neighborhood is to the color features of the flame, and the greater the edge salience, indicating that the edge features in the neighborhood are significant and the grayscale value is close to the color features of the flame, making it more likely to be the edge of a flame.

[0068] Secondly, by considering the cases where each flickering point in each grayscale image does not belong to a flickering point in different grayscale images within the association set, and combining the flickering evaluation value and edge saliency, the flame edge value is determined, specifically as follows: Count the number of times that the corresponding matching point of each flashing point in each grayscale image frame is not a flashing point in all grayscale images in the association set; It should be noted that the more times the flashing point is mentioned, the less fixed the flashing point is in the same position, the less likely it is to be caused by the shaking of leaves or light interference in the forest area, and the more it has the characteristics of the growth and spread of the edge of the flame.

[0069] The product of the number of flashes, the flicker evaluation value, and the edge saliency is used as the flame edge value of each flicker point in each frame of grayscale image; It should be noted that the larger the flame edge value, the more the flashing point matches the characteristics of the flame edge, and the more likely it is to be a pixel on the flame edge.

[0070] The segmentation threshold for the flame edge values ​​of all flashing points in each frame of grayscale image is obtained and denoted as the third segmentation threshold. In this embodiment, the process of obtaining the third segmentation threshold is as follows: the average value of the flame edge value of all flashing points in each frame of grayscale image is used as the third segmentation threshold.

[0071] Flashing points with flame edge values ​​greater than or equal to the third segmentation threshold are designated as flame edge points; The closed region formed by all flame edge points in each frame of grayscale image is taken as the flame region; Based on the identified flame area, an alarm is immediately triggered, and by combining the drone's GPS coordinates and image recognition, the fire location is accurately pinpointed on an electronic map, thereby sending fire alarm information to relevant personnel.

[0072] Furthermore, the flowchart of the method for obtaining the flame region provided in this application embodiment is as follows: Figure 2 As shown.

[0073] Based on the same inventive concept as the above methods, this application also provides a forest fire prevention and monitoring system based on multimodal data and intelligent collaboration, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-described forest fire prevention and monitoring methods based on multimodal data and intelligent collaboration.

[0074] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0075] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this application, without departing from the content of the technical solution of this application, shall fall within the protection scope of the technical solution of this application.

Claims

1. A forest fire monitoring method based on multimodal data and intelligent collaboration, characterized in that, The method includes the following steps: Acquire thermal infrared images of the forest area at various times, as well as the temperature and wind speed at different monitoring points within the forest area at various times; Analyze the temperature anomaly of each pixel in the thermal infrared image and extract all abnormal areas; Based on the attenuation distribution of pixel values ​​in the abnormal region and the rate of change of pixel values ​​in the thermal infrared images at adjacent time points, the distribution characteristic values ​​of the abnormal region are determined. By analyzing the differences in pixel values ​​within abnormal regions in thermal infrared images at adjacent time points, as well as the temperature differences and distribution characteristics at monitoring points, the heat source assessment value of the abnormal region is obtained. Combined with the temperature anomaly degree, the suspicion degree of the abnormal region is determined, and suspected regions are screened out. Collect surveillance video of the suspected area, register each grayscale image frame with historical grayscale images, and obtain the matching points of each pixel in each grayscale image frame; All matching points of each pixel are classified, and the frame order distribution and gray value differences of matching points in different categories are analyzed. The degree of change disorder of each pixel is calculated. Combining the gray value difference between each pixel and the matching point, as well as the fluctuation of wind speed at the monitoring point, the flicker evaluation value of each pixel is calculated, and flickering points are screened out. By analyzing the gradient changes and grayscale deviations of pixels in the neighborhood of each flashing point, combined with the flashing evaluation value and the matching point status of the flashing point, the flame edge value of each flashing point is determined, the flame region in each frame of grayscale image is extracted, and an alarm is triggered.

2. The forest fire monitoring method based on multimodal data and intelligent collaboration as described in claim 1, characterized in that, The analysis examines the temperature anomalies of each pixel in the thermal infrared image and extracts all abnormal regions, including: Calculate the Z-score of the pixel value of each pixel in the thermal infrared image; calculate the sum of the Z-scores of all pixels in the neighborhood of each pixel in the thermal infrared image; The temperature anomaly is the product of the Z-scores of each pixel in the thermal infrared image and their sum. Obtain the segmentation threshold of the temperature anomaly of all pixels in the thermal infrared image, and denote it as the first segmentation threshold; denote the pixels in the thermal infrared image whose temperature anomaly is greater than or equal to the first segmentation threshold as anomaly points; Connectivity analysis was performed on all abnormal points in the thermal infrared image, and the connected components were extracted as abnormal regions.

3. The forest fire monitoring method based on multimodal data and intelligent collaboration as described in claim 1, characterized in that, The determination of the distribution characteristic values ​​of the abnormal region includes: Starting from the pixel corresponding to the maximum pixel value in the abnormal region, the process extends along multiple predetermined directions, with the pixel on the boundary of the abnormal region as the endpoint. All pixels from the starting point to the endpoint in each direction are recorded as feature pixels. Calculate the difference in pixel value between each feature pixel and its neighboring feature pixels in each direction, and denote it as the first difference; Calculate the rate of change of pixel value between any feature pixel in the thermal infrared image at each time step and the corresponding pixel in the thermal infrared image at the previous time step. Calculate the difference in the rate of change between each feature pixel and its neighboring feature pixels in each direction, and denote it as the second difference; Count the number of feature pixels in each direction where both the first and second differences are greater than or equal to 0, and the total number of all feature pixels in each direction; The distribution characteristic value is the sum of the ratios of the number of each anomaly in all directions within the region to the total number.

4. The forest fire monitoring method based on multimodal data and intelligent collaboration as described in claim 1, characterized in that, The obtained heat source assessment value for the abnormal area includes: The sum of the differences between the pixel values ​​of all pixels in the abnormal region in the thermal infrared image at each time moment and the pixel values ​​of the corresponding pixels in the thermal infrared image at the previous time moment is denoted as the first sum value. The temperature difference between each monitoring point in the abnormal region and the previous time is calculated and denoted as temperature difference; the sum of the temperature differences of all monitoring points in the abnormal region at each time is denoted as the second sum value. The ratio of the first sum to the second sum is denoted as the relative ratio; the result of negative mapping of the difference between the relative ratio and the preset value is taken as the degree of synergy of the abnormal region. Calculate the average temperature of all monitoring points within the abnormal region at each time point; The heat source assessment value is the product of the average value, the distribution characteristic value, and the degree of synergy.

5. The forest fire monitoring method based on multimodal data and intelligent collaboration as described in claim 1, characterized in that, The process of determining the likelihood of an abnormal region and selecting suspected regions includes: taking the normalized result of the product of the sum of the temperature anomalies of all pixels in the abnormal region and the heat source evaluation value as the likelihood of the abnormal region; and selecting abnormal regions in the thermal infrared images at each time point whose likelihood is greater than or equal to a preset threshold as suspected regions.

6. The forest fire monitoring method based on multimodal data and intelligent collaboration as described in claim 1, characterized in that, The calculation of the variation disorder of each pixel includes: Cluster the gray values ​​of all matching points corresponding to each pixel in each grayscale image to obtain multiple clusters; Each grayscale image corresponds to a frame number. All matching points within each cluster are sorted in ascending order according to the frame number of their respective grayscale images. The difference between the frame number of each matching point within each cluster and the frame number of its adjacent matching points is taken as the time interval of each matching point. The dispersion of the time interval of all matching points within each cluster is calculated as the time dispersion of each cluster. Calculate the sum of the differences in gray values ​​between all matching points within each cluster and the matching point corresponding to the minimum time interval, and use this as the gray value difference of each cluster; calculate the product of the time dispersion and the gray value difference of each cluster. The degree of variation disorder is the sum of the product values ​​of all clusters corresponding to each pixel in each frame of grayscale image.

7. The forest fire monitoring method based on multimodal data and intelligent collaboration as described in claim 1, characterized in that, The calculation of the flicker evaluation value for each pixel and the filtering of flickering points includes: Count the number of pixels in each frame of a grayscale image whose grayscale value is different from the grayscale values ​​of all their matching pixels; The mean wind speed of all monitoring points in the suspected area at each time point is calculated and recorded as the average wind speed; the dispersion of the average wind speed at the time of acquisition corresponding to each grayscale image of the suspected area and all previous times is recorded as the wind speed fluctuation. The ratio of the quantity to the wind speed fluctuation is used as the flash intensity of each pixel. The flicker evaluation value is the product of the degree of variation disorder of each pixel in each frame of grayscale image and the flicker intensity; The segmentation threshold for obtaining the flicker evaluation value of all pixels in each frame of grayscale image is denoted as the second segmentation threshold; pixels with flicker evaluation values ​​greater than or equal to the second segmentation threshold are denoted as flickering points.

8. The forest fire monitoring method based on multimodal data and intelligent collaboration as described in claim 1, characterized in that, Determining the flame edge value of each flash point includes: Calculate the gradient intensity of each flashing point in each frame of grayscale image; obtain the maximum gradient intensity of all pixels in the neighborhood of each flashing point. Calculate the sum of the differences between the gray values ​​of all pixels in the neighborhood of each flashing point and the preset gray value, and record it as the relative difference. The ratio of the maximum gradient intensity to the relative difference is used as the edge salience of each scintillation point; Count the number of times the matching point corresponding to each flashing point does not belong to the flashing point; The flame edge value is the product of the number of times, the flicker evaluation value, and the edge salience.

9. The forest fire monitoring method based on multimodal data and intelligent collaboration as described in claim 1, characterized in that, The extraction of the flame region from each grayscale image frame includes: Obtain the segmentation threshold of the flame edge value of all flashing points in each frame of grayscale image, and denote it as the third segmentation threshold; flashing points with flame edge values ​​greater than or equal to the third segmentation threshold are regarded as flame edge points; The closed region formed by all flame edge points in each frame of grayscale image is taken as the flame region.

10. A forest fire monitoring system based on multimodal data and intelligent collaboration, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the forest fire prevention and monitoring method based on multimodal data and intelligent collaboration as described in any one of claims 1-9.

Images