Fire source trend prediction method, device, equipment, medium and product

Abstract

The embodiment of the invention provides a fire source trend prediction method and device, equipment, a medium and a product. The method comprises the following steps: acquiring multi-source data such as a visible light image, an infrared image and environmental data of a target area, respectively carrying out fire source feature extraction on the visible light image and the infrared image to obtain a visible spectrum feature and an infrared spectrum feature, then fusing the visible spectrum feature and the infrared spectrum feature to obtain a fire source fusion feature, and finally obtaining a fire source fusion feature; space-time coding fusion processing is carried out on the continuous geographic coordinates corresponding to the target area, the fire source fusion features and meteorological features in the environmental data to generate fire source space-time coding features, space-time correlation focusing processing is carried out on the fire source space-time coding features to obtain fire source space-time deepening features, and finally, the fire source space-time deepening features are obtained. And performing multi-scale space-time decoding on the fire source space-time deepening features to obtain a fire source prediction probability graph of the target area. The method is used for achieving the effect of improving the fire source trend prediction precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a fire source trend prediction method, device, equipment, medium and product. BACKGROUND

[0003] Traditional fire monitoring methods mainly rely on meteorological station data, manual patrol and experience judgment, and have problems such as limited monitoring range, slow response speed and low prediction accuracy. With the development of remote sensing technology, satellites (such as VIIRS, Landsat, MODIS) and unmanned aerial vehicles can provide multispectral, infrared, visible light and other data to provide data support for fire monitoring.

[0004] However, due to the differences in data format, resolution and observation angle of different sensors (visible light, infrared, remote sensing), it is difficult to directly fuse them. The existing methods usually perform fire prediction based on a single data modality, and have the problem of low accuracy of fire prediction results. SUMMARY

[0005] The fire source trend prediction method, device, equipment, medium and product provided by the embodiments of the present application can improve the accuracy of fire source trend prediction.

[0006] In a first aspect, the embodiments of the present application provide a fire source trend prediction method, comprising:

[0007] Obtaining multi-source data of a target region, the multi-source data at least including visible light images, infrared images and environmental data.

[0008] Performing fire feature extraction on the visible light images to obtain visible light spectrum features corresponding to the visible light images, and performing fire feature extraction on the infrared images to obtain infrared spectrum features corresponding to the infrared images.

[0009] Fusing the visible light spectrum features and the infrared spectrum features to obtain fire fusion features.

[0010] Performing spatio-temporal coding fusion processing on continuous geographic coordinates corresponding to the target region, the fire fusion features and meteorological features in the environmental data to obtain fire spatio-temporal coding features.

[0011] Performing spatio-temporal correlation focusing processing on the fire spatio-temporal coding features to obtain fire spatio-temporal deepening features.

[0012] Performing multi-scale spatio-temporal decoding on the fire spatio-temporal deepening features to obtain a fire prediction probability map of the target region.

[0013] In a possible implementation, in combination with the first aspect, the fire feature extraction on the visible light images to obtain the visible light spectrum features corresponding to the visible light images comprises:

[0014] The visible light image is subjected to deformable convolution processing to obtain a spatial attention vector corresponding to the visible light image.

[0015] The visible light image is subjected to adaptive deepening fire source related channel feature processing to obtain a channel attention vector.

[0016] The visible light image, the spatial attention vector and the channel attention vector are dynamically fused to obtain a visible spectrum feature.

[0017] In a possible implementation, in combination with the first aspect, fire source feature extraction is performed on the infrared image to obtain an infrared spectrum feature corresponding to the infrared image, including:

[0018] The fire source feature extraction is performed on the infrared image by a multi-scale thermal radiation perception network to obtain the infrared spectrum feature; wherein a thermal physical guided attention mechanism is integrated in the multi-scale thermal radiation perception network.

[0019] In a possible implementation, in combination with the first aspect, the visible spectrum feature and the infrared spectrum feature are fused to obtain a fire source fusion feature, including:

[0020] The visible spectrum feature and the infrared spectrum feature are input into a lightweight sub-network to obtain an offset.

[0021] Based on the visible spectrum feature, the infrared spectrum feature and the offset, a precise alignment feature is obtained.

[0022] The precise alignment feature is subjected to modal feature separation to obtain a visible light precise alignment feature and an infrared precise alignment feature.

[0023] The visible light precise alignment feature and the infrared precise alignment feature are subjected to semantic association processing to obtain a semantic attention score.

[0024] The infrared radiation intensity, the visible light reflectivity and the environmental temperature are subjected to physical attention calculation to obtain a physical attention score.

[0025] The second aspect, the embodiments of the present application provide a fire source trend prediction device, including:

[0026] The acquisition module is configured to acquire multi-source data of a target region, the multi-source data at least including a visible light image, an infrared image and environmental data.

[0027] The extraction module is configured to perform fire source feature extraction on the visible light image to obtain a visible spectrum feature corresponding to the visible light image, and perform fire source feature extraction on the infrared image to obtain an infrared spectrum feature corresponding to the infrared image.

[0028] a fusion module configured to fuse the visible spectrum feature and the infrared spectrum feature to obtain a fire source fusion feature.

[0029] a prediction module configured to perform spatiotemporal coding fusion processing on the continuous geographic coordinates corresponding to the target region, the fire source fusion feature, and meteorological features in the environment data to obtain a fire source spatiotemporal coding feature; perform spatiotemporal correlation focusing processing on the fire source spatiotemporal coding feature to obtain a fire source spatiotemporal deepening feature; and perform multi-scale spatiotemporal decoding on the fire source spatiotemporal deepening feature to obtain a fire source prediction probability map of the target region.

[0030] In a possible implementation, in combination with the second aspect, the extraction module performs fire source feature extraction on the visible light image to obtain a visible spectrum feature corresponding to the visible light image, specifically for:

[0031] performing deformable convolution processing on the visible light image to obtain a spatial attention vector corresponding to the visible light image.

[0032] performing adaptive deepening fire source related channel feature processing on the visible light image to obtain a channel attention vector.

[0033] dynamically fusing the visible light image, the spatial attention vector, and the channel attention vector to obtain the visible spectrum feature.

[0034] In a possible implementation, in combination with the second aspect, the extraction module performs fire source feature extraction on the infrared image to obtain an infrared spectrum feature corresponding to the infrared image, specifically for:

[0035] performing fire source feature extraction on the infrared image through a multi-scale thermal radiation perception network to obtain the infrared spectrum feature; wherein the multi-scale thermal radiation perception network integrates a thermal physics guided attention mechanism.

[0036] In a possible implementation, in combination with the second aspect, the fusion module fuses the visible spectrum feature and the infrared spectrum feature to obtain the fire source fusion feature, specifically for:

[0037] inputting the visible spectrum feature and the infrared spectrum feature into a lightweight subnetwork to obtain an offset.

[0038] obtaining precise alignment features based on the visible spectrum feature, the infrared spectrum feature, and the offset.

[0039] performing modal feature separation on the precise alignment features to obtain visible light precise alignment features and infrared precise alignment features.

[0040] performing semantic correlation processing on the visible light precise alignment features and the infrared precise alignment features to obtain semantic attention scores.

[0041] The physical attention score is calculated by performing physical attention calculation on infrared radiation intensity, visible light reflectivity, and ambient temperature.

[0042] Based on the semantic attention score, the physical attention score, the accurate alignment feature, and the dynamic weight, the feature fusion driven by semantic and physical double attention is performed to obtain the fire source fusion feature.

[0043] In a possible implementation, in combination with the second aspect, the fusion module obtains the accurate alignment feature based on the visible spectrum feature, the infrared spectrum feature, and the offset, and is specifically used for:

[0044] The visible spectrum feature, the infrared spectrum feature, and the offset are processed by deformable convolution to obtain the coarse alignment feature.

[0045] The coarse alignment feature is processed by wavelet transform and multi-scale decomposition to obtain corresponding low-frequency features and high-frequency features.

[0046] The high-frequency features are processed by deformable convolution to obtain optimized high-frequency features.

[0047] The optimized high-frequency features and the low-frequency features are processed by reconstruction to obtain the accurate alignment feature.

[0048] In a possible implementation, in combination with the second aspect, the prediction module performs spatio-temporal coding fusion processing on the continuous geographic coordinates corresponding to the target area, the fire source fusion feature, and the meteorological features in the environmental data to obtain the fire source spatio-temporal coding feature, and is specifically used for:

[0049] The continuous geographic coordinates corresponding to the target area are spatially discretized and coded to obtain a spatial position code.

[0050] The spatial position code, the fire source fusion feature, and the meteorological feature are processed by feature splicing to obtain the fire source spatio-temporal coding feature.

[0051] In a possible implementation, in combination with the second aspect, the prediction module performs spatio-temporal correlation focusing processing on the fire source spatio-temporal coding feature to obtain the fire source spatio-temporal deepening feature, and is specifically used for:

[0052] Based on the fire source spatio-temporal coding feature, a spatial neighborhood attention weight is obtained.

[0053] Based on the spatial neighborhood attention weight, the fire source spatio-temporal coding feature is processed by sparsification to obtain a fire source spatial neighborhood deepening feature.

[0054] A fire source evolution short-term time feature corresponding to the fire source spatio-temporal coding feature is determined.

[0055] The fire source spatial neighborhood deepening feature and the fire source evolution short-term time feature are fused to obtain the fire source spatio-temporal deepening feature.

[0056] In a possible implementation, in combination with the second aspect, the prediction module determines the fire evolution short-time feature corresponding to the fire spatiotemporal coding feature, and specifically for:

[0057] The fire spatiotemporal coding feature is subjected to grid splitting processing to obtain a plurality of grids corresponding to the fire spatiotemporal coding feature.

[0058] Based on the short-time window and the time sequence feature corresponding to each grid, the fire evolution grid feature of each grid is calculated.

[0059] The fire evolution grid features of all the grids are reassembled to obtain the fire evolution short-time feature.

[0060] In a possible implementation, in combination with the second aspect, the prediction module performs multi-scale spatiotemporal decoding on the fire spatiotemporal deepening feature to obtain the fire prediction probability map of the target region, and specifically for:

[0061] Based on the causal Fourier transform layer, the time sequence information in the fire spatiotemporal deepening feature is decomposed into a trend item and a periodic item.

[0062] The fire spatiotemporal deepening feature, the trend item, and the periodic item are subjected to merging processing to obtain an intermediate feature.

[0063] The intermediate feature is subjected to hole convolution and up-sampling processing to obtain the fire prediction probability map of the target region.

[0064] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory connected with the processor in communication.

[0065] The memory stores computer execution instructions.

[0066] The processor executes the computer execution instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementations of the first aspect.

[0067] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions. When the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the first aspect and / or various possible implementations of the first aspect.

[0068] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the computer program implements the first aspect and / or various possible implementations of the first aspect.

[0069] The fire source trend prediction method, device, equipment, medium and product provided by the embodiment of the application, by acquiring multi-source data such as a visible light image, an infrared image and environment data of a target region, performing fire source feature extraction on the visible light image and the infrared image respectively, obtaining visible spectrum features and infrared spectrum features, then fusing the two to obtain fire source fusion features, then performing spatio-temporal coding fusion processing on the continuous geographical coordinates corresponding to the target region, the fire source fusion features and the meteorological features in the environment data to generate fire source spatio-temporal coding features, then performing spatio-temporal correlation focusing processing on the fire source spatio-temporal coding features to obtain fire source spatio-temporal deepening features, and finally performing multi-scale spatio-temporal decoding on the fire source spatio-temporal deepening features to obtain a fire source prediction probability map of the target region, the effect of improving the fire source trend prediction accuracy is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0070] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0071] Figure 1 A scene schematic diagram of a fire source trend prediction method provided by the application;

[0072] Figure 2 A flowchart of a fire source trend prediction method provided by the application Figure 1 ;

[0073] Figure 3 A flowchart of a fire source trend prediction method provided by the application Figure 2 ;

[0074] Figure 4 A flowchart of a fire source trend prediction method provided by the application Figure 3 ;

[0075] Figure 5 A specific example diagram of a fire source trend prediction method provided by the application;

[0076] Figure 6 A structure schematic diagram of a fire source trend prediction device provided by the application;

[0077] Figure 7 A structure schematic diagram of an electronic device provided by the application.

[0078] The specific embodiments of the application have been shown by the above drawings, and will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the application by any means, but to illustrate the concept of the application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0079] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements throughout the description. The following exemplary embodiments described herein represent illustrations only. However, these are merely examples of apparatus and methods in accordance with aspects of the present application as detailed in the appended claims.

[0080] First, the terms related to the present application are explained:

[0081] Visible light image: refers to the image taken by using the visible light band, which can reflect the details of the scene visible to the human eye, and can be used to capture the visible form, position and surrounding environment of the fire source, etc. visual information, which is the basic data source for fire source monitoring.

[0082] Infrared image: refers to the image generated by detecting the infrared radiation of an object, which can reflect the temperature difference of the object, and can be used to capture the high temperature area of the fire source. Because the temperature of the fire source is high and the infrared radiation is strong, it can accurately locate the core and spread range of the fire source.

[0083] Denoising: refers to removing noise in the image caused by equipment and environment to ensure the accuracy of subsequent feature extraction and prediction.

[0084] Feature extraction: refers to the process of extracting key features that reflect the essential attributes and representative information from data, such as temperature values, shape changes, and diffusion rates from visible light images and infrared images. It provides input for the prediction model.

[0085] Lightweight subnetwork: refers to a network module in a deep learning model that has the characteristics of simple structure, small number of parameters, and small amount of calculation, which provides specific functions for the model without significantly increasing the overall computational burden of the model.

[0086] Causal Fourier transform: generally refers to the Fourier transform combined with causal relationship, focusing on the frequency characteristics of causal correlation in time series data.

[0087] Hole convolution: generally refers to a convolution method that does not need to increase the number of parameters by introducing a hole in the convolution kernel to expand the receptive field, for example, which can be used to capture larger context information in the fire source image, such as the spatial relationship between the fire source and the surrounding combustible materials.

[0088] Convolution: generally refers to a basic operation that extracts local features by sliding the convolution kernel and input data.

[0089] Deformable convolution: generally refers to a convolution method in which the shape of the convolution kernel can be adaptively adjusted according to the input data, enhancing the ability to capture irregular features.

[0090] Wavelet decomposition: generally refers to the method of decomposing signals into wavelet components of different scales and frequencies to achieve multi-resolution analysis.

[0091] Short-term time window: generally refers to a shorter time interval selected in time series analysis for observing recent data changes.

[0092] Secondly, the application background of the embodiments of the present application is explained:

[0093] In recent years, global climate change and intensified human activities have led to frequent natural disasters such as forest fires, while the expansion of urban scale and the surge in electricity demand have highlighted the potential risks of power facilities, posing a serious threat to the ecological environment and human life and property safety. Traditional fire monitoring methods mainly rely on meteorological station data, manual patrols and experience-based judgments, which have limited monitoring range, slow response speed and low prediction accuracy. With the development of remote sensing technology, satellites (such as VIIRS, Landsat, MODIS) and unmanned aerial vehicles can provide multispectral, infrared, visible light and other data to support fire monitoring. However, due to the differences in data format, resolution and observation angle of different sensors (visible light, infrared, remote sensing), it is difficult to directly fuse them. Existing methods usually perform fire prediction based on a single data modality, which has the problem of low accuracy of fire prediction results.

[0094] To solve the above problems, the inventors have studied whether multi-source data can be obtained and fused to predict fire trends based on the fused data. The inventors propose a fire trend prediction method that obtains multisource data such as visible light images and infrared images, extracts features from visible light images based on dynamic spatial and channel attention mechanisms to obtain visible spectral features, extracts features from infrared images based on a multi-scale thermal radiation perception network to obtain infrared spectral features, geometrically aligns the visible spectral features and infrared spectral features, and models the fire source fusion features based on semantic association and physical laws to focus on local spatial dependence of fire spread and capture short-term fire risk evolution trends. The fire source fusion features are spatially discretized and encoded and sparsified to obtain fire spatiotemporal deepening features, which are processed by dilated convolution and upsampling to obtain high-resolution fire prediction probability maps, thereby achieving the effect of improving the accuracy of fire trend prediction.

[0095] Taking the scenario of monitoring the transmission line corridor as an example, the specific application scenarios of the fire trend prediction method provided by the present application are described below in combination with Figure 1 Figure 1 ​As shown, the specific application scenarios of the present application include a multi-source data acquisition module 101, a spatio-temporal feature fusion module 102, and a fire source trend prediction module 103. The multi-source data acquisition module 101 obtains multi-source data from the outside, pre-processes the multi-source data, and inputs the pre-processed data into the spatio-temporal feature fusion module 102. The pre-processed data is subjected to feature extraction and feature fusion in the spatio-temporal feature fusion module 102 to obtain corresponding fire source fusion features. The fire source fusion features are input into the fire source trend prediction module 103, and a fire source prediction probability map is output, which can predict the threat of wild fire to the high-voltage transmission line and provide a data basis for triggering automatic power-off protection.

[0096] It should be noted that the application of the fire source trend prediction method provided by the present application is not limited to transmission line corridor monitoring, but can also be used in other various scenarios, for example: urban fire management, early identification of potential fire sources in fire-prone areas such as power facilities; forest fire prevention and control, obtaining multi-source data through satellites or unmanned aerial vehicles to predict the direction and speed of fire spread and provide support for evacuation and fire extinguishing decisions.

[0097] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0098] Figure 2 Flowchart of the fire source trend prediction method provided by the present application Figure 1 As shown in the flowchart, the method comprises the following steps. Figure 2

[0099] S201, obtaining multi-source data of a target area, the multi-source data at least including visible light images, infrared images, and environmental data.

[0100] In this step, after obtaining the multi-source data, the multi-source data is pre-processed, and the main pre-processing is denoising of image data.

[0101] S202, fire source feature extraction is performed on the visible light images and the infrared images respectively to obtain corresponding visible spectrum features and infrared spectrum features.

[0102] In this step, fire source feature extraction is performed on the visible light images to obtain corresponding visible spectrum features, and fire source feature extraction is performed on the infrared images to obtain corresponding infrared spectrum features.

[0103] ​Specifically, in one aspect, based on a dynamic space-channel attention mechanism, the attention is dynamically adjusted, the visible light image is processed through deformable convolution to obtain a spatial attention vector corresponding to the visible light image, the fire source region is dynamically focused, the visible light image is adaptively deepened to process the fire source related channel features to obtain a channel attention vector, the visible light image, the spatial attention vector and the channel attention vector are dynamically fused to obtain a visible spectrum feature, so as to suppress the interference regions such as vegetation and mountains, enhance the weak texture features such as smoke edges, and improve the fire source recall rate in a complex background. On the other hand, through a multi-scale thermal radiation perception network integrated with a thermal physical guided attention mechanism, the infrared image is processed to extract fire source features to obtain an infrared spectrum feature, so as to more effectively fuse local details and global thermal distribution features, and improve the ability to distinguish fire sources and high-temperature interference objects.

[0104] S203, fuse the visible spectrum feature and the infrared spectrum feature to obtain a fire source fusion feature.

[0105] In this step, the visible spectrum feature and the infrared spectrum feature are input into a lightweight sub-network to obtain an offset, and based on the visible spectrum feature, the infrared spectrum feature and the offset, a precise alignment feature is obtained. The precise alignment feature is analyzed to obtain a visible precise alignment feature and an infrared precise alignment feature, so as to solve the spatial offset problem of different sensor data. The visible precise alignment feature and the infrared precise alignment feature are further processed in terms of semantics to obtain a semantic attention score, physical attention calculation is performed on the infrared radiation intensity, the visible light reflectivity and the environmental temperature to obtain a physical attention score, based on the semantic attention score, the physical attention score, the precise alignment feature and a preset dynamic weight, semantic and physical double attention driven feature fusion is performed to obtain a fire source fusion feature, so as to suppress features that do not conform to the thermal radiation law.

[0106] S204, for the continuous geographic coordinates corresponding to the target region, the fire source fusion feature and the meteorological feature in the environmental data, time-space coding fusion processing is performed to obtain a fire source time-space coding feature.

[0107] In this step, the continuous geographic coordinate data corresponding to the target region is obtained, the continuous geographic coordinate data corresponding to the target region is processed through spatial discretization coding to obtain a spatial position code, and the spatial position code, the fire source fusion feature and the meteorological feature are processed through feature splicing to obtain a fire source time-space coding feature.

[0108] S205, for the fire source time-space coding feature, time-space correlation focusing processing is performed to obtain a fire source time-space deepening feature.

[0109] In this step, based on the fire space-time coding characteristics, the spatial neighborhood attention weight is obtained, the fire high-dimensional embedding vector is sparsified based on the spatial neighborhood attention weight, the fire spatial neighborhood feature is obtained, and the fire evolution short-term time feature corresponding to the fire space-time coding feature is determined. The fire spatial neighborhood feature and the fire evolution short-term time feature are fused to obtain the fire space-time deepening feature.

[0110] In S206, the fire source prediction probability map of the target region is obtained by performing multi-scale space-time decoding on the fire source space-time deepening feature.

[0111] In this step, the fire source space-time deepening feature is decomposed into a trend item and a periodic item based on the causal Fourier transform layer, the fire source space-time deepening feature, the trend item and the periodic item are merged to obtain an intermediate feature, and the intermediate feature is up-sampled by a hole convolution to obtain the fire source prediction probability map of the target region.

[0112] The fire source trend prediction method provided in the embodiments of the present application comprises the following steps: acquiring visible light images, infrared images and environmental data of a target region; extracting fire source features from the visible light images and the infrared images to obtain visible spectrum features and infrared spectrum features; fusing the visible spectrum features and the infrared spectrum features to obtain fire fusion features; performing space-time coding fusion processing on continuous geographical coordinates corresponding to the target region, the fire fusion features and meteorological features in the environmental data to generate fire space-time coding features; performing space-time correlation focusing processing on the fire space-time coding features to obtain fire space-time deepening features; and performing multi-scale space-time decoding on the fire space-time deepening features to obtain a fire source prediction probability map of the target region. The fire source trend prediction method provided in the embodiments of the present application can improve the accuracy of fire source trend prediction.

[0113] In the following, Figure 2 on the basis of the embodiments, the implementation process of the fire source trend prediction method provided in the present application is described in detail in the embodiments corresponding to the embodiments. Figure 3 and Figure 4 The embodiments of the present application mainly show the process of how to implement the fire source trend prediction based on the processed data. Figure 3 The embodiments of the present application mainly show the process of how to implement the fire source trend prediction based on the processed data. Figure 4

[0114] Figure 3 The flowchart of the fire source trend prediction method provided in the present application Figure 2 As shown in the figure, the embodiments of the present application are based on the embodiments, and a fire source trend prediction method is described in detail, which comprises the following steps: Figure 3 Figure 2

[0115] ​​​S301, fire source feature extraction on the visible light image is performed through a dynamic spatial and channel attention mechanism network to obtain a visible light spectrum feature.

[0116] In this step, based on the dynamic spatial and channel attention mechanism, multiple steps of feature extraction on the visible light image are constructed into a dynamic spatial and channel attention mechanism network.

[0117] Specifically, the steps of the network for fire source feature extraction on the visible light image include: performing deformable convolution processing on the visible light image, dynamically focusing on the fire source region in the image, and obtaining a spatial attention vector corresponding to the visible light image; performing adaptive deepening of fire source related channel features on the visible light image, and obtaining a channel attention vector corresponding to the visible light image; and finally, dynamically fusing the visible light image, the spatial attention vector, and the channel attention vector to obtain the visible light spectrum feature.

[0118] S302, fire source feature extraction on the infrared image is performed through a multi-scale thermal radiation perception network to obtain an infrared light spectrum feature.

[0119] In this step, based on the thermal physical guide attention mechanism, multiple steps of feature extraction on the infrared image are constructed into a multi-scale thermal radiation perception network.

[0120] Specifically, the steps of the network for fire source feature extraction on the infrared image include: based on a multi-scale residual pyramid, multi-scale thermal radiation features are extracted through convolution operations at different levels, and based on the thermal physical guide attention mechanism, the multi-scale thermal radiation features are filtered and deepened to enhance key fire source feature channels and suppress noise channels, so as to more effectively fuse local details and global thermal distribution features, improve the ability to distinguish fire sources and high-temperature interference objects, and finally obtain the infrared light spectrum feature.

[0121] S303, the visible light spectrum feature and the infrared light spectrum feature are input into a lightweight sub-network to obtain an offset.

[0122] In this step, the visible light spectrum feature and the infrared light spectrum feature are first channel spliced to obtain a spliced feature that fuses the two modal basic features, and then the spliced feature is input into the lightweight sub-network to generate an offset for correcting large-scale spatial offset through multi-layer perception machine operation learning. The above offset can be used to correct the overall misplacement caused by the difference in viewing angle, such as the overall misplacement in the image data obtained by the satellite and the ground equipment respectively.

[0123] S304, deformable convolution processing is performed on the visible light spectrum feature, the infrared light spectrum feature, and the offset to obtain a coarse alignment feature.

[0124] In this step, based on the offset obtained in S303, deformable convolution is performed on the visible spectrum features and the infrared spectrum features to obtain coarse alignment features.

[0125] S305, wavelet transform and multi-scale decomposition are performed on the coarse alignment features to obtain corresponding low-frequency features and high-frequency features.

[0126] In this step, for the coarse alignment features, a suitable wavelet transform matrix is found by calculation, and the coarse alignment features are decomposed into low-frequency features corresponding to the shape structure and high-frequency features corresponding to texture details such as smoke edges and thermal radiation gradients by wavelet decomposition.

[0127] Specifically, according to the requirements, a suitable wavelet basis function is selected to generate an initial wavelet transform matrix, and the initial wavelet transform matrix is iteratively optimized in combination with the visible spectrum features and the infrared spectrum features after coarse alignment to obtain an optimized wavelet transform matrix. Based on the optimized wavelet transform matrix, multi-layer wavelet transform decomposition is performed on the coarse alignment features until a preset number of layers is reached, and the results of the final layer are determined as low-frequency features and high-frequency features.

[0128] It should be noted that the shape corresponding to the low-frequency features obtained at this time has been basically aligned, but the high-frequency features need to be aligned again by convolution optimization.

[0129] S306, deformable convolution is performed on the high-frequency features to obtain optimized high-frequency features.

[0130] In this step, offset correction is performed on each layer of high-frequency subbands in the high-frequency features to obtain optimized high-frequency features.

[0131] Specifically, the high-frequency subbands can be analyzed by a lightweight subnetwork first to predict the detail misalignment offset, and then deformable convolution can be used to correct the subpixel-level misalignment according to the detail misalignment offset, such as the offset of the smoke edge caused by the difference in viewing angle.

[0132] S307, the optimized high-frequency features and the low-frequency features are reconstructed to obtain precise alignment features.

[0133] In this step, the optimized high-frequency features obtained in S306 and the low-frequency features obtained in S305 are spliced by layer, and inverse wavelet transform is used for reconstruction to finally obtain precise alignment features after multi-scale fusion and detail alignment.

[0134] S308, semantic association processing is performed on the visible light and infrared precise alignment features in the precise alignment features to obtain semantic attention scores.

[0135] In this step, the modal feature separation is first performed on the accurate alignment feature to obtain the visible light accurate alignment feature and the infrared accurate alignment feature, and then the semantic correlation processing is performed on the visible light accurate alignment feature and the infrared accurate alignment feature to obtain the semantic attention score.

[0136] Specifically, according to the visible light accurate alignment feature, a query vector for semantic correlation is determined, and according to the infrared accurate alignment feature, a key vector for providing semantic reference and a value vector for semantic enhancement are determined. Then, the query vector and the key vector are subjected to matrix transposition operation and scaling, and then are subjected to probability distribution, and based on the value vector, semantic enhancement is performed to complete the modeling of the semantic correlation of the visible light and infrared features.

[0137] In S309, the physical attention calculation is performed on the infrared radiation intensity, the visible light reflectivity and the environment temperature to obtain a physical attention score.

[0138] In this step, the infrared radiation intensity, the visible light reflectivity and the environment temperature are fused and mapped to convert them into a unified dimensional feature representation, and then normalized to obtain the physical attention score. The environment temperature is obtained from the initial acquired environment data.

[0139] Specifically, the infrared radiation intensity, the visible light reflectivity and the environment temperature can be fused and mapped by a multi-layer perception, and the output value is compressed to the interval [0, 1] by numerical normalization processing of the fused and mapped result to measure the constraint strength of the physical law on the feature.

[0140] In one possible implementation, the infrared radiation intensity and the visible light reflectivity are directly extracted from the originally obtained raw data, for example, the physical quantities directly collected by hardware sensors such as thermal imagers and visible light cameras.

[0141] In one possible implementation, in order to simplify the process, the infrared radiation intensity and the visible light reflectivity can also be derived and calculated from the accurate alignment feature.

[0142] In S310, based on the semantic attention score, the physical attention score, the accurate alignment feature and the dynamic weight, a fire source fusion feature is obtained.

[0143] In this step, based on the semantic attention score, the physical attention score, the accurate alignment feature and the dynamic weight, the feature fusion driven by the semantic and physical double attentions is performed to obtain the fire source fusion feature.

[0144] ​Specifically, the physical attention score is multiplied with the precise alignment feature element by element, so that the physical constraint acts on the precise alignment feature, and a dynamic weight is introduced to balance the contribution of semantic attention and physical constraint in combination with the semantic attention score. Finally, a fire source fusion feature is generated through linear weighted summation, so that the result can meet the semantic correlation alignment, such as smoke profile and edge matching, and can also meet the physical law constraint, such as thermal radiation and reflectivity rationality.

[0145] In a possible implementation, the dynamic weight can be a value preset according to specific requirements.

[0146] In a possible implementation, the dynamic weight can be an adaptive value adapted to the current scene obtained through training and learning.

[0147] The fire source trend prediction method provided in the embodiments of the present application can extract fire source features from a visible light image through a dynamic air pass convolution enhancement network to obtain a visible spectrum feature, and can extract fire source features from an infrared image through a multi-scale thermal radiation perception network to obtain an infrared spectrum feature. Then, the visible spectrum feature and the infrared spectrum feature are input into a lightweight subnetwork to obtain a displacement, and then the visible spectrum feature, the infrared spectrum feature and the displacement are subjected to deformable convolution processing to generate a coarse alignment feature. The coarse alignment feature is subjected to wavelet transform and multi-scale decomposition to obtain corresponding low-frequency features and high-frequency features. The high-frequency features are subjected to deformable convolution processing again to obtain optimized high-frequency features. Then, the optimized high-frequency features and the low-frequency features are subjected to reconstruction processing to obtain a precise alignment feature. Further, the visible light and infrared precise alignment features in the precise alignment feature are subjected to semantic correlation processing to obtain a semantic attention score. At the same time, the infrared radiation intensity, the visible light reflectivity and the ambient temperature are subjected to physical attention calculation to obtain a physical attention score. Finally, based on the semantic attention score, the physical attention score, the precise alignment feature and a dynamic weight, a fire source fusion feature is obtained, the quality of data participating in fire source trend prediction is improved, and a more accurate fire source trend prediction result is obtained.

[0148] Figure 4 Flowchart of a fire source trend prediction method provided in the present application Figure 3 As shown in Figure 4 , the embodiments of the present application are based on any of the above embodiments and a fire source trend prediction method is described in detail. The method comprises:

[0149] S401, spatially discrete coding is performed on the continuous geographic coordinates corresponding to the target region to obtain a spatial position code.

[0150] In this step, in order to realize spatial position perception, the continuous geographic coordinate data corresponding to the target area is first obtained, and then spatial discretization coding is performed to obtain the spatial position coding.

[0151] Specifically, in the target area, a plurality of grids are divided in advance, and the center coordinates of each grid are determined. Then, for the geographic coordinates of the fire source target, the similarity between the coordinates and the center coordinates of each grid is calculated through the Gaussian kernel function. Then, for the geographic coordinates of the fire source, the similarity between the coordinates and the center coordinates of each grid is calculated through the Gaussian kernel function. All similarities are accumulated, and finally the coding capable of accurately representing the spatial position relationship of the fire source is generated, and the conversion from continuous geographic coordinates to spatial position coding is completed.

[0152] In one possible implementation, the device can record the corresponding geographic coordinates synchronously through the built-in sensor when collecting image data, and the continuous geographic coordinates corresponding to the target area can be directly obtained by analyzing the initially obtained visible light image data. The geographic coordinate feature can be directly obtained by analyzing the fire source fusion feature.

[0153] In one possible implementation, there can be a geographic coordinate database in the system implementing the method, and the continuous geographic coordinates corresponding to the target area can be obtained by matching the database based on the geographic landmarks in the visible light image data.

[0154] S402, the spatial position coding, the fire source fusion feature and the weather feature are subjected to feature splicing processing to obtain the fire source space-time coding feature.

[0155] S403, the fire source space-time coding feature is subjected to sparse processing to obtain the fire source spatial neighborhood deepening feature.

[0156] In this step, based on the fire source space-time coding feature, the spatial neighborhood attention weight is obtained, and based on the spatial neighborhood attention weight, the fire source space-time coding feature is subjected to sparse processing to obtain the fire source spatial neighborhood deepening feature.

[0157] Specifically, the query and key vectors corresponding to the fire source space-time coding feature are first generated, the correlation between the query and key vectors is calculated, and the spatial neighborhood attention weight is obtained after probability distribution and adjacency matrix screening. Then, the value vector corresponding to the fire source space-time coding feature is generated, and the spatial neighborhood attention weight is weighted and fused to obtain the sparse processed fire source spatial neighborhood deepening feature. Thus, only the local spatial dependence of fire spread is focused, and the redundancy of global calculation is avoided.

[0158] S404, based on the short-term time window and the time sequence feature corresponding to each grid in the fire source space-time coding feature, the fire source evolution grid feature of each grid is calculated.

[0159] In this step, the fire source space-time coding features are grid split processed to obtain a plurality of grids corresponding to the fire source space-time coding features, and then based on a short-term time window and the time sequence features corresponding to each grid, the fire source evolution grid features of each grid are calculated.

[0160] Specifically, the fire source space-time coding features are first split according to the geographical grid dimension to obtain a time sequence feature sequence of a single grid, and then a short-term time window is set to convert the time sequence features at different time points in the window into a query vector and a key vector, so as to calculate the correlation degree of the time sequence features corresponding to the two time points, and normalize the correlation degree to obtain the fire source evolution grid features corresponding to each grid.

[0161] S405, the fire source evolution grid features of all grids are reassembled to obtain fire source evolution short-term time features.

[0162] S406, the fire source space neighborhood deepening features and the fire source evolution short-term time features are fused to obtain fire source space-time deepening features.

[0163] In this step, the fire source space neighborhood deepening features and the fire source evolution short-term time features are spliced, and then linear transformation weight fusion is used to obtain fire source space-time deepening features. The linear transformation weight can be a preset parameter matrix, or can be determined by training a model based on an existing parameter matrix.

[0164] S407, based on the causal Fourier transform layer, the time sequence information in the fire source space-time deepening features is decomposed into a trend item and a periodic item.

[0165] In this step, based on the causal Fourier transform, the time sequence information in the fire source space-time deepening features is converted from the time domain to the frequency domain. The low-frequency components in the frequency domain are retained by using a low-pass filter, and then inverse transformed back to the time domain to determine the trend item. The high-frequency components in the frequency domain are retained by using a high-pass filter, and then inverse transformed to obtain the periodic item. The trend item corresponds to a long-term trend, such as whether the overall fire is expanding or shrinking, seasonal fire changes, and slow spread, etc. The periodic item corresponds to a sudden period, such as an abnormal point of sudden fire at a certain time period and short-term dramatic changes, etc.

[0166] S408, the fire source space-time deepening features, the trend item and the periodic item are merged to obtain intermediate features.

[0167] In this step, by merging the fire source space-time deepening features, the trend item and the periodic item, the intermediate features are obtained, so as to deepen the long-term trend and the sudden period in the fire source space-time deepening features, and the fire source space-time deepening features after decomposition enhancement are determined as the intermediate features.

[0168] S409, perform hole convolution and up-sampling processing on the intermediate feature to obtain a fire source prediction probability map of the target region.

[0169] In this step, the intermediate feature is enhanced by hole convolution to obtain a high-channel low-resolution feature, and the high-channel low-resolution feature is rearranged and up-sampled to obtain a fire source prediction probability map with low-channel and high-resolution.

[0170] The fire source trend prediction method provided in the embodiments of the present application encodes the continuous geographical coordinates of the target region in space, and then splices the fire source fusion feature and the weather feature to obtain a fire source spatio-temporal coding feature; performs sparse processing on the fire source spatio-temporal coding feature to generate a fire source spatial neighborhood deepening feature, and at the same time, obtains a fire source evolution grid feature of each grid according to the time sequence feature corresponding to each grid in the short-term time window and the fire source spatio-temporal coding feature, splices the fire source evolution grid features of all grids to obtain a fire source evolution short-term time feature; fuses the fire source evolution short-term time feature and the fire source spatial neighborhood deepening feature to obtain a fire source spatio-temporal deepening feature, decomposes the time sequence information in the fire source spatio-temporal deepening feature into a trend item and a periodic item, and then combines the fire source spatio-temporal deepening feature to obtain an intermediate feature, and finally performs hole convolution and up-sampling processing on the intermediate feature to obtain a fire source prediction probability map of the target region, thereby improving the fire source trend prediction accuracy.

[0171] On the basis of any one of the above embodiments, in the following, a fire source trend prediction method is described in detail by combining Figure 5 with specific examples.

[0172] S501, acquire a large-range visible light image and a remote sensing image from a Landsat series satellite, collect an infrared image by using a MODIS sensor, and acquire a local scene visible light image by using a ground camera array.

[0173] S502, pre-process the acquired images.

[0174] In this step, the acquired images are input into a pre-processing module to obtain pre-processed image data. Denoising is an important operation in the pre-processing process.

[0175] Specifically, the pre-processing module performs denoising on the visible light image and the infrared image based on Gaussian filtering. The denoising of the visible light image can be represented by the following formula:

[0176]

[0177] The infrared image is denoised in the same way, which can be represented by the following formula:

[0178]

[0179] In the above formula, is the standard deviation of the Gaussian distribution, used to control the "fuzzy degree" of the Gaussian kernel; determines the center position of the Gaussian kernel on the image; the symbol * represents the convolution operation. Through the convolution operation, the original visible light image obtained in S501 and the original infrared image are combined with the Gaussian filter function to calculate, thereby obtaining the filtered visible light image and infrared image .

[0180] S503, the preprocessed data is sent into the space-time feature fusion module for feature extraction and multi-modal data fusion.

[0181] In this step, the preprocessed visible light image, infrared image and other multi-source heterogeneous data are input into the space-time feature fusion module for deep fusion in time and space dimensions, and a high-precision and anti-interference joint feature representation, i.e., the aforementioned fire fusion feature in the embodiment, is output. The space-time feature fusion module is composed of a visible light image feature extraction module, an infrared image feature extraction module and a fusion module, wherein the visible light image feature extraction module adopts the dynamic air convolution enhancement network in the aforementioned embodiment to process the visible light image; the infrared image feature extraction module adopts the multi-scale thermal radiation perception network in the aforementioned embodiment to process the infrared image.

[0182] Specifically, in the visible light image feature extraction module, the visible light image is first processed by deformable convolution, which can be represented by the following formula:

[0183]

[0184] wherein, represents the convolution operation; is the sampling point offset, which dynamically aligns the convolution kernel to the fire source area; is an activation function, used to generate the weight of a single channel or spatial position; the value of each position indicates the degree of attention of the fire source area to the fire source feature; is a spatial attention vector. The remaining parameters are consistent with the foregoing, and will not be described again.

[0185] The visible light image is then processed by adaptive deepening fire-related channel, which can be represented by the following formula:

[0186]

[0187] wherein, This indicates that the spatial dimension is compressed, and global average pooling is performed separately for each channel of the feature map; This indicates the performance of nonlinear transformations and cross-channel interactive learning; This is an activation function used to normalize the importance of multiple channels in channel attention; This is the channel attention vector. The remaining parameters are the same as described above and will not be repeated here.

[0188] Multiplying the spatial attention vector weights and channel attention vector weights element-wise, and using the fused weights to deepen the visible light features, we obtain the visible spectral features, which can be expressed by the following formula:

[0189]

[0190] in, This represents element-wise multiplication under a broadcast mechanism; This indicates element-wise multiplication; Indicates the characteristics of the visible spectrum.

[0191] In the infrared image feature extraction module, multi-scale fusion, channel attention enhancement, and residual connection processing are performed on the infrared image to obtain infrared spectral features, which can be expressed by the following formula:

[0192]

[0193] in, For the first Layer input features, exist ; This is a channel attention mechanism, representing global pooling of the input data. Transformation Activate the generated channel weights to enhance the features of important channels and suppress noisy channels; Indicates an upsampling operation; Indicates feature concatenation operation; These are the features after upsampling from the previous layer; Indicates splicing; Represents convolution; For the first The output features of the last layer can be identified as infrared spectral features. The remaining parameters are the same as those described above and will not be repeated here.

[0194] In the fusion module, the visible spectral features are first... and infrared spectral characteristics After concatenation, input the lightweight subnetwork to generate the corresponding offset. It can be expressed by the following formula:

[0195]

[0196] Through the deformable convolution operation, the visible spectrum features and the infrared spectrum features and the offset are used to obtain the coarse alignment features after alignment processing , which can be expressed as follows:

[0197]

[0198] , wherein, represents a deformable convolution, which is used to correct the spatial misalignment of different modal features and align the features in the spatial position. The remaining parameters are consistent with the foregoing and will not be repeated.

[0199] The coarse alignment features are further decomposed through the construction of the wavelet pyramid to obtain the low-frequency features corresponding to the smoke outline and the general shape , and the high-frequency features corresponding to the smoke edge and the thermal radiation gradient , which can be expressed as follows:

[0200]

[0201]

[0202] , wherein, is the optimized wavelet transform matrix; represents the parameter value that makes the objective function obtain the minimum value; represents the Lth layer wavelet decomposition; represents the wavelet decomposition operation; is the wavelet basis function. The remaining parameters are consistent with the foregoing and will not be repeated.

[0203] The high-frequency features are further aligned and optimized, which can be expressed as follows:

[0204]

[0205] , wherein, represents a lightweight subnetwork; represents the adjustment parameter of the high-frequency features; is the optimization of the high-frequency features. The remaining parameters are consistent with the foregoing and will not be repeated.

[0206] The low-frequency features and the optimized high-frequency features are further reconstructed, which can be expressed as follows:

[0207]

[0208] , wherein, represents the wavelet reconstruction operation, which is the inverse process of the wavelet decomposition; is the precise alignment feature; , and respectively and The final layer, i.e., the layer outputs the value. The remaining parameters are consistent with the foregoing and will not be repeated.

[0209] The semantic correlation modeling of the visible spectrum features and the infrared spectrum features can be represented as follows:

[0210]

[0211] wherein, is the visible light precise alignment feature obtained by modal feature separation on the precise alignment feature; is the query vector for semantic correlation determined according to the visible light precise alignment feature; is the infrared precise alignment feature obtained by modal feature separation on the precise alignment feature; is the key vector for providing semantic reference determined according to the infrared precise alignment feature; is the value vector for semantic enhancement determined according to the infrared precise alignment feature; , are linear transformation matrices for different uses; is a scaling factor; is the semantic attention score finally output by the above mathematical model. The remaining parameters are consistent with the foregoing and will not be repeated.

[0212] Based on the thermal radiation physical law, the infrared radiation intensity , the visible light reflectivity and the ambient temperature are fused and mapped to convert them into a unified dimensional feature representation, and then normalized to obtain the physical attention score, which can be represented by the following formula:

[0213]

[0214] Finally, the two attention scores and the precise alignment feature are dynamically fused with as the dynamic weight to obtain the fused feature , which can be represented by the following formula:

[0215]

[0216] S504, the fused data is sent into a time series Transformer prediction network to generate a feature map.

[0217] In this step, the time series Transformer prediction network takes the time series Transformer prediction module as the core component, realizes fire source trend prediction based on the hierarchical sparse attention mechanism, and finally generates a feature map about the fire source trend prediction.

[0218] Specifically, the continuous geographic coordinates corresponding to the target area are discretized into grid encoding, which can be expressed as follows:

[0219]

[0220] wherein, is the spatial position encoding, and the output is a high-dimensional vector, which is used to represent the coordinates of the grid encoding, which carries spatial proximity information; is the feature vector corresponding to the spatial coordinates; is the grid center coordinate of the th grid; is the total number of grids, which can be dynamically adjusted according to the monitoring range; is the Gaussian kernel width, and its default value is 5km.

[0221] The spatial position encoding , the fire fusion feature and the meteorological feature are then processed by feature splicing to obtain the fire spatio-temporal encoding feature , which can be expressed as follows:

[0222]

[0223] The fire spatio-temporal encoding feature is then processed by sparse processing by only calculating the attention weight of the adjacent grid, so as to reduce the calculation amount, and the fire spatial neighborhood deepening feature is obtained, which can be expressed as follows:

[0224]

[0225] wherein, is the fire spatial neighborhood deepening feature; is the linear transformation matrix; is the spatial neighborhood attention weight, which is used to focus on the local spatial dependence of fire spread, and can be calculated by the following formula:

[0226]

[0227] wherein, represents the spatial neighborhood attention weight between the grid and the grid , which measures the contribution degree of the feature of the grid to the feature update of the grid when considering the local spatial dependence; is the vector of the corresponding grid in the query vector corresponding to the fire spatio-temporal encoding feature . is the fire source spatio-temporal encoding feature corresponding grid in the key vector vector; is the corresponding grid in the adjacency matrix and grid element, when the distance between grid and grid is less than or equal to the Gaussian kernel width , otherwise, . The remaining parameters are consistent with the foregoing, and will not be repeated.

[0228] The fire source spatio-temporal encoding feature is split into the time series feature of a single grid , for each grid n, the fire source evolution grid feature between time and time is calculated , which can be expressed as follows:

[0229]

[0230] wherein, is a linear projection matrix that maps the time series feature of grid n at a certain time to the query vector and the key vector.

[0231] The fire source evolution grid features of all grids are reassembled to obtain the fire source evolution short-term time feature . The fire source spatial neighborhood deepening feature and the fire source evolution short-term time feature are fused by element-wise addition or splicing to generate the fire source spatio-temporal deepening feature that deepens the spatial and temporal dependence at the same time, which can be expressed as follows:

[0232]

[0233] wherein, is a linear transformation weight.

[0234] Finally, the fire source spatio-temporal deepening feature is restored to a high-resolution fire risk probability map through a multi-scale decoder.

[0235] In the multi-scale decoder, a causal Fourier transform layer is first introduced to decompose the fire source spatio-temporal deepening feature into a trend item and a periodic item , which can be expressed as follows:

[0236]

[0237]

[0238] wherein, is a low-pass filter for capturing long-term trends; is a high-pass filter for capturing sudden fire points.

[0239] After the hole convolution and the up-sampling processing, a high-resolution fire risk prediction probability map is obtained, which can be expressed as follows:

[0240]

[0241] wherein, is a fire risk prediction probability map; intermediate features ; represents a pixel rearrangement operation, an efficient up-sampling method; represents a hole convolution, a convolution operation for expanding the receptive field; represents that the hole rate is 3.

[0242] The high-resolution fire risk prediction probability map is a feature map generated by the prediction network.

[0243] It should be noted that the various processing steps S501-S504 shown in the Figure 5 embodiment do not constitute a specific limitation on a fire source trend prediction method. In other embodiments of the present application, a fire source trend prediction method can include more or fewer steps than Figure 5 the embodiment, for example, a fire source trend prediction method can include Figure 5 part of the steps in the embodiment, or, Figure 5 some steps in the embodiment can be replaced by steps with the same function, or, Figure 5 some steps in the embodiment can be split into multiple steps, etc.

[0244] Figure 6 The structure of a fire source trend prediction device provided in the present application is shown in FIG. 6, which includes: Figure 6

[0245] The acquisition module 601 is configured to acquire multi-source data of a target region, and the multi-source data at least includes a visible light image, an infrared image, and environmental data.

[0246] The extraction module 602 is configured to perform fire source feature extraction on the visible light image to obtain visible light spectrum features corresponding to the visible light image, and perform fire source feature extraction on the infrared image to obtain infrared light spectrum features corresponding to the infrared image.

[0247] ​The fusion module 603 is configured to fuse the visible spectrum feature and the infrared spectrum feature to obtain a fire source fusion feature.

[0248] The prediction module 604 is configured to perform spatiotemporal coding fusion processing on the continuous geographic coordinates corresponding to the target region, the fire source fusion feature, and the meteorological feature in the environment data to obtain a fire source spatiotemporal coding feature; perform spatiotemporal correlation focusing processing on the fire source spatiotemporal coding feature to obtain a fire source spatiotemporal deepening feature; and perform multi-scale spatiotemporal decoding on the fire source spatiotemporal deepening feature to obtain a fire source prediction probability map of the target region.

[0249] In a possible implementation, the extraction module 602 performs fire source feature extraction on the visible light image to obtain a visible spectrum feature corresponding to the visible light image, and specifically for:

[0250] Performing deformable convolution processing on the visible light image to obtain a spatial attention vector corresponding to the visible light image.

[0251] Performing adaptive deepening fire source related channel feature processing on the visible light image to obtain a channel attention vector.

[0252] Dynamically fusing the visible light image, the spatial attention vector, and the channel attention vector to obtain the visible spectrum feature.

[0253] In a possible implementation, the extraction module 602 performs fire source feature extraction on the infrared image to obtain an infrared spectrum feature corresponding to the infrared image, and specifically for:

[0254] Performing fire source feature extraction on the infrared image through a multi-scale thermal radiation perception network to obtain the infrared spectrum feature; wherein the multi-scale thermal radiation perception network integrates a thermal physical guided attention mechanism.

[0255] In a possible implementation, the fusion module 603 fuses the visible spectrum feature and the infrared spectrum feature to obtain the fire source fusion feature, and specifically for:

[0256] Inputting the visible spectrum feature and the infrared spectrum feature into a lightweight subnetwork to obtain an offset.

[0257] Based on the visible spectrum feature, the infrared spectrum feature, and the offset, a precise alignment feature is obtained.

[0258] Performing modal feature separation on the precise alignment feature to obtain a visible light precise alignment feature and an infrared precise alignment feature.

[0259] Performing semantic correlation processing on the visible light precise alignment feature and the infrared precise alignment feature to obtain a semantic attention score.

[0260] The physical attention score is calculated by performing physical attention calculation on infrared radiation intensity, visible light reflectivity, and environmental temperature.

[0261] Based on the semantic attention score, the physical attention score, the accurate alignment feature, and the dynamic weight, the feature fusion driven by semantic and physical double attention is performed to obtain the fire fusion feature.

[0262] In a possible implementation, the fusion module 603 obtains the accurate alignment feature based on the visible spectrum feature, the infrared spectrum feature, and the offset, and specifically for:

[0263] The visible spectrum feature, the infrared spectrum feature, and the offset are processed by deformable convolution to obtain the coarse alignment feature.

[0264] The coarse alignment feature is processed by wavelet transform and multi-scale decomposition to obtain corresponding low-frequency features and high-frequency features.

[0265] The high-frequency feature is processed by deformable convolution to obtain an optimized high-frequency feature.

[0266] The optimized high-frequency feature and the low-frequency feature are reconstructed to obtain the accurate alignment feature.

[0267] In a possible implementation, the prediction module 604 performs spatiotemporal coding fusion processing on the continuous geographic coordinates corresponding to the target region, the fire fusion feature, and the meteorological feature in the environmental data to obtain the fire spatiotemporal coding feature, and specifically for:

[0268] The continuous geographic coordinates corresponding to the target region are spatially discretized and coded to obtain a spatial position code.

[0269] The spatial position code, the fire fusion feature, and the meteorological feature are processed by feature splicing to obtain the fire spatiotemporal coding feature.

[0270] In a possible implementation, the prediction module 604 performs spatiotemporal correlation focusing processing on the fire spatiotemporal coding feature to obtain the fire spatiotemporal deepening feature, and specifically for:

[0271] Based on the fire spatiotemporal coding feature, a spatial neighborhood attention weight is obtained.

[0272] Based on the spatial neighborhood attention weight, the fire spatiotemporal coding feature is processed by sparsification to obtain a fire spatial neighborhood deepening feature.

[0273] A fire evolution short-term time feature corresponding to the fire spatiotemporal coding feature is determined.

[0274] The fire spatial neighborhood deepening feature and the fire evolution short-term time feature are fused to obtain the fire spatiotemporal deepening feature.

[0275] In a possible implementation, the prediction module 604 determines the fire source evolution short-time feature corresponding to the fire source spatiotemporal coding feature, and specifically for:

[0276] The fire source spatiotemporal coding feature is subjected to grid splitting processing to obtain a plurality of grids corresponding to the fire source spatiotemporal coding feature.

[0277] Based on the short-time window and the time sequence feature corresponding to each grid, the fire source evolution grid feature of each grid is calculated.

[0278] The fire source evolution grid features of all the grids are reassembled to obtain the fire source evolution short-time feature.

[0279] In a possible implementation, the prediction module 604 performs multi-scale spatiotemporal decoding on the fire source spatiotemporal deepening feature to obtain the fire source prediction probability map of the target region, and specifically for:

[0280] Based on the causal Fourier transform layer, the time sequence information in the fire source spatiotemporal deepening feature is decomposed into a trend item and a periodic item.

[0281] The fire source spatiotemporal deepening feature, the trend item, and the periodic item are subjected to merging processing to obtain an intermediate feature.

[0282] The intermediate feature is subjected to hole convolution and up-sampling processing to obtain the fire source prediction probability map of the target region.

[0283] The fire source trend prediction apparatus provided in this embodiment can perform the method provided in the method embodiments, and has similar implementation principles and technical effects, which will not be described here again in this embodiment.

[0284] Figure 7 A structural schematic diagram of an electronic device provided in this application is shown in FIG. 7. As shown in FIG. 7, the electronic device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, the memory 702, and the communication component 703 are connected through a bus 704. Figure 7

[0285] In the specific implementation process, the at least one processor 701 executes the computer execution instructions stored in the memory 702, so that the at least one processor 701 performs the method described above.

[0286] The specific implementation process of the processor 701 can refer to the method embodiments described above, and has similar implementation principles and technical effects, which will not be described here again in this embodiment.

[0287] ​In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0288] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0289] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0290] The present application also provides a computer program product, comprising a computer program, which is executed by a processor to implement the above method.

[0291] The present application also provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the above method is implemented.

[0292] The above readable storage medium can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0293] An example readable storage medium is coupled to the processor such that the processor can read information from the readable storage medium and can write information to the readable storage medium. Of course, the readable storage medium can also be a part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0294] The division of units is only a logical function division, and in actual implementation, another division manner can be used, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0295] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0296] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0297] If the functions are realized in the form of software function units and sold or used as independent products, the functions can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0298] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes various media capable of storing program codes, such as ROM, RAM, magnetic disk, or optical disk.

[0299] Finally, it should be noted that other embodiments of the present application will readily occur to those skilled in the art upon consideration of the specification and practice of the present application disclosed herein. The present application is intended to include all such variations, uses, or adaptations of the application in which the general principles of the application are used to best advantage and encompassed within its scope. The present application is not limited to the precise structures described and shown in the accompanying drawings and figures, and can be practiced with variation of modifications and alterations without departing from the scope of the present application. The scope of the present application is limited only by the claims appended hereto.

Claims

1. A fire source trend prediction method characterized by, The method comprises the following steps: acquiring multi-source data of a target area, wherein the multi-source data at least comprises visible light images, infrared images and environmental data; extracting fire source features from the visible light images to obtain visible light spectrum features corresponding to the visible light images, and extracting fire source features from the infrared images to obtain infrared light spectrum features corresponding to the infrared images; fusing the visible light spectrum features and the infrared light spectrum features to obtain fire source fusion features; performing spatio-temporal coding fusion processing on the continuous geographical coordinates corresponding to the target area, the fire source fusion features and the meteorological features in the environmental data to obtain fire source spatio-temporal coding features; performing spatio-temporal correlation focusing processing on the fire source spatio-temporal coding features to obtain fire source spatio-temporal deepening features; performing multi-scale spatio-temporal decoding on the fire source spatio-temporal deepening features to obtain a fire source prediction probability map of the target area.

2. The method of claim 1, wherein, The method of extracting fire source features from the visible light images to obtain visible light spectrum features corresponding to the visible light images comprises the following steps: performing deformable convolution processing on the visible light images to obtain spatial attention vectors corresponding to the visible light images; performing adaptive deepening fire source related channel feature processing on the visible light images to obtain channel attention vectors; dynamically fusing the visible light images, the spatial attention vectors and the channel attention vectors to obtain the visible light spectrum features.

3. The method of claim 1, wherein, The method of extracting fire source features from the infrared images to obtain infrared light spectrum features corresponding to the infrared images comprises the following steps: extracting fire source features from the infrared images through a multi-scale thermal radiation perception network to obtain the infrared light spectrum features; wherein the multi-scale thermal radiation perception network integrates a thermal physical guided attention mechanism.

4. The method of claim 1, wherein, The method of fusing the visible light spectrum features and the infrared light spectrum features to obtain fire source fusion features comprises the following steps: inputting the visible light spectrum features and the infrared light spectrum features into a lightweight subnetwork to obtain an offset; based on the visible light spectrum features, the infrared light spectrum features and the offset, obtaining precise alignment features; performing modal feature separation on the precise alignment features to obtain visible light precise alignment features and infrared light precise alignment features; performing semantic correlation processing on the visible light precise alignment features and the infrared light precise alignment features to obtain semantic attention scores; performing physical attention calculation on infrared radiation intensity, visible light reflectivity and environmental temperature to obtain physical attention scores; based on the semantic attention scores, the physical attention scores, the precise alignment features and dynamic weights, performing feature fusion driven by semantic and physical dual attentions to obtain the fire source fusion features.

5. The method of claim 4, wherein, The method of obtaining precise alignment features based on the visible light spectrum features, the infrared light spectrum features and the offset comprises the following steps: performing deformable convolution processing on the visible light spectrum features, the infrared light spectrum features and the offset to obtain coarse alignment features; performing wavelet transform and multi-scale decomposition on the coarse alignment features to obtain corresponding low-frequency features and high-frequency features; performing deformable convolution processing on the high-frequency features to obtain optimized high-frequency features; The optimization high-frequency feature and the low-frequency feature are reconstructed to obtain the accurate alignment feature.

6. The method of claim 1, wherein, The continuous geographical coordinates corresponding to the target region, the fire source fusion feature, and the weather feature in the environment data are subjected to spatiotemporal coding fusion processing to obtain a fire source spatiotemporal coding feature, including: The continuous geographical coordinates corresponding to the target region are subjected to spatial discretization coding to obtain the spatial position coding; The spatial position coding, the fire source fusion feature, and the weather feature are subjected to feature splicing processing to obtain the fire source spatiotemporal coding feature.

7. The method of claim 1, wherein, The fire source spatiotemporal coding feature is subjected to spatiotemporal correlation focusing processing to obtain a fire source spatiotemporal deepening feature, including: Based on the fire source spatiotemporal coding feature, a spatial neighborhood attention weight is obtained; Based on the spatial neighborhood attention weight, the fire source spatiotemporal coding feature is subjected to sparsification processing to obtain a fire source spatial neighborhood deepening feature; A fire source evolution short-term time feature corresponding to the fire source spatiotemporal coding feature is determined; The fire source spatial neighborhood deepening feature and the fire source evolution short-term time feature are fused to obtain the fire source spatiotemporal deepening feature.

8. The method of claim 7, wherein, The determination of the fire source evolution short-term time feature corresponding to the fire source spatiotemporal coding feature includes: The fire source spatiotemporal coding feature is subjected to grid splitting processing to obtain a plurality of grids corresponding to the fire source spatiotemporal coding feature; Based on a short-term time window and a time sequence feature corresponding to each grid, a fire source evolution grid feature of each grid is calculated; The fire source evolution grid features of all the grids are spliced to obtain the fire source evolution short-term time feature.

9. The method of claim 1, wherein, The fire source spatiotemporal deepening feature is subjected to multi-scale spatiotemporal decoding to obtain a fire source prediction probability map of the target region, including: Based on a causal Fourier transform layer, time sequence information in the fire source spatiotemporal deepening feature is decomposed into a trend item and a periodic item; The fire source spatiotemporal deepening feature, the trend item, and the periodic item are subjected to merging processing to obtain an intermediate feature; The intermediate feature is subjected to hole convolution and up-sampling processing to obtain the fire source prediction probability map of the target region.

10. A fire source tendency prediction apparatus characterized by comprising: It includes: An acquisition module is configured to acquire multi-source data of a target region, the multi-source data including at least visible light images, infrared images, and environment data; An extraction module is configured to extract a fire source feature from the visible light images to obtain a visible light spectrum feature corresponding to the visible light images, and extract a fire source feature from the infrared images to obtain an infrared light spectrum feature corresponding to the infrared images; A fusion module is configured to fuse the visible light spectrum feature and the infrared light spectrum feature to obtain a fire source fusion feature; A prediction module is configured to perform spatiotemporal coding fusion processing on continuous geographical coordinates corresponding to the target region, the fire source fusion feature, and a weather feature in the environment data to obtain a fire source spatiotemporal coding feature, perform spatiotemporal correlation focusing processing on the fire source spatiotemporal coding feature to obtain a fire source spatiotemporal deepening feature, and perform multi-scale spatiotemporal decoding on the fire source spatiotemporal deepening feature to obtain a fire source prediction probability map of the target region.

11. An electronic device, comprising: It includes: a processor, and a memory connected to the processor in communication; the memory stores computer-executed instructions; the processor executes the computer-executed instructions stored in the memory, so that the processor executes the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executed instructions, and the computer-executed instructions are executed by the processor to implement the method according to any one of claims 1 to 9.

13. A computer program product, characterised in that, The computer program is executed by the processor to implement the method according to any one of claims 1 to 9.

Images