Multi-source time sequence image feature collaborative integration method and system and fire early warning method
Through the multi-source timing image feature collaborative integration method, combined with RGB and infrared feature maps, a multi-source timing fusion feature map is generated, which solves the problem of fire early warning efficiency under the influence of light conditions and complex scenes in the existing technology, and improves fire monitoring and The robustness and timeliness of early warning.
Patent Information
- Application Number
- CN202411020674.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-12
AI Technical Summary
The existing fire warning technology based on RGB video images has reduced its effectiveness in darker light environments, and complex scene factors lead to false alarms or omissions, affecting the reliability and stability of the early warning system.
The multi-source timing image feature collaborative integration method is adopted, and the paired RGB feature maps and infrared feature maps are obtained, feature fusion and time encoding are performed, and multi-source timing fusion feature maps are generated, combining historical data to improve fire risk prediction and judgment.
In complex environments, it improves the robustness and timeliness of fire monitoring and early warning, reduces the false alarm rate and missed alarm rate, and enhances the accuracy of fire characteristic recognition.
Smart Images

Figure CN118941908A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of fire warning technology, and in particular to a multi-source time-series image feature collaborative integration method and system and a fire warning method. Background Art
[0002] At present, fire warning technology based on RGB video images has achieved remarkable results. However, in actual application scenarios, this fire warning technology still has several problems that need to be solved. For example, although the core technology of this fire warning technology relies on convolutional neural networks (CNN) to perform deep feature extraction on video images to identify fire signs such as flames and smoke, its performance has not reached the ideal state due to the influence of lighting conditions and scene complexity. Specifically, in a dimly lit environment, the quality of the RGB feature map decreases, resulting in a decrease in the accuracy of fire feature recognition, which in turn affects the reliability and stability of the entire warning system; at the same time, complex scene factors such as confusion between the object to be detected and the background and the presence of obstructions can easily cause false alarms or missed alarms. The former may lead to invalid waste of resources, and the latter may delay the effective response to the fire. Summary of the invention
[0003] In order to solve the technical problems existing in the background technology, the present invention proposes a multi-source time-series image feature collaborative integration method and system and a fire warning method.
[0004] The present invention proposes a method for collaboratively integrating multi-source temporal image features, comprising:
[0005] Get the paired RGB feature map and infrared feature map;
[0006] Integrate the RGB feature map and the infrared feature map to obtain the current multi-source fusion feature map;
[0007] Obtain multiple historical multi-source fusion feature maps;
[0008] According to the current multi-source fusion feature map and multiple historical multi-source fusion feature maps, a multi-source time series fusion feature map is obtained.
[0009] Preferably, the RGB feature map and the infrared feature map are integrated to obtain the current multi-source fusion feature map, which specifically includes:
[0010] Perform a maximum pooling operation on the RGB feature map features to obtain the first intermediate feature map;
[0011] Adaptively fusing the first intermediate feature map with the infrared feature map to obtain a third intermediate feature map;
[0012] Perform transposed convolution on the third intermediate feature map to obtain a fourth intermediate feature map;
[0013] Normalizing the fourth intermediate feature map to obtain a fifth intermediate feature map;
[0014] Perform matrix multiplication on the fourth feature map and the RGB feature map to obtain a sixth intermediate feature map;
[0015] Add the sixth intermediate feature map to the RGB feature map element by element to obtain a seventh intermediate feature map;
[0016] Perform feature concatenation on the seventh intermediate feature map and the third intermediate feature map to obtain an eighth intermediate feature map;
[0017] Convolve the eighth intermediate feature map to obtain the current multi-source fusion feature map.
[0018] Preferably, the first intermediate feature map is adaptively fused with the infrared feature map to obtain a third intermediate feature map, which specifically includes:
[0019] Based on cross attention, the first intermediate feature map is fused with the infrared feature map to obtain a cross attention map, and the cross attention map is used as the second intermediate feature map;
[0020] The second intermediate feature map and the infrared feature map are added element by element to obtain a third intermediate feature map.
[0021] Preferably, normalizing the fourth intermediate feature map to obtain a fifth intermediate feature map specifically includes:
[0022] The sigmoid function is used to normalize the fourth intermediate feature map to obtain the fifth intermediate feature map.
[0023] Preferably, the seventh intermediate feature map is feature concatenated with the third intermediate feature map to obtain an eighth intermediate feature map, which specifically includes:
[0024] The seventh intermediate feature map and the third intermediate feature map are feature concatenated along the dimension of the feature channel to obtain an eighth intermediate feature map.
[0025] Preferably, convolution is performed on the eighth intermediate feature map to obtain the current multi-source fusion feature map, specifically including:
[0026] The eighth intermediate feature map is convolved using a conventional convolution operation to obtain the current multi-source fusion feature map.
[0027] Preferably, after integrating the RGB feature map and the infrared feature map to obtain the current multi-source fusion feature map, the method further includes:
[0028] Store the current multi-source fusion feature map.
[0029] Preferably, a multi-source time series fusion feature map is obtained according to the current multi-source fusion feature map and a plurality of historical multi-source fusion feature maps, specifically including:
[0030] According to the current multi-source fusion feature map, a spatial attention feature map is obtained;
[0031] Perform matrix multiplication on the spatial attention feature map and multiple historical multi-source fusion feature maps respectively to obtain multiple ninth intermediate feature maps;
[0032] Using a relative time coding technique as a time encoder to encode the time of a plurality of ninth intermediate feature maps, thereby obtaining a plurality of ninth intermediate feature maps with time sequence information;
[0033] The plurality of ninth intermediate feature maps with time series information are feature concatenated with the current multi-source fusion feature map to obtain a multi-source time series fusion feature map.
[0034] Preferably, before performing feature splicing on the plurality of ninth intermediate feature maps with time series information and the current multi-source fusion feature map to obtain the multi-source time series fusion feature map, the method further includes:
[0035] The attention mechanism is used to perform weighted processing on multiple ninth intermediate feature maps with temporal information;
[0036] The weighted plurality of ninth intermediate feature maps with time series information and the current multi-source fusion feature map are feature spliced to obtain a multi-source time series fusion feature map.
[0037] Preferably, a spatial attention feature map is obtained according to the current multi-source fusion feature map, specifically including:
[0038] Perform a maximum pooling operation on the current multi-source fusion feature map in the feature channel dimension to obtain the tenth intermediate feature map;
[0039] Perform an average pooling operation on the current multi-source fusion feature map in the feature channel dimension to obtain the eleventh intermediate feature map;
[0040] Perform feature concatenation on the tenth intermediate feature map and the eleventh intermediate feature map to obtain a twelfth intermediate feature map;
[0041] Perform a convolution operation on the twelfth intermediate feature map to obtain the spatial attention feature map.
[0042] The present invention also proposes a multi-source time-series image feature collaborative integration system, comprising:
[0043] A first acquisition module is used to acquire a paired RGB feature map and an infrared feature map;
[0044] The first integration module is used to integrate the RGB feature map and the infrared feature map to obtain the current multi-source fusion feature map;
[0045] The second acquisition module is used to acquire multiple historical multi-source fusion feature maps;
[0046] The second integration module is used to obtain a multi-source time series fusion feature map according to the current multi-source fusion feature map and multiple historical multi-source fusion feature maps.
[0047] Preferably, it also includes:
[0048] The storage module is used to store the current multi-source fusion feature map.
[0049] Preferably, the first integration module is used to perform a maximum pooling operation on the RGB feature map features to obtain a first intermediate feature map;
[0050] Adaptively fusing the first intermediate feature map with the infrared feature map to obtain a third intermediate feature map;
[0051] Perform transposed convolution on the third intermediate feature map to obtain a fourth intermediate feature map;
[0052] Normalizing the fourth intermediate feature map to obtain a fifth intermediate feature map;
[0053] Perform matrix multiplication on the fourth feature map and the RGB feature map to obtain a sixth intermediate feature map;
[0054] Add the sixth intermediate feature map to the RGB feature map element by element to obtain a seventh intermediate feature map;
[0055] Perform feature concatenation on the seventh intermediate feature map and the third intermediate feature map to obtain an eighth intermediate feature map;
[0056] The eighth intermediate feature map is convolved to obtain the current multi-source fusion feature map.
[0057] Preferably, the second integration module is used to obtain a spatial attention feature map according to the current multi-source fusion feature map;
[0058] Perform matrix multiplication on the spatial attention feature map and multiple historical multi-source fusion feature maps respectively to obtain multiple ninth intermediate feature maps;
[0059] Using a relative time coding technique as a time encoder to encode the time of a plurality of ninth intermediate feature maps, thereby obtaining a plurality of ninth intermediate feature maps with time sequence information;
[0060] The plurality of ninth intermediate feature maps with time series information are feature concatenated with the current multi-source fusion feature map to obtain a multi-source time series fusion feature map.
[0061] The present invention also proposes a fire early warning method, comprising:
[0062] Get paired RGB and infrared images;
[0063] Feature extraction is performed on the RGB image and the infrared image respectively to obtain paired RGB feature maps and infrared feature maps;
[0064] Using any one of the multi-source time series image feature collaborative integration methods described above to perform feature collaborative integration on the RGB feature map and the infrared feature map to obtain a multi-source time series fusion feature map;
[0065] Perform fire detection on multi-source time series fusion feature maps to obtain fire identification results;
[0066] Based on the fire identification results, an alarm is issued.
[0067] In the present invention, the proposed multi-source time series image feature collaborative integration method and system and fire warning method, by integrating the RGB feature map and the infrared feature map to obtain the current multi-source fusion feature map, can give full play to the feature expression advantages of the RGB image and the infrared image, and effectively solve the feature matching and fusion problems caused by the difference in resolution between RGB and infrared images, which is conducive to the subsequent realization of highly robust fire monitoring and early warning, especially in complex and changeable environments. Moreover, through the current multi-source fusion feature map and multiple historical multi-source fusion feature maps, a multi-source time series fusion feature map is obtained. This embodiment can make full use of the pattern information and trend characteristics hidden in the historical data, effectively guide and enhance the prediction and judgment of the subsequent current image fire risk, and further improve the performance and timeliness of the subsequent overall early warning method. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] Figure 1 The figure is a flowchart of a method for collaboratively integrating multi-source temporal image features in one embodiment of the present invention.
[0069] Figure 2 A practical schematic diagram of a first integrated module in an embodiment of the present invention.
[0070] Figure 3 This is a schematic diagram of the use of the second integrated module in an embodiment of the present invention. DETAILED DESCRIPTION
[0071] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0072] Reference Figure 1The present invention proposes a method for collaboratively integrating multi-source temporal image features, comprising:
[0073] Get the paired RGB feature map and infrared feature map;
[0074] Integrate the RGB feature map and the infrared feature map to obtain the current multi-source fusion feature map;
[0075] Obtain multiple historical multi-source fusion feature maps;
[0076] According to the current multi-source fusion feature map and multiple historical multi-source fusion feature maps, a multi-source time series fusion feature map is obtained.
[0077] Due to the advantage of infrared images that can directly reflect the temperature changes of objects, they are not affected by the intensity of ambient light and have a high sensitivity and specificity for fire heat source detection. The present invention integrates the RGB feature map and the infrared feature map to obtain the current multi-source fusion feature map, which can give full play to the feature expression advantages of the RGB image and the infrared image, and effectively solve the feature matching and fusion problems caused by the difference in resolution between RGB and infrared images, which is conducive to the subsequent realization of highly robust fire monitoring and early warning, especially in complex and changeable environments. Moreover, by obtaining a multi-source time series fusion feature map through the current multi-source fusion feature map and multiple historical multi-source fusion feature maps, this embodiment can make full use of the pattern information and trend characteristics hidden in the historical data, effectively guide and enhance the prediction and judgment of the subsequent current image fire risk, and further improve the performance and timeliness of the subsequent overall early warning method.
[0078] In this embodiment, the RGB feature map and the infrared feature map are integrated to obtain the current multi-source fusion feature map, which specifically includes:
[0079] Perform a maximum pooling operation on the RGB feature map features to obtain the first intermediate feature map;
[0080] Adaptively fusing the first intermediate feature map with the infrared feature map to obtain a third intermediate feature map;
[0081] Perform transposed convolution on the third intermediate feature map to obtain a fourth intermediate feature map;
[0082] Normalizing the fourth intermediate feature map to obtain a fifth intermediate feature map;
[0083] Perform matrix multiplication on the fourth feature map and the RGB feature map to obtain a sixth intermediate feature map;
[0084] Add the sixth intermediate feature map to the RGB feature map element by element to obtain a seventh intermediate feature map;
[0085] Perform feature concatenation on the seventh intermediate feature map and the third intermediate feature map to obtain an eighth intermediate feature map;
[0086] The eighth intermediate feature map is convolved to obtain the current multi-source fusion feature map.
[0087] like Figure 2 As shown, this embodiment performs a max-pooling operation on the high-resolution RGB feature map to filter out key information, and performs adaptive feature fusion with the low-resolution infrared feature map to achieve mutual complementation of information. Moreover, in order to enhance the effect of the fused features, the enhanced infrared feature map is upsampled by transposed convolution to align with the high-resolution RGB feature map in the spatial dimension. In view of the temperature information carried by the infrared feature map in the spatial dimension, the sigmoid function is used to learn the spatial attention weight, so that the enhancement of the RGB feature map can be targeted and adjusted according to the temperature distribution. After the features of the two branches are enhanced, they are combined and spliced along the dimension of the feature channel, retaining multi-source information, and the spliced multi-source image features are integrated through conventional convolution operations to obtain a more robust feature representation.
[0088] In a further embodiment, the first intermediate feature map is adaptively fused with the infrared feature map to obtain a third intermediate feature map, which specifically includes:
[0089] Based on cross attention, the first intermediate feature map is fused with the infrared feature map to obtain a cross attention map, and the cross attention map is used as the second intermediate feature map;
[0090] The second intermediate feature map and the infrared feature map are added element by element to obtain a third intermediate feature map.
[0091] In this embodiment, the adaptive feature fusion of the first intermediate feature map and the infrared feature map is realized through the cross-attention mechanism, wherein the RGB feature map is responsible for extracting the linear mapping of the query information, while the infrared feature map is responsible for extracting the linear mapping of the key and value information, thereby realizing adaptive feature fusion, that is, the RGB image features can dynamically respond to and enhance the information related to the infrared image features.
[0092] In a further embodiment, normalizing the fourth intermediate feature map to obtain a fifth intermediate feature map specifically includes:
[0093] The sigmoid function is used to normalize the fourth intermediate feature map to obtain the fifth intermediate feature map.
[0094] In view of the temperature information carried by the infrared feature map in the spatial dimension, in this embodiment, the spatial attention weight is learned through the sigmoid function, so that the enhancement of the RGB features can be adjusted in a targeted manner according to the temperature distribution.
[0095] In a further embodiment, the seventh intermediate feature map is feature concatenated with the third intermediate feature map to obtain an eighth intermediate feature map, which specifically includes:
[0096] The seventh intermediate feature map is concatenated with the third intermediate feature map along the dimension of the feature channel to obtain an eighth intermediate feature map to retain multi-source information.
[0097] In a further embodiment, convolution is performed on the eighth intermediate feature map to obtain the current multi-source fusion feature map, specifically including:
[0098] The eighth intermediate feature map is convolved using a conventional convolution operation to obtain the current multi-source fusion feature map to obtain a more robust feature representation.
[0099] In this embodiment, after the RGB feature map and the infrared feature map are integrated to obtain the current multi-source fusion feature map, the following steps are also included:
[0100] The current multi-source fusion feature map is stored so that the historical multi-source fusion feature map can be obtained in time later.
[0101] like Figure 3 As shown, in this embodiment, a multi-source time series fusion feature map is obtained according to the current multi-source fusion feature map and multiple historical multi-source fusion feature maps, specifically including:
[0102] According to the current multi-source fusion feature map, a spatial attention feature map is obtained;
[0103] Perform matrix multiplication on the spatial attention feature map and multiple historical multi-source fusion feature maps respectively to obtain multiple ninth intermediate feature maps;
[0104] Using a relative time coding technique as a time encoder to encode the time of a plurality of ninth intermediate feature maps, thereby obtaining a plurality of ninth intermediate feature maps with time sequence information;
[0105] The plurality of ninth intermediate feature maps with time series information are feature concatenated with the current multi-source fusion feature map to obtain a multi-source time series fusion feature map.
[0106] This embodiment uses the current multi-source fusion feature map to generate a spatial attention feature map, which not only highlights the key information of the current frame, but also effectively integrates and interacts with the historical multi-source fusion feature map. In addition, relative time coding is introduced to enable the historical multi-source fusion feature map to be adaptively retrieved and applied according to the time relationship. This embodiment integrates the current features with the historical features in a collaborative way to form a multi-source time series fusion feature map, optimizes the representation of the features, and can provide a more comprehensive feature foundation for the subsequent fire warning system.
[0107] In a further embodiment, before performing feature splicing on the plurality of ninth intermediate feature maps with time series information and the current multi-source fusion feature map to obtain the multi-source time series fusion feature map, the method further includes:
[0108] The attention mechanism is used to perform weighted processing on multiple ninth intermediate feature maps with temporal information;
[0109] The weighted plurality of ninth intermediate feature maps with time series information and the current multi-source fusion feature map are feature spliced to obtain a multi-source time series fusion feature map.
[0110] Because not all historical data are equally important and some may have greater predictive value, in this embodiment, a plurality of ninth intermediate feature maps with time series information are weighted through an attention mechanism; then the weighted plurality of ninth intermediate feature maps with time series information and the current multi-source fusion feature map are collaboratively integrated to form a multi-source time series fusion feature map, which further optimizes the feature representation and can provide a more comprehensive feature basis for subsequent fire warning systems.
[0111] In a further embodiment, a spatial attention feature map is obtained according to the current multi-source fusion feature map, specifically including:
[0112] Perform a maximum pooling operation on the current multi-source fusion feature map in the feature channel dimension to obtain the tenth intermediate feature map;
[0113] Perform an average pooling operation on the current multi-source fusion feature map in the feature channel dimension to obtain the eleventh intermediate feature map;
[0114] Perform feature concatenation on the tenth intermediate feature map and the eleventh intermediate feature map to obtain a twelfth intermediate feature map;
[0115] Perform a convolution operation on the twelfth intermediate feature map to obtain the spatial attention feature map.
[0116] In this process, the maximum pooling and summing in the feature channel direction are used. Average pooling ( avg-pooling )The operation compresses the feature dimension, which helps to extract more representative information. At the same time, the convolution operation is used to learn the spatial attention weights, so that the attention mechanism can focus on important feature areas more intelligently.
[0117] The present invention also proposes a multi-source time-series image feature collaborative integration system, comprising:
[0118] A first acquisition module is used to acquire a paired RGB feature map and an infrared feature map;
[0119] The first integration module is used to integrate the RGB feature map and the infrared feature map to obtain the current multi-source fusion feature map;
[0120] The second acquisition module is used to acquire multiple historical multi-source fusion feature maps;
[0121] The second integration module is used to obtain a multi-source time series fusion feature map according to the current multi-source fusion feature map and multiple historical multi-source fusion feature maps.
[0122] The present invention integrates the RGB feature map and the infrared feature map through the first integration module to obtain the current multi-source fusion feature map, which can give full play to the feature expression advantages of the RGB image and the infrared image, and effectively solve the feature matching and fusion problems caused by the difference in resolution between RGB and infrared images, which is conducive to the subsequent realization of highly robust fire monitoring and early warning, especially in complex and changeable environments. Moreover, the current multi-source fusion feature map and multiple historical multi-source fusion feature maps are integrated through the second integration module to obtain a multi-source time series fusion feature map, which can make full use of the pattern information and trend characteristics hidden in the historical data, effectively guide and enhance the prediction and judgment of the subsequent current image fire risk, and further improve the performance and timeliness of the subsequent overall early warning method.
[0123] In this embodiment, it also includes:
[0124] The storage module is used to store the current multi-source fusion feature map so that the historical multi-source fusion feature map can be obtained in time in the future. It is conducive to the in-depth mining of the time series features in the video image, and can better understand and predict the object movement and temperature rise process, which is particularly important for early fire detection.
[0125] Wherein, during the integration process, the first integration module performs a maximum pooling operation on the RGB feature map features to obtain a first intermediate feature map;
[0126] Adaptively fusing the first intermediate feature map with the infrared feature map to obtain a third intermediate feature map;
[0127] Perform transposed convolution on the third intermediate feature map to obtain a fourth intermediate feature map;
[0128] Normalizing the fourth intermediate feature map to obtain a fifth intermediate feature map;
[0129] Perform matrix multiplication on the fourth feature map and the RGB feature map to obtain a sixth intermediate feature map;
[0130] Add the sixth intermediate feature map to the RGB feature map element by element to obtain a seventh intermediate feature map;
[0131] Perform feature concatenation on the seventh intermediate feature map and the third intermediate feature map to obtain an eighth intermediate feature map;
[0132] The eighth intermediate feature map is convolved to obtain the current multi-source fusion feature map.
[0133] Among them, during the integration process, the second integration module obtains a spatial attention feature map according to the current multi-source fusion feature map;
[0134] Perform matrix multiplication on the spatial attention feature map and multiple historical multi-source fusion feature maps respectively to obtain multiple ninth intermediate feature maps;
[0135] Using a relative time coding technique as a time encoder to encode the time of a plurality of ninth intermediate feature maps, thereby obtaining a plurality of ninth intermediate feature maps with time sequence information;
[0136] The plurality of ninth intermediate feature maps with time series information are feature concatenated with the current multi-source fusion feature map to obtain a multi-source time series fusion feature map.
[0137] The present invention also proposes a fire early warning method, comprising:
[0138] Get paired RGB and infrared images;
[0139] Feature extraction is performed on the RGB image and the infrared image respectively to obtain paired RGB feature maps and infrared feature maps;
[0140] Using any one of the multi-source time series image feature collaborative integration methods described above to perform feature collaborative integration on the RGB feature map and the infrared feature map to obtain a multi-source time series fusion feature map;
[0141] Perform fire detection on multi-source time series fusion feature maps to obtain fire identification results;
[0142] Based on the fire identification results, an alarm is issued.
[0143] Due to the advantages of integrating RGB and infrared images, the present invention can adapt to a wider range of environmental conditions, not only improving the accuracy and robustness of predictions, but also helping to more deeply understand and predict the development of fires. The present invention helps to reduce the false alarm rate and missed alarm rate of fire warning systems. This is crucial to ensure public safety and reduce unnecessary interference.
[0144] In the process of feature extraction of RGB images, a larger convolution kernel size and a relatively large convolution step size are first used to learn shallow features. This not only effectively reduces the resolution of image features, but also significantly reduces the time consumption of subsequent convolution operations. Then, by stacking standardized convolution modules layer by layer, the dimension of the feature channel is continuously increased. At the same time, the input and output results of different layers are connected using residual learning and feature identity mapping modes, ensuring the smooth transmission of gradient information and obtaining rich and accurate semantic feature expressions.
[0145] In the process of feature extraction of infrared images, since infrared images contain rich temperature gradient information, a convolution operation with a step size of 1 is always used in the feature extraction process to ensure that the spatial resolution of the infrared image is not affected and the spatial information of the original image is retained; in order to balance the extraction effect and computational efficiency, the depthwise separable convolution technology is used. This convolution method can effectively reduce the computational complexity while ensuring the feature extraction effect, making the entire feature extraction process both efficient and accurate.
[0146] In this embodiment, fire detection is performed on the multi-source time series fusion feature map to obtain a fire identification result, which specifically includes:
[0147] The target detection network is used to detect fires on multi-source time series fusion feature maps to obtain fire recognition results.
[0148] Specifically, the target detection network is a target detection network commonly used in the prior art, such as YOLO and DETR series.
[0149] Of course, before using the target detection network to detect the multi-source time series fusion feature map, it is necessary to use Focal Loss and Smooth L1 Loss loss functions to optimize the training process of the target detection network under the guidance of supervised learning. In the detection stage of the target detection network, post-processing techniques such as non-maximum suppression (NMS) are introduced to eliminate the overlap of detection boxes, thereby improving the accuracy of detection.
[0150] In this embodiment, the fire identification result includes the target object temperature and the ambient temperature of consecutive frames within a preset time window.
[0151] Among them, according to the fire identification results, an alarm is issued, including:
[0152] Calculate the difference between the target object temperature and the ambient temperature in each frame;
[0153] When the difference is greater than a preset first threshold, marking the frame as an abnormal temperature frame;
[0154] Calculate the number of consecutive abnormal temperature frames within a preset time window;
[0155] When the number is greater than a preset second threshold, it is determined that a fire has occurred and an alarm is issued.
[0156] Of course, in this embodiment, the existing fire alarm strategy can also be used to issue an alarm based on the fire identification result.
[0157] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A method for collaborative integration of multi-source temporal image features. It is characterized in that include: Get the paired RGB feature map and infrared feature map; Integrate the RGB feature map and the infrared feature map to obtain the current multi-source fusion feature map; Obtain multiple historical multi-source fusion feature maps; According to the current multi-source fusion feature map and multiple historical multi-source fusion feature maps, a multi-source time series fusion feature map is obtained.
2. According to claim 1, the method for collaborative integration of multi-source temporal image features, It is characterized in that Integrate the RGB feature map and the infrared feature map to obtain the current multi-source fusion feature map, which specifically includes: Perform a maximum pooling operation on the RGB feature map features to obtain the first intermediate feature map; Adaptively fusing the first intermediate feature map with the infrared feature map to obtain a third intermediate feature map; Perform transposed convolution on the third intermediate feature map to obtain a fourth intermediate feature map; Normalizing the fourth intermediate feature map to obtain a fifth intermediate feature map; Perform matrix multiplication on the fourth feature map and the RGB feature map to obtain a sixth intermediate feature map; Add the sixth intermediate feature map to the RGB feature map element by element to obtain a seventh intermediate feature map; Perform feature concatenation on the seventh intermediate feature map and the third intermediate feature map to obtain an eighth intermediate feature map; Convolve the eighth intermediate feature map to obtain the current multi-source fusion feature map; Preferably, the first intermediate feature map is adaptively fused with the infrared feature map to obtain a third intermediate feature map, which specifically includes: Based on cross attention, the first intermediate feature map is fused with the infrared feature map to obtain a cross attention map, and the cross attention map is used as the second intermediate feature map; Adding the second intermediate feature map and the infrared feature map element by element to obtain a third intermediate feature map; Preferably, normalizing the fourth intermediate feature map to obtain a fifth intermediate feature map specifically includes: The fourth intermediate feature map is normalized using the sigmoid function to obtain a fifth intermediate feature map; Preferably, the seventh intermediate feature map is feature-concatenated with the third intermediate feature map to obtain an eighth intermediate feature map, which specifically includes: Perform feature concatenation on the seventh intermediate feature map and the third intermediate feature map along the dimension of the feature channel to obtain an eighth intermediate feature map; Preferably, convolving the eighth intermediate feature map to obtain the current multi-source fusion feature map specifically includes: The eighth intermediate feature map is convolved using a conventional convolution operation to obtain the current multi-source fusion feature map.
3. The method for collaborative integration of multi-source temporal image features according to claim 1, It is characterized in that After the RGB feature map and the infrared feature map are integrated to obtain the current multi-source fusion feature map, the method further includes: storing the current multi-source fusion feature map.
4. The method for collaborative integration of multi-source temporal image features according to claim 1, It is characterized in that According to the current multi-source fusion feature map and multiple historical multi-source fusion feature maps, a multi-source time series fusion feature map is obtained, which specifically includes: According to the current multi-source fusion feature map, a spatial attention feature map is obtained; Perform matrix multiplication on the spatial attention feature map and multiple historical multi-source fusion feature maps respectively to obtain multiple ninth intermediate feature maps; Using a relative time coding technique as a time encoder to encode the time of a plurality of ninth intermediate feature maps, thereby obtaining a plurality of ninth intermediate feature maps with time sequence information; Perform feature splicing on the plurality of ninth intermediate feature maps with time series information and the current multi-source fusion feature map to obtain a multi-source time series fusion feature map; Preferably, before performing feature splicing on the plurality of ninth intermediate feature maps with time series information and the current multi-source fusion feature map to obtain the multi-source time series fusion feature map, the method further includes: The attention mechanism is used to perform weighted processing on multiple ninth intermediate feature maps with temporal information; The weighted plurality of ninth intermediate feature maps with time series information and the current multi-source fusion feature map are feature spliced to obtain a multi-source time series fusion feature map.
5. The method for collaborative integration of multi-source temporal image features according to claim 4, It is characterized in that According to the current multi-source fusion feature map, the spatial attention feature map is obtained, which includes: Perform a maximum pooling operation on the current multi-source fusion feature map in the feature channel dimension to obtain the tenth intermediate feature map; Perform an average pooling operation on the current multi-source fusion feature map in the feature channel dimension to obtain the eleventh intermediate feature map; Perform feature concatenation on the tenth intermediate feature map and the eleventh intermediate feature map to obtain a twelfth intermediate feature map; Perform a convolution operation on the twelfth intermediate feature map to obtain the spatial attention feature map.
6. A multi-source time-series image feature collaborative integration system, It is characterized in that include: A first acquisition module is used to acquire a paired RGB feature map and an infrared feature map; The first integration module is used to integrate the RGB feature map and the infrared feature map to obtain the current multi-source fusion feature map; The second acquisition module is used to acquire multiple historical multi-source fusion feature maps; The second integration module is used to obtain a multi-source time series fusion feature map according to the current multi-source fusion feature map and multiple historical multi-source fusion feature maps.
7. The method for collaboratively integrating multi-source temporal image features according to claim 6, It is characterized in that Also includes: The storage module is used to store the current multi-source fusion feature map.
8. The method for collaboratively integrating multi-source temporal image features according to claim 6, It is characterized in that The first integrated module is used to perform a maximum pooling operation on the RGB feature map to obtain a first intermediate feature map; perform adaptive feature fusion on the first intermediate feature map and the infrared feature map to obtain a third intermediate feature map; perform transposed convolution on the third intermediate feature map to obtain a fourth intermediate feature map; normalize the fourth intermediate feature map to obtain a fifth intermediate feature map; perform matrix multiplication on the fourth feature map and the RGB feature map to obtain a sixth intermediate feature map; The sixth intermediate feature map is added element by element to the RGB feature map to obtain the seventh intermediate feature map; the seventh intermediate feature map is feature concatenated with the third intermediate feature map to obtain the eighth intermediate feature map; the eighth intermediate feature map is convolved to obtain the current multi-source fusion feature map.
9. The method for collaborative integration of multi-source temporal image features according to claim 6, It is characterized in that The second integration module is used to obtain a spatial attention feature map according to the current multi-source fusion feature map; perform matrix multiplication on the spatial attention feature map and multiple historical multi-source fusion feature maps respectively to obtain multiple ninth intermediate feature maps; use the relative time coding technology as a time encoder to encode the time of the multiple ninth intermediate feature maps to obtain multiple ninth intermediate feature maps with time sequence information; The plurality of ninth intermediate feature maps with time series information are feature concatenated with the current multi-source fusion feature map to obtain a multi-source time series fusion feature map.
10. A fire early warning method, It is characterized in that include: Get paired RGB and infrared images; Feature extraction is performed on the RGB image and the infrared image respectively to obtain paired RGB feature maps and infrared feature maps; Using the multi-source time series image feature collaborative integration method as described in any one of claims 1 to 5, the RGB feature map and the infrared feature map are collaboratively integrated to obtain a multi-source time series fusion feature map; Perform fire detection on multi-source time series fusion feature maps to obtain fire identification results; Based on the fire identification results, an alarm is issued.
Citation Information
Cited By
Forest fire spreading and backtracking prediction method based on multi-modal deep learning
CN119539178A
Lightweight model-driven unmanned aerial vehicle forest fire detection method
CN122574710A
