Weak fire source detection method

Abstract

The invention discloses a weak fire source detection method, which comprises an initial feature extraction module for obtaining initial features of visible light and infrared images; the cross-modal global correlation constrained knowledge migration module is used for modeling global correlation in different modal initial features respectively, and realizing inter-modal knowledge migration under the constraint of the cross-modal global correlation so as to obtain cross-modal knowledge compensation features of visible light and infrared images; the adaptive intra-modal feature enhancement module is used for generating attention weights of the initial features and the cross-modal knowledge compensation features in different modals respectively, and fusing the initial features and the cross-modal knowledge compensation features by utilizing the attention weights to obtain enhanced features; the inter-modal feature fusion module for prompting learning guidance constructs a learnable prompt according to the cross-modal enhancement feature, and generates a modal complementary feature under the guidance of a prompting learning strategy to obtain a final fusion feature; and the weak fire source detection head completes weak fire source detection by taking the fusion features as input.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and object detection, and in particular to a weak fire detection method. BACKGROUND

[0002] As a typical complex disaster, fire can cause a variety of impacts such as casualties, property losses and ecological environment destruction. Effective detection of weak fire can provide timely warning in the early stage of fire formation, thereby avoiding the occurrence of major fire accidents. Due to the ability of visible light cameras in capturing appearance details and the ability of infrared cameras in sensing thermal radiation, compared with single modal images, joint use of visible light and infrared images is beneficial to achieve more accurate weak fire detection. Therefore, multi-modal weak fire detection based on visible light and infrared images has important research significance and application value.

[0003] In recent years, with the development of deep neural networks, multi-modal fire detection based on deep learning has become the mainstream research direction. Mu et al. adopted a pixel-level fusion strategy to fuse visible light and infrared images, and used the fused images to train an existing object detector to achieve fire detection. Zhang et al. used channel attention mechanism to enhance the feature maps of visible light and infrared images, thereby capturing the dependency between the deep channels of the network and improving the multi-modal fire detection capability of the network. Liu et al. designed different detection networks for visible light and infrared images, and fused the prediction results of the two networks to achieve effective multi-modal fire detection.

[0004] The above methods have made certain research progress in the field of multi-modal fire detection. However, these methods usually use a simple way to extract and fuse multi-modal information, lack of in-depth mining of modal specific information and in-depth exploration of complementary information between modalities, which limits the further improvement of the performance of weak fire detection. SUMMARY

[0005] The present application proposes a weak fire detection method, which realizes knowledge transfer between modalities under the constraint of cross-modal global correlation, thereby obtaining cross-modal knowledge compensation features of visible light and infrared images, and jointly obtains enhanced features with modal initial features, effectively mining specific representations within modalities; in addition, the method introduces a prompt learning strategy, constructs a learnable prompt according to cross-modal information, to adaptively generate features complementary to modalities, thereby promoting sufficient fusion of features between modalities, achieving more accurate weak fire detection, as described in detail below:

[0006] A weak fire detection method, the method comprising:

[0007] An initial feature extraction module uses different modal backbone networks to obtain initial features of visible light and infrared images respectively;

[0008] The knowledge transfer module with cross-modal global correlation constraint obtains a global correlation matrix of the initial features of the visible light and infrared images by using a correlation perception unit; cross-modal knowledge compensation features of the visible light and infrared images are obtained by using knowledge transfer networks of different modalities respectively; a global correlation matrix of the cross-modal knowledge compensation features of the visible light and infrared images is obtained by using the correlation perception unit; a consistency constraint is constructed between the global correlation matrix of the cross-modal knowledge compensation features of the visible light image and the global correlation matrix of the initial features of the infrared image; a consistency constraint is constructed between the global correlation matrix of the cross-modal knowledge compensation features of the infrared image and the global correlation matrix of the initial features of the visible light image;

[0009] The adaptive intra-modal feature enhancement module obtains attention weights of the initial features and the cross-modal knowledge compensation features by using a global attention unit in the visible light and infrared modalities respectively; the initial features and the cross-modal knowledge compensation features in the modalities are fused by using the attention weights to obtain enhanced features;

[0010] The prompt learning guided inter-modal feature fusion module obtains a learnable prompt as input by using the cross-modal enhanced features, the learnable prompt is taken as a query vector to obtain a query feature by using a self-attention layer, the query feature and the enhanced features of the corresponding modalities are taken as inputs to obtain interactive features by using a cross-attention layer, and the interactive features are taken as inputs to obtain modal complementary features by using a feedforward layer; the modal complementary features of the two modalities are spliced to obtain final fusion features;

[0011] The weak fire detection head takes the fusion features as input to complete weak fire detection.

[0012] The cross-modal knowledge transfer process includes:

[0013] A consistency constraint is constructed between the global correlation matrix of the cross-modal knowledge compensation features of the visible light image and the global correlation matrix of the initial features of the infrared image; a consistency constraint is constructed between the global correlation matrix of the cross-modal knowledge compensation features of the infrared image and the global correlation matrix of the initial features of the visible light image.

[0014] The query feature is obtained as:

[0015] The cross-modal enhanced features are taken as input, a learnable prompt is generated by using a prompt generator, the learnable prompt is taken as a query vector, and a query feature is obtained by using a self-attention layer.

[0016] The global correlation matrix of the initial features of the visible light and infrared images is obtained by using the correlation perception unit as:

[0017]

[0018]

[0019] wherein, and denote the correlation-aware units of the first and the second visible light and infrared modalities, respectively, and denote the corresponding global correlation matrices, denote the initial features of the first and the second visible light and infrared images, respectively.

[0020] wherein, the global correlation matrix of the visible light and infrared image cross-modal knowledge compensation features obtained by the correlation-aware units is:

[0021]

[0022]

[0023] wherein, and denote the correlation-aware units of the first and the second visible light and infrared modalities, respectively, and denote the corresponding global correlation matrices, denote the cross-modal knowledge compensation features of the first and the second visible light and infrared images, respectively.

[0024] wherein, the consistency constraint is:

[0025]

[0026] wherein, denotes the cross-modal consistency loss, denotes the Euclidean distance, denotes the gradient stop operation.

[0027] wherein, the attention weights of the initial features and the cross-modal knowledge compensation features are:

[0028]

[0029]

[0030] wherein, , and denote the global attention units of the initial features and the cross-modal knowledge compensation features, respectively,​​​​​​​​​​ and respectively represent initial features and cross-modal knowledge compensation features.

[0031] wherein the modal complementary features are:

[0032]

[0033] wherein, , represent a learnable prompt, represent a self-attention layer, represent a cross-attention layer, represent a feed-forward layer, represent the obtained modal complementary features.

[0034] The technical scheme provided by the present application has the beneficial effects that:

[0035] 1. The present application effectively alleviates the problem of insufficient utilization of multi-modal information in existing methods by deeply mining specific information within multi-modal images and complementary information between modalities, thereby improving the performance of multi-modal weak fire detection.

[0036] 2. The present application designs a knowledge transfer module with cross-modal global correlation constraints, which realizes knowledge transfer between modalities under the constraint of cross-modal global correlation, thereby obtaining cross-modal knowledge compensation features of visible light and infrared images.

[0037] 3. The present application designs an inter-modal feature fusion module guided by prompt learning, which adaptively generates modal complementary features through a prompt learning strategy under the guidance of cross-modal information, to effectively fuse different modal features, thereby obtaining robust multi-modal fusion features. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a flowchart of a weak fire detection method. DETAILED DESCRIPTION

[0039] To make the purpose, technical scheme and advantages of the present application clearer, the following further describes the embodiments of the present application in detail.

[0040] I. Constructing an initial feature extraction module

[0041] The pair of visible light and infrared images to be detected have the same resolution, and the spatial size of the two images is , and the number of image channels is 3. The initial feature extraction module takes the pair of visible light and infrared images as input, and uses the backbone network of the visible light modality to obtain the initial features of the visible light image, and simultaneously uses the backbone network Obtaining initial features of infrared images The initial features include multiple scales, wherein represents the number of multi-scale detection features, and is set to 3. The features of different scales have different sizes of receptive fields to adapt to targets of different sizes in the image. The spatial sizes of the three initial features are specifically set to , and . The backbone network structures of the two modalities are the same, and both use CSPDarknet and FPN. CSPDarknet and FPN are well known to those skilled in the art, and embodiments of the present application do not repeat them.

[0042] II. Designing a knowledge transfer module with cross-modal global correlation constraints

[0043] On the basis of the initial feature extraction module, the embodiments of the present application design a knowledge transfer module with cross-modal global correlation constraints. The module realizes inter-modal knowledge transfer under the constraint of cross-modal global correlation, thereby obtaining cross-modal knowledge compensation features of visible light and infrared images. First, the knowledge transfer module with cross-modal global correlation constraints designed by the present application uses a correlation perception unit to obtain the global correlation matrix of the initial features of different modalities.

[0044] The formula of the above process is as follows:

[0045]

[0046]

[0047] wherein, and represent the correlation perception units of the first visible light and infrared modalities, respectively, and represent the global correlation matrices of and , respectively, and represent the initial features of the first visible light and infrared images, respectively. The correlation perception units of the visible light and infrared modalities adopt the same structure, which includes a layer of Non-local layer. In the embodiments of the present application, is set to 2.

[0048] Then, the designed module uses the knowledge transfer network of the visible light modality to obtain the cross-modal knowledge compensation features of the visible light image, and simultaneously uses the knowledge transfer network of the infrared modality to obtain the cross-modal knowledge compensation features of the infrared image where the knowledge transfer networks of two modalities have the same backbone network structure. For images of different modalities, the number of cross-modal knowledge compensation features is set to 3. The spatial size of 3 features is specifically set to , and .

[0049] Subsequently, the global correlation matrix of the cross-modal knowledge compensation features of the visible light and infrared images is obtained by using the correlation perception unit. The formula representation of the above process is as follows:

[0050]

[0051]

[0052] wherein, and represent the correlation perception units of the th visible light and infrared modalities, and represent the corresponding global correlation matrices of and , respectively, and represent the cross-modal knowledge compensation features of the th visible light and infrared images.

[0053] To ensure that the cross-modal knowledge compensation features of the visible light images can effectively learn the knowledge in the initial features of the infrared images , a consistency constraint is constructed between the global correlation matrix of the cross-modal knowledge compensation features of the visible light images and the global correlation matrix of the initial features of the infrared images . Similarly, to ensure that the cross-modal knowledge compensation features of the infrared images can effectively learn the knowledge in the initial features of the visible light images , a consistency constraint is constructed between the global correlation matrix of the cross-modal knowledge compensation features of the infrared images and the global correlation matrix of the initial features of the visible light images . The formula representation of the above process is as follows:

[0054]

[0055] wherein, represents the cross-modal consistency loss, represents the Euclidean distance, represents the gradient stop operation.

[0056] III. Construction of adaptive intra-modal feature enhancement module

[0057] In the adaptive intra-modal feature enhancement module, the initial intra-modal feature and the cross-modal knowledge compensation feature are adaptively fused to realize intra-modal feature enhancement. The module first obtains the attention weights of the initial feature and the cross-modal knowledge compensation feature by using a global attention unit, and the formula is as follows:

[0058]

[0059]

[0060] wherein, , and respectively represent the global attention units of the initial feature and the cross-modal knowledge compensation feature, and respectively represent the initial feature and the cross-modal knowledge compensation feature. The two global attention units have the same structure and are both composed of 3 convolutional layers and an activation layer. Among them, the convolution kernel size of the convolutional layer is set to , and the activation layer uses activation function to realize. and respectively represent the corresponding attention weights of and .

[0061] Then, the attention weights are used to fuse the intra-modal two features to obtain the enhanced feature. The specific calculation process is as follows:

[0062]

[0063] wherein, represents the enhanced feature, represents the element multiplication operation.

[0064] Four, design prompt learning guided inter-modal feature fusion module

[0065] In order to fully fuse the enhanced features of the two modalities to obtain the final multi-modal fusion feature, the embodiment of the application designs a prompt learning guided inter-modal feature fusion module. The module first takes the cross-modal enhanced feature as input, and uses a prompt generator to generate a learnable prompt .

[0066]

[0067] wherein, and . represents the prompt generator, and the dimensions of the learnable prompts of different modalities are the same, which is set to .

[0068] Then, the module learns the complementary feature representation with learnable cues As the query vector, the query Transformer network is used to learn more complementary feature representations. The query Transformer contains three parts: self-attention layers, cross-attention layers and feedforward layers. The query vector First, the query feature is obtained through the self-attention layer. The query feature and the enhanced feature of the corresponding modality are input into the cross-attention layer to obtain the interaction feature. Finally, the interaction feature is input into the feedforward layer to obtain the modality complementary feature. The above calculation process is represented as follows:

[0069]

[0070] wherein, , represents the learnable cue, represents the self-attention layer, represents the cross-attention layer, represents the feedforward layer, represents the obtained modality complementary feature.

[0071] Finally, the modality complementary features of the two modalities are spliced to obtain the final fusion feature. The above calculation process is shown as follows:

[0072]

[0073] wherein, and . represents the feature splicing operation, represents the fused feature, and represent the modality complementary features of the two modalities, respectively. In the embodiment of the present application, contains three scales, and the spatial dimensions are specifically set as , and .

[0074] Five, construct a weak fire detection head

[0075] The weak fire detection head takes the fusion feature as input to complete the weak fire detection. Specifically, the classification branch is used to predict whether the target is a weak fire, and the regression branch is used to predict the position of the weak fire, which is represented by the formula as follows:

[0076]

[0077]

[0078] wherein, and denote the classification branch and the regression branch, respectively. and denote the weak fire and its location, respectively, denotes the number of detected weak fires.

[0079] Six, training the weak fire detection network

[0080] To ensure that the designed network can effectively detect weak fires, the embodiments of the present application train the network in an end-to-end manner to obtain the trained model weights. During the training process, the distance between the predicted weak fire and the real weak fire is measured using the constructed loss function to constrain the network training. The formula of the constructed loss function is as follows:

[0081]

[0082] wherein, denotes the constructed cross-modal consistency loss, denotes the classification loss, denotes the regression loss, , and denote the weight coefficients of the cross-modal consistency loss, the classification loss and the regression loss, respectively. The classification loss and the regression loss are well known to those skilled in the art, and the embodiments of the present application do not repeat them.

[0083] Seven, weak fire detection application

[0084] After obtaining the trained model weights, the designed weak fire detection network loads the model weights to realize the recognition of weak fires in images and the positioning of the locations of weak fires, and can be effectively applied to forest fire monitoring scenarios. Specifically, a UAV equipped with visible light and infrared cameras takes pictures of forest scenes to obtain multi-modal images. Based on the multi-modal images of the forest scene, the designed method detects whether there is a weak fire in the image. If a weak fire is detected in the multi-modal image, the staff can determine the location of the weak fire in the real scene through the location of the UAV and the location of the weak fire in the image, so as to realize timely warning and accurate extinguishing in the early stage of fire formation, thereby avoiding the occurrence of major fire accidents.

[0085] In summary, the embodiments of the present application propose a weak fire detection method, which aims to learn the specific features within the modalities of multi-modal images and effectively fuse the features of different modalities based on the prompt learning strategy, so as to obtain more robust multi-modal fusion features and improve the performance of weak fire detection.

[0086] Those skilled in the art can understand that the drawings are only schematic illustrations of a preferred embodiment, and the above-mentioned embodiment numbers of the application are only for description, not representing the advantages and disadvantages of the embodiments.

[0087] The above description is only the preferred embodiment of the application, and is not used to limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A method for detecting a weak fire source, characterized in that: The method comprises: The initial feature extraction module uses the backbone networks of different modalities to obtain the initial features of visible light and infrared images respectively; The knowledge transfer module with cross-modal global correlation constraints uses a correlation perception unit to obtain the global correlation matrix of the initial features of visible light and infrared images; uses knowledge transfer networks of different modalities to obtain cross-modal knowledge compensation features of visible light and infrared images respectively; uses the correlation perception unit to obtain the global correlation matrix of the cross-modal knowledge compensation features of visible light and infrared images; constructs consistency constraints between the global correlation matrix of the cross-modal knowledge compensation features of visible light images and the global correlation matrix of the initial features of infrared images; and constructs consistency constraints between the global correlation matrix of the cross-modal knowledge compensation features of infrared images and the global correlation matrix of the initial features of visible light images. The adaptive intra-modal feature enhancement module uses global attention units to obtain the attention weights of the initial features and cross-modal knowledge compensation features in the visible light and infrared modalities respectively; and uses the attention weights to fuse the initial features and cross-modal knowledge compensation features in the modality to obtain enhanced features. The inter-modal feature fusion module guided by prompt learning uses cross-modal enhanced features as input to obtain learnable prompts. The learnable prompts are used as query vectors and passed through the self-attention layer to obtain query features. The query features and the enhanced features of the corresponding modality are passed through the cross-attention layer to obtain interactive features. The interactive features are passed through the feedforward layer to obtain modal complementary features. The modal complementary features of the two modalities are spliced ​​to obtain the final fusion features. The faint fire source detection head uses the fused features as input to complete the faint fire source detection.

2. A weak fire source detection method according to claim 1, characterized in that: The cross-modal knowledge transfer process includes: A consistency constraint is constructed between the global correlation matrix of the cross-modal knowledge compensation features of the visible light image and the global correlation matrix of the initial features of the infrared image; a consistency constraint is constructed between the global correlation matrix of the cross-modal knowledge compensation features of the infrared image and the global correlation matrix of the initial features of the visible light image.

3. A weak fire source detection method according to claim 1, characterized in that: The acquisition query features are: Taking the cross-modal enhanced features as input, the prompt generator generates learnable prompts, and the learnable prompts are used as query vectors to obtain query features through the self-attention layer.

4. A weak fire source detection method according to claim 1, characterized in that: The global correlation matrix of the initial features of the visible light and infrared images obtained by the correlation perception unit is: ; ; in, and Respectively represent A correlation sensing unit for visible light and infrared modalities, and Respectively and The corresponding global correlation matrix, and Respectively represent Initial features of visible and infrared images.

5. A method for detecting a weak fire source according to claim 1, characterized in that: The global correlation matrix of the cross-modal knowledge compensation features of visible light and infrared images obtained by using the correlation perception unit is: ; ; in, and Respectively represent A correlation sensing unit for visible light and infrared modalities, and Respectively and The corresponding global correlation matrix, and Respectively represent Cross-modal knowledge compensation features of visible light and infrared images.

6. A method for detecting a weak fire source according to claim 2, characterized in that: The consistency constraints are: ; in, represents the cross-modal consistency loss, represents the Euclidean distance, Indicates a gradient stop operation.

7. A weak fire source detection method according to claim 1, characterized in that: The attention weights of the initial features and cross-modal knowledge compensation features are: ; ; in, , and The global attention units representing the initial features and cross-modal knowledge compensation features, and Represent the initial features and cross-modal knowledge compensation features respectively.

8. A weak fire source detection method according to claim 1, characterized in that: The modal complementarity characteristics are: ; in, , Indicates a learning hint. represents the self-attention layer, represents the cross attention layer, represents the feed-forward layer, represents the obtained modal complementary features.

Images