Training method of fire detection model based on dual-spectrum image fusion

Abstract

The present application relates to a training method for a fire detection model of dual-spectrum image fusion, comprising: obtaining a deep neural network model to be trained, the deep neural network model comprising two parallel high-resolution networks, the two high-resolution networks being used to process different spectral images respectively, and each of the high-resolution networks comprising a first branch, a second branch and a third branch; performing a first training on the deep neural network model through dual-spectrum non-fire images; performing a second training on the neural network model after the first training by alternating single and dual-spectrum fire images; the deep neural network model after the second training is a fire detection model. The training method for the fire detection model provided by the present invention can be trained and improved using single-spectrum fire images, and there is no problem of difficulty in model convergence, which solves the problem that the current training method for the fire detection model cannot effectively use single-spectrum fire images to improve the training effect.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and in particular to a training method for a fire detection model of dual-spectrum image fusion. Background Art

[0002] Fire detection based on dual-spectrum information fusion generally uses the complementary information between visible light images and infrared (near infrared / thermal imaging) images to improve the accuracy of fire detection. Since the two spectral images are taken by visible light and infrared imaging devices respectively, even if the two images can have exactly the same camera internal parameters (such as focal length, pixel size, etc., equivalent to being taken by exactly the same device) through cropping and scaling of the images, there is still parallax between the dual-spectrum images due to the different locations of the devices (the same object is in different positions in the two images). The dual-spectrum images that have achieved internal parameter normalization still have parallax due to different external parameters of the device (placement location). Since the dual-spectrum images cannot be completely aligned, the existing dual-spectrum information fusion fire detection models usually adopt decision-level fusion or high-level feature fusion methods, but currently the decision-level fusion or high-level feature fusion are in the late stage of the model and cannot achieve early interaction of multi-spectral information. Early interaction of multi-spectral information can promote the model to learn more discriminative multi-spectral features and promote the improvement of model performance.

[0003] However, early and deep information fusion will make the neural network highly coupled with the dual-spectrum input. In the absence of any spectral input, the network will be greatly disturbed and unable to extract reliable single-spectrum features. This phenomenon makes it impossible to participate in the training of historically accumulated single-spectrum fire data. If the fusion model is simply input with single-spectrum data during the training phase, it will be difficult to converge and the model will be unstable because of the huge gap between the extracted single-spectrum features and the dual-spectrum features in the feature space.

[0004] There is currently no effective solution to the problem that the current training method of dual-spectrum fire detection models cannot effectively utilize single-spectrum fire images to improve training results. Summary of the invention

[0005] The present invention provides a training method for a dual-spectrum image fusion fire detection model to solve the problem that the current dual-spectrum fire detection model training method cannot effectively utilize single-spectrum fire images to improve the training effect.

[0006] In a first aspect, the present invention provides a training method for a fire detection model of dual-spectrum image fusion, comprising:

[0007] Obtaining a deep neural network model to be trained, the deep neural network model comprising two parallel high-resolution networks, the two high-resolution networks are respectively used to process different spectral images, and each of the high-resolution networks comprises a first branch, a second branch, and a third branch;

[0008] Performing a first training on the deep neural network model through the dual-spectrum non-fire image, wherein in the first training of the deep neural network model, the first feature maps of the two second branches, the first feature maps of the two third branches, and the last feature maps of the two third branches are all aligned through an alignment module;

[0009] The deep neural network model after the first training is subjected to a second training by alternating single-spectrum and dual-spectrum fire images. In the second training of the deep neural network model, when the input of the neural network model is a dual-spectrum fire image, the first feature maps of two second branches and the first feature maps of two third branches are fused with each other through a fusion module, and the first feature map of each second branch is fused with the first feature map of another second branch as a branch reasoning basis, and the first feature map of each third branch is fused with the first feature map of another third branch as a branch reasoning basis;

[0010] The deep neural network model after the second training is a fire detection model.

[0011] In a second aspect, the present invention provides a fire detection method, comprising:

[0012] Inputting a target image to be identified into a trained fire detection model, wherein the target image is a single-spectrum image or a dual-spectrum image;

[0013] The target image is subjected to image recognition by the fire detection model, and whether there is a flame in the target image is determined according to the image recognition result. The fire detection model is obtained by the training method of the dual-spectrum image fusion fire detection model described in the first aspect.

[0014] In a third aspect, the present invention provides a training device for a fire detection model of dual-spectrum image fusion, comprising:

[0015] A model acquisition module, used to acquire a deep neural network model to be trained, wherein the deep neural network model includes two parallel high-resolution networks, the two high-resolution networks are used to process different spectral images respectively, and each of the high-resolution networks includes a first branch, a second branch and a third branch;

[0016] A first training module is used to perform a first training on the deep neural network model through a dual-spectrum non-fire image, wherein in the first training of the deep neural network model, the first feature maps of the two second branches, the first feature maps of the two third branches, and the last feature maps of the two third branches are aligned through an alignment module;

[0017] A second training module is used to perform a second training on the deep neural network model after the first training by alternating single-spectrum and dual-spectrum fire images. In the second training of the deep neural network model, when the input of the neural network model is a dual-spectrum fire image, the first feature maps of two second branches and the first feature maps of two third branches are fused with each other through a fusion module, and the first feature map of each second branch is fused with the first feature map of another second branch as a branch reasoning basis, and the first feature map of each third branch is fused with the first feature map of another third branch as a branch reasoning basis;

[0018] The second trained deep neural network model is a fire detection model.

[0019] In a fourth aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the training method of the fire detection model of dual-spectrum image fusion described in the first aspect or the fire detection method described in the second aspect.

[0020] In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method of the fire detection model of dual-spectrum image fusion described in the first aspect or the steps of the fire detection method described in the second aspect.

[0021] Compared with the related art, the training method of the dual-spectrum image fusion fire detection model provided by the present invention can use single-spectrum fire images for training and be improved, and there is no problem of model difficulty in convergence. It solves the problem that the current training method of the dual-spectrum fire detection model cannot effectively use single-spectrum fire images to improve the training effect.

[0022] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of a training method for a fire detection model using dual-spectrum image fusion provided in an embodiment of the present invention;

[0024] Figure 2 is a feature processing flow chart of the fire detection model provided in an embodiment of the present invention when performing the first training;

[0025] Figure 3 is a feature processing flow chart of an alignment module provided in an embodiment of the present invention;

[0026] Figure 4 is a feature processing flow chart of the fire detection model provided in an embodiment of the present invention when performing the second training;

[0027] Figure 5 It is a feature processing flow chart of the fusion module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0029] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0030] In an embodiment of the present invention, a training method for a fire detection model of dual-spectrum image fusion is provided. Figure 1 FIG. 1 is a flow chart of a training method for a fire detection model using dual-spectrum image fusion provided in an embodiment of the present invention. Figure 1As shown, the process includes the following steps:

[0031] Step S110, obtaining a deep neural network model to be trained, the deep neural network model includes two parallel high-resolution networks (HRNet), the two high-resolution networks are used to process different spectral images respectively, and each high-resolution network includes a first branch, a second branch and a third branch.

[0032] Specifically, the deep neural network model is a neural network model based on deep learning. The deep neural network model for flame detection used in the present invention is built based on a high-resolution network. The high-resolution network is characterized by combining multiple branches of different resolutions in parallel and fusing these branches multiple times in the network to maintain a high-resolution feature map while also introducing richer high-level information.

[0033] For example, the high-resolution network used is based on a convolutional neural network HRNet proposed in Deep High-Resolution Representation Learning for Human Pose Estimation. Figure 2 ,The high-resolution network mainly includes a first branch, a second branch and a third branch in parallel.

[0034] In the first branch, the first feature map is obtained by convolving the network input, the second feature map is obtained by convolving the first feature map, the third feature map is obtained by convolving the second feature map, the fourth feature map is obtained by convolving the third feature map, the fifth feature map is obtained by adding the convolution result of the fourth feature map and the upsampling result of the third feature map in the second branch, the sixth feature map is obtained by convolving the fifth feature map, the seventh feature map is obtained by adding the convolution result of the sixth feature map and the upsampling result of the fifth feature map in the second branch, the eighth feature map is obtained by convolving the seventh feature map, and the ninth feature map (the last feature map) is obtained by adding the convolution result of the eighth feature map, the upsampling result of the seventh feature map in the second branch, and the upsampling result of the third feature map in the third branch.

[0035] In the second branch, the first feature map is obtained by downsampling the second feature map in the first branch, the second feature map is obtained by convolving the first feature map, the third feature map is obtained by adding the convolution result of the second feature map and the downsampling result of the fourth feature map in the first branch, the fourth feature map is obtained by convolving the third feature map, the fifth feature map is obtained by adding the convolution result of the fourth feature map and the downsampling result of the sixth feature map in the first branch, the sixth feature map is obtained by convolving the fifth feature map, and the seventh feature map (the last feature map) is obtained by adding the convolution result of the sixth feature map, the downsampling result of the eighth feature map in the first branch, and the upsampling result of the second feature map in the third branch.

[0036] In the third branch, the first feature map is obtained by adding the downsampling result of the sixth feature map in the first branch and the downsampling result of the fourth feature map in the second branch, the second feature map is obtained by convolving the first feature map, and the third feature map (the last feature map) is obtained by adding the convolution result of the second feature map, the downsampling result of the eighth feature map in the first branch, and the downsampling result of the sixth feature map in the second branch.

[0037] It should be noted that, in each branch, the first feature map is the basis for branch reasoning. At the same time, since HRNet is an existing mature network, the above is only an explanation of the relationship between some feature maps in HRNet, and the actual HRNet can introduce more convolutional layers on this basis. For example, taking the first branch as an example, there may not be only two convolutional layers between the second feature map and the fourth feature map, and there may be more convolutional layers. Similarly, there may be a large number of convolutional layers between the fifth and sixth feature maps. The present invention focuses on the application of two HRNets.

[0038] The two high-resolution networks process images of different spectra respectively. For example, if one of them processes visible light images, the other can process infrared light images. That is, the dual-spectrum images are the input of the first branches of the two high-resolution networks respectively.

[0039] Step S120, performing a first training on the deep neural network model using the dual-spectrum non-fire image. In the first training of the deep neural network model, the first feature maps of the two second branches, the first feature maps of the two third branches, and the last feature maps of the two third branches are aligned through an alignment module.

[0040] Specifically, considering that there are relatively few paired dual-spectrum fire images, in order to improve the feature extraction capability of the model, that is, to allow the model to extract the same feature distribution from paired dual-spectrum images, the present invention first performs a first training (pre-training) on ​​the deep neural network model using dual-spectrum non-fire images. During the pre-training process, by setting an alignment module, the first feature maps of the second branches in the two high-resolution networks are aligned as much as possible, and the first feature maps of the third branches in the two high-resolution networks are aligned as much as possible.

[0041] In some of the embodiments, each alignment module includes two parallel network channels, each network channel includes multiple downsampling convolutional layers, fully connected layers and normalization layers connected in sequence, parameters are shared between the corresponding downsampling convolutional layers of the two network channels, and parameters are shared between the fully connected layers of the two network channels; the inputs of the two network channels are feature maps in the two second branches or the two third branches, respectively, and the output product of the two network channels is the output of the alignment module; the first training adopts a first loss function, and the first loss function is used to calculate the loss between the output of the alignment module and its label.

[0042] Specifically, the output of the alignment module is the correlation matrix of the outputs of the two network channels (usually an N×N correlation two-dimensional matrix), expressed as:

[0043]

[0044] in, f C and f I The output of the two network channels respectively.

[0045] Since the corresponding downsampling convolutional layers of the two network channels share parameters and the fully connected layers of the two network channels share parameters, and thus the mapping relationship between the input and output of each network channel is the same, the greater the correlation between the outputs of the two network channels, the more similar the feature distributions between the inputs of the two network channels. For paired bispectral images, the labels here use a matrix with greater correlation, and for unpaired bispectral images, the labels here use a matrix with less correlation. Therefore, by minimizing the loss between the output of the alignment module and its label to train the model, the model can extract feature maps with more similar distributions from paired bispectral images.

[0046] Exemplarily, the loss function of the first training is:

[0047]

[0048] in, r and c Represent the row and column indices respectively,N Indicates the total number of rows and columns, Cor r and Cor c They represent the output of the alignment module. r Row and c List, and Respectively represent the label r Row and c List, It means that A is the predicted vector and B is the true vector to calculate the cross entropy loss.

[0049] In this embodiment, the total loss during training is the superposition of the row-column cross entropy loss between the output of the alignment module and its label. Of course, in other embodiments, the row cross entropy loss or column cross entropy loss between the output of the alignment module and its label can also be used as the total loss during training. The corresponding loss functions are as follows:

[0050]

[0051] Step S130, performing a second training on the deep neural network model after the first training by alternating single-spectrum and dual-spectrum fire images. In the second training of the deep neural network model, when the input of the neural network model is a dual-spectrum fire image, the first feature maps of the two second branches and the first feature maps of the two third branches are fused with each other through a fusion module, and the first feature map of each second branch is fused with the first feature map of another second branch as the basis for branch reasoning, and the first feature map of each third branch is fused with the first feature map of another third branch as the basis for branch reasoning.

[0052] The pre-trained neural network model already has good feature extraction capabilities, especially when facing paired dual-spectrum images, it extracts feature maps with similar distribution. On this basis, the single and dual-spectrum fire images are used alternately to perform the second training on the neural network model. In the second training, when the input of the neural network model is a dual-spectrum fire image, a fusion module is used to fuse the first feature maps of the two second branches, and the fused feature maps are used as the reasoning basis of the second branch; and another fusion module is used to fuse the first feature maps of the two third branches, and the fused feature maps are used as the reasoning basis of the third branch. Through the fusion module, the reasoning basis of the second branch and the third branch in any high-resolution network can be integrated with the image features (the features of another spectral image) in another high-resolution network. Among them, it is more important that through pre-training, the feature distribution alignment before the fusion of the feature map has been achieved, and then any high-resolution network will not cause changes in its own feature space due to the feature fusion of the features of another high-resolution network, so the two high-resolution networks can still be used independently (the distribution of the feature space is the same when used independently and jointly). When the input of the neural network model is a single-spectrum fire image, the single-spectrum fire image is used as the sample data of the corresponding high-resolution network, and the corresponding high-resolution network is trained separately. During the training process, the high-resolution network will not integrate the features of another high-resolution network.

[0053] In summary, the pre-trained neural network model can use single-spectrum fire images to train the corresponding high-resolution network, or use dual-spectrum images to train the entire deep neural network model, thereby improving the recognition accuracy of the network model for single-spectrum images and dual-spectrum images, respectively.

[0054] In some of the embodiments, the feature processing flow of the fusion module is as follows:

[0055]

[0056] Among them, two high-resolution networks are defined as a first high-resolution network and a second high-resolution network. n represents the batch index of the feature map, C and I Corresponding to different spectra, and They represent the feature maps obtained by merging the width and height of the feature maps from the first high-resolution network and the second high-resolution network, respectively. T represents the matrix transpose, A r and A c They represent the matrices obtained by performing the softmax operation on matrix A in both row and column directions. and They are used as the corresponding feature maps in the first high-resolution network and the second high-resolution network, respectively.

[0057] Specifically, and Corresponding, when is the feature map obtained by merging the width and height of the first high-resolution network feature map, then As the basis for reasoning about the corresponding branch in the first high resolution; and Corresponding, when is the feature map obtained by merging the width and height of the feature map from the second high-resolution network, then Serves as the basis for reasoning about the corresponding branch in the second highest resolution.

[0058] The second trained deep neural network model is a fire detection model.

[0059] It should be noted that the fire detection model adopted by the present invention is a combination and improvement of the high-resolution network. In the traditional high-resolution network, the reasoning basis of the second branch and the third branch is usually the first feature map, and the feature map is obtained by downsampling the feature map in the previous branch. In the second training stage of the model, the present invention fuses the first feature map of the second branch or the third branch in the two high-resolution networks through a fusion module and uses it as the reasoning basis of the corresponding branch, assuming the role of the first feature map. Therefore, from the perspective of the feature map function, the output of the fusion module can be defined as the new first feature map of the corresponding branch. From the perspective of the sequence of feature maps, the output of the fusion module can also be defined as the second feature map of the corresponding branch. At this time, the sequence number of the original second feature map to the last feature map in the branch needs to be postponed by one, but the relationship between each feature map and other feature maps remains unchanged, and only the sequence number is postponed.

[0060] For the first training, the model is mainly trained to optimize feature extraction from different feature distribution angles, and for the second training, the model is mainly trained to optimize feature extraction from the feature recognition angle. Specifically, in the second training, the outputs of the two high-resolution networks are respectively used for image recognition through image recognition heads, and the output of each high-resolution network includes the last feature map of its first branch, second branch, and third branch; the second training uses the second loss function, which is used to calculate the loss between the output of the image recognition head and its label. Among them, a variety of different image recognition tasks can be used to identify whether there is a flame in the image. Different image recognition tasks correspond to different image recognition heads.

[0061] The image recognition head is one of the target detection head, semantic segmentation head, instance segmentation head and image classification head. The target detection head is used to implement the target detection task, which is used to identify whether there are fire-related targets such as flames and smoke in the detection image. It usually outputs a selected box to locate the target and gives the category. The semantic segmentation head is used to implement the semantic segmentation task, classify the pixels in the image, and determine the attribution of different pixels. The attribution can be background, flame, smoke, etc. The instance segmentation head is used to implement the instance segmentation task and identify the entities in the image. Similar to semantic segmentation, instance segmentation is also pixel-level segmentation. The difference between it and semantic segmentation is that semantic segmentation only gives the entity type, but does not distinguish entities of the same type. For example, if there are two flames in the image, semantic segmentation will give whether each pixel belongs to the background or the flame, while instance segmentation will give whether each pixel belongs to the background, flame 1 or flame 2. In addition, instance segmentation can distinguish entities of the same type and give the corresponding number of entities. The image classification head is used to implement the image classification task and classify the image. Images containing flames and / or smoke can be defined as fire images, and images that do not contain flames and smoke can be defined as non-flame images. Therefore, different image recognition tasks can be used to identify whether an image contains flames or smoke, thereby determining whether a fire has occurred in the area of ​​the image.

[0062] It should be noted that the specific loss function used in the second training corresponds to the image recognition head, that is, the loss function matching the current image recognition head can be used. The present invention does not involve specific model optimization of the image recognition head and its loss function.

[0063] In summary, the present invention provides a new training method for a fire detection model with dual-spectrum image fusion. The fire detection model uses two parallel high-resolution networks. During the training process, the deep neural network model is first trained with dual-spectrum non-fire images so that the two high-resolution networks have a feature space distribution that tends to be consistent. Then, the neural network model after the first training is second trained alternately with single and dual-spectrum fire images so that the two high-resolution networks can extract features that are beneficial to image recognition from the image, thereby improving the image recognition accuracy of the model. Among them, it is more important that, through pre-training, the feature distribution alignment before the fusion of feature maps has been achieved, and then any high-resolution network will not cause changes in its own feature space due to the feature fusion of another high-resolution network, so the two high-resolution networks can still be used independently (the distribution of the feature space is basically the same when used independently and in combination). Therefore, the training method for the fire detection model with dual-spectrum image fusion provided by the present invention can be trained and improved using single-spectrum fire images, and there is no problem of difficult model convergence, which solves the problem that the current training method of the dual-spectrum fire detection model cannot effectively use single-spectrum fire images to improve the training effect.

[0064] As follows, the training method of the dual-spectrum image fusion fire detection model provided by the present invention is further described through a specific embodiment.

[0065] Reference Figures 2 to 4 , which is the fire detection model used in this embodiment. The two high-resolution networks are used to process visible light images and infrared images respectively, and can be defined as visible light network and infrared network respectively. The feature map markings in the figure are shown as follows:

[0066] 1. Three levels of feature maps input to the alignment module in pre-training in the visible light network: , , Among them, the subscript C represents visible light, and the superscripts {1, 2, 3} represent the feature maps of three levels (corresponding to alignment module 1, alignment module 2, and alignment module 3). N Represents the batch-size of images input sequentially during pre-training, H and W Represents the height and width of the feature map, D Represents the feature dimension.

[0067] 2. Three levels of feature maps in the infrared network that are input to the alignment module during pre-training: , , Among them, the subscript I stands for infrared, and the other markings are the same as in the visible light network.

[0068] 3. In the alignment module, there is no {1, 2, 3} superscript and represents an input feature map of any level (there can only be one at a time, depending on which level of feature map the current module is used to align), k is the number of downsampling convolutions in the alignment module 1 (used to align the first feature maps of the second branch of the two high-resolution networks) k =2 , alignment modules 2 & 3 (used to align the first feature map of the third branch in the two high-resolution networks and the last feature map of the third branch in the two high-resolution networks, respectively) k =1. f C and f I It is the feature vector obtained by passing the visible light and infrared feature maps through the fully connected network layer.

[0069] 4. , , , It is the visible light and infrared feature map enhanced by the fusion module.

[0070] 5. In the fusion module, since there is no operation on the batch-size dimension (the feature maps of different batch-size dimensions are fused independently), only N The slices on it are represented as follows: , , where the subscript n It is the index value of the batch-size dimension slice.

[0071] 6. is the adjacency matrix, A r , A c Sequentially A It is the attention map after softmax by row and column.

[0072] Specifically, the model structure and training are described as follows:

[0073] 1. Backbone network.

[0074] This embodiment provides a deep neural network structure Dual-HRNet that can fuse dual-spectrum image information for fire detection. Its basis is a convolutional neural network HRNet proposed in "Deep High-Resolution Representation Learning for HumanPose Estimation". Figure 1 It can be seen that the three feature maps used for pre-training , , (Z=C or I) are the first feature map of the second branch of HRNet, the first feature map of the third branch, and the last feature map of the third branch. During the training process (Figure 4), only the first two feature maps are used for dual-spectral feature fusion.

[0075] 2. Pre-training phase.

[0076] 2.1 Pre-training dataset

[0077] Use a visible light-infrared dual-spectrum camera to synchronously trigger dual-spectrum image acquisition in time to obtain dual-spectrum image pairs. Build a massive, unlabeled (here the label refers to the absence of an image identification label, such as whether the image contains a detection target) dual-spectrum dataset.

[0078] 2.2 Alignment module design.

[0079] The dual-spectrum images are input into the visible light network HRNet-RGB and the infrared network HRNet-Infrared respectively to obtain three levels of “feature map pairs”:

[0080]

[0081]

[0082]

[0083] Take any of the “feature graph pairs” Take the example to describe the mechanism of the alignment module.

[0084] like Figure 3 As shown, the two feature maps of the input First experience k Downsampling convolutional layers to reduce the spatial dimension (for , k=2; for the other two feature map pairs, k=1), and then pass through the fully connected layer and the normalization layer to obtain the feature vector and . Referring to the classic multimodal pre-training strategy in CLIP (Contrastive Language-Image Pre-Training), Get an N×N correlation two-dimensional matrix Cor .

[0085] 2.3 Pre-training loss function.

[0086] Due to the input N The images are dual-band paired data, and the network can be pre-trained by using the prior knowledge (as a label) of "Do the dual-spectrum images come from the same image pair?" Therefore, Cor The true value of the prediction , which is the N-dimensional unit matrix (indicating that the bispectral images are paired and have strong correlation). Referring to CLIP, the cross entropy loss function is constructed in both row and column directions:

[0087]

[0088] in, r , c are row and column indices respectively, It means taking A as the prediction vector and B as the true value vector to calculate the cross entropy loss.

[0089] 2.4 Feature map distribution alignment.

[0090] According to the above pre-training strategy, the dual-spectrum features extracted by the network can be fC and f I have the same distribution. However, both in the training phase and in the model application phase, and How to ensure that the two feature maps have the same distribution? As shown in Figure 3, by controlling the downsampling convolution layer and the fully connected layer of the dual-spectrum branch to have shared parameters, the dual-spectrum features have the same The mapping relationship between the two spectra is ensured, thereby ensuring that the network can learn the dual-spectrum feature maps with the same distribution.

[0091] 3. Training phase.

[0092] 3.1 Dual-spectrum fusion module.

[0093] As shown in Figure 4, and Perform dual-spectral feature fusion on the two-level feature maps to obtain the enhanced feature map and Replace the original feature map. According to the network structure of HRNet, the fused feature map distributes the multispectral information to the feature extraction process of other stages through the multi-scale information fusion mechanism of HRNet, realizing the deep fusion of multispectral information at multiple feature levels.

[0094] like Figure 5 As shown in the figure, in the fusion module, since there is no operation on the batch dimension involved, only slices on the batch are used to represent it, such as: , , where the subscript n is the index value of the batch dimension slice. First, the height and width dimensions of the feature map are merged into one dimension and obtained by reshape operation and Then, the neighbor matrix is ​​obtained by matrix multiplication ,right A Perform softmax operations in both row and column directions to obtain A r and A c Finally, the enhanced dual-spectrum feature is obtained:

[0095]

[0096] The enhanced features replace the original features, that is, replace the role of the original first feature map, as the basis for subsequent reasoning, and continue to participate in subsequent network reasoning.

[0097] 3.2 Training tasks

[0098] The core of the present invention lies in the structural design of the dual-spectrum backbone network and the (pre) training method. There is no restriction on the fire detection task connected to the backbone network, which can be a variety of optional tasks such as image classification, target detection, semantic segmentation, instance segmentation, etc. The following will be described according to the fire target detection task. Figure 4 As shown in the figure, after the two HRNet backbone networks, two independent YOLO detection heads are designed to detect fire targets (such as smoke, fire, etc.) in visible light and infrared images respectively. The network structure and loss function of the detection head refer to the design ideas of the classic YOLO (unlimited version). The YOLO detection head here can be replaced with other target detection heads, such as Faster R-CNN, etc.; the target detection head can also be replaced with an image classification head (to determine whether there is smoke, fire, etc. in the image), and the loss function is also changed to a classification loss function accordingly, which will not be repeated here.

[0099] 3.3 Dual-spectrum training and single-spectrum training.

[0100] Bispectral training: Input a bispectral image, perform bispectral information fusion through Dual-HRNet, and extract features. The bispectral features enter two independent task heads, such as the YOLO detection head, to obtain detection results on two spectral images. According to the manually annotated labels, compare the detection results, calculate the training cost, and use the bispectral joint cost to optimize the parameters of the entire network.

[0101] Single spectrum training: For the single spectrum training set, the fusion module is not considered, and the corresponding HRNet and the corresponding task head are trained separately.

[0102] 4. Model application.

[0103] The model that has undergone the pre-training and training stages can be put into testing and business applications. The model of the present invention has multiple application modes: dual spectrum, visible light single spectrum, infrared single spectrum. One model is applicable to multiple camera hardware configurations and has unique advantages.

[0104] In summary, the above embodiments provide a deep neural network structure Dual-HRNet that can fuse dual-spectrum image information for fire detection. Its basis is a convolutional neural network HRNet proposed in "Deep High-Resolution Representation Learning for Human Pose Estimation". This embodiment designs an independent HRNet to extract features for dual-spectrum images, so it is called Dual-HRNet. The Dual-HRNet provided by the present invention is divided into two learning stages: a pre-training stage and a training stage.

[0105] In the pre-training stage, a large number of unlabeled, non-fire visible light-infrared image pairs are used as input. Through the alignment module and alignment loss, the features learned by the two HRNets have the same feature distribution in the feature space to bridge the "gap" between multi-spectral features, that is, to achieve the alignment of multi-spectral features. Moreover, non-fire visible light-infrared image pairs are relatively easy to obtain. Massive data pre-training can significantly improve the generalization ability of the model.

[0106] During the training phase, a fusion module is designed to achieve enhanced replacement of visible light features and infrared features. The consistency in the distribution of dual-spectrum features enables the enhanced features to maintain a distribution similar to the original features. Therefore, three types of data can be input during the training phase: the first is a pair of visible light and infrared images. Dual-HRNet aggregates, fuses and distributes the dual-spectrum information, and finally outputs the fire detection results on each of the two networks to achieve joint training of the model. The second and third types are visible light / infrared single-spectrum fire images, respectively. The detection results are generated separately through the visible light / infrared sub-networks, and the corresponding sub-networks are optimized.

[0107] The advantages of the Dual-HRNet and its pre-training and training methods provided in this embodiment are that they can achieve both deep fusion of dual-spectrum information and relative independence of dual-spectrum feature extraction and fire detection sub-networks; they can fully utilize the advantages of deep complementary information of dual-spectrum fusion and the advantages of historically accumulated single-spectrum fire image data. The details are as follows:

[0108] 1. Multi-level and structured information fusion of dual-spectral information. The provided Dual-HRNet realizes information fusion at two feature levels with spatial structure, and cleverly utilizes the multi-level feature fusion characteristics of HRNet to distribute the fused information to other levels of features.

[0109] 2. Adopt a dual-spectrum pre-training strategy and use massive, unlabeled dual-spectrum image pairs to pre-train the network to improve the generalization performance of the model.

[0110] 3. Through the dual-spectrum feature alignment training in pre-training, the distribution alignment of the dual spectrum in the feature space is realized. Based on the aligned features, through the design of the fusion module, the "unbiased" fusion of the dual-spectrum fusion features and the single-spectrum features is realized. This provides a basis for training a single sub-network using the existing single-spectrum fire data.

[0111] In an embodiment of the present invention, a fire detection method is also provided, which includes:

[0112] Step S210, inputting a target image to be identified into the trained fire detection model, where the target image is a single-spectrum image or a dual-spectrum image;

[0113] Step S220, performing image recognition on the target image through the fire detection model, and determining whether there is a flame in the target image according to the image recognition result. The fire detection model is obtained through the training method of the fire detection model of dual-spectrum image fusion provided by the present invention.

[0114] Since the training method of the dual-spectrum image fusion fire detection model provided by the present invention can use single and dual-spectrum fire images for training, it has the advantage of better training effect. Therefore, the fire detection model obtained by the training method is used to recognize the image, which can more accurately recognize the image and determine whether the image contains physical targets related to the fire.

[0115] In an embodiment of the present invention, a training device for a fire detection model of dual-spectrum image fusion is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0116] The training device of the fire detection model of dual-spectrum image fusion provided in the embodiment of the present invention comprises:

[0117] The model acquisition module is used to obtain a deep neural network model to be trained. The deep neural network model includes two parallel high-resolution networks. The two high-resolution networks are used to process different spectral images respectively. Each high-resolution network includes a first branch, a second branch and a third branch.

[0118] The first training module is used to perform a first training on the deep neural network model using the dual-spectrum non-fire image. During the first training of the deep neural network model, the first feature maps of the two second branches, the first feature maps of the two third branches, and the last feature maps of the two third branches are aligned through the alignment module.

[0119] The second training module is used to perform a second training on the neural network model after the first training by alternating single-spectrum and dual-spectrum fire images. In the second training of the deep neural network model, when the input of the neural network model is a dual-spectrum fire image, the first feature maps of the two second branches and the first feature maps of the two third branches are fused with each other through a fusion module. The first feature map of each second branch is fused with the first feature map of another second branch and used as the basis for branch reasoning. The first feature map of each third branch is fused with the first feature map of another third branch and used as the basis for branch reasoning.

[0120] The second trained deep neural network model is a fire detection model.

[0121] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0122] In an embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the training method or fire detection method of the dual-spectrum image fusion fire detection model provided by the present invention.

[0123] In an embodiment of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the training method of the fire detection model of dual-spectrum image fusion or the steps of the fire detection method provided by the present invention are implemented.

[0124] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0125] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

Claims

1. A training method for a fire detection model based on dual-spectrum image fusion, characterized in that: include: Obtaining a deep neural network model to be trained, the deep neural network model comprising two parallel high-resolution networks, the two high-resolution networks are respectively used to process different spectral images, and each of the high-resolution networks comprises a first branch, a second branch, and a third branch; Performing a first training on the deep neural network model through the dual-spectrum non-fire image, wherein in the first training of the deep neural network model, the first feature maps of the two second branches, the first feature maps of the two third branches, and the last feature maps of the two third branches are all aligned through an alignment module; The deep neural network model after the first training is subjected to a second training by alternating single-spectrum and double-spectrum fire images. In the second training of the deep neural network model, when the input of the neural network model is a double-spectrum fire image, the first feature maps of two second branches and the first feature maps of two third branches are fused with each other through a fusion module. The first feature map of each second branch is fused with the first feature map of another second branch as a branch reasoning basis, and the first feature map of each third branch is fused with the first feature map of another third branch as a branch reasoning basis. When the input of the neural network model is a single-spectrum fire image, the single-spectrum fire image is used as sample data of the corresponding high-resolution network, and the corresponding high-resolution network is trained separately. The deep neural network model after the second training is a fire detection model.

2. The training method of the fire detection model of dual-spectrum image fusion according to claim 1 is characterized in that: Each alignment module includes two parallel network channels, each network channel includes a plurality of downsampling convolution layers, a fully connected layer and a normalization layer connected in sequence, parameters are shared between the corresponding downsampling convolution layers of the two network channels, and parameters are shared between the fully connected layers of the two network channels; Inputs of the two network channels are feature maps in two of the second branches or two of the third branches, respectively, and the output product of the two network channels is the output of the alignment module; The first training adopts a first loss function, and the first loss function is used to calculate the loss between the output of the alignment module and its label.

3. The training method of the fire detection model of dual-spectrum image fusion according to claim 2 is characterized in that: The loss function of the first training is: ， in, r and c Represent the row and column indices respectively, N Indicates the total number of rows and columns, Cor r and Cor c They represent the output of the alignment module. r Row and c List, and Respectively represent the label r Row and c List, It means that A is the predicted vector and B is the true vector to calculate the cross entropy loss.

4. The training method of the fire detection model of dual-spectrum image fusion according to claim 1 is characterized in that: The feature processing flow of the fusion module is as follows: ， Among them, two high-resolution networks are defined as a first high-resolution network and a second high-resolution network. n represents the batch index of the feature map, C and I Corresponding to different spectra, and They represent the feature maps obtained by merging the width and height of the feature maps from the first high-resolution network and the second high-resolution network, respectively. T represents the matrix transpose, A r and A c They represent the matrices obtained by performing the softmax operation on matrix A in both row and column directions. and They serve as the corresponding branch reasoning basis in the first high-resolution network and the second high-resolution network respectively.

5. The training method of the fire detection model of dual-spectrum image fusion according to claim 4 is characterized in that: In the second training, the outputs of the two high-resolution networks are respectively subjected to image recognition by image recognition heads, and the output of each high-resolution network includes the last feature map of its first branch, second branch and third branch; The second training adopts a second loss function, which is used to calculate the loss between the output of the image recognition head and its label.

6. The training method of the fire detection model of dual-spectrum image fusion according to claim 5 is characterized in that: The image recognition head is one of a target detection head, a semantic segmentation head, an instance segmentation head and an image classification head.

7. A fire detection method, characterized in that: include: Inputting a target image to be identified into a trained fire detection model, wherein the target image is a single-spectrum image or a dual-spectrum image; The target image is subjected to image recognition by the fire detection model, and whether there is a flame in the target image is determined according to the image recognition result. The fire detection model is obtained by the training method of the dual-spectrum image fusion fire detection model described in any one of claims 1-6.

8. A training device for a fire detection model based on dual-spectrum image fusion, characterized in that: include: A model acquisition module, used to acquire a deep neural network model to be trained, wherein the deep neural network model includes two parallel high-resolution networks, the two high-resolution networks are used to process different spectral images respectively, and each of the high-resolution networks includes a first branch, a second branch and a third branch; A first training module is used to perform a first training on the deep neural network model through a dual-spectrum non-fire image, wherein in the first training of the deep neural network model, the first feature maps of the two second branches, the first feature maps of the two third branches, and the last feature maps of the two third branches are aligned through an alignment module; A second training module is used to perform a second training on the deep neural network model after the first training by alternating single-spectrum and double-spectrum fire images. In the second training of the deep neural network model, when the input of the neural network model is a double-spectrum fire image, the first feature maps of two second branches and the first feature maps of two third branches are fused with each other through a fusion module. The first feature map of each second branch is fused with the first feature map of another second branch as a branch reasoning basis, and the first feature map of each third branch is fused with the first feature map of another third branch as a branch reasoning basis. When the input of the neural network model is a single-spectrum fire image, the single-spectrum fire image is used as sample data of the corresponding high-resolution network, and the corresponding high-resolution network is trained separately; The second trained deep neural network model is a fire detection model.

9. An electronic device, comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the training method of the fire detection model of dual-spectrum image fusion according to any one of claims 1 to 6 or the fire detection method according to claim 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the training method of the fire detection model of dual-spectrum image fusion described in any one of claims 1 to 6 or the steps of the fire detection method described in claim 7 are implemented.

Images