Multi-modal flame AI identification method based on FFDNET

Abstract

The invention discloses a multi-modal flame AI recognition method based on FFDNET, and the method comprises a denoising model training method and a flame recognition method, the denoising model training comprises the steps: inputting a preprocessed image and a predicted noise level map into FFDNet, carrying out the sub-channel down-sampling and dimension reduction of the input image on RGB channels, and obtaining sub-images; respectively carrying out denoising processing on the sub-images on the three channels by utilizing FFDNet, correcting the denoising process according to quality evaluation, and carrying out training according to the denoising process to obtain a noise level map; the flame identification method comprises the following steps: obtaining a blue flame part by using RGB channel enhancement and HSV features, extracting a yellow flame part of a target image according to the HSV features, and picking up a flame region by an original image flame contour line drawn by overlapping a contour; the denoising processing of the recognition method reduces the calculation burden of the network in the spatial dimension, the denoising effect is more ideal, and the flame recognition method can obtain an accurate and complete flame image; the method can be widely applied to the field of image processing, and is suitable for the fields of real-time flame monitoring, fire prediction and the like of kilns and boilers.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a multimodal flame AI recognition method based on FFDNET. Background Art

[0002] In the field of combustion, constant volume bombs are used to study combustion characteristics, and the flame images taken have different levels and types of noise. There are many factors affecting the noise in the flame image inside the constant volume bomb. First, affected by the experimental boundary inside the constant volume bomb, the temperature, pressure and the amount of oil beam all affect the level of noise; second, affected by the external environment of the constant volume bomb, the brightness of the external light source and the type of light source all affect the level and type of noise; third, the flame image of the constant volume bomb is a color image and the noise is complex. The noise is composed of a combination of multiple types of noise, not a single type of noise. In the subsequent analysis, noise has a major problem in data extraction. The noise level in different batches of flame images is also different, and the noise level needs to be adjusted each time;

[0003] A Chinese invention patent application with publication number CN117994711A discloses a method for identifying flames in engine plume images, in which the patent identifies the RGB features and HSV features corresponding to each pixel in each frame of plume image, and obtains the color feature vector of each frame image by the difference between the HSV features of each frame image and the first frame image. The color feature vector is input into the flame recognition model to obtain the flame recognition result corresponding to each frame; a Chinese invention patent application with publication number CN116309122A discloses a method for suppressing speckle noise in phase fringe images based on deep learning. The patent combines spatial pyramid pooling with a non-local attention mechanism to obtain more feature information of speckle noise images, integrating asymmetric fusion of non-local blocks to obtain global information. The global information is then input into the FFDNet network to train speckle noise images with different noise levels. An edge extraction module is set at the output to obtain an edge map, which is back-propagated to the FFDNet network to guide the network's noise suppression process. The above two patents are too simplistic in their recognition and training of noise levels and types, and have poor denoising effects on other types of noise or overly complex noise.

[0004] The flame image in a constant-volume bomb has a significantly different noise complexity due to the different external environments and internal combustion boundaries. The temperature inside the bomb and the brightness of the external light source both determine the noise level. The flame image in the bomb is not composed of a single noise, but a mixture of multiple noises. Therefore, it is necessary to develop an adaptive AI method for multimodal flame recognition in constant-volume bombs based on FFDNet in deep learning. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects in the prior art and provide a multimodal flame AI recognition method based on FFDNET to overcome one or more problems caused by the limitations and defects of the relevant technology to a certain extent.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] A multimodal flame AI recognition method based on FFDNET, including denoising model training and denoising image flame recognition method, wherein:

[0008] Training the denoising model consists of the following steps:

[0009] S1: The collected original flame images are preprocessed, and the preprocessed image dataset is divided into a training set and a test set. The clean images obtained after denoising the original images of the training set are used as training targets, and the clean images obtained after denoising the original images of the test set are used as evaluation images;

[0010] S2: perform noise estimation on the preprocessed image to obtain a predicted noise level map;

[0011] S3: The preprocessed image and its predicted noise level map in step S2 are input to FFDNet. FFDNet performs channel-wise spatial downsampling on the input image on the RGB channels, shifting the spatial resolution of the input image to the channel dimension. After dimensionality reduction, several sub-images are obtained. The convolutional layers, ReLU functions, and multi-level residual blocks of FFDNet are used to denoise the sub-images on the three channels.

[0012] S4: Recombining the sub-images on the three RGB channels to obtain a denoised training image, performing a quality assessment on the denoised training image based on the training objective, modifying the convolution kernel, and repeating step S3 until the quality assessment is satisfied;

[0013] S5: Denoise all preprocessed images of the training set according to steps S2, S3, and S4, train the denoising model, and obtain a noise level map;

[0014] S6: Input the preprocessed image of the test set into the trained FFDNet model, and input the noise level map obtained during training. Complete the denoising process according to the FFDNet-based denoising method in step S3, reconstruct the denoised test image, use the evaluation image to evaluate the quality of the denoised test image, verify and save the optimal noise level map, and complete the training of the denoising model.

[0015] After preprocessing, the original flame image is input into the trained denoising model, and the optimal noise level map is input. The denoising process is performed according to step S3 to obtain a denoised image. The flame recognition method of the denoised image includes the following steps:

[0016] P1: The flame features of the denoised image are divided into the blue flame part and the high temperature flame part for identification and extraction respectively;

[0017] For the blue flame part, first, the denoised image is enhanced on the RGB channels to extract the blue sub-image on the B channel. Then, the first HSV feature parameter range is set to extract the blue flame area of ​​the blue sub-image. Finally, the blue flame outline of the blue flame area is extracted.

[0018] For the high-temperature flame part, first, the denoised image is processed to obtain an easily recognizable target image. Then, the second HSV feature parameter range is set to extract the yellow flame area of ​​the target image. Finally, the yellow flame outline of the yellow flame area is extracted.

[0019] P2: Superimpose the blue flame outline and the yellow flame outline to draw the original flame outline. Determine the continuity between the two and adjust the first HSV feature parameter range and the second HSV feature parameter range respectively until the two parts of the outline overlap without gaps. The original flame outline drawing is completed. The flame is extracted and recognized based on the original flame outline, and the flame recognition information is output.

[0020] Furthermore, in step S3, the local features extracted by the convolution operations and activation functions on the R, G, and B channels are not only transferred to the next layer of operations on their respective channels, but also synchronously share the learning information of the same layer with the remaining channels.

[0021] Furthermore, the noise estimation in step S2 also includes marking the key denoising area based on experience, and increasing the convolution kernel weight in the key denoising area in step S3.

[0022] Furthermore, the method further includes preliminary denoising before the deep denoising in step S3, and uses an LMS filter to adjust the denoising intensity of different areas on the RGB three channels to achieve preliminary denoising.

[0023] Furthermore, evaluation parameters for image quality assessment include peak signal-to-noise ratio and structural similarity index.

[0024] Furthermore, the image processing of the high-temperature flame portion in step S6 includes background difference processing and inter-frame difference processing, and the recognition processed images obtained by the two processes are superimposed proportionally to obtain a target image whose HSV features are easily recognized.

[0025] Compared with the prior art, the multimodal flame AI recognition method based on FFDNET of the present invention has the following beneficial effects:

[0026] This recognition method uses the flame image in the constant volume bomb as the carrier, and performs RGB channel-by-channel spatial downsampling based on deep learning FFDNet to process image noise, reducing the network's computational burden in the spatial dimension. At the same time, the denoising effect of the channel-by-channel noise processing is more ideal, and the image details and information can be well preserved on the basis of accurate noise removal. In addition, the blue flame and yellow flame are extracted and recognized respectively using RGB feature extraction and HSV feature automatic extraction methods. The target image obtained by proportionally superimposing the images obtained using the inter-frame difference method and the background difference method has a clear outline and color gradient, thereby ensuring the accurate extraction and recognition of the yellow flame. Finally, the flame outline of the original image is obtained by superposition, and the flame area is picked up based on the flame outline, which is conducive to obtaining an accurate and complete flame image.

[0027] This recognition method is not only suitable for the recognition of engine combustion flames, but the denoising method and recognition means it provides can also be widely used in the field of image processing. In addition, this technology can be applied to the fields of real-time flame recognition and monitoring of furnaces and boilers, as well as fire recognition and prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a training flow chart of the FFDNet-based denoising model disclosed in the present invention;

[0029] Figure 2 The denoising flow chart of the original noisy flame image disclosed in the present invention;

[0030] Figure 3 This is a flow chart of flame recognition of denoised images disclosed in the present invention. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only the best embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0032] This embodiment provides a multimodal flame AI recognition method based on FFDNet, which aims to train a denoising model that can adaptively evaluate the noise level through FFDNet, remove the noise of the flame image collected in the constant volume bomb, output a clear flame image, and extract and recognize multimodal flames from the flame image, such as Figure 1-Figure 3 As shown, the recognition method includes the training of the denoising model and the flame recognition method of the denoised image, wherein,

[0033] Training the denoising model based on FFDNET includes the following steps:

[0034] S1: In order to improve the adaptability of the denoising model, the collected flame original images should be preset and combined with multiple possible noise types. The flame original images are preprocessed in FFDNET. The preprocessing includes image pixel size unification and flame base point alignment. The preprocessed image dataset is divided into training set and test set in an 8:2 ratio. The clean images obtained after denoising the original images of the training set are used as training targets, and the clean images obtained after denoising the original images of the test set are used as evaluation images.

[0035] S2: FFDNET performs noise estimation on the preprocessed image. Based on the clean image, FFDNET estimates the noise intensity and noise type in the flame image, records the parameter selection of the denoising process, and forms a predicted noise level map. The noise estimation can also empirically estimate and mark the key denoising areas to guide the weight selection of deep denoising;

[0036] Next, a simple preliminary denoising is performed using an LMS filter, and the noise information is recorded in a noise level map. The LMS filter adjusts its own weight by minimizing the sum of squared errors between the output signal and the expected signal. It is an efficient and simple adaptive filter. Therefore, the LMS filter can be used to automatically adjust the denoising intensity of different areas on the RGB three channels to perform simple preliminary denoising.

[0037] S3: Denoising is the core step of FFDNet, which mainly uses the neural network in deep learning to denoise the input noisy image. The preliminary denoised image of the training set is input into FFDNet, and the noise level map of the predicted image is input. FFDNet performs channel-by-channel spatial downsampling on the three RGB channels, such as Figure 2 As shown in Figure 1, the spatial resolution of the input image is moved to the channel dimension, and several sub-images are obtained after dimensionality reduction. The sub-images on each channel are denoised separately.

[0038] FFDNet constructs multiple convolutional layers, and the convolution operation is performed simultaneously on R, G, and B. Each convolutional layer consists of the input layer and convolution kernel of each channel. The size of the input layer is generally H×W×C (H is the height of the image, W is the width of the image, and C is the number of channels of the image). After the channel processing operation, downsampling is performed at the spatial level, so the input layer becomes H / 2×W / 2×4C. Each convolutional layer is composed of multiple convolution kernels, and the size of each convolution kernel is generally k×k (generally k is an odd number). The convolution kernel performs convolution operations on the image. Generally, each convolution kernel only learns a specific image feature, so the convolution layer is constructed. The convolution layer of the previous layer uses the activation function (ReLU) to introduce nonlinear characteristics after the convolution operation, captures and extracts some complex local features under the channel, and converts these local feature information into features extracted by the information of the next layer of the channel. These local information is also synchronously shared with the same layer operations of other channels. The ReLU function can provide a linear correction for each input value. When the input is positive, it is directly passed to the next layer. If it is negative, the output is 0.

[0039] Mathematical model of convolution operation:

[0040] O(i, j) = (I*K)(i, j) = ∑ m，n I(i+m,j+n)·K(m,n)+b

[0041] Where: I is the input image, K is the convolution kernel;

[0042] I(i, j) is the pixel value at position i, j in the input image;

[0043] K(m,n) is the weight of the convolution kernel at position m,n, and the weight of the marked key denoising area is appropriately increased;

[0044] b is the bias term;

[0045] O(i, j) is the pixel point of the output feature map.

[0046] The residual block ensures that information in each convolutional layer is not missed and gradients do not vanish during image denoising. The multi-level residual block can perform different feature extraction scales and different convolution operations on each convolutional layer and combine these results through jump connections. The residual block combines the input of each convolutional layer with the output after the operation of the layer, avoiding the problem of gradient vanishing during the convolution calculation process of each channel.

[0047] Residual block formula expression:

[0048] y=F(x,{W i})+x

[0049] Where: x is the input;

[0050] F(x,{W i}) is the result obtained by processing the input through convolution operations, etc.

[0051] y is the output, which is obtained by directly adding the processed result to the input.

[0052] The final convolutional layer is responsible for summarizing the details of the image gradually restored by multiple convolutional layers and residual blocks, and is mainly responsible for generating the final denoising result. The residual block connection adds the input image and the "residual" after multiple layers of convolution and feature extraction of each channel, and finally passes the input information to the output layer through the skip connection.

[0053] S4: Recombine the sub-images on the RGB channels and output the denoised training image. Perform a quality assessment on the denoised training image based on the training objective, modify the convolution kernel, and repeat step S3 until the assessment requirements are met. Record all types of noise information during the denoising training process.

[0054] Image quality assessment is a quality assessment of the denoised image. It is mainly evaluated through two parameters: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). PSNR is used to measure the quality of the denoised image, and SSIM is used to evaluate the structural similarity between the denoised image and the original image. These two evaluation parameters can reduce noise interference and ensure that the output denoised image retains the structure and details of the original image.

[0055] Peak Signal-to-Noise Ratio (PSNR):

[0056] PSNR=10*log 10 MAX 2 / MSE

[0057] Where: MAX is the maximum value of the image pixel;

[0058] MSE is the average of the mean difference between the pixel values ​​of the original image and the denoised image.

[0059] Structural Similarity Index (SSIM):

[0060]

[0061] in: The average brightness of image x and y;

[0062] The brightness variance of image x and y;

[0063] σ xy : covariance between images x and y;

[0064] C1, C1: Constant used to avoid the denominator being zero.

[0065] S5: Refer again Figure 1 , denoising all input images of the training set according to step S2, step S3 and step S4, training the denoising model and obtaining a noise level map;

[0066] S6: After preliminary denoising of the original image preprocessed in the test set, the image is input into the trained FFDNet model, the noise level map obtained during training is input, and denoising is completed according to the FFDNet-based channel denoising method in step S3, and the denoised test image is reconstructed. The quality of the denoised test image is evaluated based on the evaluation image using the above-mentioned peak signal-to-noise ratio and structural similarity index, the optimal noise level map is verified and saved, and the training of the denoising model is completed.

[0067] The recognition of multi-mode flames mainly involves the recognition of blue flames and high-temperature yellow flames, such as Figure 2 As shown, the original noisy flame image is preprocessed and preliminarily denoised, then input into the trained denoising model, and the optimal noise level map is input, and channel denoising is performed according to step S3. After combination, a denoised image is obtained. The flame recognition method of the denoised image includes the following steps:

[0068] P1: The flame features of the denoised image are divided into the blue flame part and the high temperature flame part for identification and extraction respectively;

[0069] For the blue flame part, first, the denoised image is enhanced with RGB channels, and the blue sub-image on the B channel is extracted. Then, the first HSV feature parameter range is set to extract the blue flame area of ​​the blue sub-image, and then the blue flame outline of the blue flame area is obtained.

[0070] For the high-temperature yellow flame part, due to the insufficient clarity of the high-temperature flame outline on the denoised image, the denoised image needs to be processed to obtain an easily recognizable target image. The image processing includes background difference processing and inter-frame difference processing. The background difference method is conducive to accurately finding the target, but its flame color change is not sensitive. The inter-frame difference method highlights dynamic changes, but the boundary blur is not obvious. Therefore, the denoised image can obtain a clear flame boundary through the background difference method, and a clear color gradient can be obtained by the inter-frame difference method. Then, the recognition processing images obtained by the two processings are superimposed in a certain proportion to obtain a target image with a clear flame color gradient and a clear boundary, which is conducive to HSV feature recognition and extraction. At the same time, due to the complex flame morphology, the superposition of the two can further verify the morphology and details of the flame; then, the second HSV feature parameter range is set to extract the yellow flame area of ​​the target image, and then the yellow flame outline of the yellow flame area is extracted;

[0071] P2: Superimpose the blue flame outline and the yellow flame outline, and draw the original flame outline; the blue flame outline and the yellow flame outline should be closely connected to each other. After superimposing the two, their continuity is judged. If there is a gap, the first HSV feature parameter range and the second HSV feature parameter range are adjusted respectively until the two parts of the outline completely overlap without gaps. The original flame outline is drawn, and the flame area is extracted and identified based on the original flame outline, and the relevant information data of the extracted and identified flame area is output.

[0072] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal flame AI recognition method based on FFDNET, characterized in that: The invention includes the training of denoising model and flame recognition method of denoising image, wherein, Training the denoising model consists of the following steps: S1: The collected original flame images are preprocessed, and the preprocessed image dataset is divided into a training set and a test set. The clean images obtained after denoising the original images of the training set are used as training targets, and the clean images obtained after denoising the original images of the test set are used as evaluation images; S2: perform noise estimation on the preprocessed image to obtain a predicted noise level map; S3: The preprocessed image and its predicted noise level map in step S2 are input to FFDNet. FFDNet performs channel-wise spatial downsampling on the input image on the RGB channels, shifting the spatial resolution of the input image to the channel dimension. After dimensionality reduction, several sub-images are obtained. The convolutional layers, ReLU functions, and multi-level residual blocks of FFDNet are used to denoise the sub-images on the three channels. S4: Recombining the sub-images on the three RGB channels to obtain a denoised training image, performing a quality assessment on the denoised training image based on the training objective, modifying the convolution kernel, and repeating step S3 until the quality assessment is satisfied; S5: Denoise all pre-processed images of the training set according to steps S2, S3, and S4, train the denoising model, and obtain a noise level map; S6: Input the preprocessed image of the test set into the trained FFDNet model, and input the noise level map obtained during training. Complete the denoising process according to the FFDNet-based denoising method in step S3, reconstruct the denoised test image, use the evaluation image to evaluate the quality of the denoised test image, verify and save the optimal noise level map, and complete the training of the denoising model. After preprocessing, the original flame image is input into the trained denoising model, and the optimal noise level map is input. The denoising process is performed according to step S3 to obtain a denoised image. The flame recognition method of the denoised image includes the following steps: P1: The flame features of the denoised image are divided into the blue flame part and the high temperature flame part for identification and extraction respectively; For the blue flame part, first, the denoised image is enhanced on the RGB channels to extract the blue sub-image on the B channel. Then, the first HSV feature parameter range is set to extract the blue flame area of ​​the blue sub-image. Finally, the blue flame outline of the blue flame area is extracted. For the high-temperature flame part, first, the denoised image is processed to obtain an easily recognizable target image. Then, the second HSV feature parameter range is set to extract the yellow flame area of ​​the target image. Finally, the yellow flame outline of the yellow flame area is extracted. P2: Superimpose the blue flame outline and the yellow flame outline to draw the original flame outline. Determine the continuity between the two and adjust the first HSV feature parameter range and the second HSV feature parameter range respectively until the two parts of the outline overlap without gaps. The original flame outline drawing is completed. The flame is extracted and recognized based on the original flame outline, and the flame recognition information is output.

2. The multimodal flame AI recognition method based on FFDNET according to claim 1, wherein: In step S3, the local features extracted by the convolution operations and activation functions on the R, G, and B channels are not only transferred to the next layer of operations on their respective channels, but also synchronously share the learning information of the same layer with the remaining channels.

3. The multimodal flame AI recognition method based on FFDNET according to claim 1, wherein: The noise estimation in step S2 also includes marking the key denoising area based on experience and increasing the convolution kernel weight in the key denoising area in step S3.

4. The multimodal flame AI recognition method based on FFDNET according to claim 3, characterized in that: It also includes preliminary denoising before the deep denoising in step S3, using an LMS filter to adjust the denoising intensity of different areas on the RGB three channels to achieve preliminary denoising.

5. The multimodal flame AI recognition method based on FFDNET according to claim 1, characterized in that: The evaluation parameters for image quality assessment include peak signal-to-noise ratio and structural similarity index.

6. The multimodal flame AI recognition method based on FFDNET according to any one of claims 1 to 5, characterized in that: The image processing of the high-temperature flame portion in step P1 includes background difference processing and inter-frame difference processing. The recognition processed images obtained by the two processes are superimposed proportionally to obtain a target image with HSV features that is easy to identify.

Images