Multi-modal flame ai recognition method based on ffdnet

Abstract

This invention discloses a multimodal flame AI recognition method based on FFDNet, including a denoising model training and a flame recognition method. The denoising model training involves inputting a preprocessed image and a predicted noise level map into FFDNet, performing channel-wise downsampling on the RGB three channels to obtain sub-images, and then using FFDNet to denoise the sub-images on each of the three channels. The denoising process is corrected based on quality assessment, and the noise level map is obtained through this training. The flame recognition method includes using RGB channel enhancement and HSV features to obtain the blue flame portion, extracting the yellow flame portion of the target image from the HSV features, and picking the flame area using the flame outline drawn by superimposing the contours of the original image. This recognition method not only reduces the computational burden on the network in the spatial dimension but also achieves a more ideal denoising effect. Furthermore, the flame recognition method can obtain accurate and complete flame images. It can be widely applied in the field of image processing and is suitable for real-time flame monitoring and fire prediction in furnaces and boilers.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to a multimodal flame AI recognition method based on FFDNET. Background Technology

[0002] In the field of combustion, constant-volume bombs are used to study combustion characteristics, and the captured flame images exhibit varying levels and types of noise. The noise in flame images from constant-volume bombs is influenced by several factors. First, the experimental boundaries within the constant-volume bomb affect the noise level due to factors such as temperature, pressure, and the amount of fuel jet. Second, the external environment of the constant-volume bomb, including the brightness and type of external light sources, influences both the noise level and type. Third, constant-volume bomb flame images are color images with complex noise, composed of combinations of various noise types, not a single noise category. In subsequent analysis, noise poses a significant challenge to data extraction, and the noise levels vary between different batches of flame images, requiring adjustment of the noise level each time.

[0003] Chinese invention patent application CN117994711A discloses a method for identifying flames from engine plume images. This patent identifies the RGB and HSV features of each pixel in each frame of the plume image. It then obtains the color feature vector for each frame by calculating the difference between the HSV features of each frame and the first frame. This color feature vector is input into a flame recognition model to obtain the flame recognition result for each frame. Another Chinese invention patent application CN116309122A discloses a method for suppressing speckle noise in phase fringe images based on deep learning. The method described in this patent, in order to obtain more feature information from speckle noise images, integrates spatial pyramid pooling and nonlocal attention mechanisms into an asymmetric fusion nonlocal block to obtain global information. This global information is then input into an FFDNet network to train speckle noise images with different noise levels. An edge extraction module is set at the output end to obtain an edge map, which is then backpropagated to the FFDNet network to guide the network's noise suppression process. However, both of these patents are too simplistic in their identification and training of noise levels and types, resulting in poor denoising effects for other types of noise and overly complex noise.

[0004] The noise complexity of flame images in constant-volume projectiles varies greatly due to differences in the external environment and internal combustion boundaries. The temperature inside the constant-volume projectile and the brightness of the external light source both determine the noise level. The flame image inside the constant-volume projectile is not composed of a single type of noise, but rather a mixture of multiple types of noise. Therefore, it is necessary to develop an adaptive AI method for multimodal flame recognition inside constant-volume projectiles based on FFDNet in deep learning. Summary of the Invention

[0005] The purpose of this invention is to overcome the deficiencies in the prior art and provide a multimodal flame AI recognition method based on FFDNET, so as to overcome one or more problems caused by the limitations and defects of related technologies to a certain extent.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A multimodal flame AI recognition method based on FFDNET includes training a denoising model and a flame recognition method for denoised images, wherein...

[0008] Training a denoising model involves the following steps:

[0009] S1: The collected raw flame images are preprocessed. The preprocessed image dataset is divided into a training set and a test set. The clean images obtained after denoising the original images in the training set are used as training targets, and the clean images obtained after denoising the original images in the test set are used as evaluation images.

[0010] S2: Perform noise estimation on the preprocessed image to obtain a predicted noise level map;

[0011] S3: Input the preprocessed image and its predicted noise level map from step S2 into FFDNet. FFDNet performs channel-specific spatial downsampling on the input image in the RGB three channels, shifting the spatial resolution of the input image to the channel dimension. After dimensionality reduction, several sub-images are obtained. The convolutional layers, ReLU function, and multi-level residual blocks of FFDNet are used to denoise the sub-images in the three channels respectively.

[0012] S4: Recombine the sub-images on the RGB three channels to obtain the denoised training image. Based on the training objective, evaluate the quality of the denoised training image, correct the convolution kernel, and repeat step S3 until the quality evaluation is satisfied.

[0013] S5: Denoise all preprocessed images in the training set according to steps S2, S3 and S4, train the denoising model and obtain the noise level map.

[0014] S6: Input the preprocessed image of the test set into the trained FFDNet model, and input the noise level map obtained during training. Perform denoising processing according to the FFDNet-based denoising method in step S3, reconstruct to obtain the denoised test image, use the evaluation image to evaluate the quality of the denoised test image, verify and save the best noise level map, and complete the training of the denoising model.

[0015] The original flame image is preprocessed and then input into the trained denoising model. The optimal noise level map is input, and denoising is performed according to step S3 to obtain a denoised image. The flame recognition method for the denoised image includes the following steps:

[0016] P1: The flame features of the denoised image are divided into blue flame and high-temperature flame, which are then identified and extracted separately; among them,

[0017] For the blue flame part, firstly, the RGB channels of the denoised image are enhanced, and the blue sub-image on the B channel is extracted. Then, the blue flame region is extracted from the blue sub-image by setting the first HSV feature parameter range. Finally, the blue flame outline of the blue flame region is extracted.

[0018] For the high-temperature flame part, firstly, the denoised image is processed to obtain an easily identifiable target image. Then, the yellow flame region of the target image is extracted by setting the range of the second HSV feature parameters. Finally, the yellow flame outline of the yellow flame region is extracted.

[0019] P2: Overlay the blue flame outline and the yellow flame outline to draw the original flame outline. Determine the continuity between the two and adjust the range of the first HSV feature parameter and the range of the second HSV feature parameter respectively until the two parts of the outline overlap without gaps. Then the drawing of the original flame outline is completed. Based on the original flame outline, the flame is extracted and identified, and the flame identification information is output.

[0020] Furthermore, in step S3, the convolution operations and local features extracted by the activation functions on the R, G, and B channels are not only transformed into the next layer operations on their respective channels, but also synchronously share the learning information of the same layer with the other channels.

[0021] Furthermore, the noise estimation in step S2 also includes marking key denoising regions based on experience and increasing the weight of the convolution kernel in the key denoising regions in step S3.

[0022] Furthermore, it also includes preliminary denoising before the deep denoising in step S3, which uses an LMS filter to adjust the denoising intensity of different regions on the RGB three channels to achieve preliminary denoising.

[0023] Furthermore, the evaluation parameters for image quality assessment include peak signal-to-noise ratio and structural similarity index.

[0024] Furthermore, the image processing of the high-temperature flame part in step S6 includes background difference processing and inter-frame difference processing. The recognition images obtained by the two processing methods are superimposed proportionally to obtain a target image with HSV features that are easy to identify.

[0025] Compared with existing technologies, the multimodal flame AI recognition method based on FFDNET of the present invention has the following advantages:

[0026] This recognition method uses the flame image inside a fixed-volume projectile as a carrier. Based on deep learning's FFDNet, it performs RGB channel-wise spatial downsampling to process image noise, reducing the computational burden on the network in the spatial dimension. At the same time, the denoising effect of channel-wise noise processing is more ideal, and it can preserve image details and information well while accurately removing noise. In addition, RGB feature extraction and HSV feature automatic extraction methods are used to extract and recognize blue flames and yellow flames respectively. The target image obtained by superimposing the images obtained by inter-frame difference and background difference methods according to the ratio has a clear outline and color gradient, thus ensuring the accurate extraction and recognition of yellow flames. Finally, the original flame outline is obtained by superimposing the images, and the flame region is picked out by the flame outline, which is conducive to obtaining an accurate and complete flame image.

[0027] This identification method is not only applicable to the identification of engine combustion flames, but its denoising method and identification means can also be widely used in the field of image processing. In addition, this technology can be applied to the fields of real-time flame identification and monitoring of furnaces and boilers, as well as fire identification and prediction. Attached Figure Description

[0028] Figure 1 This is a flowchart of the training process for the FFDNet-based denoising model disclosed in this invention.

[0029] Figure 2 This is a flowchart of the denoising process for the original noisy flame image disclosed in this invention;

[0030] Figure 3 This is a flowchart of the flame recognition process for denoised images disclosed in this invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely the best embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] This embodiment provides a multimodal flame AI recognition method based on FFDNet. The aim is to train a denoising model using FFDNet that can adaptively evaluate noise levels, remove noise from flame images acquired within a fixed-volume projectile, output clear flame images, and extract and recognize multimodal flames from these images. Figures 1-3 As shown, the recognition method includes training a denoising model and a flame recognition method for denoised images, wherein,

[0033] Training a denoising model based on FFDNET includes the following steps:

[0034] S1: In order to improve the adaptability of the denoising model, the original flame images should be pre-set and include multiple possible noise types. The original flame images are pre-processed in FFDNET. The pre-processing includes image pixel size unification and flame base point alignment. The pre-processed image dataset is divided into training set and test set in an 8:2 ratio. The clean images obtained after denoising the original images in the training set are used as training targets, and the clean images obtained after denoising the original images in the test set are used as evaluation images.

[0035] S2: FFDNET performs noise estimation on the preprocessed image. Based on the clean image, FFDNET estimates the noise intensity and type in the flame image, records the parameter selection in the denoising process, and forms a predicted noise level map. In the noise estimation, it can also empirically predict and mark key denoising areas to guide the selection of weights for deep denoising.

[0036] Next, a simple preliminary denoising is performed using an LMS filter, and the noise information is recorded in the noise level map. The LMS filter adjusts its own weight by minimizing the sum of squared errors between the output signal and the desired signal. It is an efficient and simple adaptive filter. Therefore, the LMS filter can be used to perform denoising of different intensities for different noise types and intensities. The denoising intensity of different regions on the RGB three channels can be automatically adjusted to perform simple preliminary denoising.

[0037] S3: Noise removal is the core step of FFDNet. It mainly uses a neural network in deep learning to denoise the input noisy image. The initially denoised images from the training set are input into FFDNet along with the noise level map of the predicted image. FFDNet performs channel-wise spatial downsampling on the input image across the RGB channels, such as... Figure 2 As shown, the spatial resolution of the input image is shifted to the channel dimension, and several sub-images are obtained after dimensionality reduction. Denoising processing is then performed on the sub-images in each channel.

[0038] FFDNet constructs multiple convolutional layers, with convolution operations performed simultaneously on the R, G, and B axes. Each convolutional layer consists of an input layer for each channel and a convolutional kernel. The input layer size is typically H×W×C (H is the image height, W is the image width, and C is the number of image channels). After channel-wise processing and downsampling at the spatial level, the input layer becomes H / 2×W / 2×4C. Each convolutional layer consists of multiple convolutional kernels, each typically k×k (k is usually an odd number). The convolutional kernels perform convolution operations on the image. Generally, each convolutional kernel learns only one specific image feature, thus completing the construction of the convolutional layer. The previous convolutional layer, after convolution operation, uses the ReLU activation function to introduce non-linear characteristics, capturing and extracting some complex local features in that channel, and transforming these local feature information into features extracted by the next layer of information in that channel. This local information is also shared synchronously with the same layer operations of other channels. The ReLU function can provide a linear correction for each input value. When the input is positive, it is directly passed to the next layer; if it is negative, the output is 0.

[0039] Mathematical model of convolution operation:

[0040] O(i,j)=(I*K)(i,j)=∑ m，n I(i+m,j+n)·K(m,n)+b

[0041] Where: I is the input image, and K is the convolution kernel;

[0042] I(i,j) is the pixel value at position i,j in the input image;

[0043] K(m, n) is the weight of the convolution kernel at positions m and n. The weights are appropriately increased for the marked key denoising regions.

[0044] b is the bias term;

[0045] O(i,j) represents the pixels in the output feature map.

[0046] Residual blocks ensure that information from each convolutional layer is not lost and gradient vanishing is avoided during image denoising. Multi-level residual blocks can perform different feature extraction scales and different convolution operations on each convolutional layer and combine these results through skip connections. Residual blocks combine the input of each convolutional layer with the output of that layer, avoiding the gradient vanishing problem during the convolution calculation of each channel.

[0047] Residual block formula expression:

[0048] y = F(x, {W i})+x

[0049] Where: x is the input;

[0050] F(x,{W i}) is the result obtained by processing the input through operations such as convolution;

[0051] y is the output, which is obtained by adding the processed result to the input.

[0052] The final convolutional layer summarizes the details of the image gradually restored through multiple convolutional layers and residual blocks, and is mainly responsible for generating the final denoising result; the residual block connection adds the input image with the "residual" after each channel has undergone multiple convolutions and feature extraction, and finally passes the input information to the output layer through skip connections;

[0053] S4: Recombine the sub-images on the RGB three channels to output the denoised training image. Evaluate the quality of the denoised training image based on the training objective, correct the convolution kernel, repeat step S3 until the evaluation requirements are met, and record various noise information during the denoising training process.

[0054] Image quality assessment is the evaluation of the quality of the denoised image. It is mainly evaluated through two parameters: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). PSNR is used to measure the quality of the denoised image, while SSIM is used to evaluate the structural similarity between the denoised image and the original image. These two evaluation parameters can reduce noise interference and ensure that the output denoised image retains the structure and details of the original image.

[0055] Peak Signal-to-Noise Ratio (PSNR):

[0056] PSNR = 10 * log 10 MAX 2 / MSE

[0057] Where: MAX is the maximum value of the image pixels;

[0058] MSE is the average difference between the pixel values ​​of the original image and the denoised image.

[0059] Structural Similarity Index (SSIM):

[0060]

[0061] in: Average brightness of the image at x and y;

[0062] The brightness variance of the image in x and y directions;

[0063] σ xy : Covariance between the x and y axes of the image;

[0064] C1, C1: constant, used to avoid the denominator being zero.

[0065] S5: Refer again Figure 1 Denoise all input images in the training set according to steps S2, S3 and S4, train the denoising model, and obtain the noise level map.

[0066] S6: After performing preliminary denoising on the preprocessed original images of the test set, input them into the trained FFDNet model. Input the noise level map obtained during training, and complete the denoising process according to the FFDNet-based channel denoising method in step S3. Reconstruct the denoised test image, and use the peak signal-to-noise ratio and structural similarity index mentioned above to evaluate the quality of the denoised test image based on the evaluation image. Verify and save the best noise level map, and complete the training of the denoising model.

[0067] Multimodal flame identification primarily involves recognizing blue flames and high-temperature yellow flames, such as... Figure 2 As shown, the original noisy flame image, after preprocessing and initial denoising, is input into the trained denoising model along with the optimal noise level map. Channel-by-channel denoising is then performed according to step S3, and the resulting image is combined to obtain the denoised image. The flame recognition method for the denoised image includes the following steps:

[0068] P1: The flame features of the denoised image are divided into blue flame and high-temperature flame, which are then identified and extracted separately; among them,

[0069] For the blue flame part, firstly, the RGB channel of the denoised image is enhanced and the blue sub-image on the B channel is extracted. Then, the blue flame region of the blue sub-image is extracted by setting the first HSV feature parameter range, and then the blue flame outline of the blue flame region is obtained.

[0070] For the high-temperature yellow flame portion, due to insufficient clarity of the flame outline in the denoised image, image processing is required to obtain an easily identifiable target image. Image processing includes background subtraction and inter-frame subtraction. Background subtraction is beneficial for accurately locating the target, but it is not sensitive to changes in flame color. Inter-frame subtraction highlights dynamic changes, but the boundaries are blurred and not obvious. Therefore, the denoised image can obtain a clear flame boundary through background subtraction, and a clear color gradient can be obtained through inter-frame subtraction. Then, the recognition images obtained by the two processes are superimposed at a certain ratio to obtain a target image with a clear flame color gradient and a clear boundary, which is beneficial for HSV feature recognition and extraction. At the same time, due to the complex flame shape, the superposition of the two can further verify the shape and details of the flame. Next, a second HSV feature parameter range is set to extract the yellow flame region of the target image, and then the yellow flame outline of the yellow flame region is extracted.

[0071] P2: Overlay the blue and yellow flame outlines and draw the original flame outline. The blue and yellow flame outlines should be closely connected. After overlaying, determine their continuity. If there are gaps, adjust the range of the first HSV feature parameter and the range of the second HSV feature parameter respectively until the two outlines completely overlap without gaps. Then the original flame outline is drawn. Based on the original flame outline, the flame area is extracted and identified, and the relevant information data of the extracted and identified flame area is output.

[0072] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-modal flame AI recognition method based on FFDNET, characterized in that, This includes training a denoising model and a flame recognition method for denoised images, among which... Training a denoising model involves the following steps: S1: The collected raw flame images are preprocessed. The preprocessed image dataset is divided into a training set and a test set. The clean images obtained after denoising the original images in the training set are used as training targets, and the clean images obtained after denoising the original images in the test set are used as evaluation images. S2: Perform noise estimation on the preprocessed image to obtain a predicted noise level map; S3: Input the preprocessed image and its predicted noise level map from step S2 into FFDNet. FFDNet performs channel-specific spatial downsampling on the input image in the RGB three channels, shifting the spatial resolution of the input image to the channel dimension. After dimensionality reduction, several sub-images are obtained. The convolutional layers, ReLU function, and multi-level residual blocks of FFDNet are used to denoise the sub-images in the three channels respectively. S4: Recombine the sub-images on the RGB three channels to obtain the denoised training image. Based on the training objective, evaluate the quality of the denoised training image, correct the convolution kernel, and repeat step S3 until the quality evaluation is satisfied. S5: Denoise all preprocessed images in the training set according to steps S2, S3 and S4, train the denoising model and obtain the noise level map. S6: Input the preprocessed image of the test set into the trained FFDNet model, and input the noise level map obtained during training. Perform denoising processing according to the FFDNet-based denoising method in step S3, reconstruct to obtain the denoised test image, use the evaluation image to evaluate the quality of the denoised test image, verify and save the best noise level map, and complete the training of the denoising model. The original flame image is preprocessed and then input into the trained denoising model. The optimal noise level map is input, and denoising is performed according to step S3 to obtain a denoised image. The flame recognition method for the denoised image includes the following steps: P1: The flame features of the denoised image are divided into blue flame and high-temperature flame, which are then identified and extracted separately; among them, For the blue flame part, firstly, the RGB channels of the denoised image are enhanced, and the blue sub-image on the B channel is extracted. Then, the blue flame region is extracted from the blue sub-image by setting the first HSV feature parameter range. Finally, the blue flame outline of the blue flame region is extracted. For the high-temperature flame part, firstly, the denoised image is processed, including background subtraction and inter-frame subtraction. The recognition images obtained by the two processes are superimposed proportionally to obtain a target image with easily recognizable HSV features. Then, the yellow flame region of the target image is extracted by setting the range of the second HSV feature parameters. Finally, the yellow flame outline of the yellow flame region is extracted. P2: Overlay the blue flame outline and the yellow flame outline to draw the original flame outline. Determine the continuity between the two and adjust the range of the first HSV feature parameter and the range of the second HSV feature parameter respectively until the two parts of the outline overlap without gaps. Then the drawing of the original flame outline is completed. Based on the original flame outline, the flame is extracted and identified, and the flame identification information is output.

2. The multimodal flame AI recognition method based on FFDNET according to claim 1, characterized in that: In step S3, the convolution operations and local features extracted by the activation functions on the R, G, and B channels are not only transformed into the next layer operations on their respective channels, but also synchronously share the learning information of the same layer with the other channels.

3. The multimodal flame AI recognition method based on FFDNET according to claim 1, characterized in that: The noise estimation in step S2 also includes marking key denoising regions based on experience and increasing the weight of the convolution kernel in the key denoising regions in step S3.

4. The multimodal flame AI recognition method based on FFDNET according to claim 3, characterized in that: It also includes preliminary denoising before the deep denoising in step S3, which uses an LMS filter to adjust the denoising intensity of different regions on the RGB three channels to achieve preliminary denoising.

5. The multimodal flame AI recognition method based on FFDNET according to claim 1, characterized in that: The evaluation parameters for image quality assessment include peak signal-to-noise ratio and structural similarity index.

Images