An intelligent detection method and system for hydrocarbon combustible flames integrating industrial vision and infrared detection technology

Abstract

The present invention relates to the field of flame detection technology, specifically to an intelligent detection method and system for hydrocarbon combustible flames that integrates industrial vision and infrared detection technology, including: obtaining infrared video and visible light video of a target area through an infrared camera and an RGB camera; performing oscillation frequency detection on the infrared video to obtain a frequency amplitude-frequency curve corresponding to the infrared video, and matching the frequency amplitude-frequency curve with a preset standard frequency amplitude-frequency curve to obtain the matching degree between the two. The present invention can accurately detect flames in multiple dimensions by utilizing the frequency amplitude-frequency curve obtained by an infrared camera and the image frames obtained by a visible light camera. The infrared video provides temperature information, and the visible light video provides morphology and structure information. The combination of the two enables the system to more accurately identify flames and avoid false alarms and missed alarms.

Description

Technical Field

[0001] The present invention relates to the technical field of flame detection, and in particular to an intelligent detection method and system for hydrocarbon combustible flames that integrates industrial vision and infrared detection technologies. Background Art

[0002] Hydrocarbon combustibles are mainly composed of carbon and hydrogen elements and are flammable. Such compounds include but are not limited to petroleum products, natural gas, coal and its derivatives, and other fuel oils. Hydrocarbon combustibles release energy during combustion and are therefore widely used as a source of energy, such as for power generation, heating, and fuel.

[0003] Industrial vision technology uses computer vision and image processing technology to analyze and process images or videos to extract useful information for control, detection or identification. Infrared detection technology uses the characteristics of infrared radiation to achieve object detection, temperature measurement, imaging or identification.

[0004] Traditional methods usually rely on a single sensor, such as a single visible light camera or a single infrared camera, which limits the diversity and comprehensiveness of the system's acquisition of flame information, which may lead to inaccurate and low reliability of detection. In addition, a single sensor often finds it difficult to accurately capture the characteristics of the flame and is easily affected by background interference or lighting conditions. Traditional methods usually perform flame detection based on fixed threshold settings. This method is difficult to cope with the changing characteristics of flames in different environments and may lead to false alarms or missed alarms, especially in strong lighting or complex backgrounds. Summary of the invention

[0005] The purpose of the present invention is to address the problems existing in the background technology and to propose an intelligent detection method and system for hydrocarbon combustible flames that integrates industrial vision and infrared detection technology.

[0006] The technical solution of the present invention is a method for intelligently detecting hydrocarbon combustible flames by integrating industrial vision and infrared detection technology, comprising:

[0007] Acquire infrared video and visible light video of the target area through infrared camera and RGB camera;

[0008] Performing oscillation frequency detection on the infrared video to obtain a frequency amplitude-frequency curve corresponding to the infrared video, and matching the frequency amplitude-frequency curve with a preset standard frequency amplitude-frequency curve to obtain a matching degree between the two;

[0009] If the matching degree is lower than a preset matching degree threshold, temperature segmentation is performed on the infrared video to obtain a set of infrared video frames to be detected of the infrared video;

[0010] Based on the infrared video frame set to be detected of the infrared video, a corresponding visible light video frame set to be detected of the visible light video is acquired, and the infrared video frame set to be detected is combined with the visible light video frame set to be detected to obtain a set of video frame pairs to be detected, wherein the infrared video frame set to be detected corresponds one-to-one to the visible light video frame set to be detected;

[0011] Performing video frame pair fusion on the set of video frame pairs to be detected by using a trained image fusion model to obtain a fused image set, performing video frame pair detection on the set of video frame pairs to be detected by using a trained first detection model to obtain a first detection result, and performing fused image detection on the set of fused images by using a trained second detection model to obtain a second detection result;

[0012] The first detection result and the second detection result are combined to obtain a third detection result.

[0013] Preferably, performing oscillation frequency detection on the infrared video to obtain a frequency amplitude-frequency curve corresponding to the infrared video includes:

[0014] Obtaining the grayscale value of each pixel in a plurality of infrared video frames in the infrared video;

[0015] The similarity between each adjacent infrared video frame in the infrared video is calculated to obtain a similarity sequence of the infrared video. The similarity calculation formula between the adjacent infrared video frames is as follows:

[0016]

[0017] Among them, R m,n represents the similarity between adjacent infrared video frames at (m,n) pixels, Cov represents covariance, Var represents variance, and A m,n and B m,n Represents the grayscale value of the (m,n) pixel in the current and adjacent infrared video frames in the infrared video of the target area;

[0018] Based on the similarity sequence of the infrared video and the time interval between adjacent infrared video frames, a frequency amplitude-frequency curve corresponding to the infrared video is constructed by discrete Fourier transform. The frequency amplitude-frequency curve is as follows:

[0019]

[0020] Wherein, x(t) represents the frequency amplitude-frequency curve, x(n) represents the nth similarity sample point in the similarity sequence, N represents the length of the similarity sequence, and k represents the index of the time interval between adjacent infrared video frames.

[0021] Preferably, performing temperature segmentation on the infrared video to obtain an infrared video frame to be detected in the infrared video includes:

[0022] The adaptive grayscale difference threshold of multiple infrared video frames in the infrared video is calculated, and the adaptive grayscale difference threshold calculation formula of the infrared video frame is as follows:

[0023]

[0024] in, represents the adaptive grayscale difference threshold of the infrared video frame, T th represents the preset grayscale difference threshold of the infrared video, f(x,y) represents the total value of the pixel area with the coordinate (x,y) in the infrared video frame, represents the average pixel value of the pixel area in the infrared video frame, u represents the total number of pixel areas in the infrared video frame, and R represents the total set of coordinates in the infrared video frame;

[0025] If the absolute value of the difference between the total pixel value of the pixel area of ​​the plurality of infrared video frames in the infrared video and the average pixel value of the pixel area in the infrared video frame is not greater than the adaptive grayscale difference threshold of the infrared video frame, the infrared video frame is used as the infrared video frame to be detected;

[0026] The above operation is repeated until a plurality of infrared video frames in the infrared video are traversed to obtain a set of infrared video frames to be detected in the infrared video.

[0027] Preferably, obtaining the corresponding set of visible light video frames to be detected of the visible light video based on the set of infrared video frames to be detected of the infrared video includes:

[0028] Obtaining frame rate and timestamp information of the infrared video and the visible light video;

[0029] Correspondingly associating each video frame in the infrared video and the visible light video based on the frame rate and the timestamp information to obtain an association mapping table;

[0030] Based on the infrared video frame set to be detected of the infrared video, the visible light video frame set to be detected of the visible light video is obtained through the association mapping table.

[0031] Preferably, the image fusion model includes a preprocessing layer, a three-layer encoder and a decoder in sequence. The preprocessing layer converts the video frame pair to be detected from a single channel to 32 channels through a 1×1 convolution layer, and outputs the result through a batch layer and an activation layer. The encoder includes an infrared attention module, a visible light attention module, an infrared dominant module and a visible light dominant module. The video frame pair to be detected is input into the infrared dominant module and the visible light dominant module of the first layer after passing through the preprocessing layer, and the absolute value of the gradient of the video frame to be detected is input into the infrared attention module and the visible light attention module respectively, and the outputs of the infrared attention module and the visible light attention module are point-by-point multiplied with the infrared dominant module and the visible light dominant module respectively, and the results of the point-by-point multiplication are input into the infrared dominant module and the visible light dominant module of the second layer, and the outputs of the infrared dominant module and the visible light dominant module of the third layer are point-by-point multiplied with the outputs of the infrared attention module and the visible light attention module of the third layer, and the results of the point-by-point multiplication are cascaded, and the cascaded results are input into the decoder, and the decoder is used to reconstruct the image to obtain a fused image.

[0032] Preferably, the infrared dominant module and the visible light dominant module include two local residual networks and one global residual network, the local residual network includes two convolutional layers, two batch processing layers and two activation layers, the global residual network includes one convolutional layer, one batch processing layer and one activation layer, the infrared attention module and the visible light attention module include convolutional layers, batch processing layers and Sigmoid layers, the infrared attention module and the visible light attention module are used to calculate the weights of the output features of each layer of the infrared dominant module and the visible light dominant module respectively, and the decoder includes 4 layers of features A retention module, wherein the feature retention module includes a 3×3 convolutional neural network, 3 parallel paths and a 1×1 convolutional neural network. The feature retention module first converts the input into 64 channels through the 3×3 convolutional neural network, extracts features of multiple receptive fields through the 3 parallel paths, the 3 parallel paths include 3, 2 and 1 convolutional layers respectively, and the outputs of the 3 parallel paths are cascaded and input into the 1×1 convolutional neural network, and the output of the 1×1 convolutional neural network is processed through a residual network and then through a Relu activation layer to obtain a final output.

[0033] Preferably, the output characteristic expressions of the infrared dominant module and the visible light dominant module are as follows:

[0034]

[0035] in, and Respectively represent the output characteristics of the infrared dominant module and the visible light dominant module, and They represent the outputs of the infrared-dominated module and the visible-light-dominated module of the previous layer, respectively. and Respectively represent the output features of the infrared attention module and the visible light attention module of the previous layer;

[0036] The output expression of the third layer encoder is as follows:

[0037]

[0038] Where E represents the output of the third layer encoder, and They represent the outputs of the infrared dominant module and the visible light dominant module of the third layer respectively. and They represent the outputs of the infrared attention module and the visible light attention module of the third layer respectively, and concat represents cascade;

[0039] The input expression of the feature retention module is as follows:

[0040]

[0041] in, represents the input of the Nth layer feature preservation module, and Represent the outputs of the 1st and N-1th layer feature retention modules respectively;

[0042] The output expression of the encoder is as follows:

[0043]

[0044] Where D represents the output of the encoder, and Represent the outputs of the detail-preserving modules at layers 1, 2, 3, and 4, respectively.

[0045] Preferably, the first detection model includes a feature extraction module, a feature fusion module and a Transformer-based encoding-decoding module, the feature extraction module uses two ResNet50s to respectively extract the infrared feature map and the visible light feature map of the infrared video frame to be detected and the visible light video frame to be detected in the video frame pair to be detected, the feature fusion module is used to calculate the first weight map and the second weight map of the visible light feature map through average pooling and maximum pooling operations, and reduce the dimension through the convolution kernel, and activate through the activation function to obtain the third weight map of the first weight map and the second weight map, splice the third weight map and the infrared feature map to obtain the infrared-visible light feature map, obtain the fourth weight map of the infrared-visible light feature map through the channel attention module, activate the fourth weight map and the infrared-visible light feature map through the activation function to obtain the fusion feature, and the attention module uses the ECANet model.

[0046] Preferably, the second detection model adopts the YOLOv5 model.

[0047] The technical solution of the present invention is: a hydrocarbon combustible flame intelligent detection system integrating industrial vision and infrared detection technology, which is applicable to the hydrocarbon combustible flame intelligent detection method integrating industrial vision and infrared detection technology, including:

[0048] A data acquisition module, wherein the data acquisition module is used to acquire infrared video and visible light video of a target area;

[0049] A curve matching module, which is used to detect the oscillation frequency of the infrared video, obtain a frequency amplitude-frequency curve, and match it with a preset standard frequency amplitude-frequency curve to evaluate the matching degree;

[0050] A video processing module, wherein the video processing module is used to perform temperature segmentation on the infrared video to obtain a set of infrared video frames to be detected when the matching degree is lower than a preset matching degree threshold;

[0051] A frame pair combination module, the frame pair combination module is used to obtain a corresponding visible light video frame set to be detected of the visible light video based on the infrared video frame set to be detected, and to combine the infrared video frames to be detected and the visible light video frames to be detected into a video frame pair set to be detected in a one-to-one correspondence;

[0052] A flame detection module, the flame detection module is used to use the trained image fusion model to perform video frame pair fusion on the set of video frame pairs to be detected to generate a fused image set, use the trained first detection model to detect the set of video frame pairs to be detected to obtain a first detection result, and use the trained second detection model to detect the fused image set to obtain a second detection result;

[0053] The decision fusion module is used to perform decision fusion on the first detection result and the second detection result to generate a final third detection result.

[0054] Compared with the prior art, the above technical solution of the present invention has the following beneficial technical effects:

[0055] 1. The present invention uses the frequency amplitude-frequency curve obtained by the infrared camera and the image frame obtained by the visible light camera to accurately detect flames in multiple dimensions. The infrared video provides temperature information, and the visible light video provides morphology and structure information. The combination of the two enables the system to more accurately identify flames and avoid false alarms and missed alarms. The frequency oscillation detection and temperature segmentation can quickly determine the set of video frames that need to be detected. Combined with the image fusion model and the trained detection model, flame detection can be performed in real time and the detection results can be quickly generated, which is of great significance for timely response to emergency situations such as fires.

[0056] 2. The present invention can effectively resist these interferences and improve the stability and reliability of detection by combining infrared and visible light information for detection. By adopting multiple detection models and decision fusion strategies, multi-level strategies can comprehensively utilize the advantages of different models, reduce false detection rates and missed detection rates, and improve the overall performance of the detection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic flow chart of an overall method in an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of the overall system flow in an embodiment of the present invention.

[0059] Figure numerals: 1. Data acquisition module; 2. Curve matching module; 3. Video processing module; 4. Frame pair combination module; 5. Flame detection module; 6. Decision fusion module. DETAILED DESCRIPTION

[0060] Embodiment 1, as Figure 1 As shown, the present invention proposes an intelligent detection method for hydrocarbon combustible flames that integrates industrial vision and infrared detection technology, including:

[0061] S1, obtain infrared video and visible light video of the target area through infrared camera and RGB camera;

[0062] S2, performing oscillation frequency detection on the infrared video to obtain a frequency amplitude-frequency curve corresponding to the infrared video, and matching the frequency amplitude-frequency curve with a preset standard frequency amplitude-frequency curve to obtain a matching degree between the two;

[0063] S3. If the matching degree is lower than a preset matching degree threshold, temperature segmentation is performed on the infrared video to obtain a set of infrared video frames to be detected of the infrared video;

[0064] S4. Based on the infrared video frame set to be detected of the infrared video, a corresponding visible light video frame set to be detected of the visible light video is obtained, and the infrared video frame set to be detected is combined with the visible light video frame set to be detected to obtain a set of video frame pairs to be detected, where the infrared video frame set to be detected corresponds to the visible light video frame set to be detected in one-to-one correspondence;

[0065] S5. Performing video frame pair fusion on the set of video frame pairs to be detected by using the trained image fusion model to obtain a fused image set, performing video frame pair detection on the set of video frame pairs to be detected by using the trained first detection model to obtain a first detection result, and performing fused image detection on the fused image set by using the trained second detection model to obtain a second detection result;

[0066] S6. Decision-making fusion is performed on the first detection result and the second detection result to obtain a third detection result.

[0067] In the present invention, the infrared camera is capable of detecting infrared radiation emitted by an object and converting it into a thermal image; the RGB camera is a traditional color camera used to obtain visible light images; the first detection result and the second detection result are weighted averaged according to their own confidence levels through confidence weighted averaging to obtain the second detection result.

[0068] Embodiment 2, the present invention proposes a method for intelligently detecting hydrocarbon combustible flames that integrates industrial vision and infrared detection technology. Compared with embodiment 1, this embodiment further includes: performing oscillation frequency detection on the infrared video to obtain a frequency amplitude-frequency curve corresponding to the infrared video, including:

[0069] A1. Obtaining the grayscale value of each pixel in a plurality of infrared video frames in the infrared video;

[0070] A2. Calculate the similarity between each adjacent infrared video frame in the infrared video to obtain a similarity sequence of the infrared video. The similarity calculation formula between adjacent infrared video frames is as follows:

[0071]

[0072] Among them, R m,n represents the similarity between adjacent infrared video frames at (m,n) pixels, Cov represents covariance, Var represents variance, and A m,n and B m,n Represents the grayscale value of the (m,n) pixel in the current and adjacent infrared video frames in the infrared video of the target area;

[0073] A3. Based on the similarity sequence of infrared video and the time interval between adjacent infrared video frames, the frequency amplitude-frequency curve corresponding to the infrared video is constructed by discrete Fourier transform. The frequency amplitude-frequency curve is as follows:

[0074]

[0075] Wherein, x(t) represents the frequency amplitude-frequency curve, x(n) represents the nth similarity sample point in the similarity sequence, N represents the length of the similarity sequence, and k represents the index of the time interval between adjacent infrared video frames.

[0076] In an optional embodiment, performing temperature segmentation on the infrared video to obtain an infrared video frame to be detected in the infrared video includes:

[0077] B1. Calculate the adaptive grayscale difference threshold of multiple infrared video frames in the infrared video. The adaptive grayscale difference threshold calculation formula of the infrared video frame is as follows:

[0078]

[0079] in, represents the adaptive grayscale difference threshold of the infrared video frame, T th represents the preset grayscale difference threshold of the infrared video, f(x,y) represents the total value of the pixel area with the coordinate (x,y) in the infrared video frame, represents the average pixel value of the pixel area in the infrared video frame, u represents the total number of pixel areas in the infrared video frame, and R represents the total set of coordinates in the infrared video frame;

[0080] B2. If the absolute value of the difference between the total pixel value of the pixel area of ​​the plurality of infrared video frames in the infrared video and the average pixel value of the pixel area in the infrared video frame is not greater than the adaptive grayscale difference threshold of the infrared video frame, the infrared video frame is used as the infrared video frame to be detected;

[0081] B3. Repeat the above operation until multiple infrared video frames in the infrared video are traversed to obtain a set of infrared video frames to be detected in the infrared video.

[0082] It should be noted that the adaptive grayscale difference threshold refers to a threshold dynamically calculated based on the pixel statistical characteristics of each infrared video frame; the pixel area refers to a pixel area in the infrared video frame, which may be a rectangular or other shaped area, used for pixel value statistics.

[0083] In an optional embodiment, obtaining a corresponding visible light video frame set to be detected of a visible light video based on a set of infrared video frames to be detected of an infrared video includes:

[0084] C1. Obtain the frame rate and timestamp information of infrared video and visible light video;

[0085] C2. Correspondingly associating each video frame in the infrared video and the visible light video based on the frame rate and timestamp information to obtain an association mapping table;

[0086] C3. Obtaining the set of visible light video frames to be detected of the visible light video through an association mapping table based on the set of infrared video frames to be detected of the infrared video.

[0087] In an optional embodiment, the image fusion model includes a preprocessing layer, a three-layer encoder and a decoder in sequence. The preprocessing layer converts the video frame pair to be detected from a single channel to 32 channels through a 1×1 convolution layer, and outputs the result through a batch layer and an activation layer. The encoder includes an infrared attention module, a visible light attention module, an infrared dominant module and a visible light dominant module. The video frame pair to be detected is input into the infrared dominant module and the visible light dominant module of the first layer after passing through the preprocessing layer, and the absolute value of the gradient of the video frame to be detected is input into the infrared attention module and the visible light attention module respectively, and the outputs of the infrared attention module and the visible light attention module are point-by-point multiplied with the infrared dominant module and the visible light dominant module respectively, and the results of the point-by-point multiplication are input into the infrared dominant module and the visible light dominant module of the second layer, the outputs of the infrared dominant module and the visible light dominant module of the third layer are point-by-point multiplied with the outputs of the infrared attention module and the visible light attention module of the third layer, and the results of the point-by-point multiplication are cascaded, and the cascaded results are input into the decoder. The decoder is used to reconstruct the image to obtain a fused image.

[0088] It should be noted that the batch layer is a technology used to accelerate the training process of deep neural networks and improve the convergence speed by normalizing the distribution of each feature during training; the activation layer is used to introduce nonlinear characteristics to increase the expressive power of the model; point-by-point multiplication refers to the operation of multiplying the corresponding elements of two tensors (such as feature maps) with the same dimension to fuse different feature representations; cascading refers to connecting multiple tensors along a certain dimension, usually on the channel dimension of the feature map, to obtain richer feature representations.

[0089] In an optional embodiment, the infrared dominant module and the visible light dominant module include two local residual networks and one global residual network, the local residual network includes two convolutional layers, two batch processing layers and two activation layers, the global residual network includes one convolutional layer, one batch processing layer and an activation layer, the infrared attention module and the visible light attention module include convolutional layers, batch processing layers and Sigmoid layers, the weights of the output features of each layer of the infrared dominant module and the visible light dominant module are calculated respectively through the infrared attention module and the visible light attention module, the decoder includes a 4-layer feature retention module, the feature retention module includes a 3×3 convolutional neural network, 3 parallel paths and a 1×1 convolutional neural network, the feature retention module first converts the input into 64 channels through a 3×3 convolutional neural network, extracts features of multiple receptive fields through 3 parallel paths, the 3 parallel paths include 3, 2 and 1 convolutional layers respectively, and the outputs of the 3 parallel paths are cascaded and input into a 1×1 convolutional neural network, the output of the 1×1 convolutional neural network is processed through the residual network and then through the Relu activation layer to obtain the final output.

[0090] In an optional embodiment, the output characteristic expressions of the infrared dominant module and the visible light dominant module are as follows:

[0091]

[0092] in, and Respectively represent the output characteristics of the infrared dominant module and the visible light dominant module, and They represent the outputs of the infrared-dominated module and the visible-light-dominated module of the previous layer, respectively. and Respectively represent the output features of the infrared attention module and the visible light attention module of the previous layer;

[0093] The output expression of the third layer encoder is as follows:

[0094]

[0095] Where E represents the output of the third layer encoder, and They represent the outputs of the infrared dominant module and the visible light dominant module of the third layer respectively. and They represent the outputs of the infrared attention module and the visible light attention module of the third layer respectively, and concat represents cascade;

[0096] The input expression of the feature preservation module is as follows:

[0097]

[0098] in, represents the input of the Nth layer feature preservation module, and Represent the outputs of the 1st and N-1th layer feature retention modules respectively;

[0099] The output expression of the encoder is as follows:

[0100]

[0101] Where D represents the output of the encoder, and Represent the outputs of the detail-preserving modules at layers 1, 2, 3, and 4, respectively.

[0102] In an optional embodiment, the first detection model includes a feature extraction module, a feature fusion module and a Transformer-based encoding-decoding module. The feature extraction module uses two ResNet50s to respectively extract the infrared feature map and the visible light feature map of the infrared video frame to be detected and the visible light video frame to be detected in the video frame pair to be detected. The feature fusion module is used to calculate the first weight map and the second weight map of the visible light feature map through average pooling and maximum pooling operations, and reduce the dimension through the convolution kernel, and activate through the activation function to obtain a third weight map of the first weight map and the second weight map, splice the third weight map and the infrared feature map to obtain the infrared-visible light feature map, obtain the fourth weight map of the infrared visible light feature map through the channel attention module, activate the fourth weight map and the infrared visible light feature map through the activation function to obtain the fusion feature, and the attention module uses the ECANet model.

[0103] It should be noted that ResNet is a deep convolutional neural network architecture proposed by Microsoft Research. ResNet50 is a variant of it, which contains 50 convolutional layers and fully connected layers, as well as a structure called "Residual Blocks". The design of residual blocks allows the neural network to be deeper, while reducing the gradient vanishing problem, which helps to train deep networks more effectively; ECANet is an attention mechanism model for enhancing the performance of deep neural networks. It is specially designed to enhance the effect of channel attention in convolutional neural networks, thereby improving the network's perception of important features.

[0104] In an optional embodiment, the second detection model adopts the YOLOv5 model.

[0105] It should be noted that YOLO is a real-time target detection algorithm. Its core idea is to regard the target detection task as a regression problem and directly predict the bounding box and category probability at the image level. YOLOv5 is one of the latest versions of the YOLO series, developed by Ultralytics. Compared with previous versions such as YOLOv4, YOLOv5 improves the model structure and performance while maintaining efficient real-time target detection capabilities.

[0106] Embodiment three, as Figure 2 As shown, a hydrocarbon combustible flame intelligent detection system integrating industrial vision and infrared detection technology is applicable to the above-mentioned hydrocarbon combustible flame intelligent detection method integrating industrial vision and infrared detection technology, including:

[0107] Data acquisition module 1, data acquisition module 1 is used to acquire infrared video and visible light video of the target area;

[0108] Curve matching module 2, which is used to detect the oscillation frequency of the infrared video, obtain a frequency amplitude-frequency curve, and match it with a preset standard frequency amplitude-frequency curve to evaluate the matching degree;

[0109] The video processing module 3 is used to perform temperature segmentation on the infrared video to obtain a set of infrared video frames to be detected when the matching degree is lower than a preset matching degree threshold;

[0110] The frame pair combination module 4 is used to obtain a corresponding visible light video frame set to be detected of the visible light video based on the infrared video frame set to be detected, and to combine the infrared video frames to be detected and the visible light video frames to be detected into a video frame pair set to be detected in a one-to-one correspondence;

[0111] The flame detection module 5 is used to use the trained image fusion model to perform video frame fusion on the set of video frame pairs to be detected to generate a fused image set, use the trained first detection model to detect the set of video frame pairs to be detected to obtain a first detection result, and use the trained second detection model to detect the fused image set to obtain a second detection result;

[0112] The decision fusion module 6 is used to perform decision fusion on the first detection result and the second detection result to generate a final third detection result.

[0113] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. An intelligent detection method for hydrocarbon combustible flames that integrates industrial vision and infrared detection technology, characterized in that: include: Acquire infrared video and visible light video of the target area through infrared camera and RGB camera; Performing oscillation frequency detection on the infrared video to obtain a frequency amplitude-frequency curve corresponding to the infrared video, and matching the frequency amplitude-frequency curve with a preset standard frequency amplitude-frequency curve to obtain a matching degree between the two; If the matching degree is lower than a preset matching degree threshold, temperature segmentation is performed on the infrared video to obtain a set of infrared video frames to be detected of the infrared video; Based on the infrared video frame set to be detected of the infrared video, a corresponding visible light video frame set to be detected of the visible light video is acquired, and the infrared video frame set to be detected is combined with the visible light video frame set to be detected to obtain a set of video frame pairs to be detected, wherein the infrared video frame set to be detected corresponds one-to-one to the visible light video frame set to be detected; Performing video frame pair fusion on the set of video frame pairs to be detected by using a trained image fusion model to obtain a fused image set, performing video frame pair detection on the set of video frame pairs to be detected by using a trained first detection model to obtain a first detection result, and performing fused image detection on the set of fused images by using a trained second detection model to obtain a second detection result; Performing decision fusion on the first detection result and the second detection result to obtain a third detection result; The image fusion model includes a preprocessing layer, a three-layer encoder and a decoder in sequence. The preprocessing layer converts the video frame pair to be detected from a single channel to 32 channels through a 1×1 convolution layer, and outputs the result through a batch layer and an activation layer. The encoder includes an infrared attention module, a visible light attention module, an infrared dominant module and a visible light dominant module. The video frame pair to be detected is input into the infrared dominant module and the visible light dominant module of the first layer after passing through the preprocessing layer, and the absolute value of the gradient of the video frame to be detected is input into the infrared attention module and the visible light attention module respectively, and the outputs of the infrared attention module and the visible light attention module are point-by-point multiplied with the infrared dominant module and the visible light dominant module respectively, and the results of the point-by-point multiplication are input into the infrared dominant module and the visible light dominant module of the second layer, and the outputs of the infrared dominant module and the visible light dominant module of the third layer are point-by-point multiplied with the outputs of the infrared attention module and the visible light attention module of the third layer, and the results of the point-by-point multiplication are cascaded, and the cascaded results are input into the decoder. The decoder is used to reconstruct the image to obtain a fused image.

2. According to claim 1, a method for intelligent detection of hydrocarbon combustible flames integrating industrial vision and infrared detection technology is characterized by: Performing oscillation frequency detection on the infrared video to obtain a frequency amplitude-frequency curve corresponding to the infrared video includes: Obtaining the grayscale value of each pixel in a plurality of infrared video frames in the infrared video; The similarity between each adjacent infrared video frame in the infrared video is calculated to obtain a similarity sequence of the infrared video. The similarity calculation formula between the adjacent infrared video frames is as follows: Among them, R m,n represents the similarity between adjacent infrared video frames at (m,n) pixels, Cov represents covariance, Var represents variance, and A m,n and B n,n Represents the grayscale value of the (m,n) pixel in the current and adjacent infrared video frames in the infrared video of the target area; Based on the similarity sequence of the infrared video and the time interval between adjacent infrared video frames, a frequency amplitude-frequency curve corresponding to the infrared video is constructed by discrete Fourier transform. The frequency amplitude-frequency curve is as follows: Wherein, x(t) represents the frequency amplitude-frequency curve, x(n) represents the nth similarity sample point in the similarity sequence, N represents the length of the similarity sequence, and k represents the index of the time interval between adjacent infrared video frames.

3. According to claim 1, a method for intelligent detection of hydrocarbon combustible flames integrating industrial vision and infrared detection technology is characterized in that: Performing temperature segmentation on the infrared video to obtain an infrared video frame to be detected in the infrared video includes: The adaptive grayscale difference threshold of multiple infrared video frames in the infrared video is calculated, and the adaptive grayscale difference threshold calculation formula of the infrared video frame is as follows: in, represents the adaptive grayscale difference threshold of the infrared video frame, T th represents the preset grayscale difference threshold of the infrared video, f(x,y) represents the total value of the pixel area with the coordinate (x,y) in the infrared video frame, represents the average pixel value of the pixel area in the infrared video frame, u represents the total number of pixel areas in the infrared video frame, and R represents the total set of coordinates in the infrared video frame; If the absolute value of the difference between the total pixel value of the pixel area of ​​the plurality of infrared video frames in the infrared video and the average pixel value of the pixel area in the infrared video frame is not greater than the adaptive grayscale difference threshold of the infrared video frame, the infrared video frame is used as the infrared video frame to be detected; The above operation is repeated until a plurality of infrared video frames in the infrared video are traversed to obtain a set of infrared video frames to be detected in the infrared video.

4. According to claim 1, a method for intelligent detection of hydrocarbon combustible flames integrating industrial vision and infrared detection technology is characterized in that: Acquiring a corresponding set of visible light video frames to be detected of the visible light video based on the set of infrared video frames to be detected of the infrared video, comprising: Obtaining frame rate and timestamp information of the infrared video and the visible light video; Correspondingly associating each video frame in the infrared video and the visible light video based on the frame rate and the timestamp information to obtain an association mapping table; Based on the infrared video frame set to be detected of the infrared video, the visible light video frame set to be detected of the visible light video is obtained through the association mapping table.

5. According to claim 1, a method for intelligent detection of hydrocarbon combustible flames integrating industrial vision and infrared detection technology is characterized in that: The infrared dominant module and the visible light dominant module include two local residual networks and one global residual network, the local residual network includes two convolutional layers, two batch processing layers and two activation layers, the global residual network includes one convolutional layer, one batch processing layer and one activation layer, the infrared attention module and the visible light attention module include convolutional layers, batch processing layers and Sigmoid layers, the weights of the output features of each layer of the infrared dominant module and the visible light dominant module are calculated respectively by the infrared attention module and the visible light attention module, and the decoder includes 4 layers of feature retention A module, wherein the feature retention module includes a 3×3 convolutional neural network, 3 parallel paths and a 1×1 convolutional neural network. The feature retention module first converts the input into 64 channels through the 3×3 convolutional neural network, extracts features of multiple receptive fields through the 3 parallel paths, the 3 parallel paths include 3, 2 and 1 convolutional layers respectively, and the outputs of the 3 parallel paths are cascaded and input into the 1×1 convolutional neural network, and the output of the 1×1 convolutional neural network is processed through a residual network and then through a Relu activation layer to obtain a final output.

6. The method for intelligent detection of hydrocarbon combustible flames integrating industrial vision and infrared detection technology according to claim 5 is characterized in that: The output characteristic expressions of the infrared dominant module and the visible light dominant module are as follows: in, and Respectively represent the output characteristics of the infrared dominant module and the visible light dominant module, and They represent the outputs of the infrared-dominated module and the visible-light-dominated module of the previous layer, respectively. and Respectively represent the output features of the infrared attention module and the visible light attention module of the previous layer; The output expression of the third layer encoder is as follows: Where E represents the output of the third layer encoder, and They represent the outputs of the infrared dominant module and the visible light dominant module of the third layer respectively. and They represent the outputs of the infrared attention module and the visible light attention module of the third layer respectively, and concat represents cascade; The input expression of the feature retention module is as follows: in, represents the input of the Nth layer feature preservation module, and Represent the outputs of the 1st and N-1th layer feature retention modules respectively; The output expression of the encoder is as follows: Where D represents the output of the encoder, and Represent the outputs of the detail-preserving modules at layers 1, 2, 3, and 4, respectively.

7. The method for intelligent detection of hydrocarbon combustible flames integrating industrial vision and infrared detection technology according to claim 1 is characterized in that: The first detection model includes a feature extraction module, a feature fusion module and a Transformer-based encoding-decoding module. The feature extraction module uses two ResNet50s to respectively extract the infrared feature map and the visible light feature map of the infrared video frame to be detected and the visible light video frame to be detected in the video frame pair to be detected. The feature fusion module is used to calculate the first weight map and the second weight map of the visible light feature map through average pooling and maximum pooling operations, and reduce the dimension through the convolution kernel, and activate through the activation function to obtain the third weight map of the first weight map and the second weight map, splice the third weight map and the infrared feature map to obtain the infrared-visible light feature map, obtain the fourth weight map of the infrared-visible light feature map through the channel attention module, activate the fourth weight map and the infrared-visible light feature map through the activation function to obtain the fusion feature, and the attention module uses the ECANet model.

8. The method for intelligent detection of hydrocarbon combustible flames integrating industrial vision and infrared detection technology according to claim 1 is characterized in that: The second detection model adopts the YOLOv5 model.

9. An intelligent detection system for hydrocarbon combustible flames integrating industrial vision and infrared detection technology, which is applicable to an intelligent detection method for hydrocarbon combustible flames integrating industrial vision and infrared detection technology as described in any one of claims 1 to 8, characterized in that: include: A data acquisition module (1), the data acquisition module (1) is used to acquire infrared video and visible light video of the target area; A curve matching module (2), the curve matching module (2) is used to detect the oscillation frequency of the infrared video, obtain a frequency amplitude-frequency curve, match it with a preset standard frequency amplitude-frequency curve, and evaluate the matching degree; A video processing module (3), wherein the video processing module (3) is used to perform temperature segmentation on the infrared video when the matching degree is lower than a preset matching degree threshold, so as to obtain a set of infrared video frames to be detected; A frame pair combination module (4), the frame pair combination module (4) is used to obtain a corresponding visible light video frame set to be detected of the visible light video based on the infrared video frame set to be detected, and to combine the infrared video frames to be detected and the visible light video frames to be detected into a video frame pair set to be detected in a one-to-one correspondence; A flame detection module (5), the flame detection module (5) is used to use a trained image fusion model to perform video frame pair fusion on a set of video frame pairs to be detected, to generate a fused image set, to use a trained first detection model to detect the set of video frame pairs to be detected, to obtain a first detection result, and to use a trained second detection model to detect the fused image set, to obtain a second detection result; A decision fusion module (6), wherein the decision fusion module (6) is used to perform decision fusion on the first detection result and the second detection result to generate a final third detection result.

Images