A method of smoke and flame detection for a factory pipeline based on multi-modal data

Abstract

The application discloses a kind of smoke and flame detection methods of factory pipeline based on multi-modal data, obtain the multi-modal data of different types of smoke and flame, and are preprocessed;For visible image part, SIFT descriptor and SURF descriptor are used to extract image local feature points, and visual features are extracted by fusing two kinds of descriptors;For infrared image, gradient mode extraction method is used;For smoke data, directly use smoke concentration features as model input;First, use element-by-element operation to fuse two kinds of images, then use attention mechanism to weight the fused data;After multi-modal data fusion, the features and smoke data fused by hybrid model are sent to the input end of Yolov7 algorithm for target detection.The application fuses the multi-modal data of visible light camera and infrared camera, and improves the accuracy and reliability of factory pipeline smoke and flame detection through feature enhancement.

Description

TECHNICAL FIELD

[0001] The present invention belongs to the technical field, and particularly relates to a method for detecting smoke and flame in a factory pipeline based on multi-modal data. BACKGROUND

[0002] In industrial production processes, the detection of smoke and flame in factory pipelines is crucial for ensuring the safe operation of factories. Currently, there are some technologies for detecting smoke and flame, including the use of visible light cameras, smoke sensors, etc. However, these existing technologies have some shortcomings and deficiencies, limiting their performance and effectiveness in practical applications. The single data source used in existing technologies, such as visible light cameras, has certain limitations in smoke and flame detection. These methods can only rely on single-modal information for detection, and cannot fully utilize the rich information of multi-modal data to improve detection accuracy and reliability. Existing target detection technologies can detect smoke and flame targets, but cannot provide real-time judgment and early warning of potential dangerous situations. The existing technology has the following main problems to be solved: lack of methods to fully utilize multi-modal data, inability to achieve accurate detection of smoke and flame, lack of judgment and timely warning ability for potential dangerous situations.

[0003] The present invention aims to solve the above problems and proposes a method for detecting smoke and flame in a factory pipeline based on multi-modal data fusion and feature enhancement. This invention combines the multi-modal data of visible light cameras and infrared cameras, and uses the fused features to enhance the robustness and reliability of the target detection algorithm. It improves the accuracy and reliability of smoke and flame detection, fully utilizes the feature information of visible light and infrared images through multi-modal data fusion, can more accurately judge potential fire risks, and timely trigger warning measures; enhances the real-time, stability and operability of factory pipeline smoke and flame detection. By combining multi-modal data fusion and feature enhancement in the YOLOv7 algorithm, the limitations and deficiencies of existing technologies are solved. It can provide more accurate, reliable and efficient detection of smoke and flame in factory pipelines, and has wide application prospects and commercial value. SUMMARY

[0004] The purpose of the present invention is to propose a method for detecting smoke and flame in a factory pipeline based on multi-modal data, which combines the multi-modal data of visible light cameras and infrared cameras, and improves the accuracy and reliability of factory pipeline smoke and flame detection through feature enhancement.

[0005] Technical solution: The method for detecting smoke and flame in a factory pipeline based on multi-modal data according to the present invention, specifically includes the following steps:

[0006] (1) Pre-acquire multi-modal data of different types of smoke and flame in the factory pipeline, and pre-process the multi-modal data;

[0007] (2) Convert different types of data into a format suitable for YOLOv7 input;

[0008] (3) For the visible image part, use SIFT descriptor and SURF descriptor to extract image local feature points and calculate corresponding feature descriptors, and extract visual features by fusing the two descriptors;

[0009] (4) For infrared images, use gradient mode extraction method to capture texture in the image to calculate edge information and extract hot spots and object boundaries in infrared images;

[0010] (5) For smoke data, directly use smoke concentration features as model input;

[0011] (6) First, use element-wise operations to fuse two images, and then use attention mechanism to weight the fused data;

[0012] (7) After multi-modal data fusion, use the fused features of the hybrid model and smoke data to input into the input end of the Yolov7 algorithm for target detection, i.e. to detect smoke and flame in the factory pipeline.

[0013] Further, the multi-modal data acquisition process of step (1) is as follows:

[0014] For visible light images, use a visible light camera to collect image data; for infrared images, use a thermal imager to collect infrared data; for smoke data, use a smoke sensor to collect smoke concentration.

[0015] Further, the data pre-processing process of step (1) is as follows:

[0016] Synchronize and align the data using timestamps, which have two data sources Data1 and Data2, and timestamps t_Data1 and t_Data2. Take the timestamp of data source Data1 as the reference time, then the timestamp of data source Data2 needs to be:

[0017]

[0018] In the formula, t_Data2_aligned represents the timestamp of the aligned data source t_Data2, δ represents the time delay of data source t_Data2 relative to data source Data1; data source Data1 is a data set from smoke and flame in the network, data source t_Data2 is a data set from smoke and flame shot inside the factory, and Data represents the total data set;

[0019] For the visible light image data in the total data set Data, a Gaussian filter is used to denoise the image; this is achieved by performing a weighted average of the pixels in the neighborhood of each pixel in the image captured by the visible light camera, the Gaussian filter using a two-dimensional Gaussian kernel to smooth the image:

[0020]

[0021] In the formula, K(x, y) represents the value of the Gaussian kernel, x and y represent the pixel position coordinates, and sigma represents the standard deviation of the Gaussian kernel;

[0022] For each pixel in the image, take its neighborhood pixels, and perform a weighted average of the neighborhood pixels according to the weight of the Gaussian kernel to obtain the new value of the pixel; the Gaussian filter denoising process is:

[0023] D'(x, y) = å(D(x+i, y+j)*K(i, j)) (3)

[0024] In the formula, D'(x, y) represents the pixel value of the image after Gaussian filtering, D(x+i, y+j) represents the original value of each field pixel in the image, K(i, j) represents the value of the Gaussian kernel, and [i, j] represents the coordinate position of the kernel relative to the center point.

[0025] Random flipping, random rotation, and random cropping enhancement operations are performed on the image data, the image data after data conversion is randomly rotated in the horizontal or vertical direction, the view transformation of the image data is enhanced, so that the model has better recognition ability for different directions of the image:

[0026]

[0027] In the formula, F_H represents the horizontally flipped image, F_V represents the vertically flipped image, and random(0, 1) represents a random number, according to which the image is flipped left and right; if the random number is 1, the image is flipped left and right, otherwise no processing is performed.

[0028] Random rotation is also implemented while random flipping is performed, the image is randomly rotated at a random angle; a rotation angle is randomly generated within a specified angle range, and the image is rotated according to the generated random angle; an image rotation function is used for operation, the image rotation is implemented through an affine transformation matrix, for a two-dimensional image, the affine transformation matrix is:

[0029]

[0030] In the formula, theta represents the rotation angle, t x and ty represents the number of pixels of translation, t x and t y can be set to zero; the pixel coordinates on the original image are (x, y), and the new coordinates after affine transformation are (x', y'):

[0031]

[0032] where M 11 , M 12 , M 13 , M 21 , M 22 , M 23 are elements of the affine transformation matrix (5), for image rotation, M 11 and M 22 are the cosine and sine values of rotation, respectively, while M 12 and M 21 are the negative sine and cosine values, respectively, and M 13 and M 23 are used to adjust the translation after rotation.

[0033] Further, the step (2) is implemented as follows:

[0034] convert the pixel values of the image into numerical data, use floating-point data to represent the pixel values, and map the pixel values to the range [0, 1]; convert the channel order of the image from the usual RGB to the required BGR order for the model; generate the converted image data as a tensor represented by a multi-dimensional array; for batch image input, form a 4-dimensional tensor with the shape [batch size, height, width, channel number], and let the original image be Pic, with the size:

[0035] Pic = High * Weight * Count (7)

[0036] where High represents the height, Weight represents the width, and Count represents the channel number, then the image is converted into a tensor as:

[0037]

[0038] where Num pic represents the number of images;

[0039] For images with different pixel value ranges, normalization operation is also needed to scale the pixel values to the range [0, 1];

[0040] For the data of smoke, the sensor data is converted into numerical data of float type, the data is linearly scaled according to the measurement accuracy and range of the sensor, so that the data is mapped to a suitable range; the original sensor data is Gas, and numerical processing is performed on the data by using a data type conversion method:

[0041] Gas'=(Gas1,Gas2,...Gas Num_Gas ) (9)

[0042] In the formula, Gas' represents the converted gas data, Gas1, Gas2,...Gas Num_Gas represent the numerical format of Num_Gas gas samples.

[0043] Further, the step (3) is implemented as follows:

[0044] It can be seen that the image part uses SIFT descriptor and SURF descriptor to extract image local feature points and calculate corresponding feature descriptors, and by fusing the respective advantages of the two descriptors, visual features are further extracted:

[0045]

[0046] In the formula, weight represents a weight function for weighting the contribution of different position pixels, s represents the scale of the feature point, grad magnitude represents the gradient intensity of the pixel, grad orientation represents the gradient direction of the pixel, grad matrix represents the Hessian matrix of the pixel, including the second order derivative information;

[0047] The neural network is used for feature extraction and fusion, and the specific operation process includes forward propagation and loss function calculation; first, a neural network architecture is constructed, and SIFT and SURF feature descriptors are received as input layer; a hidden layer is set, which contains a certain number of neurons and activation functions; finally, the fused feature descriptors are generated as output layer; the selected SIFT feature descriptor is SIFT data =[sift1,sift2,...sift n ], the selected SURF feature descriptor is SURF data =[surf1,surf2,...surf n ], the input of the neural network is set as IN, wherein IN=[SIFT,SURF] represents that SIFT and SURF feature descriptors are connected together as input; the output of the neural network is OUT, and the calculation formula of forward propagation is:

[0048]

[0049] In the formula, Z1 represents a hidden layer input, A1 represents a hidden layer activation function, Z2 represents an output layer input, OUT represents an output layer activation function, w1 and w2 represent weight matrices of the hidden layer and the output layer respectively, b1 and b2 represent bias vectors of the hidden layer and the output layer respectively, and activation_function represents an activation function.

[0050] A cross-entropy loss function is selected to measure the difference between the neural network output and the target output, assuming that the neural network model output is OUT pred , which represents the prediction probability of the model for two categories, and the true probability is OUT true , which represents the true category of the feature descriptor, and its calculation formula is:

[0051]

[0052] In the formula, N represents the number of samples; when OUT true = 1, the first contributes to the cross-entropy loss, and the smaller the cross-entropy loss, the closer the prediction probability OUT pred of the fused feature description by the neural network model to 1, the more accurate the prediction is;

[0053] The weights and biases of the neural network model are updated by the forward propagation algorithm to minimize the loss function, and the fused feature value is obtained from the output layer, and its calculation formula is:

[0054]

[0055] The fused feature value Voutput obtained by training and adjusting the parameters of the neural network model can better adapt to the input of the YOLOv7 algorithm.

[0056] Further, the step (4) is implemented as follows:

[0057] In infrared image processing, the gradient pattern extraction method is used to calculate edge information by capturing the texture in the image, which has a good response to hot spots and object boundaries in infrared images; first, the Sobel operator is used to calculate the gradient value and direction gradient of the infrared image, and 0-360 degrees are divided into 8 directions, then the gradient direction of each pixel is qualitatively defined as one of the 8 directions, and finally, the gradient frequency distribution of each direction in the image is counted to obtain the gradient pattern feature; for a gray image I(x,y), the gradient calculation can be realized by convolution in the following two directions, as shown in formula (14):

[0058]

[0059] In the formula, G_x(x',y') represents the gradient in the horizontal direction, and G_y(x',y') represents the gradient in the vertical direction;

[0060] Then, the gradient magnitude Mag(x',y') and gradient direction Angle(x',y') are calculated, as shown in equation (15):

[0061]

[0062] The gradient direction Angle(x',y') is divided into multiple directional intervals. For example, 0 to 360 degrees is divided into 8 directions: 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°. The gradient direction Angle(x',y') of each pixel is quantized into one of these directions, that is, continuous angles are mapped to discrete directional labels. The gradient frequency distribution of each direction in the image is statistically analyzed to obtain the gradient pattern feature C. For each pixel, the frequency of its gradient direction is recorded and represented in the form of a histogram. In this way, the gradient pattern feature of the image is obtained, which is used to describe the texture edge information and hotspot locations in the infrared image.

[0063] Furthermore, the implementation process of step (6) is as follows:

[0064] The features of visible light and infrared images are weighted pixel by pixel. The feature vector of the visible light image is V = [v1, v2, ... v2]. n The feature vector of the infrared image is C = [c1, c2, ... c]. n Then the fused feature vector Fusion is:

[0065] Fusion=[v1*α+c1*(1-α),v2*α+c2*(1-α),...,v n *α+c n *(1-α)] (16)

[0066] In the formula, α is an adjustable weighting parameter used to control the importance of features in visible light and infrared images, and its value ranges from [0,1].

[0067] After data fusion, an attention mechanism is introduced to weight the fused feature vector, calculating the correlation and weights between the data. The fused feature vector is Fusion = [fu1, fu2, ..., fu...]. n Use an attention weight vector K = [k1, k2, ..., k n The feature vectors are weighted to obtain the final fused feature vector Fusion'; a self-attention mechanism is used to train the attention weights K:

[0068]

[0069] In the formula, Score(fu_m, Fusion) represents the similarity score between the mth feature vector and the entire feature vector set Fusion, · represents the dot product operation of the matrix, Fusion T represents the transpose of the feature vector set Fusion, fu_m represents the mth fused feature vector, K represents the attention weight of the mth feature vector, and Fusion' represents the final weighted fusion feature vector.

[0070] Advantages: Compared with the prior art, the advantages of the present application: the present application provides more rich and accurate target detection information through multi-modal data fusion and feature enhancement; multi-modal data fusion and feature enhancement can fully utilize the advantages of different data sources, improve the robustness of the algorithm, make it work stably in different environments and conditions, reduce the response time, the present application can more accurately judge the fire risk, effectively warn and trigger the corresponding emergency response measures. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is the flowchart of the present application; DETAILED DESCRIPTION

[0072] The present application will be further described in detail below in combination with the drawings.

[0073] As Figure 1 shown, the present application proposes a method for detecting smoke and flame in factory pipelines based on multi-modal data, which specifically includes the following steps:

[0074] Step 1: Pre-acquire multi-modal data of different types of smoke and flame in the factory pipeline, and pre-process the multi-modal data.

[0075] In the factory, due to the different shapes and colors of smoke and flame, different types of smoke and flame need to be collected for their multi-modal data. For visible light images, visible light cameras are used to collect image data; for infrared images, thermal imagers and other devices are used to collect infrared data; for smoke data, smoke sensors are used to collect smoke concentration.

[0076] Since different types of data sources may have certain positional bias or time difference, data alignment is needed to make different data have the same timestamp or spatial position. The present application uses the timestamp method to synchronize and align the data, which has two data sources Data1 and Data2, and their timestamps are t_Data1 and t_Data2 respectively. Taking the timestamp of data source Data1 as the reference time, the timestamp of data source Data2 can be calculated by formula (1):

[0077]

[0078] where t_Data2_aligned represents the timestamp of the aligned data source t_Data2, and δ represents the time delay of the data source t_Data2 relative to the data source Data1. The data source Data1 is a data set from smoke and flame in the network, the data source t_Data2 is a data set from smoke and flame photographed in the factory, and Data represents the total data set.

[0079] For the visible light image data in the total data set Data, a Gaussian filter is used to denoise the image. This is achieved by performing a weighted average of the pixels in the image captured by the visible light camera. The Gaussian filter uses a two-dimensional Gaussian kernel to smooth the image, and its calculation formula (2) is as follows:

[0080]

[0081] where K(x, y) represents the value of the Gaussian kernel, x and y represent the pixel position coordinates, and σ represents the standard deviation of the Gaussian kernel.

[0082] For each pixel in the image, its neighborhood pixels are taken and a weighted average of the neighborhood pixels is performed according to the weights of the Gaussian kernel to obtain the new value of the pixel. The size of the kernel and the standard deviation are two important parameters of the Gaussian filter. The size of the kernel determines the degree of smoothing, and the standard deviation determines the range of the weight distribution. The process of Gaussian filter denoising can be represented by formula (3) as follows:

[0083] D'(x, y) = ∑(D(x+i, y+j)*K(i, j)) (3)

[0084] where D'(x, y) represents the pixel value of the image after Gaussian filtering, D(x+i, y+j) represents the original value of each field pixel in the image, K(i, j) represents the value of the Gaussian kernel, and [i, j] represents the coordinate position of the kernel relative to the center point. By removing the noise in the image through Gaussian filtering, the image is enhanced, the image quality and visual features are improved, and a good condition basis is laid for subsequent feature fusion.

[0085] In order to increase the diversity and robustness of the data samples, data augmentation techniques are used. For image data, random flipping, random rotation, random cropping and other enhancement operations can be performed. The image data after data conversion is randomly rotated in the horizontal or vertical direction to enhance the perspective transformation of the image data, so that the model has better recognition ability for different directions of the image, and its calculation formula (4) is as follows:

[0086]

[0087] In the formula, F H represents the horizontally flipped image, F V represents the vertically flipped image, and random(0, 1) represents a random number. The image is flipped left and right according to the generated random number. If the random number is 1, the image is flipped left and right, otherwise no processing is performed.

[0088] The operation of random rotation is also implemented while performing random flipping, and the image is rotated by a random angle. A rotation angle is randomly generated within a specified angle range, and the image is rotated according to the generated random angle. The operation is performed using an image rotation function, and image rotation can be implemented through an affine transformation matrix. For a two-dimensional image, the affine transformation matrix is as shown in formula (5):

[0089]

[0090] In the formula, θ represents the rotation angle, t x and t y represent the number of pixels of translation. When the rotation center is the image center, t x and t y can be set to zero. The pixel coordinates on the original image are (x, y), and the new coordinates after affine transformation are (x', y'), which can be calculated by the following equation (6):

[0091]

[0092] where M 11 , M 12 , M 13 , M 21 , M 22 , M 23 are elements of the affine transformation matrix (5). For image rotation, M 11 and M 22 are the cosine and sine values of rotation, respectively, while M 12 and M 21 are the negative sine and cosine values, respectively. M 13 and M 23 can be used to adjust the translation after rotation.

[0093] Step 2: Convert different types of data into a format suitable for YOLOv7 input.

[0094] After data alignment, denoising, and data enhancement, the different types of data are converted into a format suitable for model input. The pixel values of the image are converted to numerical data, the pixel values are represented using floating-point data, and the pixel values are mapped to the range [0, 1]. The channel order of the image is converted from the usual RGB (red, green, blue) to the BGR order required by the model. The converted image data is generated as a tensor represented by a multi-dimensional array. For batch image input, a 4-dimensional tensor can be formed with a shape of [batch size, height, width, channel number]. Let the original image be Pic, and its size be formula (7):

[0095] Pic=High*Weight*Count (7)

[0096] In the formula, High represents the height, Weight represents the width, and Count represents the channel number. Then the image is converted to a tensor as formula (8):

[0097]

[0098] In the formula, Num pic represents the number of images. For images with different pixel value ranges, normalization operation is also needed to scale the pixel values to the range [0, 1] to facilitate model training.

[0099] For smoke data, the original physical quantity data is obtained by sensor equipment and preprocessed, such as calibration, filtering, and denoising, etc., and the sensor data is converted to floating-point numerical data. According to the measurement accuracy and range of the sensor, the data is linearly scaled so that the data is mapped to a suitable range. Let the original sensor data be Gas, and the numerical processing of the data is performed by data type conversion, and the conversion formula (9) is:

[0100] Gas'=(Gas1,Gas2,...Gas Num_Gas ) (9)

[0101] In the formula, Gas' represents the converted gas data, Gas1, Gas2,..., Gas Num_Gas represent the numerical format of Num_Gas gas samples.

[0102] Step 3: For visible image part, SIFT descriptor and SURF descriptor are used to extract image local feature points, and corresponding feature descriptors are calculated, and visual features are extracted by fusing two kinds of descriptors.

[0103] The visible image part adopts SIFT descriptor and SURF descriptor to extract image local feature points and calculate corresponding feature descriptors. By fusing the respective advantages of the two descriptors, the strong performance of SIFT in scale invariance and rotation invariance and the advantages of SURF in calculation speed and robustness are exerted, so as to further extract visual features, and the calculation formula (10) is as follows:

[0104]

[0105] In the formula, weight represents a weight function for weighting the contribution of pixels at different positions, s represents the scale of the feature point, grad magnitude represents the gradient intensity of the pixel, grad orientation represents the gradient direction of the pixel, and grad matrix represents the Hessian matrix of the pixel, including second-order derivative information.

[0106] The neural network is used for feature extraction and fusion, and the specific operation process includes forward propagation and loss function calculation. First, a neural network architecture is constructed, and SIFT and SURF feature descriptors are received as input layers; a hidden layer is set, which contains a certain number of neurons and an activation function; and finally, the fused feature descriptors are generated as output layers. Assuming that the selected SIFT feature descriptor is SIFT data =[sift1, sift2,... sift n ], the selected SURF feature descriptor is SURF data =[surf1, surf2,... surf n ], the input of the neural network is set as IN, wherein IN=[SIFT, SURF] indicates that the SIFT and SURF feature descriptors are connected together as input. The output of the neural network is OUT, and the calculation formula (11) of the forward propagation is as follows:

[0107]

[0108] In the formula, Z1 represents the hidden layer input, A1 represents the hidden layer activation function, Z2 represents the output layer input, OUT represents the output layer activation function, w1 and w2 represent the weight matrices of the hidden layer and the output layer respectively, b1 and b2 represent the bias vectors of the hidden layer and the output layer respectively, and activation_function represents the activation function.

[0109] The cross-entropy loss function is selected to measure the difference between the output of the neural network and the target output. Assuming that the output of the neural network model is OUT pred , which represents the prediction probability of the model for two categories, and the true probability is OUT true, which represents the true class of the feature descriptor (0 or 1), and its calculation formula (12) is:

[0110]

[0111] where N represents the number of samples. When OUT true = 1, the first contribution to the cross-entropy loss, and the smaller the cross-entropy loss, the closer the predicted probability OUT pred of the fused feature value to 1, the more accurate the prediction.

[0112] The weights and biases of the neural network model are updated by the forward propagation algorithm to minimize the loss function, and the fused feature value is obtained from the output layer, and its calculation formula (13) is:

[0113]

[0114] The fused feature value Voutput obtained by training and adjusting the parameters of the neural network model can better adapt to the input of the YOLOv7 algorithm.

[0115] Step 4: Use the gradient pattern extraction method for infrared images to calculate edge information by capturing texture in the image, extract hot spots and object boundaries in the infrared image.

[0116] The gradient pattern extraction method is used in infrared image processing to calculate edge information by capturing texture in the image, which has a good response to hot spots and object boundaries in the infrared image. First, the Sobel operator is used to calculate the gradient value and direction gradient of the infrared image, and 0-360 degrees are divided into 8 directions, then the gradient direction of each pixel is qualitatively defined as one of the 8 directions, and finally, the gradient frequency distribution of each direction in the image is counted to obtain the gradient pattern feature. For a gray image I(x,y), the gradient can be calculated by convolution in the following two directions, as formula (14):

[0117]

[0118] where G_x(x',y') represents the horizontal gradient and G_y(x',y') represents the vertical gradient.

[0119] Then, calculate the gradient magnitude Mag(x',y') and gradient direction Angle(x',y'), as formula (15):

[0120]

[0121] The gradient direction Angle(x', y') is divided into multiple direction intervals, for example, 0-360 degrees are divided into 8 directions: 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°. The gradient direction Angle(x', y') of each pixel is quantized into one of these directions, that is, the continuous angle is mapped to a discrete direction label. The gradient frequency distribution of each direction in the image is counted to obtain the gradient pattern feature C. For each pixel, the frequency of its gradient direction is recorded in the form of a histogram. In this way, the gradient pattern feature of the image is obtained, which is used to describe the texture edge information and hot spot position in the infrared image.

[0122] Step 5: For smoke data, directly use the smoke concentration feature as the model input.

[0123] For smoke data, the smoke concentration feature detected by the smoke sensor can be directly used as input without additional calculation, and these results are directly used as input for the detection algorithm. In the detection of multi-modal data, other sensors and image data are combined to improve the accuracy and reliability of the smoke and flame detection algorithm.

[0124] Step 6: First, use element-wise operations to fuse the two images, and then use attention mechanisms to weight the fused data.

[0125] Different fusion strategies are used in combination with multi-modal data. First, element-wise operations are used to fuse the two images, and then attention mechanisms are used to weight the fused data. The features of the visible light image and the infrared image are weighted pixel by pixel. Assuming that the feature vector of the visible light image is V = [v1, v2,... v n ], and the feature vector of the infrared image is C = [c1, c2,... c n ], then the fused feature vector Fusion can be represented as formula (16):

[0126] Fusion = [v1*α + c1*(1-α), v2*α + c2*(1-α),..., v n *α + c n *(1-α)] (16)

[0127] In the formula, α is an adjustable weight parameter used to control the importance of the visible light image and the infrared image features, and its value range is [0, 1].

[0128] After data fusion, an attention mechanism is introduced to weight the fused feature vector, and the correlation and weight between the data are calculated. In the fused feature vector Fusion = [fu1, fu2,..., fu n] using an attention weight vector K = [k1, k2,..., k n ] to weight the feature vectors to obtain the final fusion feature vector Fusion'. The attention weight vector K can be learned by an attention model. Common attention models include self-attention mechanism, squeeze-and-excitation (SE) module, etc. In this patent, the self-attention mechanism is used to train the attention weight K, and the calculation formula (17) is as follows:

[0129]

[0130] In the formula, Score(fu_m, Fusion) represents the similarity score between the mth feature vector and the entire feature vector set Fusion, · represents the dot product operation of the matrix, Fusion T represents the transpose of the feature vector set Fusion, fu_m represents the mth fused feature vector, K represents the attention weight of the mth feature vector, and Fusion' represents the weighted final fusion feature vector.

[0131] After multi-modal data fusion, the mixed model fused features and smoke data are delivered to the input end of the Yolov7 algorithm for target detection, i.e. detecting smoke and flame in the factory pipeline. According to the detection result and feature information, it is judged whether there is a dangerous situation, and the fire warning of the factory pipeline is given.

Claims

1. A method for smoke and flame detection in a factory pipeline based on multi-modal data, characterized in that, The method comprises the following steps: (1) pre-acquire multi-modal data of different types of smoke and fire in the factory pipeline, and pre-process the multi-modal data; (2) convert different types of data into a format suitable for YOLOv7 input; (3) for the visible image part, SIFT descriptor and SURF descriptor are used to extract image local feature points, and corresponding feature descriptors are calculated, and visual features are further extracted by fusing the two descriptors; (4) for the infrared image, a gradient mode extraction method is used to calculate edge information by capturing the texture in the image, and hot spots and object boundaries in the infrared image are extracted; (5) for smoke data, smoke concentration features are directly used as model input; (6) the two images are fused by using element-by-element operation, and the fused data are weighted by using attention mechanism; (7) after multi-modal data fusion, the fused features and smoke data are input into the input end of the Yolov7 algorithm for target detection, that is, smoke and fire in the factory pipeline are detected; The step (3) is implemented as follows: SIFT descriptor and SURF descriptor are used to extract image local feature points, and corresponding feature descriptors are calculated, and visual features are further extracted by fusing the respective advantages of the two descriptors: where weight denotes a weight function that weights the contribution of different position pixels, (x',y') is the new coordinate of the original image (x,y) after affine transformation, s denotes the scale of the feature point, grad magnitude denotes the gradient intensity of the pixel, grad orientation denotes the gradient direction of the pixel, grad matrix denotes the Hessian matrix of the pixel, including the second order partial derivative information; The feature extraction and fusion using the neural network includes a forward propagation and a loss function calculation. First, a neural network architecture is constructed, and SIFT and SURF feature descriptors are received as input layers. A hidden layer is set, and a certain number of neurons and an activation function are included. Finally, a fused feature descriptor is generated as an output layer. The selected SIFT feature descriptor is SIFT data =[sift1, sift2,... sift n ], the selected SURF feature descriptor is SURF data =[surf1, surf2,... surf n ], the input of the neural network is set as IN, wherein IN=[SIFT, SURF] indicates that the SIFT and SURF feature descriptors are connected together as the input; the output of the neural network is OUT, and a calculation formula of the forward propagation is: In the formula, Z1 represents hidden layer input, A1 represents hidden layer activation function, Z2 represents output layer input, OUT represents output layer activation function, w1 and w2 represent weight matrices of the hidden layer and the output layer respectively, b1 and b2 represent bias vectors of the hidden layer and the output layer respectively, and activation_function represents an activation function; A cross-entropy loss function is selected to measure the difference between the neural network output and the target output, assuming the neural network model output is OUT pred , representing the model's predicted probability for the two classes, and the true probability is OUT true , representing the true class of the feature descriptor, which is calculated as: In the formula, N represents the number of samples; when OUT true = 1, the first cross-entropy loss, and the smaller the cross-entropy loss, the closer the predicted probability OUT pred of the neural network model to 1 after fusion of the feature description, the more accurate the prediction. The weights and biases of the neural network model are updated by using a forward propagation algorithm to minimize the loss function, and then the fused feature values are obtained from the output layer, and the calculation formula is as follows: The fused feature values V are obtained through training and parameter adjustment of the neural network model.

2. The method of claim 1, wherein, The multi-modal data acquisition process of step (1) is as follows: For visible light images, a visible light camera is used to collect image data; for infrared images, a thermal imager is used to collect infrared data; and for smoke data, a smoke sensor is used to collect smoke concentration.

3. The method of claim 1, wherein, The data pre-processing process of step (1) is as follows: Data synchronization alignment is performed in a time stamp manner, there are two data sources Data1 and Data2, the time stamps are t_Data1 and t_Data2, and the time stamp of the data source Data2 is: In the formula, t_Data2_aligend represents the time stamp of the aligned data source t_Data2, and δ represents the time delay of the data source t_Data2 relative to the data source Data1; the data source Data1 is a data set of smoke and fire from the network, the data source t_Data2 is a data set of smoke and fire shot in the factory, and Data represents the total data set; For the visible light image data in the total data set Data, a Gaussian filter is used for image denoising processing; the Gaussian filter is realized by weighted average of each pixel in the image collected by the visible light camera and the surrounding pixels, and a two-dimensional Gaussian kernel is used for image smoothing processing: In the formula, K(x, y) represents the value of the Gaussian kernel, x and y represent the pixel position coordinates, and sigma represents the standard deviation of the Gaussian kernel; For each pixel in the image, the surrounding neighborhood pixels are taken, and the neighborhood pixels are weighted and averaged according to the weight of the Gaussian kernel to obtain the new value of the pixel; the Gaussian filter denoising processing is: D'(x, y) = ∑(D(x+i, y+j)*K(i, j)) (3) In the formula, D'(x, y) represents the image pixel value after Gaussian filtering, D(x+i, y+j) represents the original value of each field pixel in the image, K(i, j) represents the value of the Gaussian kernel, and [i, j] represents the coordinate position of the kernel relative to the center point. Random flip, random rotation, and random crop enhancement operations are performed on the image data, the image data after data conversion is randomly rotated in the horizontal or vertical direction, the view angle transformation of the image data is enhanced, and the model has better recognition ability for different directions of the image: In the formula, F_H represents the horizontally flipped image, F_V represents the vertically flipped image, and random(0, 1) represents a random number. According to the generated random number, the image is flipped left and right; if the random number is 1, the image is flipped left and right, otherwise no processing is performed; The random rotation operation is also realized while the random flip is performed, and the image is randomly rotated at an angle. A rotation angle is randomly generated from a specified angle range, and the image is rotated according to the generated random angle. The image rotation function is used for operation, and the image rotation is realized through an affine transformation matrix. For a two-dimensional image, the affine transformation matrix is: In the formula, θ represents the rotation angle, and t x and t y t represents the number of pixels translated, when the rotation center is the image center. x and t y Set to zero; the pixel coordinates on the original image are (x, y), and the new coordinates after the affine transformation are (x', y'): where M 11 , M 12 , M 13 , M 21 , M 22 , M 23 are elements of the affine transformation matrix (5), for image rotation M 11 and M 22 are the cosine and sine values of the rotation, respectively, while M 12 and M 21 are the negative sine and cosine values, respectively, and M 13 and M 23 are used to adjust the translation after rotation.

4. The method of claim 1, wherein, The implementation process of step (2) is as follows: The pixel value of the image is converted into numerical data, the pixel value is represented by a floating-point data, and the pixel value is mapped to the range of [0, 1]; the channel order of the image is converted from the usual RGB to the BGR order required by the model; the converted image data is generated as a tensor represented by a multidimensional array; for batch image input, a 4-dimensional tensor is formed with a shape of [batch size, height, width, channel number], and the original image is Pic with a size of: Pic = High * Weight * Count (7) In the formula, High represents the height, Weight represents the width, and Count represents the channel number. The image is converted into a tensor as follows: Num pic represents the number of images; For images with different pixel value ranges, normalization operation is also needed to scale the pixel value to the range of [0, 1]; For the smoke data, the sensor data is converted into floating-point numerical data, and the data is linearly scaled according to the measurement accuracy and range of the sensor to map the data to a suitable range; the original sensor data is Gas, and the data type conversion method is used for numerical processing: Gas' = (Gas1, Gas2,... Gas Num_Gas ) (9) In the formula, Gas' represents converted gas data, Gas1, Gas2,... Gas Num_Gas Num_Gas represents the number of gas samples.

5. The method of claim 1, wherein, The implementation process of step (4) is as follows: The gradient pattern extraction method is used in infrared image processing, which has a good response to hot spots and object boundaries in infrared images by calculating edge information from the texture in the captured image. Firstly, the Sobel operator is used to calculate the gradient value and the directional gradient of the infrared image, and 0-360 degrees are divided into 8 directions, then the gradient direction of each pixel is qualitatively defined as one of the 8 directions, finally, the gradient frequency distribution of each direction in the image is counted to obtain the gradient pattern feature. For the gray image I(x, y), the calculation of the gradient is realized through the convolution of the following two directions, as shown in formula (14): In the formula, G_x(x', y') represents the horizontal direction gradient, and G_y(x', y') represents the vertical direction gradient; Then, the gradient amplitude Mag(x', y') and the gradient direction Angle(x', y') are calculated, as shown in formula (15): The gradient direction Angle(x', y') is divided into multiple direction intervals, 0-360 degrees are divided into 8 directions: 0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°, and the gradient direction Angle(x', y') of each pixel is quantified as one of these directions, that is, the continuous angle is mapped to a discrete direction label; the gradient frequency distribution of each direction in the image is counted to obtain the gradient pattern feature C. For each pixel, the frequency of its gradient direction is recorded in the form of a histogram; the gradient pattern feature of the image is obtained, which is used to describe the texture edge information and hot spot position in the infrared image.

6. The method of claim 1, wherein, The step (6) is realized as follows: The features of the visible light image and the infrared image are pixel by pixel weighted, the feature vector of the visible light image is V=[v1, v2,...v n ], the feature vector of the infrared image is C=[c1, c2,...c n ], and the fused feature vector Fusion is: Fusion=[v1*α+c1*(1-α),v2*α+c2*(1-α),...,v n *a+c n *(1-a)] (16) In the formula, α is an adjustable weight parameter used to control the importance of the visible light image and the infrared image feature, and its value range is [0, 1]; After data fusion, an attention mechanism is introduced to weight the fused feature vector, calculate the correlation and weight between data; in the fused feature vector Fusion=[fu1, fu2,..., fu n ] uses an attention weight vector K=[k1, k2,..., k n ] to weight the feature vector to obtain the final fused feature vector Fusion'; the self-attention mechanism is used to train the attention weight K: wherein Score(fu_m, Fusion) represents the similarity score between the m-th feature vector and the whole feature vector set Fusion, · denotes the dot product operation of matrices, Fusion T denotes the transpose of the feature vector set Fusion, fu_m denotes the m-th fused feature vector, K denotes the attention weight of the m-th feature vector, and Fusion' denotes the weighted final fused feature vector.

Images