Intelligent flame monitoring system and method based on industrial vision technology

Through the flame intelligent monitoring system based on industrial vision technology, using video surveillance and feature fusion technology, the problem of insufficient sensitivity and accuracy of traditional systems is solved, accurate flame identification and rating evaluation is achieved, and fire response efficiency and historical analysis capabilities are improved. .

CN118912528APending Publication Date: 2024-11-08WUXI INSTITUTE OF TECHNOLOGY
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410957740.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional flame monitoring systems rely on manual inspection and simple sensors, with low sensitivity and accuracy, prone to missed or false alarms, and in-depth analysis and historical records, resulting in poor performance in complex environments.

Method used

Adopt a flame intelligent monitoring system based on industrial vision technology, including video surveillance unit, object detection unit, flame evaluation unit, abnormal alarm unit and visualization unit. Through real-time video surveillance, keyframe extraction, edge detection and feature fusion, the accuracy of flame is achieved. Identify and grade evaluation and generate abnormal alarm signals and history.

Benefits of technology

It realizes timely identification and positioning of flames, improves fire response efficiency, reduces losses and risks, and supports post-event accident analysis and safety management through historical video analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118912528A_ABST
    Figure CN118912528A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of flame monitoring, in particular to an intelligent flame monitoring system and method based on the industrial vision technology, and the system comprises a flame positioning unit, a target reinspection unit, a flame evaluation unit, an abnormity alarm unit and a visualization unit, and further comprises a video monitoring unit which is used for carrying out the video monitoring of a target area, and the video processing unit is used for processing the target area to obtain a real-time monitoring video in the target area, and transmitting the real-time monitoring video in the target area to the key frame extraction unit and the target detection unit. The monitoring video in the target area is acquired in real time through the video monitoring unit, the flame is quickly identified and positioned through the target detection unit and the key frame extraction unit, the real-time performance enables the system to respond to a fire event in time, and loss and harm possibly caused by the fire can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of flame monitoring, and in particular to an intelligent flame monitoring system and method based on industrial vision technology. Background Art

[0002] Industrial vision technology refers to the use of computer vision technology and image processing technology to acquire, process and analyze visual information in the industrial production process. It mainly obtains real-time images or video streams through cameras or sensors, and then uses various image processing algorithms and machine learning methods to analyze and interpret these images.

[0003] Traditional systems usually rely on manual inspections or simple sensor alarms, which are prone to missed alarms or false alarms because manual inspections may not be timely or accurate, and the sensitivity and accuracy of sensors are relatively low. In addition, traditional systems' flame recognition usually relies on simple smoke or heat-sensitive elements, which have limited flame recognition capabilities and are easily affected by environmental factors such as smoke and light changes, resulting in false alarms or missed alarms. Traditional systems often lack the ability to deeply analyze and process surveillance videos, and are unable to perform pixel-level analysis or multi-feature fusion flame recognition and evaluation. Therefore, they perform poorly in complex environments or multi-target scenarios. Traditional systems are usually unable to effectively record and analyze historical surveillance videos, so they have limited support for subsequent fire analysis and safety improvements. Summary of the invention

[0004] The purpose of the present invention is to address the problems existing in the background technology and to propose a flame intelligent monitoring system and method based on industrial vision technology.

[0005] The technical solution of the present invention: The flame intelligent monitoring system based on industrial vision technology includes a flame positioning unit, a target re-inspection unit, a flame evaluation unit, an abnormal alarm unit and a visualization unit, and also includes:

[0006] A video monitoring unit, the video monitoring unit is used to perform video monitoring on the target area to obtain real-time monitoring video in the target area, and transmit the real-time monitoring video in the target area to the key frame extraction unit and the target detection unit;

[0007] A key frame extraction unit, wherein the key frame extraction unit receives the real-time monitoring video in the target area transmitted by the video monitoring unit, and performs key frame extraction on the real-time monitoring video in the target area by a key frame extraction method to obtain a plurality of key frames corresponding to the real-time monitoring video, and transmits the plurality of key frames corresponding to the real-time monitoring video to a target review unit;

[0008] A target detection unit receives the real-time monitoring video in the target area transmitted by the video monitoring unit, obtains the latest frame of the real-time monitoring video in the target area, and performs target detection on the latest frame through a trained target detection model to determine whether there is a target flame in the latest frame. If there is a target flame, the target frame position of the target flame and the category of the target in the latest frame are detected, and the latest frame with the target flame is transmitted to the flame evaluation unit, and the target frame position of the target flame and the category of the target in the latest frame are transmitted to the abnormal alarm unit and the visualization unit, and the real-time monitoring video with the target flame is obtained, and the real-time monitoring video is historically recorded to obtain a historical monitoring video, and the historical monitoring video is transmitted to the flame positioning unit.

[0009] Preferably, the flame assessment unit receives the latest frame in which the target flame exists transmitted by the target detection unit, and performs edge detection on the latest frame in which the target flame exists by an edge detection algorithm to obtain a target flame image, and performs feature extraction on the target flame image to obtain an image feature set of the target flame image, wherein the image feature set includes color features, texture features, shape features and wavelet energy features, and performs multi-feature fusion on the image feature set to obtain fusion features of the target flame image, and classifies the fusion features of the target flame image and the image feature set by a trained support vector machine to obtain a flame level corresponding to the target flame, and transmits the flame level corresponding to the target flame to the abnormal alarm unit and the visualization unit.

[0010] Preferably, the target review unit receives multiple key frames corresponding to the real-time monitoring video transmitted by the key frame extraction unit, and performs target detection on the multiple key frames through a trained target detection model to determine whether there is a target flame in the multiple key frames. If there is a target flame, the target frame position of the target flame and the target category in the multiple key frames are detected, and the target frame position of the target flame and the target category in the multiple key frames are transmitted to the abnormal alarm unit and the visualization unit.

[0011] Preferably, the flame locating unit receives the historical monitoring video transmitted by the target detection unit, and detects the video segment containing the target flame in the historical monitoring video through a trained flame detection model to obtain the time boundary of the video segment, and transmits the time boundary of the video segment to the visualization unit.

[0012] Preferably, the abnormal alarm unit receives the target frame position and target category of the target flame in the latest frame transmitted by the target detection unit, the flame level corresponding to the target flame transmitted by the flame evaluation unit, and the target frame position and target category of the target flame in the multiple key frames transmitted by the target review unit, and generates a corresponding abnormal alarm signal based on the target frame position and target category of the target flame in the latest frame, the target frame position and target category of the target flame in the multiple key frames, and the flame level corresponding to the target flame, and makes an abnormal alarm through the alarm device based on the abnormal alarm signal.

[0013] Preferably, the visualization unit receives the target frame position and target category of the target flame in the latest frame transmitted by the target detection unit, the flame level corresponding to the target flame transmitted by the flame evaluation unit, the target frame position and target category of the target flame in the multiple key frames transmitted by the target review unit, and the time boundary of the video segment transmitted by the flame positioning unit, and displays the target flame in the latest frame, the target flame in the multiple key frames, and the flame level corresponding to the target flame, and locates the historical monitoring video based on the time boundary of the video segment to obtain the video segment containing the target flame in the historical monitoring video, and displays the video segment containing the target flame in the historical monitoring video.

[0014] Preferably, the key frame extraction method comprises the following steps:

[0015] A1. Divide the real-time monitoring video in the target area into a plurality of continuous video frames, and calculate the mutual information of all adjacent video frames in the real-time monitoring video;

[0016] A2, constructing a sliding window of a preset size, comparing the mutual information of each of the video frames in the sliding window, and determining whether the mutual information of the currently compared video frame is the minimum value of the mutual information of the video frames in the sliding window, if it is the minimum value, moving the sliding window to the right, otherwise, executing step A3;

[0017] A3. Calculate the local adaptive threshold of the sliding window. The calculation formula of the local adaptive threshold is as follows:

[0018]

[0019] Among them, u i represents the mean, σ i represents the standard deviation, T i represents the local adaptive threshold, I t,t+1represents the mutual information, γ represents the preset parameter, 2n+1 represents the size of the sliding window, and t represents the sequence number of the video frame;

[0020] A4. Compare the mutual information of the video frame currently being compared with the local adaptive threshold. If the mutual information is less than the local adaptive threshold, it indicates that the video frame currently being compared is a mutation frame, and move the sliding window to the right until the mutual information of the video frame currently being compared is not less than the local adaptive threshold, thereby obtaining a gradual end frame, and use the mutation frame and the gradual end frame as key frames.

[0021] Preferably, performing feature extraction on the target flame image to obtain an image feature set of the target flame image comprises the following steps:

[0022] B1. Perform color feature extraction on the target flame image to obtain the color feature of the target flame image. The color feature extraction formula is as follows:

[0023]

[0024] Among them, V, S and H represent hue, saturation and brightness respectively, and R, G and B represent the RGB values ​​of the target flame image;

[0025] B2. Extracting texture features of the target flame image to obtain texture features of the target flame image. The texture feature extraction formula is as follows:

[0026]

[0027] Among them, L B x(x c ,y c ) represents texture features, m represents the number of pixels, g i Represents the gray value of adjacent pixels, g c represents the center gray value, (x c ,y c ) represents the center pixel, represents a symbolic function, and

[0028] B3. Perform shape feature extraction on the target flame image to obtain the shape feature of the target flame image. The shape feature extraction formula is as follows:

[0029]

[0030] Among them, G(x,y) represents the gradient amplitude at the pixel point (x,y), α(x,y) represents the gradient direction at the pixel point (x,y), and G x(x,y) and G y (x,y) represents the gradient magnitude of the pixel point (x,y) in the x and y directions respectively, and H(x,y) represents the pixel value at the pixel point (x,y);

[0031] B4. Perform wavelet energy feature extraction on the target flame image to obtain the wavelet energy feature of the target flame image. The wavelet energy feature extraction formula is as follows:

[0032]

[0033] Where e represents the wavelet energy feature, M and N represent the size of the channel coefficient matrix, and x(m,n) represents the value at row m and column n in the channel coefficient matrix;

[0034] B5. Combining the color feature, the texture feature, the shape feature and the wavelet energy feature to obtain an image feature set of the target flame image.

[0035] Preferably, the image feature set is subjected to multi-feature fusion to obtain the fusion feature of the target flame image, comprising the following steps:

[0036] C1. Perform block recognition on the image feature set, and select feature blocks based on the block feature dimension to obtain multi-dimensional sample features. The feature block selection formula is as follows:

[0037]

[0038] Among them, R(C), R(S), R(E) and R(t) represent the recognition rates of the final selected feature blocks of color, shape, wavelet energy and texture respectively. i )、R(S j )、R(E u ) and R(T k ) represent the recognition rates of color, shape, wavelet energy and texture feature blocks respectively, D(t) and D(T) represent the dimensions of texture candidate and final feature blocks respectively, and τ represents the feature block tolerance;

[0039] C2. Extract the optimal principal component from the multidimensional sample features by principal component analysis method. The optimal principal component extraction requirement formula is as follows:

[0040]

[0041] Among them, C(y o )、R(y o ) and V(y o) represent the contribution rate, recognition rate and rising speed of the principal component, T represents the threshold of the contribution rate of the principal component, η represents the tolerance, R(y s ) and R(y r ) represents the recognition rate of the first s and first r principal components;

[0042] C3. Calculate the weights of the optimal principal components of color, shape, wavelet energy and texture. The weight formula of the optimal principal components is as follows:

[0043]

[0044] Among them, ω θ represents the weights of the optimal principal components of color, shape, wavelet energy and texture, Represents the recognition rate of color, shape, wavelet energy and texture principal components;

[0045] C4. Performing weighted averaging on the color, shape, wavelet energy and texture features based on the weights of the optimal principal components of the color, shape, wavelet energy and texture to obtain a fusion feature of the target flame image.

[0046] Preferably, the flame detection model includes a feature extraction module, a first prediction module and a second prediction module, the feature extraction module uses a three-dimensional convolutional neural network to extract the spatiotemporal features of the historical monitoring video, the first prediction module is used to roughly predict the time boundary of the historical monitoring video to obtain a rough time boundary, the second prediction module is used to finely predict the time boundary of the video segment within the rough time boundary roughly predicted by the first prediction module, and the first prediction module converts the spatiotemporal features f into a coarse time boundary through a fully connected layer. l Divide into positioning features and categorical features And the positioning features are obtained through two temporal convolution layers. and the classification features For each time step in , we predict and generate the rough temporal boundary of the video segment instance for the i-th time step. and classification scores The coarse time boundary is converted into a true coarse time boundary through a time scale scaling module. The second prediction module transforms the positioning features through a fully connected layer Divide into starting features and termination features Based on the true coarse time boundary Extract two control regions separately The boundary pooling module is used to control the two control areas. Find the maximum value of the starting feature within and the termination characteristic maximum The starting feature maximum value and the termination characteristic maximum value splicing to obtain a composite feature, reducing the dimension of the composite feature through a temporal convolution layer to obtain a feature sequence, predicting the offset of the time boundary of the video segment based on the feature sequence, and predicting the offset of the time boundary and the true coarse time boundary based on the offset of the time boundary Gets the fine time boundary.

[0047] Preferably, the true coarse time boundary calculation formula is as follows:

[0048]

[0049] Among them, i represents the i-th time step of the termination feature in the total step length, and l represents the l-th video segment in the historical surveillance video;

[0050] The calculation formulas for the starting feature and the ending feature are as follows:

[0051]

[0052] Among them, σ represents the RELU activation function, GN represents the batch normalization operation, and Conv represents the convolutional layer;

[0053] The control area calculation formula is as follows:

[0054]

[0055] Among them, σ represents the RELU activation function, GN represents the batch normalization operation, and Conv represents the convolutional layer.

[0056] The technical solution of the present invention: a flame intelligent monitoring method based on industrial vision technology, which is applicable to the flame intelligent monitoring system based on industrial vision technology, is characterized in that it includes the following steps:

[0057] S1, real-time monitoring of the target area, obtaining real-time monitoring video, and extracting key frames from the real-time monitoring video;

[0058] S2, use the trained target detection model to analyze the latest monitoring frame, detect whether there is a flame, and locate the position and category of the flame;

[0059] S3, performing edge detection and feature extraction on the detected flame image to obtain an image feature set, fusing the image feature set to obtain a fusion feature, and classifying the fusion feature and the image feature set through a trained support vector machine to determine the level of the flame;

[0060] S4, performing target detection again on multiple key frames to detect whether there is a flame, and locate the position and type of the flame;

[0061] S5, analyzing historical surveillance videos, detecting video segments containing flames, and marking their time boundaries;

[0062] S6. Generate a corresponding abnormal alarm signal based on the detected flame and the flame level, and visualize the monitored flame location, category and historical video segment.

[0063] Compared with the prior art, the above technical solution of the present invention has the following beneficial technical effects:

[0064] 1. The present invention obtains monitoring videos in the target area in real time through the video monitoring unit, and quickly identifies and locates the flame through the target detection unit and the key frame extraction unit. This real-time performance enables the system to respond to fire events in a timely manner, which helps to reduce the losses and hazards that may be caused by the fire. The flame evaluation unit uses the edge detection algorithm and the support vector machine to extract and classify the target flame image, which can effectively distinguish the type and level of the flame. This classification helps to accurately evaluate and process the fire situation, and improves the decision-making efficiency of responding to the fire.

[0065] 2. The present invention generates an abnormal alarm signal based on the target detection result and the level information of the flame assessment through the abnormal alarm unit, and realizes timely alarm notification through the alarm device. This function can immediately detect and respond to the fire in the early stage, which is helpful to quickly take emergency measures and reduce the losses and risks caused by the fire. Through the visualization unit combined with the time boundary information of the flame positioning unit, the historical monitoring video is located and displayed, and the occurrence and development process of the fire incident can be traced back. This function not only helps post-accident analysis and responsibility tracing, but also provides valuable experience and data support for future fire prevention and safety management. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of the overall system flow in an embodiment of the present invention;

[0067] Figure 2 The figure is a flow chart of an overall method in one embodiment of the present invention.

[0068] Figure numerals: 1. Video monitoring unit; 2. Key frame extraction unit; 3. Target detection unit; 4. Flame positioning unit; 5. Target re-inspection unit; 6. Flame evaluation unit; 7. Abnormal alarm unit; 8. Visualization unit. DETAILED DESCRIPTION

[0069] Embodiment 1, as Figure 1As shown, the flame intelligent monitoring system based on industrial vision technology proposed by the present invention includes a flame positioning unit 4, a target re-inspection unit 5, a flame evaluation unit 6, an abnormal alarm unit 7 and a visualization unit 8, and also includes:

[0070] The video monitoring unit 1 is used to perform video monitoring on the target area to obtain real-time monitoring video in the target area, and transmit the real-time monitoring video in the target area to the key frame extraction unit 2 and the target detection unit 3;

[0071] The key frame extraction unit 2 receives the real-time monitoring video in the target area transmitted by the video monitoring unit 1, and extracts key frames from the real-time monitoring video in the target area by a key frame extraction method to obtain a plurality of key frames corresponding to the real-time monitoring video, and transmits the plurality of key frames corresponding to the real-time monitoring video to the target re-inspection unit 5;

[0072] The target detection unit 3 receives the real-time monitoring video in the target area transmitted by the video monitoring unit 1, obtains the latest frame of the real-time monitoring video in the target area, and performs target detection on the latest frame through the trained target detection model to determine whether there is a target flame in the latest frame. If there is a target flame, the target frame position of the target flame and the category of the target in the latest frame are detected, and the latest frame with the target flame is transmitted to the flame evaluation unit 6, and the target frame position of the target flame in the latest frame and the category of the target are transmitted to the abnormal alarm unit 7 and the visualization unit 8, and the real-time monitoring video with the target flame is obtained, and the real-time monitoring video is historically recorded to obtain the historical monitoring video, and the historical monitoring video is transmitted to the flame positioning unit 4.

[0073] In an optional embodiment, the flame evaluation unit 6 receives the latest frame with the target flame transmitted by the target detection unit 3, and performs edge detection on the latest frame with the target flame through an edge detection algorithm to obtain a target flame image, and performs feature extraction on the target flame image to obtain an image feature set of the target flame image, the image feature set includes color features, texture features, shape features and wavelet energy features, and performs multi-feature fusion on the image feature set to obtain a fusion feature of the target flame image, and classifies the fusion feature and the image feature set of the target flame image through a trained support vector machine to obtain a flame level corresponding to the target flame, and transmits the flame level corresponding to the target flame to the abnormal alarm unit 7 and the visualization unit 8.

[0074] It should be noted that support vector machine is a supervised learning model widely used in machine learning, mainly used for classification and regression tasks. Its basic idea is to find a hyperplane (a straight line in two-dimensional space and a hyperplane in higher-dimensional space) to separate data points of different categories, while making the distance from this hyperplane to the nearest data point as far as possible. These nearest data points are called support vectors.

[0075] In an optional embodiment, the target review unit 5 receives multiple key frames corresponding to the real-time monitoring video transmitted by the key frame extraction unit 2, and performs target detection on the multiple key frames through a trained target detection model to determine whether there is a target flame in the multiple key frames. If there is a target flame, the target frame position of the target flame and the target category in the multiple key frames are detected, and the target frame position of the target flame and the target category in the multiple key frames are transmitted to the abnormal alarm unit 7 and the visualization unit 8.

[0076] It should be noted that target detection models usually use deep learning methods, especially convolutional neural networks, including but not limited to FasterR-CNN, YOLO, SSD and MaskR-CNN.

[0077] In an optional embodiment, the flame locating unit 4 receives the historical monitoring video transmitted by the target detection unit 3, and detects the video segment containing the target flame in the historical monitoring video through a trained flame detection model to obtain the time boundary of the video segment, and transmits the time boundary of the video segment to the visualization unit 8.

[0078] In an optional embodiment, the abnormal alarm unit 7 receives the target frame position and target category of the target flame in the latest frame transmitted by the target detection unit 3, the flame level corresponding to the target flame transmitted by the flame evaluation unit 6, and the target frame position and target category of the target flame in multiple key frames transmitted by the target review unit 5, and generates a corresponding abnormal alarm signal based on the target frame position and target category of the target flame in the latest frame, the target frame position and target category of the target flame in multiple key frames, and the flame level corresponding to the target flame, and makes an abnormal alarm through the alarm device based on the abnormal alarm signal.

[0079] In an optional embodiment, the visualization unit 8 receives the target frame position of the target flame and the target category in the latest frame transmitted by the target detection unit 3, the flame level corresponding to the target flame transmitted by the flame evaluation unit 6, the target frame position of the target flame in multiple key frames transmitted by the target review unit 5 and the target category, and the time boundary of the video segment transmitted by the flame positioning unit 4, and displays the target flame in the latest frame, the target flame in multiple key frames and the flame level corresponding to the target flame, and locates the historical monitoring video based on the time boundary of the video segment to obtain the video segment containing the target flame in the historical monitoring video, and displays the video segment containing the target flame in the historical monitoring video.

[0080] Embodiment 2, the flame intelligent monitoring system based on industrial vision technology proposed by the present invention, compared with embodiment 1, this embodiment also includes: a key frame extraction method, comprising the following steps:

[0081] A1. Divide the real-time monitoring video in the target area into multiple continuous video frames, and calculate the mutual information of all adjacent video frames in the real-time monitoring video;

[0082] A2, constructing a sliding window of a preset size, comparing the mutual information of each video frame in the sliding window, and determining whether the mutual information of the currently compared video frame is the minimum value of the mutual information of the video frames in the sliding window. If it is the minimum value, moving the sliding window to the right, otherwise, executing step A3;

[0083] A3. Calculate the local adaptive threshold of the sliding window. The calculation formula of the local adaptive threshold is as follows:

[0084]

[0085] Among them, u i represents the mean, σ i represents the standard deviation, T i represents the local adaptive threshold, I t,t+1 represents the mutual information, γ represents the preset parameter, 2n+1 represents the size of the sliding window, and t represents the sequence number of the video frame;

[0086] A4. Compare the mutual information of the currently compared video frame with the local adaptive threshold. If it is less than the local adaptive threshold, it indicates that the currently compared video frame is a mutation frame, and move the sliding window to the right until the mutual information of the currently compared video frame is not less than the local adaptive threshold, thereby obtaining a gradual end frame, and taking the mutation frame and the gradual end frame as key frames.

[0087] In this embodiment, the mutual information is a statistic that measures the degree of association between two random variables and is used to evaluate the similarity or degree of change between video frames; the sliding window of preset size is a window of fixed size, which slides on the video frame sequence and is used to compare the mutual information of different video frames; the local adaptive threshold represents a threshold calculated based on the mutual information in the sliding window, which is used to determine whether the currently compared video frame meets the condition of a key frame; the mutation frame represents a frame in the video frame where a significant change occurs, and the mutual information of the previous and next frames is usually quite different; the gradual change end frame represents a frame in the video frame where the gradual change process ends, and its mutual information begins to meet the local adaptive threshold.

[0088] In an optional embodiment, performing feature extraction on the target flame image to obtain an image feature set of the target flame image includes the following steps:

[0089] B1. Perform color feature extraction on the target flame image to obtain the color feature of the target flame image. The color feature extraction formula is as follows:

[0090]

[0091] Among them, V, S and H represent hue, saturation and brightness respectively, and R, G and B represent the RGB values ​​of the target flame image;

[0092] B2. Extract texture features of the target flame image to obtain the texture features of the target flame image. The texture feature extraction formula is as follows:

[0093]

[0094] Among them, L BP (x c ,y c ) represents texture features, m represents the number of pixels, g i Represents the gray value of adjacent pixels, g c represents the center gray value, (x c ,y c ) represents the center pixel, represents a symbolic function, and

[0095] B3. Perform shape feature extraction on the target flame image to obtain the shape feature of the target flame image. The shape feature extraction formula is as follows:

[0096]

[0097] Among them, G(x,y) represents the gradient amplitude at the pixel point (x,y), α(x,y) represents the gradient direction at the pixel point (x,y), and G x(x,y) and G y (x,y) represents the gradient magnitude of the pixel point (x,y) in the x and y directions respectively, and H(x,y) represents the pixel value at the pixel point (x,y);

[0098] B4. Perform wavelet energy feature extraction on the target flame image to obtain the wavelet energy feature of the target flame image. The wavelet energy feature extraction formula is as follows:

[0099]

[0100] Where e represents the wavelet energy feature, M and N represent the size of the channel coefficient matrix, and x(m,n) represents the value at row m and column n in the channel coefficient matrix;

[0101] B5. Combine color features, texture features, shape features and wavelet energy features to obtain an image feature set of the target flame image.

[0102] It should be noted that the wavelet energy feature represents a channel coefficient matrix obtained by wavelet transform, and the channel coefficient matrix represents energy distribution in different frequencies and directions.

[0103] In an optional embodiment, the image feature set is subjected to multi-feature fusion to obtain fusion features of the target flame image, including the following steps:

[0104] C1. Perform block recognition on the image feature set and select feature blocks based on the block feature dimension to obtain multi-dimensional sample features. The feature block selection formula is as follows:

[0105]

[0106] Among them, R(C), R(S), R(E) and R(t) represent the recognition rates of the final selected feature blocks of color, shape, wavelet energy and texture respectively. i )、R(S j )、R(E u ) and R(T k ) represent the recognition rates of color, shape, wavelet energy and texture feature blocks respectively, D(t) and D(T) represent the dimensions of texture candidate and final feature blocks respectively, and τ represents the feature block tolerance;

[0107] C2. Extract the optimal principal component from the multidimensional sample features through the principal component analysis method. The optimal principal component extraction requirement formula is as follows:

[0108]

[0109] Among them, C(y o )、R(y o) and V(y o ) represent the contribution rate, recognition rate and rising speed of the principal component, T represents the threshold of the contribution rate of the principal component, η represents the tolerance, R(y s ) and R(y r ) represents the recognition rate of the first s and first r principal components;

[0110] C3. Calculate the weights of the optimal principal components of color, shape, wavelet energy and texture. The weight formula of the optimal principal component is as follows:

[0111]

[0112] Among them, ω θ represents the weights of the optimal principal components of color, shape, wavelet energy and texture, Represents the recognition rate of color, shape, wavelet energy and texture principal components;

[0113] C4. Based on the weights of the optimal principal components of color, shape, wavelet energy and texture, the color, shape, wavelet energy and texture features are weighted averaged to obtain the fusion features of the target flame image.

[0114] In an optional embodiment, the flame detection model includes a feature extraction module, a first prediction module and a second prediction module. The feature extraction module uses a three-dimensional convolutional neural network to extract the spatiotemporal features of the historical monitoring video. The first prediction module is used to make a rough prediction of the time boundary of the historical monitoring video to obtain a rough time boundary. The second prediction module is used to make a fine prediction of the time boundary of the video segment within the rough time boundary roughly predicted by the first prediction module. The first prediction module converts the spatiotemporal features f l Divide into positioning features and categorical features And through two temporal convolutional layers to locate the features and categorical features For each time step in , we predict and generate the rough temporal boundary of the video segment instance for the i-th time step. and classification scores The coarse time boundary is converted into a true coarse time boundary through the time scale scaling module The second prediction module uses a fully connected layer to locate the features Divide into starting features and termination features Based on true rough time boundary Extract two control regions separately Through the boundary pooling module in the two control areas Find the maximum value of the starting feature within and the termination characteristic maximum Set the starting feature maximum value and the termination characteristic maximum The composite features are concatenated to obtain composite features, and the composite features are reduced in dimension through the temporal convolution layer to obtain a feature sequence. The offset of the time boundary of the video segment is predicted based on the feature sequence, and the offset of the time boundary and the true coarse time boundary are predicted based on the time boundary offset. Gets the fine time boundary.

[0115] In an optional embodiment, the true coarse time boundary calculation formula is as follows:

[0116]

[0117] Among them, i represents the i-th time step of the termination feature in the total step length, and l represents the l-th video segment in the historical surveillance video;

[0118] The calculation formulas for the start and end features are as follows:

[0119]

[0120] Among them, σ represents the RELU activation function, GN represents the batch normalization operation, and Conv represents the convolutional layer;

[0121] The control area calculation formula is as follows:

[0122]

[0123] Among them, σ represents the RELU activation function, GN represents the batch normalization operation, and Conv represents the convolutional layer.

[0124] Embodiment three, as Figure 2 As shown, the flame intelligent monitoring method based on industrial vision technology, which is applicable to the flame intelligent monitoring system based on industrial vision technology, includes the following steps:

[0125] S1, real-time monitoring of the target area, obtaining real-time monitoring video, and extracting key frames from the real-time monitoring video;

[0126] S2, use the trained target detection model to analyze the latest monitoring frame, detect whether there is a flame, and locate the position and category of the flame;

[0127] S3, performing edge detection and feature extraction on the detected flame image to obtain an image feature set, fusing the image feature set to obtain a fused feature, and classifying the fused feature and the image feature set by a trained support vector machine to determine the level of the flame;

[0128] S4, performing target detection again on multiple key frames to detect whether there is a flame, and locate the position and type of the flame;

[0129] S5, analyzing historical surveillance videos, detecting video segments containing flames, and marking their time boundaries;

[0130] S6. Generate a corresponding abnormal alarm signal based on the detected flame and the flame level, and visualize the monitored flame location, category and historical video segment.

[0131] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A flame intelligent monitoring system based on industrial vision technology, comprising a flame positioning unit (4), a target rechecking unit (5), a flame evaluation unit (6), an abnormal alarm unit (7) and a visualization unit (8), characterized in that: A video monitoring unit (1), the video monitoring unit (1) is used to perform video monitoring on a target area to obtain a real-time monitoring video in the target area, and transmit the real-time monitoring video in the target area to a key frame extraction unit (2) and a target detection unit (3); A key frame extraction unit (2), the key frame extraction unit (2) receives the real-time monitoring video in the target area transmitted by the video monitoring unit (1), and extracts key frames from the real-time monitoring video in the target area using a key frame extraction method to obtain a plurality of key frames corresponding to the real-time monitoring video, and transmits the plurality of key frames corresponding to the real-time monitoring video to a target re-inspection unit (5); A target detection unit (3) receives the real-time monitoring video in the target area transmitted by the video monitoring unit (1), obtains the latest frame of the real-time monitoring video in the target area, and performs target detection on the latest frame using a trained target detection model to determine whether the latest frame has a target flame. If the target flame exists, the target frame position of the target flame and the target category in the latest frame are detected, and the latest frame with the target flame is transmitted to the flame evaluation unit (6), and the target frame position of the target flame and the target category in the latest frame are transmitted to the abnormal alarm unit (7) and the visualization unit (8), and the real-time monitoring video with the target flame is obtained, and the real-time monitoring video is recorded in history to obtain a historical monitoring video, and the historical monitoring video is transmitted to the flame positioning unit (4).

2. The flame intelligent monitoring system based on industrial vision technology according to claim 1 is characterized in that: The flame evaluation unit (6) receives the latest frame in which the target flame exists, which is transmitted by the target detection unit (3), and performs edge detection on the latest frame in which the target flame exists by using an edge detection algorithm to obtain a target flame image, and performs feature extraction on the target flame image to obtain an image feature set of the target flame image, wherein the image feature set includes color features, texture features, shape features and wavelet energy features, performs multi-feature fusion on the image feature set to obtain fusion features of the target flame image, and classifies the fusion features of the target flame image and the image feature set by using a trained support vector machine to obtain a flame level corresponding to the target flame, and transmits the flame level corresponding to the target flame to the abnormal alarm unit (7) and the visualization unit (8); The target recheck unit (5) receives a plurality of key frames corresponding to the real-time monitoring video transmitted by the key frame extraction unit (2), and performs target detection on the plurality of key frames using a trained target detection model to determine whether a target flame exists in the plurality of key frames. If a target flame exists, the target frame position of the target flame and the target category in the plurality of key frames are detected, and the target frame position of the target flame and the target category in the plurality of key frames are transmitted to the abnormal alarm unit (7) and the visualization unit (8).

3. The flame intelligent monitoring system based on industrial vision technology according to claim 2 is characterized in that: The flame positioning unit (4) receives the historical monitoring video transmitted by the target detection unit (3), and detects the video segment containing the target flame in the historical monitoring video by using the trained flame detection model to obtain the time boundary of the video segment, and transmits the time boundary of the video segment to the visualization unit (8); The abnormal alarm unit (7) receives the target frame position and target category of the target flame in the latest frame transmitted by the target detection unit (3), the flame level corresponding to the target flame transmitted by the flame evaluation unit (6), and the target frame position and target category of the target flame in the multiple key frames transmitted by the target re-inspection unit (5), and generates a corresponding abnormal alarm signal based on the target frame position and target category of the target flame in the latest frame, the target frame position and target category of the target flame in the multiple key frames, and the flame level corresponding to the target flame, and performs an abnormal alarm through an alarm device based on the abnormal alarm signal.

4. The flame intelligent monitoring system based on industrial vision technology according to claim 3 is characterized in that: The visualization unit (8) receives the target frame position and target category of the target flame in the latest frame transmitted by the target detection unit (3), the flame level corresponding to the target flame transmitted by the flame evaluation unit (6), the target frame position and target category of the target flame in the multiple key frames transmitted by the target recheck unit (5), and the time boundary of the video segment transmitted by the flame positioning unit (4), and displays the target flame in the latest frame, the target flame in the multiple key frames, and the flame level corresponding to the target flame, and locates the historical monitoring video based on the time boundary of the video segment to obtain the video segment containing the target flame in the historical monitoring video, and displays the video segment containing the target flame in the historical monitoring video.

5. The flame intelligent monitoring system based on industrial vision technology according to claim 1 is characterized in that: The key frame extraction method comprises the following steps: A1. Divide the real-time monitoring video in the target area into a plurality of continuous video frames, and calculate the mutual information of all adjacent video frames in the real-time monitoring video; A2, constructing a sliding window of a preset size, comparing the mutual information of each of the video frames in the sliding window, and determining whether the mutual information of the currently compared video frame is the minimum value of the mutual information of the video frames in the sliding window, if it is the minimum value, moving the sliding window to the right, otherwise, executing step A3; A3. Calculate the local adaptive threshold of the sliding window. The calculation formula of the local adaptive threshold is as follows: Among them, u i represents the mean, σ i represents the standard deviation, T i represents the local adaptive threshold, I t,t+1 represents the mutual information, γ represents the preset parameter, 2n+1 represents the size of the sliding window, and t represents the sequence number of the video frame; A4. Compare the mutual information of the video frame currently being compared with the local adaptive threshold. If the mutual information is less than the local adaptive threshold, it indicates that the video frame currently being compared is a mutation frame, and move the sliding window to the right until the mutual information of the video frame currently being compared is not less than the local adaptive threshold, thereby obtaining a gradual end frame, and use the mutation frame and the gradual end frame as key frames.

6. The flame intelligent monitoring system based on industrial vision technology according to claim 2 is characterized in that: Performing feature extraction on the target flame image to obtain an image feature set of the target flame image comprises the following steps: B1. Perform color feature extraction on the target flame image to obtain the color feature of the target flame image. The color feature extraction formula is as follows: Among them, V, S and H represent hue, saturation and brightness respectively, and R, G and B represent the RGB values ​​of the target flame image; B2. Extracting texture features of the target flame image to obtain texture features of the target flame image. The texture feature extraction formula is as follows: Among them, L BP (x c ,y c ) represents texture features, m represents the number of pixels, g i Represents the gray value of adjacent pixels, g c represents the center gray value, (x c ,y c ) represents the center pixel, represents a symbolic function, and B3. Perform shape feature extraction on the target flame image to obtain the shape feature of the target flame image. The shape feature extraction formula is as follows: Among them, G(x,y) represents the gradient amplitude at the pixel point (x,y), α(x,y) represents the gradient direction at the pixel point (x,y), and G x (x,y) and G y (x,y) represents the gradient magnitude of the pixel point (x,y) in the x and y directions respectively, and H(x,y) represents the pixel value at the pixel point (x,y); B4. Perform wavelet energy feature extraction on the target flame image to obtain the wavelet energy feature of the target flame image. The wavelet energy feature extraction formula is as follows: Where e represents the wavelet energy feature, M and N represent the size of the channel coefficient matrix, and x(m,n) represents the value at row m and column n in the channel coefficient matrix; B5. Combining the color feature, the texture feature, the shape feature and the wavelet energy feature to obtain an image feature set of the target flame image.

7. The flame intelligent monitoring system based on industrial vision technology according to claim 2 is characterized in that: The image feature set is subjected to multi-feature fusion to obtain the fusion feature of the target flame image, comprising the following steps: C1. Perform block recognition on the image feature set, and select feature blocks based on the block feature dimension to obtain multi-dimensional sample features. The feature block selection formula is as follows: Among them, R(C), R(S), R(E) and R(t) represent the recognition rates of the final selected feature blocks of color, shape, wavelet energy and texture respectively. i )、R(S j )、R(E u ) and R(T k ) represent the recognition rates of color, shape, wavelet energy and texture feature blocks respectively, D(t) and D(T) represent the dimensions of texture candidate and final feature blocks respectively, and τ represents the feature block tolerance; C2. Extract the optimal principal component from the multidimensional sample features by principal component analysis method. The optimal principal component extraction requirement formula is as follows: Among them, C(y o )、R(y o ) and V(y o ) represent the contribution rate, recognition rate and rising speed of the principal component, T represents the threshold of the contribution rate of the principal component, η represents the tolerance, R(y s ) and R(y r ) represents the recognition rate of the first s and first r principal components; C3. Calculate the weights of the optimal principal components of color, shape, wavelet energy and texture. The weight formula of the optimal principal components is as follows: Among them, ω θ represents the weights of the optimal principal components of color, shape, wavelet energy and texture, Represents the recognition rate of color, shape, wavelet energy and texture principal components; C4. Performing weighted averaging on the color, shape, wavelet energy and texture features based on the weights of the optimal principal components of the color, shape, wavelet energy and texture to obtain a fusion feature of the target flame image.

8. The flame intelligent monitoring system based on industrial vision technology according to claim 3 is characterized by: The flame detection model includes a feature extraction module, a first prediction module and a second prediction module. The feature extraction module uses a three-dimensional convolutional neural network to extract the spatiotemporal features of the historical monitoring video. The first prediction module is used to roughly predict the time boundary of the historical monitoring video to obtain a rough time boundary. The second prediction module is used to finely predict the time boundary of the video segment within the rough time boundary roughly predicted by the first prediction module. The first prediction module converts the spatiotemporal features f l Divide into positioning features and categorical features And the positioning features are obtained through two temporal convolution layers. and the classification features For each time step in , we predict and generate the rough temporal boundary of the video segment instance for the i-th time step. and classification scores The coarse time boundary is converted into a true coarse time boundary through a time scale scaling module. The second prediction module transforms the positioning features through a fully connected layer Divide into starting features and termination features Based on the true coarse time boundary Extract two control regions separately The boundary pooling module is used to control the two control areas. Find the maximum value of the starting feature within and the termination characteristic maximum The starting feature maximum value and the maximum value of the termination characteristic splicing to obtain a composite feature, reducing the dimension of the composite feature through a temporal convolution layer to obtain a feature sequence, predicting the offset of the time boundary of the video segment based on the feature sequence, and predicting the offset of the time boundary and the true coarse time boundary based on the offset of the time boundary Gets the fine time boundary.

9. The flame intelligent monitoring system based on industrial vision technology according to claim 8 is characterized in that: The true coarse time boundary calculation formula is as follows: Among them, i represents the i-th time step of the termination feature in the total step length, and l represents the l-th video segment in the historical surveillance video; The calculation formulas for the starting feature and the ending feature are as follows: Among them, σ represents the RELU activation function, GN represents the batch normalization operation, and Conv represents the convolutional layer; The control area calculation formula is as follows: Among them, σ represents the RELU activation function, GN represents the batch normalization operation, and Conv represents the convolutional layer.

10. The flame intelligent monitoring method based on industrial vision technology is applicable to the flame intelligent monitoring system based on industrial vision technology according to any one of claims 1 to 9, characterized in that: The following steps are involved: S1, real-time monitoring of the target area, obtaining real-time monitoring video, and extracting key frames from the real-time monitoring video; S2, use the trained target detection model to analyze the latest monitoring frame, detect whether there is a flame, and locate the position and category of the flame; S3, performing edge detection and feature extraction on the detected flame image to obtain an image feature set, fusing the image feature set to obtain a fused feature, and classifying the fused feature and the image feature set by a trained support vector machine to determine the level of the flame; S4, performing target detection again on multiple key frames to detect whether there is a flame, and locate the position and type of the flame; S5, analyzing historical surveillance videos, detecting video segments containing flames, and marking their time boundaries; S6. Generate a corresponding abnormal alarm signal based on the detected flame and the flame level, and visualize the monitored flame location, category and historical video segment.

Citation Information

Cited By

  • Monitoring video intelligent inspection management method and system based on artificial intelligence

    CN119131702A

  • Boiler operation monitoring method and device, electronic equipment and storage medium

    CN119532978A

  • A boiler operation monitoring method, device, electronic device and storage medium

    CN119532978B

  • Video anomaly recognition processing method and system based on artificial intelligence

    CN119904785A

  • Crystallization anomaly identification method based on unsupervised learning

    CN120543498A