Early tiny flame detection method based on deep aerial convolution and kalman filtering

Abstract

The application discloses a kind of early small flame detection methods based on deep air-to-space convolution and Kalman filtering, belong to computer vision target detection technical field.The method of the application includes the following steps: after video stream input, the target area of video frame is cut to determine;Deep air-to-space convolution method SPD is used to replace part of convolution layer and part of maximum pooling layer in convolutional neural network;Fire dynamic capture is carried out by frame difference intersection-over-union IOU method calculation, and secondary discrimination of flame is carried out;Improved Kalman filter algorithm is used, and the detection and identification of early small flame are optimized.The application applies deep air-to-space convolution method, frame difference intersection-over-union IOU calculation method and Kalman filter flame detection method, improves the accuracy and speed of flame detection, and is applied to complex scenes or low frame rate scenes, to realize early small flame intelligent detection and identification in complex environment.

Description

Technical Field

[0001] This invention belongs to the field of computer vision target detection technology, specifically relating to an early fine flame detection method based on deep-to-space convolution and Kalman filtering. Background Technology

[0002] Fire is one of the most destructive and frequent disasters, threatening people's lives and property. Traditionally, fire detection mainly relies on smoke, fog, and light detectors. However, these methods are severely limited by external conditions such as weather, temperature, and location, and the detection results are often unsatisfactory.

[0003] To prevent major accidents caused by fires, computer vision-based flame detection systems have attracted considerable attention. Traditional computer vision fire detection methods mostly rely on algorithms based on the color features of flames, using manually set thresholds to determine the presence of flames. Currently, most color models for flame detection use RGB, HIS, YUV, and Lab. Later, to improve the accuracy of flame recognition, researchers proposed combining color features with dynamic flame characteristics. Chen et al. proposed a video flame recognition method based on combining multiple flame features, using RGB and HIS models to extract candidate flame regions and combining them with a dynamic flickering algorithm for fire detection. Alternatively, flames can be identified through changes in the flame's angle and area. A method combining brightness thresholds and changes in flame roundness enables graded fire alarms. Based on the flame's position... The characteristics of color space representation are combined with the texture features of flames to identify forest fires. In recent years, with the rapid development of computer processing speed, deep learning methods have received widespread attention from scholars, and research on flame recognition based on deep learning has become a hot research direction.

[0004] In recent years, various network flame detection technologies based on YOLO have emerged, achieving good accuracy and speed for large-scale flame detection. However, when applied to scenarios with small flames or complex backgrounds, problems have become apparent, such as excessive false positives, low frame rates, difficulty in detecting flames, and frame drops in some detections.

[0005] Existing literature: Zhang Binbin, Pazilai Mahmuti. An improved flame target detection algorithm based on YOLOv3 [J]. Progress in Laser & Optoelectronics, 2021, 058(024): 281-288, proposes an improved flame target detection algorithm based on YOLOv3 (You onlook once, v3). By reusing local information through an improved feature pyramid network, a large-scale fully convolutional module is designed to obtain global spatial information at different scales. Then, complete intersection-union ratio (CIoU) and Focal Loss are used as loss functions to alleviate the imbalance in binary classification of the dataset, improving the detection accuracy and speed of the target. However, this algorithm has a large number of network parameters and high computational cost, and does not construct a more efficient lightweight network. The detection efficiency and real-time performance need further improvement.

[0006] The existing patent application number CN201911355379.7 proposes a fire detection method based on deep learning. It uses deformable convolution and pooling operations to extract fire features and uses a skip connection method to fuse low-level features with high-level features, which effectively improves the reliability and applicability of the fire detection method. It also has good detection results for fires in different scenarios and fires of different sizes. However, when the fire is very small or there is a target that is very similar to the fire, false detection will still occur. The detection accuracy and precision need to be improved.

[0007] In summary, current flame detection methods still have some problems when applied to scenarios with small flames or complex backgrounds. Therefore, it is necessary to propose an early method for detecting small flames based on deep-to-space convolution and Kalman filtering. Summary of the Invention

[0008] Purpose of the invention: In order to overcome the shortcomings of existing technologies, such as the low detection accuracy and limited application in complex background scenes of computer vision flame detection methods, this invention proposes an early fine flame detection method based on deep-to-space convolution and Kalman filtering. This method aims to improve the accuracy and speed of flame detection and to be applied in complex or low frame rate scenes.

[0009] Technical Solution: To solve the above-mentioned technical problems, the present invention proposes an early fine flame detection method based on deep-to-space convolution and Kalman filtering, comprising the following steps:

[0010] Step 1: Obtain the video stream for the target area;

[0011] Step 2: Preprocess the video stream, extract the corresponding video frames, and determine the target area;

[0012] Step 3: Input the video frames after data preprocessing into a lightweight YOLO network that applies the deep-to-space convolution method;

[0013] Step 4: Perform preliminary target detection on the video frames after data preprocessing by applying a lightweight YOLO network with deep-to-space convolution method. If a suspicious target is detected, retain and store the location information of all detected suspicious targets.

[0014] Furthermore, in step 2, the video stream is preprocessed using a video frame region of interest determination method to extract the corresponding video frames and determine the target region, thereby reducing network computation and the computational load of subsequent steps.

[0015] Furthermore, in step 4, the lightweight YOLO network of the deep-to-space convolution method uses SPD network blocks for preliminary target detection, specifically including the following steps:

[0016] First, set the SPD network blocks to intermediate feature maps of arbitrary size. The intermediate feature mapping The size is ;

[0017] Then the intermediate features are mapped Perform network block partitioning to obtain Each sub-feature mapping is X( , (C) It is a constant;

[0018] Next, the sub-feature maps obtained through the above steps are connected and merged along the channel dimension. After connection and merging, a new feature map is obtained, which is: ( , , );

[0019] Next, the new feature map obtained by the above connection merging is processed... Non-stride convolution, after The feature map formed after non-stepping convolution is: , , ,in The number of kernels in non-straddle convolutions is taken as... ;

[0020] The final output is , , The feature map, the feature map , , The content in the corresponding video frame is the detected suspicious target, and the location information of all detected suspicious targets is retained and stored.

[0021] SPD network blocks are used for preliminary target detection. By dividing the feature map and performing fusion convolution in the channel direction, small and easily overlooked feature points in the image are preserved, which improves the recognition rate of small targets when detecting suspicious targets.

[0022] Furthermore, the above The range of values ​​for is ： .by scale=2 Taking a single channel with a feature mapping channel dimension C=1 as an example, the specific steps include:

[0023] First of all Figure 2 (a) Intermediate feature mapping sequence diagram of SPD (Space-to-depth) network blocks set as single-channel (C=1) ( , ,1);

[0024] Then as Figure 2 As shown in (b), the intermediate feature is mapped The network is divided into blocks, resulting in... Each sub-network block, of which scale=2 ,get The sub-feature mapping sequence diagrams are as follows: , , , ,like Figure 2 (c) shows that the sub-feature sequence diagrams corresponding to each sub-block are all X ( , ,1);

[0025] Then as Figure 2 (c) and Figure 2 As shown in (d), the four sub-feature maps obtained through the above steps are connected and merged along the channel dimension to obtain a new feature map, which is: ( , , );

[0026] Immediately afterwards Figure 2 (d) and Figure 2 As shown in (e), the new feature map obtained by the above connection merging is... ( , , )conduct Non-stride convolution (i.e., convolution with a stride of 1), after After non-stepping convolution, the new feature map is: , , ,in The number of kernels in non-straddle convolutions is taken as... ;

[0027] Finally, as Figure 2 As shown in (e), the output yields the new feature map. , , This feature mapping , , The corresponding feature information is the image information content in the video frame. It is selected as the detected suspicious target, and the location information of all detected suspicious targets is retained and stored.

[0028] Furthermore, after the lightweight YOLO network using the deep-to-space convolution method performs preliminary target detection and detects suspicious targets, the following steps are also included:

[0029] Step 5: Perform preliminary target detection using a lightweight YOLO network with deep-to-space convolution method to determine whether a suspicious target is detected. If a suspicious target is detected, retain and store the location information of all detected suspicious targets and proceed to step 6; otherwise, proceed to step 8.

[0030] Step 6: Determine whether the content in the previous video frame is a suspicious target. If yes, proceed to step 7; otherwise, proceed to step 10.

[0031] Step 7: Perform secondary detection on the suspicious target to determine whether the suspicious target is the optimal suspicious target. Specifically, the frame difference inter-interference-comparison (IOU) method is used for secondary detection. If the suspicious target is the optimal suspicious target, proceed to step 9; otherwise, proceed to step 10.

[0032] Step 8: If no suspicious target is detected in Step 5, check if there is a suspicious target in the previous frame of the current frame. If there is, use the Kalman filter method to predict the target position in the current frame to obtain the optimal detection box selection information and proceed to Step 9. Otherwise, return to Step 2 to continue extracting video frames.

[0033] Step 9: Map the optimal suspicious target location information obtained in Step 7, or the target location information predicted by the Kalman filter in Step 8, onto the original video frame for box selection.

[0034] Step 10: Determine if the current frame is the last frame. If yes, end the process; otherwise, return to step 2 and continue extracting video frames to continue the detection until the process ends.

[0035] Furthermore, in step 7, the frame difference inter-interference-union (IOU) method is used for secondary detection, specifically including the following steps:

[0036] First, calculate the position coordinates of the intersection between the current frame's target box and the previous frame's target box. The calculation formula is as follows:

[0037]

[0038] In formula (1) above, max() is the maximum value function and min() is the minimum value function. This represents the coordinates of the top-left corner of the target bounding box 'a' in the current frame. This represents the coordinates of the bottom right corner of the target bounding box 'a' in the current frame. The coordinates of the top-left corner of the target bounding box b in the previous frame. The coordinates of the lower right corner of the target bounding box b in the previous frame are obtained through the above formula (1). This represents the coordinates of the top-left corner of the intersection between the current frame's bounding box and the previous frame's bounding box. This is the coordinate of the bottom right corner of the intersection between the current frame's target bounding box and the previous frame's target bounding box;

[0039] Then calculate the area of ​​the intersection between the current frame's target box and the previous frame's target box. The formula for this intersection area is:

[0040] Formula (2)

[0041] In the above formula (2), The area of ​​the intersecting part. This is the x-coordinate of the top-left corner of the intersection between the current frame's bounding box and the previous frame's bounding box. This is the x-coordinate of the bottom right corner of the intersection between the current frame's target bounding box and the previous frame's target bounding box. This is the y-coordinate of the top-left corner of the intersection between the current frame's bounding box and the previous frame's bounding box. This is the y-coordinate of the bottom right corner of the intersection between the current frame's target bounding box and the previous frame's target bounding box. Represents coordinates ( The vector value of ) Represents coordinates ( The vector value of ).

[0042] Next, the area of ​​the union of the current frame's bounding box and the previous frame's bounding box is calculated. The formula for this area is:

[0043] Formula (3)

[0044]

[0045]

[0046] In the above formula (3), It is calculated using formula (2). It is calculated using formula (4). This represents the area of ​​the target bounding box in the current frame. It is calculated using formula (5). This represents the area of ​​the target bounding box in the previous frame;

[0047] Next, the frame difference inter-frame union (IOU) between the current frame's target box and the previous frame's target box is calculated using the following formula:

[0048]

[0049] In the above formula (6), It is calculated using formula (2). It is calculated using formula (3);

[0050] Finally, a threshold is determined for the Intersection over Union (IOU) of the frame difference between the target box in the current frame and the target box in the previous frame. If the IOU is greater than the threshold, the suspected target is the optimal suspected target and proceed to step 9; otherwise, proceed to step 10.

[0051] By using the frame difference inter-union comparison (IOU) method to perform secondary detection of flames, the false detection rate of small flames in the early stage is reduced, the flame detection accuracy is improved, and the detection speed is also increased.

[0052] Furthermore, the above frame differences are interleaved and compared. The threshold is set to k , k The range of values ​​for is ...

[0053] Furthermore, step 8 involves predicting the target in the current frame using a Kalman filter, specifically including the following steps:

[0054] First, we set up the state equation and the covariance equation. The formula for calculating the state equation is as follows:

[0055]

[0056] In the above formula (7), A Here is the state transition matrix. This is the selection information for the suspicious target from the previous frame. B For the control variable matrix, Indicates observation noise. Indicates process noise;

[0057] The formula for calculating the covariance equation is as follows:

[0058]

[0059] In the above formula (8), The covariance of the prior prediction for the current frame. A Here is the state transition matrix. Let be the covariance of the optimal predicted target at the previous time step. State transition matrix A The transpose of the matrix, Q Here is the system noise covariance matrix;

[0060] Next, set the system status. X ,as follows:

[0061] Formula (9)

[0062] In the above formula (9), x is the x-axis coordinate of the center point of the video frame, and y is the y-axis coordinate of the center point of the video frame. This represents the change in the x-axis coordinate of the current frame's center point compared to the x-axis coordinate of the center point in the previous frame. This represents the change in the y-axis coordinate of the current frame's center point compared to the y-axis coordinate of the center point in the previous frame. It is the width of the video frame. The height of the video frame. Transpose of a matrix

[0063] Then, the state equation (7) and covariance equation (8) are updated, and the updated calculation formula for state equation (7) is:

[0064]

[0065] The updated calculation formula for the covariance equation (8) is as follows:

[0066]

[0067] In the above formula (10), It is calculated using formula (7). The Kalman gain at time k, To reduce the weight of the YOLO network in detecting suspicious targets from the previous frame, the bounding box information is used. H It is the transformation matrix from the state vector to the measurement vector;

[0068] In the above formula (11), I It is the identity matrix. The Kalman gain at time k, H It is the transformation matrix from the state vector to the measurement vector. It is calculated using formula (8);

[0069] The above The Kalman gain at time k is calculated using the following formula:

[0070]

[0071] In the above formula (12), It is calculated using formula (8). for H transpose matrix, H It is the transformation matrix from the state vector to the measurement vector. for H transpose matrix, R It is the covariance matrix of the measured noise;

[0072] Finally, the results obtained through the above steps As the optimal prediction state, this optimal prediction state is the optimal detection bounding box information of the system, and proceeds to step 9; if no target in the previous frame of the current frame is detected, return to step 2 to continue extracting video frames.

[0073] The Mann filter algorithm is used for prediction, which solves the common problem of missing suspicious target frames in flame detection. It can detect small flames while ensuring accurate detection and improving detection speed.

[0074] Furthermore, set the control variable matrix. B Let the zero matrix be the state transition matrix. A for:

[0075] .

[0076] Beneficial effects: Compared with the prior art, the advantages of this invention are:

[0077] 1. Compared with the prior art, the present invention addresses the problem of excessive misjudgments in the detection process caused by the prior art. The present invention adopts a detection method based on deep-space convolution and Kalman filter. The deep-space convolution method solves the problem of excessive misjudgments in the detection process, improves the accuracy of early detection of small flames, and also increases the detection speed.

[0078] 2. Compared with existing technologies, when a video stream is detected as input, before feeding it into the neural network, a lightweight YOLO network model using the deep-to-hollow convolution method is applied. Based on the data preprocessing, the maximum trapezoidal area in the video frame is extracted by filtering, or the target region is directly cropped from the original video frame. The target region is then determined, and finally, the target region cut from the video frame is divided into video region sub-blocks, thereby reducing the target region. This solves the traditional problems of excessive parameters and large computational load, effectively reduces the subsequent computational load, improves the network operation speed, and enhances the real-time performance of detection.

[0079] 3. Compared with existing technologies, the core of the CNN (Convolutional Neural Network) architecture is the convolutional layer, mainly using strided convolution, followed by position output through pooling layers. Common pooling methods include max pooling and average pooling. Max pooling leads to a high loss rate when detecting small objects, while average pooling easily causes the network to fuse useless features. This invention applies a lightweight YOLO network model using deep-to-depth convolution and uses SPD (Space-to-depth) network blocks to replace some convolutional layers and some max pooling layers in the convolutional neural network. This optimizes the problems of max pooling discarding potentially needed feature points, average pooling fusing unnecessary feature points, or directly using convolutional layers to increase parameters, thereby reducing the number of neural network parameters and correcting the discarding of certain feature points by max pooling. This optimizes the detection performance for small objects and objects in low-resolution conditions, and improves the detection rate of small flames in low-resolution, complex scenes.

[0080] 4. Compared with existing technologies, this invention combines traditional methods with deep learning. After outputting from a lightweight neural network, it employs the Intersection of Union (IOU) algorithm for dynamic flame detection, thereby reducing false positives. Specifically, when performing secondary detection of suspicious targets using the IOU method, it combines traditional methods with deep learning to dynamically capture flames after outputting from a lightweight neural network. This effectively improves the problem of image holes appearing during differencing when there are large areas of similar grayscale values ​​on the surface of moving targets in inter-frame difference modeling. It also optimizes the problem of excessive false positives in Gaussian mixture background modeling and provides secondary flame detection, thereby reducing excessive false positives caused by single-judgment methods and improving the accuracy of early detection of small flames.

[0081] 5. Compared with the prior art, in order to improve detection accuracy and reduce false judgments in flame detection, when the current video frame is detected as a suspicious target, and the content of the previous video frame is also detected as a suspicious target, in order to determine the optimal suspicious target among the two suspicious targets, the suspicious target is further discriminated to determine whether the suspicious target is the optimal suspicious target among the two suspicious targets, thereby reducing false judgments and improving detection accuracy.

[0082] 6. Compared with existing technologies, which struggle to detect small flames in complex or low frame rate scenarios, and suffer from the loss of some suspicious target frames during flame detection, this invention employs a Kalman filter to predict the target in the current frame when suspicious target frames are lost. It predicts the value of the current frame using the optimal result of the previous frame and corrects the current frame's value using the observed value, thus obtaining the optimal result for the current frame. This achieves accurate detection of small flames while improving detection speed.

[0083] 7. Compared with existing technologies, this invention adopts a method that combines traditional computer vision with deep learning. Specifically, it combines deep-to-space convolution and Kalman filter algorithms to map targets that have completed secondary detection or targets that have completed intelligent prediction through Kalman filters to the original video frames, thereby ensuring the safety of detection operations. Compared with single detection methods, it has a more efficient effect, reduces the loss of small targets in the early stages, and improves the detection accuracy and precision in complex scenes. It can detect flames in the early stages of a fire in a timely manner, greatly reducing the damage to the surrounding environment and reducing the economic losses caused by fires.

[0084] 8. Compared with existing technologies, this invention combines the deep-to-space convolution method and the Kalman filter algorithm. Specifically, it applies a lightweight YOLO network based on the deep-to-space convolution method. By using a lightweight backbone, removing some parameters to extract network features, and replacing the original network blocks, the network's computation speed is improved, thereby increasing the accuracy of small target detection.

[0085] 9. Compared with existing technologies, this invention combines the deep-to-space convolution method and the Kalman filter algorithm. Compared with the color space, dynamic capture, and deep learning methods, this invention has higher accuracy in early flame detection and the detection effect is significantly better than the other three methods. It can more accurately capture small flame targets and reduce detection errors.

[0086] 10. Compared with the prior art, the present invention adopts a method that combines traditional computer vision with deep learning, specifically by combining the deep-to-space convolution method and the Kalman filter algorithm. The method of the present invention has high detection accuracy and fast detection speed, and is suitable for flame detection in various scenarios, especially for early small flame scenarios or application scenarios with complex backgrounds. It realizes intelligent detection and recognition of early small flames in complex environments and has wide applicability. Attached Figure Description

[0087] Figure 1 This is a flowchart of the present invention.

[0088] Figure 2 This is a diagram of the SPD (Space-to-depth) network block structure.

[0089] Figure 3 This is a diagram of the frame difference inter-joint ratio (IOU) detection method.

[0090] Figure 4 This is a comparison chart of the detection performance of the method of the present invention with other methods under the same detection environment.

[0091] Figure 5 This is a bar chart showing the performance of the method of this invention compared to other methods under the same detection environment.

[0092] Figure 6 This is a line graph showing the statistical performance of the method of this invention compared to other methods under the same detection environment. Detailed Implementation

[0093] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0094] Example 1

[0095] The early fine flame detection method based on deep-to-space convolution and Kalman filtering in this embodiment refers to... Figure 1 The method includes the following steps:

[0096] Step 1: Obtain the video stream for the target area;

[0097] Step 2: Preprocess the video stream, extract the corresponding video frames, and determine the target area;

[0098] Step 3: Input the video frames after data preprocessing into a lightweight YOLO network that applies the deep-to-space convolution method;

[0099] Step 4: Perform preliminary target detection on the video frames after data preprocessing by applying a lightweight YOLO network with deep-to-space convolution method. If a suspicious target is detected, retain and store the location information of all detected suspicious targets.

[0100] This embodiment preprocesses the video stream of the target area and extracts the corresponding video frames. The video frames can be obtained through video surveillance, position sensors, etc. By applying a lightweight YOLO model with deep-to-space convolution method, preliminary target detection is performed on the video frames after data preprocessing, which can reduce network computation, reduce computational costs, and thus improve detection speed.

[0101] This embodiment uses a lightweight YOLO network employing the deep-to-space convolution method to perform preliminary target detection on video frames after data preprocessing. Suspicious targets are detected, and the location coordinates of all detected suspicious targets are stored so that the background can view the suspicious targets and their location coordinates, thereby achieving real-time and accurate detection of target flames.

[0102] Example 2

[0103] This embodiment of the method for detecting early, fine flames based on deep-to-space convolution and Kalman filtering, based on Embodiment 1, obtains the video stream of the target area through video surveillance, position sensors, etc., and then preprocesses the video stream in step 2 using a video frame region of interest determination method to extract the corresponding video frames and determine the target area, i.e., the target flame in the video stream. Specifically, it includes the following steps:

[0104] First, read the input video stream;

[0105] Then, the nth frame image of the video frame in the video stream is obtained, and the region of the nth frame image is set as the region of interest. For example, the nth frame image can be the 1st frame image or other frame images.

[0106] Next, the position coordinates of the region of interest are normalized (this normalization process can be such that the coordinates are all between 0 and 1 after normalization).

[0107] Finally, the target region is cropped from the original video frame (using conventional image processing operations to select the object or region of interest for processing), and the resulting video frame image information is used as the preprocessed video frame image information for subsequent processing steps.

[0108] This embodiment performs target region cropping on the original video frames, identifying the target region to reduce network computation and the computational load of subsequent steps. This embodiment processes the nth frame of the video stream by selecting a fixed region of interest for cropping, and then uses this preprocessed video frame as the preprocessed video frame for subsequent flame detection. After detection, the detection information of the region of interest (i.e., the nth frame) is mapped back to the original video frame.

[0109] This embodiment uses a video frame region of interest determination method to preprocess the video stream and extract the corresponding video frames, thereby reducing the target area for detecting flames. This solves the traditional problems of too many parameters and large computational load, effectively reducing the subsequent computational load, improving network computing speed, and enhancing detection real-time performance.

[0110] Example 3

[0111] This embodiment presents an early fine flame detection method based on deep-to-depth convolution and Kalman filtering. Building upon Embodiment 2, step 4 inputs the preprocessed video frame into a lightweight YOLO network employing deep-to-depth convolution. This embodiment uses SPD (Space-to-depth) network blocks to perform preliminary target detection on the preprocessed video frame. By finely dividing the feature map of the preprocessed video frame, small, easily overlooked feature points are preserved. Currently, when using convolutional neural networks to detect very fine targets or low-resolution images, detection performance rapidly declines. This embodiment optimizes the detection of small particles and low-resolution objects, preserving small, easily overlooked feature points to detect small, suspicious targets in the video frame.

[0112] This embodiment is based on Embodiment 2. The preprocessed video frames are input into a lightweight YOLO network model that applies the deep-to-depth convolution method. The SPD (Space-to-depth) network block in the lightweight YOLO network using the deep-to-depth convolution method performs preliminary target detection on the image information in the preprocessed video frames. Specifically, this includes the following steps:

[0113] First, set the SPD (Space-to-depth) network blocks to intermediate feature maps of arbitrary size. ( , (C), the intermediate feature map The size is .

[0114] Specifically, in a CNN (Convolutional Neural Network), the data in the convolutional layers exists in a three-dimensional form. Each layer generates a feature map (i.e., a feature value matrix obtained by convolutional operations on image information) to represent the features of the image information in the input video frame. The aforementioned intermediate feature map... ( , In C), This represents the size of the feature map, which indicates how the image is divided into... Each cell is used for output, and the output is also in cell units. S is a constant, and C is a constant, representing the feature mapping channel dimension.

[0115] Then the intermediate features are mapped The network is divided into blocks, resulting in... Each sub-network block corresponds to a sub-feature mapping, thus obtaining... Each sub-feature mapping is X( , (C), where is a constant and C is a constant, representing the dimension of the feature mapping channels, and the sequence of sub-feature mappings corresponding to each sub-feature mapping is as follows: ,Right now

[0116]

[0117]

[0118]

[0119]

[0120]

[0121]

[0122]

[0123] Specifically, the sub-feature mapping contains features from the image information, and the above sub-feature mapping sequence S is a constant. x The range of values ​​is , y The range of values ​​is , is a constant and .

[0124] Next, the sub-feature maps obtained through the above steps are connected and merged along the channel dimension. After connection and merging, a new feature map is obtained, which is: ( , , S is a constant. C is a constant, where C represents the dimension of the feature mapping channel;

[0125] Next, the new feature map obtained by the above connection merging is processed. Non-stretched convolutions (i.e., convolutions with a stride of 1) preserve the largest feature point and possible related feature point information within the selected region. After non-stepping convolution, the new feature map is , , Non-stepping convolution is used to preserve as much discriminative feature information as possible in the video frame. The number of kernels in non-straddle convolutions is taken as... , is a constant and S is a constant, C is a constant, and C represents the dimension of the feature mapping channel;

[0126] The final output is , , The feature map, the feature map , , The corresponding feature information is the image information content in the video frame. It is selected as the detected suspicious target, and the location information of all detected suspicious targets is retained and stored.

[0127] This embodiment employs a lightweight YOLO network based on the deep-to-space convolution method, using SPD (Space-to-depth) network blocks for initial target detection. In conventional CNN (Convolutional Neural Network) networks, the core of the system structure is the convolution layer, which mainly uses strided convolution and pooling layers to output location information for suspicious targets. CNN networks typically use max pooling and average pooling to reduce the size of the feature map, facilitating input into deeper networks. Max pooling selects the feature point with the largest feature value within the target box and uses it as the feature point of the selected region, easily ignoring other effective feature points (especially small features). Average pooling takes the average value of the feature points within the target box and uses it as the feature point of the selected region, easily leading to the inclusion of features unrelated to the target object in the feature map. In this embodiment, the Space-to-depth (SPD) method is successfully applied to optimize the above problems. By dividing the feature map and performing fusion convolution in the channel direction, the maximum feature point and possible related feature points in the area selected by the target box are successfully preserved, so that small and easily ignored feature points in the image can be preserved, and the recognition rate of small targets is improved when detecting suspicious targets.

[0128] This embodiment applies the Space-to-depth (SPD) network block, which solves the problems encountered in the application of CNNs (Convolutional Neural Networks). These problems include the discarding of some potentially needed feature points when applying max pooling in CNNs, the fusion of some unnecessary feature points by average pooling, and the problem of adding too many parameters by directly using convolutional layers. On the other hand, it optimizes the detection of small particles and low-resolution objects, thereby improving the detection accuracy of small particles and low-resolution objects, and improving the detection accuracy of fine targets.

[0129] Example 4

[0130] This embodiment presents an early fine flame detection method based on deep-to-depth convolution and Kalman filtering, based on Embodiment 3. The lightweight YOLO network using deep-to-depth convolution employs SPD (Space-to-depth) network blocks, such as... Figure 2 As shown.

[0131] This embodiment uses Feature mapping channel dimension Taking a single channel as an example, in practical applications, any channel can be used. The preprocessed video frames are input into a lightweight YOLO network model applying deep-to-holographic convolution for initial detection and judgment. The obtained preliminary detection information of suspicious targets is stored for subsequent operations. (Reference) Figure 2 Specifically, it includes the following steps:

[0132] First of all Figure 2 (a) The SPD (Space-to-depth) network block is set to single-channel ( Intermediate feature mapping sequence diagram ( , ,1);

[0133] Then as Figure 2 As shown in (b), the intermediate feature is mapped The network is divided into blocks, resulting in... Each sub-network block, of which ,get The sub-feature mapping sequence diagrams are as follows: , , , ,like Figure 2 (c) shows that the sub-feature sequence diagrams corresponding to each sub-block are all X ( , ,1);

[0134] Then as Figure 2(c) and Figure 2 As shown in (d), the four sub-feature maps obtained through the above steps are connected and merged along the channel dimension to obtain a new feature map, which is: ( , , );

[0135] Immediately afterwards Figure 2 (d) and Figure 2 As shown in (e), the new feature map obtained by the above connection merging is... ( , , )conduct Non-stride convolution (i.e., convolution with a stride of 1), after After non-stepping convolution, the new feature map is: , , ,in The number of kernels in non-straddle convolutions is taken as... ;

[0136] Finally, as Figure 2 As shown in (e), the output yields the new feature map. , , This feature mapping , , The corresponding feature information is the image information content in the video frame. It is selected as the detected suspicious target, and the location information of all detected suspicious targets is retained and stored.

[0137] In this embodiment, a lightweight YOLO network using the deep-to-depth convolution method is employed. SPD (Space-to-depth) network blocks are used to perform preliminary detection of image information in the preprocessed video frames. If the image information in the preprocessed video frame is determined to contain suspicious target information, the detection result of the suspicious target video frame is retained, and this suspicious target video frame is input into subsequent steps to further determine the fire situation.

[0138] Example 5

[0139] This embodiment presents an early fine flame detection method based on deep-space convolution and Kalman filtering. Building upon Embodiment 3, it combines traditional methods with deep learning. A lightweight YOLO network using deep-space convolution is applied for initial target detection. After detecting a suspicious target, the Intersection of Union (IOU) method is used for dynamic flame capture, performing a secondary detection on the suspicious target. This reduces excessive false positives caused by a single judgment and improves the accuracy of early fine flame detection. The specific steps include:

[0140] Step 5: Perform preliminary target detection using a lightweight YOLO network with deep-to-space convolution method to determine whether a suspicious target is detected. If a suspicious target is detected, retain and store the location information of all detected suspicious targets and proceed to step 6; otherwise, proceed to step 8.

[0141] Step 6: Determine whether the content in the previous video frame is a suspicious target. If yes, proceed to step 7; otherwise, proceed to step 10.

[0142] Step 7: Perform secondary detection on the suspicious target to determine whether the suspicious target is the optimal suspicious target. Specifically, the frame difference inter-interference-comparison (IOU) method is used for secondary detection. If the suspicious target is the optimal suspicious target, proceed to step 9; otherwise, proceed to step 10.

[0143] Step 8: If no suspicious target is detected in Step 5, check if there is a suspicious target in the previous frame of the current frame. If there is, use the Kalman filter method to predict the target position in the current frame to obtain the optimal detection box selection information and proceed to Step 9. Otherwise, return to Step 2 to continue extracting video frames.

[0144] Step 9: Map the optimal suspicious target location information obtained in Step 7, or the target location information predicted by the Kalman filter in Step 8, onto the original video frame for box selection to ensure the safety of the detection operation.

[0145] Step 10: Determine if the current frame is the last frame. If yes, end the process; otherwise, return to step 2 and continue extracting video frames until the process ends.

[0146] In this embodiment, in order to improve detection accuracy and reduce false positives in flame detection, when the current video frame is detected as a suspicious target, and the content of the previous video frame is also detected as a suspicious target, in order to determine the optimal suspicious target among the two suspicious targets, it is necessary to perform a secondary discrimination on the suspicious target to determine whether the suspicious target is the optimal suspicious target among the two suspicious targets, thereby reducing false positives and improving detection accuracy.

[0147] This embodiment employs the Frame Differential Intersection Over Union (IOU) method to further evaluate suspicious targets detected by the lightweight YOLO neural network. The IOU method calculates the IOU value between the bounding box of a suspicious target in the previous frame and the bounding box of a suspicious target in the current frame. If the IOU value is not lower than a set threshold... Then, the location coordinates of the suspicious target in the current frame are stored, and the suspicious target in the current frame is mapped to the original video frame as the optimal suspicious target for selection.

[0148] This embodiment uses a lightweight YOLO network model based on deep spatial convolution to detect flames in target areas based on video frames. Knowing that existing technologies suffer from excessive false positives, this embodiment employs the Intersection over Union (IOU) method for secondary flame detection, reducing the false positive rate for early-stage, small flames and improving both detection accuracy and speed.

[0149] Example 6

[0150] This embodiment presents an early fine flame detection method based on deep-to-space convolution and Kalman filter algorithm. Building upon Embodiment 5, it employs the Intersection over Union (IOU) method to perform secondary detection on suspicious targets, thereby determining whether the suspicious target is the optimal suspicious target. The specific method is as follows:

[0151] The frame difference inter-union ratio (IOU) method is used to determine whether the IOU value between the suspected target in the previous frame and the initially detected suspected target in the current frame is greater than a threshold k. If it is greater than the threshold, the location information of the suspected target in the current frame is retained as the optimal target location information, and then used as a reference. Figure 3 After selecting the target bounding box from the video frame information of the two suspicious targets mentioned above, Figure 3 The top left corner box represents the target bounding box of the current frame, and the bottom right corner box represents the target bounding box of the previous frame. The gray area represents the overlapping area of ​​the two target bounding boxes. The specific steps include:

[0152] First, calculate the position coordinates of the intersection between the current frame's target bounding box and the previous frame's target bounding box. The calculation formula is as follows:

[0153]

[0154] In formula (1) above, max() is the maximum value function and min() is the minimum value function. This represents the coordinates of the top-left corner of the target bounding box 'a' in the current frame. This represents the coordinates of the bottom right corner of the target bounding box 'a' in the current frame. The coordinates of the top-left corner of the target bounding box b in the previous frame. The coordinates of the lower right corner of the target bounding box b in the previous frame are obtained through the above formula (1). This represents the coordinates of the top-left corner of the intersection between the current frame's bounding box and the previous frame's bounding box. The coordinates of the bottom right corner of the intersection between the current frame's bounding box and the previous frame's bounding box;

[0155] Then calculate the area of ​​the intersection between the current frame's target box and the previous frame's target box. The formula for this intersection area is:

[0156] Formula (2)

[0157] In the above formula (2), The area of ​​the intersecting part. This is the x-coordinate of the top-left corner of the intersection between the current frame's bounding box and the previous frame's bounding box. This is the x-coordinate of the bottom right corner of the intersection between the current frame's target bounding box and the previous frame's target bounding box. This is the y-coordinate of the top-left corner of the intersection between the current frame's bounding box and the previous frame's bounding box. This is the y-coordinate of the bottom right corner of the intersection between the current frame's target bounding box and the previous frame's target bounding box. Represents coordinates ( The vector value of ) Represents coordinates ( The vector value of ).

[0158] Next, the area of ​​the union of the current frame's bounding box and the previous frame's bounding box is calculated. The formula for this area is:

[0159] Formula (3)

[0160]

[0161]

[0162] In the above formula (3), It is calculated using formula (2). It is calculated using formula (4). This represents the area of ​​the target bounding box in the current frame. It is calculated using formula (5). This represents the area of ​​the target bounding box in the previous frame;

[0163] Next, the frame difference inter-frame union (IOU) between the current frame's target box and the previous frame's target box is calculated using the following formula:

[0164]

[0165] In the above formula (6), It is calculated using formula (2). It is calculated using formula (3);

[0166] Finally, a threshold is determined based on the Intersection over Union (IOU) between the current frame's target box and the previous frame's target box. If the IOU is greater than the threshold... k , k The range of values ​​for is If so, then the current suspicious target is the optimal suspicious target.

[0167] This embodiment employs the Intersection over Union (IOU) method for secondary detection of suspicious targets. Conventional dynamic capture methods, such as inter-frame difference modeling and Gaussian mixture background modeling, suffer from image voids during the difference process when the surface of the moving target to be detected has large areas of similar grayscale values. To address this, existing dynamic discrimination methods are combined with deep learning. The flame is then re-evaluated using the IOU method after the output of a lightweight YOLO neural network.

[0168] The frame difference inter-interference-union (IOU) method used in this embodiment differs from conventional dynamic capture methods. To optimize the problem of excessive misjudgment during modeling, when calculating the intersection area and union area of ​​the current frame target box and the previous frame target box, the original coordinates in the original formula (2) and union area formulas (4) and (5) are modified. and Each coordinate value in the graph is incremented by 1 to reduce the impact of overlapping pixels between two target boxes on the area being calculated, thereby improving accuracy and calculation precision, and also improving the accuracy of flame detection results.

[0169] Example 7

[0170] This embodiment presents an early fine flame detection method based on deep-to-space convolution and the Kalman filter algorithm. Building upon Embodiment 6, it employs the Intersection over Union (IOU) method for secondary detection of suspicious targets. Figure 3 The top-left box represents the target bounding box of the current frame, and the bottom-right box represents the target bounding box of the previous frame. The gray area represents the overlapping area of ​​the two target bounding boxes, where the IOU threshold is... The value range is set to This indicates that the target bounding box in the previous frame overlaps with the target bounding box in the previous frame, and the detection result is acceptable.

[0171] Optimally, take ,when At that time, the overlap between the target bounding box in the previous frame and the target bounding box in the previous frame accounted for more than half, compared to The threshold range is narrowed, and the detection accuracy is improved;

[0172] Optimally, take ,when Ideally, the target bounding box in the current frame completely overlaps with the target bounding box in the previous frame.

[0173] Optimally, take ,when At that time, the overlap between the target box in the previous frame and the target box in the previous frame accounts for half;

[0174] The above analysis shows that when the IOU value is not lower than the set threshold... At that time, the location coordinates of the suspected target in the current frame are stored, and the suspected target bounding box in the current frame is used as the optimal target bounding box in the current frame. When the threshold is reached... At that time, the test results were acceptable; when This represents the ideal state; the closer the threshold k is to 1, the higher the overlap between the two target boxes, indicating higher detection accuracy and more accurate results.

[0175] Example 8

[0176] This embodiment presents an early fine flame detection method based on deep-space convolution and Kalman filter algorithm. Based on Embodiment 6, after initial target detection using a lightweight YOLO network applying deep-space convolution, if no suspicious target is detected, the method checks if a suspicious target exists in the previous frame. If so, it uses the Kalman filter method to predict the target position in the current frame to obtain the optimal detection bounding box information. Specifically, this includes the following steps:

[0177] First, we set up the state equation and the covariance equation. The formula for calculating the state equation is as follows:

[0178]

[0179] In the above formula (7), k for k time, A Here is the state transition matrix. This is the selection information for the suspicious target from the previous frame. B Set the control variable matrix. B It is a zero matrix. Indicates observation noise. Indicates process noise;

[0180] The formula for calculating the covariance equation is as follows:

[0181]

[0182] In the above formula (8), The covariance of the prior prediction for the current frame. A Here is the state transition matrix. Let be the covariance of the optimal predicted target at the previous time step. State transition matrix A The transpose of the matrix, Q Let be the system noise covariance matrix, and let be the state transition matrix. A for:

[0183] ;

[0184] Next, set the system status. X ,as follows:

[0185] Formula (9)

[0186] In the above formula (9), x is the x-axis coordinate of the center point of the video frame, and y is the y-axis coordinate of the center point of the video frame. This represents the change in the x-axis coordinate of the current frame's center point compared to the x-axis coordinate of the center point in the previous frame. This represents the change in the y-axis coordinate of the current frame's center point compared to the y-axis coordinate of the center point in the previous frame. It is the width of the video frame. The height of the video frame. Transpose of a matrix

[0187] Then, the state equation (7) and covariance equation (8) are updated, and the updated calculation formula for state equation (7) is:

[0188]

[0189] The updated calculation formula for the covariance equation (8) is as follows:

[0190]

[0191] In the above formula (10), It is calculated using formula (7). The Kalman gain at time k, To reduce the weight of the YOLO network in detecting suspicious targets from the previous frame, the bounding box information is used. H It is the transformation matrix from the state vector to the measurement vector;

[0192] In the above formula (11), I It is the identity matrix. The Kalman gain at time k, H It is the transformation matrix from the state vector to the measurement vector. It is calculated using formula (8);

[0193] The above The Kalman gain at time k is calculated using the following formula:

[0194]

[0195] In the above formula (12), It is calculated using formula (8). for H transpose matrix, H It is the transformation matrix from the state vector to the measurement vector. for H transpose matrix, R It is the covariance matrix of the measured noise;

[0196] Finally, the results obtained through the above steps As the optimal prediction state, this optimal prediction state is the optimal detection bounding box information of the system, and proceeds to step 9; if no target in the previous frame of the current frame is detected, return to step 2 to continue extracting video frames.

[0197] This embodiment addresses the difficulty of detecting small flame targets in complex or low frame rate scenarios using existing technologies, as well as the phenomenon of missing suspicious target frames during flame detection. This invention employs a Kalman filter to predict the target in the current frame when suspicious target frames are lost. For detecting small flames, the hyperparameters of the Kalman filter are adjusted before detection to increase the weight of prior values ​​in the calculation. R (The covariance matrix of the measurement noise) is large, so set... The state equation at time 0 is 0, and the covariance is set to 1. The value of the current frame is predicted by the optimal result of the previous frame, and the value of the current frame is corrected by the observation value of the current frame. The optimal predicted state calculated by the Kalman filter is used as the box selection information for the optimal detection of the system to obtain the optimal result of the current frame. This solves the common phenomenon of missing suspicious target frames in flame detection, realizes the detection of small flames while ensuring accurate detection of small flames, and improves the detection speed.

[0198] This embodiment employs the Kalman filter algorithm, especially the Kalman filter algorithm for prediction and selection of the optimal target, thereby solving the common problem of missing frames of suspicious targets with small flames in flame detection, improving detection performance in complex or low frame rate scenarios, and enhancing the real-time performance of small flame detection.

[0199] Example 9

[0200] This embodiment presents an early fine flame detection method based on deep-space convolution and Kalman filter algorithm. Building upon Embodiment 8, compared to using a single detection method, this embodiment combines deep learning with traditional methods. Specifically, it combines deep-space convolution and Kalman filter algorithms to ensure normal detection operation. The details are as follows:

[0201] First, in complex background environments, when a fire is in its early stages, the flames are very small and difficult to detect. After the monitoring video or sensors collect information about the target area, the deep-to-space convolution method is used, and the quantized neural network SPD is used to perform preliminary detection of the target. The detected small suspicious target information is stored.

[0202] Then, in low frame rate scenarios, if a suspected target frame is lost, prediction is made using a Kalman filter;

[0203] Next, for the suspicious targets output by the lightweight neural network, a second detection is performed using IOU to further determine whether the suspicious target is a flame;

[0204] Finally, the targets that have completed secondary detection or have undergone intelligent prediction using a Kalman filter are mapped back to the original video frames to ensure safe operation of the detection process, reduce early loss of small targets, and improve detection accuracy and precision in complex scenes.

[0205] This embodiment is more efficient than a single detection method. By adding an SPD network block to the neural network, it improves the recognition of small targets or low frame rate scenes in complex scenarios. Furthermore, the subsequent Kalman filter algorithm improves the phenomenon of missing suspicious target frames in such scenarios, greatly enhancing the real-time performance of flame detection and improving the real-time performance of early small flame detection. This allows for timely detection of flames in the early stages of a fire, significantly reducing damage to the surrounding environment and minimizing economic losses caused by fires.

[0206] Example 10

[0207] This embodiment of the early fine flame detection method based on deep-space convolution and Kalman filter algorithm is based on Embodiment 2. It applies a lightweight YOLO network using deep-space convolution. This improves network processing speed by employing a lightweight backbone, removing some parameters to extract network features, and replacing existing network blocks. The specific steps include:

[0208] First, the convolutional layers in the backbone network are reduced, retaining only some network blocks;

[0209] Next, the retained C3 (Concentrated-Comprehensive Convolution Block) network block is replaced with the SKNet (Selective Kernel Networks) attention mechanism network block;

[0210] Finally, a deep-to-hollow convolution method (SPD-Conv) is inserted between each ordinary convolutional layer and the SKNet network block for processing.

[0211] This embodiment employs a lightweight YOLO network using the deep-to-space convolution method. It utilizes a lightweight backbone and addresses the issue that the size of the receptive field (the area mapped to the pixel in the feature map output by each layer of the CNN on the preprocessed video frame image) significantly impacts the accuracy of target detection in existing convolutional neural networks (CNNs). This embodiment utilizes the dynamic selection mechanism of convolutional kernels in CNNs, employing SKNet (Selective Kernel Networks) to adaptively adjust the size of its receptive field (convolutional kernel) based on the multi-scale information of the input video frame image. Neurons in SKNet can capture target objects of different scales, thereby improving the computational speed of the lightweight network and increasing the accuracy of detecting small targets.

[0212] Example 11

[0213] This embodiment presents an early fine flame detection method based on deep-to-space convolution and Kalman filter algorithm. Based on Embodiment 9, a simulation experiment was conducted in a complex background environment with the target being a fine flame. Figure 4 This is an experimental detection image simulating a real-world application where a worker holds a small flame in a complex background environment. The image shows the flame detection using this invention and three methods: color space, motion capture, and deep learning. The white frame represents the flame area detected by the four methods.

[0214] Figure 4 (a) is a diagram showing the location of the flame detected by the method of the present invention. Figure 4 (b) is a map showing the location of the flames detected using the color space method. Figure 4 (c) is a map showing the location of the flames detected using the dynamic capture method. Figure 4 (d) is a map showing the location of the flames detected using a single deep learning network. (By...) Figure 4 It can be seen that, under the current detection environment, this method can accurately detect the location of a small handheld flame, while the other three methods all have deviations and large false detection rates. This shows that the detection method is more accurate and can more accurately detect the location of a small flame.

[0215] Table 1 shows the performance data of the four detection methods in terms of detection accuracy and false detection rate under this experimental environment:

[0216] Table 1 Performance of different detection methods

[0217] name Color space Motion capture Deep learning This method precise 2419 2661 1762 2972 False positives 3401 1382 933 69

[0218] Figure 5 and Figure 6 Table 1 is a statistical chart of data from the present invention and three other methods for flame detection.

[0219] Figure 5 This is a bar chart comparison. The horizontal axis represents four methods: color space, motion capture, deep learning, and the deep-to-space convolution method and Kalman filter method of this invention. The vertical axis represents numerical values. The bars marked with diagonal lines in the chart represent accuracy, and the bars marked with horizontal lines represent false detection rate.

[0220] Figure 6 This is a line graph comparison. The horizontal axis represents four methods: color space, motion capture, deep learning, and the deep-to-space convolution method and Kalman filter method of this invention. The vertical axis represents numerical values. Solid lines represent accuracy, and dashed lines represent false detection rate.

[0221] Through Table 1 and Figure 5 as well as Figure 6 The performance comparison shows that the detection error of this method is only 69, which is much lower than the other three detection methods. The accuracy is 2972, which is also the highest among the four methods. It can be seen that the deep-space convolution method and Kalman filter method of this invention are significantly better than the other three methods, and can more accurately capture small flame targets and reduce detection errors.

[0222] Based on the above analysis and simulation practice, the deep-to-space convolution method and Kalman filter method proposed in this invention have better performance in flame detection, improve the detection accuracy of small flames, and reduce the error rate. This invention can be widely used in actual flame detection scenarios.

[0223] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting early, fine flames based on deep-space convolution and Kalman filtering, characterized in that, Includes the following steps: Step 1: Obtain the video stream for the target area; Step 2: Preprocess the video stream, extract the corresponding video frames, and determine the target region; Step 2 applies the video frame region of interest determination method to preprocess the video stream, extract the corresponding video frames, and determine the target region. Step 3: Input the video frames after data preprocessing into a lightweight YOLO network that applies the deep-to-space convolution method; Step 4: Perform preliminary target detection on the video frames after data preprocessing by applying a lightweight YOLO network with deep-to-space convolution method. If a suspicious target is detected, retain and store the location information of all detected suspicious targets. In step 4, the lightweight YOLO network of the deep-to-space convolution method uses SPD network blocks for preliminary target detection, specifically including the following steps: First, set the SPD network block as an intermediate feature map X0 of arbitrary size, with the size of the intermediate feature map X0 being S×S×C; Then, the intermediate feature map X0 is used to divide the network into blocks to obtain the scale. 2 There are ×C sub-feature maps, each of which is... scale is a constant; Next, the sub-feature maps obtained through the above steps are connected and merged along the channel dimension. After connection and merging, a new feature map is obtained, which is: Next, a 1×1 non-staggered convolution is performed on the new feature map obtained by the above connection merging. The feature map formed after the 1×1 non-staggered convolution is: Where C1 is the number of kernels in the non-straddle convolution, taking... The final output is The feature map, the feature map The content in the corresponding video frame is the detected suspicious target, and the location information of all detected suspicious targets is retained and stored; It also includes the following steps: Step 5: Perform preliminary target detection using a lightweight YOLO network with deep-to-space convolution method to determine whether a suspicious target is detected. If a suspicious target is detected, retain and store the location information of all detected suspicious targets and proceed to step 6; otherwise, proceed to step 8. Step 6: Determine whether the content in the previous video frame is a suspicious target. If yes, proceed to step 7; otherwise, proceed to step 10. Step 7: Perform secondary detection on the suspicious target to determine whether the suspicious target is the optimal suspicious target. Specifically, the frame difference inter-interference-comparison (IOU) method is used for secondary detection. If the suspicious target is the optimal suspicious target, proceed to step 9; otherwise, proceed to step 10. Step 8: If no suspicious target is detected in Step 5, check if there is a suspicious target in the previous frame of the current frame. If there is, use the Kalman filter method to predict the target position in the current frame to obtain the optimal detection box selection information and proceed to Step 9. Otherwise, return to Step 2 to continue extracting video frames. Step 9: Map the optimal suspicious target location information obtained in Step 7, or the target location information predicted by the Kalman filter in Step 8, onto the original video frame for box selection. Step 10: Determine if the current frame is the last frame. If yes, end the process; otherwise, return to step 2 and continue extracting video frames to continue the detection until the process ends.

2. The method for detecting early fine flames based on deep-to-space convolution and Kalman filtering according to claim 1, characterized in that, The range of values ​​for scale is: scale ≥ 2.

3. The method for detecting early fine flames based on deep-to-space convolution and Kalman filtering according to claim 1, characterized in that, Step 7, which uses the Inter-Frame Difference Interchange-Union (IOU) method for secondary detection, specifically includes the following steps: First, calculate the position coordinates of the intersection between the current frame's target bounding box and the previous frame's target bounding box. The calculation formula is as follows: In formula (1) above, max() is the maximum value function, min() is the minimum value function, (x a1 y a1 (x) represents the top-left corner coordinates of the target bounding box a in the current frame. a2 y a2 (x) represents the coordinates of the bottom right corner of the target bounding box a in the current frame. b1 y b1 (x) represents the coordinates of the top-left corner of the target bounding box b in the previous frame. b2 y b2 (x1, y1) is the coordinate value of the lower right corner of the target box b in the previous frame; (x2, y2) is the coordinate value of the upper left corner of the intersection of the target box in the current frame and the target box in the previous frame, obtained by formula (1) above; (x2, y2) is the coordinate value of the lower right corner of the intersection of the target box in the current frame and the target box in the previous frame. Then calculate the area of ​​the intersection between the current frame's target box and the previous frame's target box. The formula for the area of ​​this intersection is: intersection=|x2-x1+1.0,0|·|y2-y1+1.0,0| Formula (2) In the above formula (2), intersection is the area of ​​the intersection part, x1 is the x-axis coordinate of the upper left corner of the intersection part of the current frame target box and the previous frame target box, x2 is the x-axis coordinate of the lower right corner of the intersection part of the current frame target box and the previous frame target box, y1 is the y-axis coordinate of the upper left corner of the intersection part of the current frame target box and the previous frame target box, y2 is the y-axis coordinate of the lower right corner of the intersection part of the current frame target box and the previous frame target box, |x2-x1+1.0,0| represents the vector value of coordinate (x2-x1+1.0,0), and |y2-y1+1.0,0| represents the vector value of coordinate (y2-y1+1.0,0). Next, the area of ​​the union of the current frame's bounding box and the previous frame's bounding box is calculated. The formula for this area is: union=S A +S B -intersection formula (3) S A =(x a2 -x a1 +1.0)·(y a2 -y a1 +1.0) Formula (4) S B =(x b2 -x b1 +1.0)·(y b2 -y b1 +1.0) Formula (5) In formula (3) above, intersection is calculated using formula (2), and S A S is calculated using formula (4). A S represents the area of ​​the target bounding box in the current frame. B S is calculated using formula (5). B This represents the area of ​​the target bounding box in the previous frame; Next, the frame difference inter-frame union (IOU) between the current frame's target box and the previous frame's target box is calculated using the following formula: In the above formula (6), the intersection is calculated using formula (2), and the union is calculated using formula (3); Finally, a threshold is determined for the Intersection over Union (IOU) of the frame difference between the target box in the current frame and the target box in the previous frame. If the IOU is greater than the threshold, the suspected target is the optimal suspected target and proceed to step 9; otherwise, proceed to step 10.

4. The method for detecting early fine flames based on deep-to-space convolution and Kalman filtering according to claim 3, characterized in that, Set the threshold for the Inter-Frame Union (IOU) ratio to k, where k ranges from 0 to 1.

5. The method for detecting early fine flames based on deep-to-space convolution and Kalman filtering according to claim 4, characterized in that, Step 8 involves predicting the target in the current frame using a Kalman filter, specifically including the following steps: First, we set up the state equation and the covariance equation. The formula for calculating the state equation is as follows: In the above formula (7), A is the state transition matrix, X k-1 This represents the bounding box information of the suspected target from the previous frame, where B is the control variable matrix, and u... k-1 w represents observation noise. k-1 Indicates process noise; The formula for calculating the covariance equation is as follows: In the above formula (8), Let P be the covariance of the prior prediction for the current frame, A be the state transition matrix, and P be the variance of the prior prediction. k-1 Let A be the covariance of the optimal predicted target at the previous time step. T Let Q be the transpose of the state transition matrix A, and let Q be the system noise covariance matrix. Next, set the system status X as follows: X = [x, y, w, h, dx, dy] T Formula (9) In the above formula (9), x is the x-axis coordinate of the center point of the video frame, y is the y-axis coordinate of the center point of the video frame, dx is the change in the x-axis coordinate of the current frame center point compared to the x-axis coordinate of the previous frame center point, dy is the change in the y-axis coordinate of the current frame center point compared to the y-axis coordinate of the previous frame center point, w is the width of the video frame, and h is the height of the video frame. T Transpose of a matrix Then, the state equation (7) and covariance equation (8) are updated, and the updated calculation formula for state equation (7) is: The updated calculation formula for the covariance equation (8) is as follows: In the above formula (10), K is calculated using formula (7). k Z is the Kalman gain at time k. k To reduce the weight of the YOLO network in detecting suspicious targets in the previous frame, H is the transformation matrix from the state vector to the measurement vector. In the above formula (11), I is the identity matrix, K k Let K be the Kalman gain at time k, and H be the transformation matrix from the state vector to the measurement vector. It is calculated using formula (8); The above K k The Kalman gain at time k is calculated using the following formula: In the above formula (12), H is calculated using formula (8). T Let H be the transpose matrix, where H is the transformation matrix from the state vector to the measurement vector. T Let H be the transpose of the matrix, and R be the covariance matrix of the measurement noise; Finally, X obtained through the above steps k As the optimal prediction state, this optimal prediction state is the optimal detection bounding box information of the system, and proceeds to step 9; if no target in the previous frame of the current frame is detected, return to step 2 to continue extracting video frames.

6. The method for detecting early fine flames based on deep-to-space convolution and Kalman filtering according to claim 5, characterized in that, Set the control variable matrix B to zero, and define the state transition matrix A as follows:

Images