Visual smoke detection method and system based on spatial relation characteristics

Abstract

The invention relates to the technical field of visual smoke detection, in particular to a visual smoke detection method and system based on spatial relation characteristics. The method aims at improving the accuracy and robustness of smoke detection, space relation characteristics of smoke in an image are analyzed, special-shaped convolution different from traditional convolution is used for collecting space information in different directions, information vectors are integrated through specific aggregation operation, and a system is guided to extract representative visual characteristics. A multilevel feature extraction module is designed, and the module can analyze the shape and motion of smoke and the relation between the smoke and the surrounding environment from different scales and angles. In addition, the invention further provides an adjustable spatial feature selection mechanism which can dynamically adjust the extracted feature types according to the features of different scenes so as to optimize the detection performance. Finally, the effectiveness of the method in a complex environment is verified through a system, and a novel solution is provided for visual smoke detection.

Description

Technical Field

[0001] The present invention relates to the field of target detection and neural network, and in particular to a visual smoke and flame detection method and system based on spatial relationship features. Background Art

[0002] In the era of big data and intelligence, feature extraction and collection technology plays an increasingly important role in computer vision, natural language processing and other fields. With the rapid development of deep learning and machine learning algorithms, efficient and accurate collection and extraction of features has become a hot topic of research. Among them, spatial relationship features, as an important feature representation, can effectively capture the relationship and structural information between different objects in the data, and are widely used in many fields such as image recognition, scene understanding and human-computer interaction.

[0003] Visual smoke detection is an important research direction in computer vision and image processing, and is widely used in fire monitoring, security prevention, and environmental monitoring. With the acceleration of urbanization, traditional smoke detectors have gradually exposed their limitations, such as long response time and strong sensitivity to environmental factors. Therefore, vision-based smoke detection technology has emerged, which uses cameras to capture video streams and monitor smoke in real time through image analysis.

[0004] At present, visual smoke detection methods mainly rely on image processing and pattern recognition technology. Traditional methods usually identify smoke by analyzing changes in image pixels based on color and shape features. However, these methods often face problems such as low recognition rate and high false alarm rate under complex backgrounds, lighting changes and diverse smoke types. In recent years, the rapid development of deep learning has brought new opportunities for visual smoke detection. Methods based on convolutional neural networks (CNNs) can automatically extract deep features of images and significantly improve the accuracy and robustness of detection. At the same time, dynamic detection technology combined with time series information can better capture the dynamic changes of smoke and improve the real-time and reliability of detection.

[0005] Although research has made some progress, visual smoke detection still faces challenges, such as insufficient data set diversity, high real-time requirements, and adaptability in different environments. Currently, many feature collection methods still rely on traditional manual feature extraction and lack automation and intelligent means, resulting in low feature collection efficiency and poor accuracy. Especially when dealing with complex scenes and large-scale data, traditional methods are difficult to meet the requirements of real-time and efficiency. In addition, existing methods often ignore the importance of spatial relationship features and cannot fully reflect the interactions and dependencies between objects, which limits the improvement of model performance.

[0006] In order to solve these problems, a feature collection method and system based on spatial relationship features is proposed. This method extracts more comprehensive and accurate features by analyzing the spatial relationship between objects in the data, thereby improving the efficiency and effect of feature collection. Combined with deep learning algorithms, this method can automatically learn spatial relationship features, promote the optimization of spatial relationship feature extraction for non-rigid objects such as smoke, and provide stronger support for the application of visual smoke and flame detection systems. Summary of the invention

[0007] In view of the shortcomings of the existing smoke detection system, such as poor recognition effect, weak robustness, and insufficient versatility, the present invention proposes and designs a feature collection method based on spatial relationship features to solve the above problems. The invention mainly includes two parts: 1. Spatial relationship feature extraction part: It mainly designs a universal spatial feature extraction method, which can be used for the target detection model to extract spatial relationship information in the relevant layer, and perform specific processing on information in different directions according to specific circumstances. 2. Multi-level spatial feature fusion part: It designs a method for spatial relationship information interaction between layers, which can not only enhance the model learning ability through spatial relationship features, but also overcome the problem of insufficient stability of the model when dealing with different scenarios. The extraction, fusion and guidance of spatial relationship features are realized. The specific contents are as follows:

[0008] (1) A spatial feature collection method is designed, which uses narrow convolution to collect spatial information in the horizontal and vertical directions respectively and retains them in two vectors.

[0009] (2) A method for aggregating spatial relationship feature information vectors is designed to integrate spatial relationship information in different directions into the original feature map scale.

[0010] (3) A spatial context information fusion module is designed. After inputting feature maps of different scales, the spatial context information is extracted and fused, and then output to a specific level.

[0011] (4) A method for separating spatial context features is designed. During the process of spatial context separation and integration, the feature vectors in two directions are processed using preset weights to separate the spatial context weights in different directions.

[0012] (5) Export the trained model to a format suitable for the device and deploy it in the device environment to achieve real-time detection.

[0013] Compared with the existing smoke detection model, the advantages of the present invention are:

[0014] (1) The design performance of the present invention is better. It uses a module based on spatial relationship context, which makes the model more capable of perceiving and learning spatial context. Important indicators of the model, such as average precision and recall rate, are higher than those of other existing models.

[0015] (2) The present invention has the capability of multi-scale detection and uses a detection head with multi-scale detection capability, so that the detection system has good detection effect on objects of different sizes.

[0016] (3) The present invention reduces the resource consumption of model training. Although the additional spatial information module slightly increases the cost of a single training, the overall cost is reduced because the model converges faster and the number of training times is reduced.

[0017] (4) The present invention has wider applicability. By separating the weights of vectors with spatial relationship information, it can achieve better effects in different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 :System overall structure diagram

[0019] Figure 2 :The specific structure diagram of the multi-scale spatial context information module

[0020] Figure 3 :Spatial information extraction and merging operation flow chart DETAILED DESCRIPTION

[0021] At present, in most scenarios, people use edge devices with cameras to detect smoke and flames. Edge devices integrate cameras, processors, memory and network interfaces. By running a trained target detection model on the device, an alarm and location can be issued when a fire occurs. The running target detection model is limited by the computing power of the edge device and the requirement of real-time detection. Usually, a single-stage target detection algorithm based on a convolutional neural network is used. These models usually consist of three parts: Backbone, Neck, and Head. It is a fast and accurate target detection algorithm. The core idea is to convert the target detection problem into a regression problem, and directly predict the category and position of the target through a neural network. The present invention designs a target detection method for non-rigid objects such as smoke and flames that combines spatial relationship features. Users can use this detector more flexibly on edge devices to make the detection performance better and suitable for more scenarios. The specific implementation method is as follows:

[0022] 1. Overall system structure specifications

[0023] The present invention is used for smoke and flame detection, aiming to improve the overall performance through the guidance of the intermediate layer. Therefore, the overall process structure is standardized, and the feature graphs of each convolutional neural network layer are respectively passed through the Backbone network and the Neck network, and are guided and fused by the intermediate spatial context information layer. Then, the result of this detection is output in the Head network and compared with the true value. Figure 1 The specific instructions are as follows:

[0024] (1) The convolutional layer in the backbone network generally uses classic convolutional structures such as VGG and ResNet according to the specific computing power requirements and the actual situation of the detection object. It is usually composed of operations such as convolution, pooling, upsampling, and downsampling. It has the ability to extract and integrate image information. This paper uses the network structure of CSPDarknet53, which is mainly composed of multiple residual network blocks. This network has better feature extraction capabilities and attempts to perform feature fusion at each stage, which is more conducive to the recognition of non-rigid objects such as smoke and flames.

[0025] (2) The role of the Neck network is to fuse multi-level features, which is usually composed of convolution operations, upsampling, downsampling and splicing operations. The present invention adopts the network structure of BiFPN, which is mainly composed of top-down feature propagation, bottom-up feature propagation and lateral connection operations. The purpose is to build a feature pyramid in the middle layer and strengthen the fusion of features at different levels. In the feature fusion process, BiFPN also performs feature screening and feature expansion. Feature screening selects important information in the feature map in a nonlinear way, retains features that are beneficial to target detection, and eliminates unimportant features. Feature expansion can increase the flexibility of each feature map, making it better adapted to target detection tasks.

[0026] (3) The Head part extracts information from the feature map provided by Neck and outputs the category and bounding box information of each target. The loss function is calculated based on this information, and the parameters in the model are updated through gradient descent and back propagation algorithms. The present invention uses a multi-scale prediction structure to directly predict on feature maps of different scales, thereby improving the detection performance of targets of different sizes.

[0027] 2. Original picture processing:

[0028] The present invention has the ability of multi-scale detection. The original image will be cropped, filled, filtered, RGB channel separated and other operations before being input into the detection system. The purpose is to make the input image consistent with the system input format, and also increase the amount of data. The original data training was labeled before training. The smoke and flame objects in the image were labeled with corresponding labels respectively. Other objects were not specifically labeled, indicating that they were identified as background in the system. At the same time, some easily confused objects were added to the data set, such as red lights, red flags, fog, etc., which were added to the data set and used as negative examples for learning, which is conducive to enhancing the detection ability of the system.

[0029] 3. Use of Multi-Scale Spatial Context Module

[0030] Due to the characteristics of smoke as a non-rigid object, the effect of conventional models is limited. According to the spatial relationship characteristics of smoke affected by gravity in reality, the present invention adds this module to the model to extract and fuse spatial relationship information at different levels to guide the updated model parameters through the back propagation algorithm. The specific composition of this module is as follows: Figure 2 As shown, the specific implementation method is as follows:

[0031] (1) The input of this module is provided by the third and fourth layers of the Backbone network. First, the spatial information extraction and merging operation is performed on the fourth layer input feature map with lower resolution. After the output results are normalized and activated, the number of channels is halved using convolution. One of the outputs is upsampled and then concatenated with the feature map of the third layer input of the Backbone network to obtain a new set of feature maps with more channels and higher resolution. After that, the spatial information extraction and merging operation is performed again. After obtaining a matrix with multi-level spatial information, it is normalized and the corresponding activation function is applied. Finally, it is downsampled and concatenated with the original feature map that has undergone 1*1 convolution.

[0032] (2) The output of this module concatenates the outputs of the two levels together and concatenates them with the feature map obtained through the main process in the middle layer 2 of the Neck network, so that the feature map has more reliable spatial relationship information and plays a role in guiding learning.

[0033] 4. Use of spatial information extraction and merging operations

[0034] This operation is used in the above module, mainly to perform corresponding convolution and merging operations on the feature map input to it, such as Figure 3 As shown in the figure, this part uses two sets of 1*n and n*1 special-shaped convolutions to extract vectors with spatial information in two directions, and then obtains the feature map with spatial information to be output through a specific aggregation operation.

[0035] 5. Use of spatial vector aggregation operations:

[0036] Regarding the differences in various needs for smoke detection, as the main monitoring method, users hope that the detection system has a higher recall rate, that is, fewer missed detections. They may also use this system as a supplement to other smoke detection methods and hope to have higher accuracy. There are many ways to aggregate spatial vectors. We found that different aggregation methods affect the results of the detection system. For example, directly performing an inner product operation on two vectors can increase the recall rate and reduce missed detections at the expense of a little accuracy, and taking large values ​​of the horizontal and vertical coordinates can increase detection accuracy. Through practice, it is found that taking the average value of the horizontal and vertical coordinates has the best overall effect. The reason is that taking the average value can keep the values ​​in the spatial information feature map stable and will not have large deviations.

[0037] 6. Use of Spatial Information Weighted Separation Operation

[0038] In reality, smoke detection may involve different scenarios. The importance of spatial relationship information in the horizontal and vertical directions of the identified object varies in different scenarios. For example, from a drone’s bird’s-eye view, the system is not sensitive to the vertical spatial features of the smoke. Therefore, in the averaging operation during spatial vector aggregation, we add different weights to separate the weights of the spatial information vectors in two different directions to adapt to the needs of different scenarios.

[0039] 7. Deploy detection systems on edge devices

[0040] Convert the trained YOLO model to ONNX or other intermediate formats, install the compiler, dependent libraries, and frameworks required to run the YOLO model, such as GCC, OpenCV, Pytorch, ONNX Runtime, etc., and deploy the converted model to the edge device.

Claims

1. A visual smoke and flame detection method and system based on spatial relationship features, characterized by: The following steps are involved: (1) receiving an image of the area to be detected and performing preprocessing on the received image, including denoising, enhancement and other operations, to improve image quality; (2) Performing convolution, upsampling, downsampling, and splicing operations on the image tensor in a detection system with a spatial relationship feature module to extract overall image features; (3) extracting spatial relationship features in the feature map output by a specific convolutional layer in the spatial context information module, including but not limited to the shape, size, position of the smoke and flame area, and its relative relationship with other objects; (4) Based on the extracted spatial relationship features, the system learning ability is optimized, a smoke detection model is constructed, and accurate detection of smoke and flames is achieved by training and optimizing model parameters; (5) Input the image to be detected into the trained smoke detection model and output the smoke detection result.

2. The detection system having a spatial relationship feature module according to claim 1, characterized in that: (1) It has the structure of Backbone, Neck, Head and spatial context information module; (2) Its structure has the characteristics of multi-level connections. For example, the output of one layer serves as the input of other layers, and the input of one layer may come from the output of different layers. (3) The detection head uses a multi-scale detection structure.

3. The spatial relationship context module according to claim 2, characterized in that: (1) A structure with two input channels of high and low resolution and a single characteristic channel as output; (2) Feature maps of different scales are fused and spliced ​​within the module; (3) The spatial relationship feature extraction and merging operation is used within the module.

4. The spatial relationship feature extraction and merging operation according to claim 3 is characterized in that: (1) A narrow and long shaped convolution is used to extract the spatial relationship information vector of the feature map; (2) A variety of optional operations are used to aggregate the feature vectors to obtain a spatial relationship information matrix; (3) In the aggregation operation, different vectors can be weighted aggregated.

Images