Smoke detection device based on panoramic optical flow video analysis

Abstract

The invention discloses a smoke detection device based on panoramic optical flow, which is used for judging whether fire smoke is generated or not and comprises a fisheye camera unit, an image conversion unit, an optical flow analysis unit and a smoke detection unit. The invention provides a smoke detection method based on panoramic optical flow. The method comprises the following steps: acquiring a 2D fisheye image from a fisheye camera; calculating an optical flow field of the fisheye image; the optical flow is converted into an ERP optical flow through equidistant columnar projection; converting the fisheye image into an ERP image through equidistant columnar projection; the ERP image and the ERP optical flow are fused; and calculating target frame position coordinates and confidence of the smoke through the target detection network. According to the invention, the fisheye camera is adopted to capture information of a three-dimensional space, and compared with a pinhole camera, the fisheye camera can cover the whole scene; the bimodal input based on image and optical flow fusion makes full use of the motion information of the object, and the performance of the target detection network is further improved.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a visual smoke detection device based on panoramic optical flow. Background Art

[0002] Smoke detection algorithm is an important research direction in the field of computer vision and image processing. Smoke detection is of great significance for early warning of fire. The goal of smoke detection is to accurately detect smoke or smoke-related phenomena such as fire and burning in the scene by analyzing image or video data.

[0003] The smoke detection algorithm mainly includes: Traditional methods based on color and texture features: Early smoke detection methods mainly rely on color and texture features. These methods are usually based on the grayscale range and texture features of smoke and the difference between the background, by setting thresholds or using machine learning classifiers to achieve smoke detection. However, these methods have limited effect on smoke detection under lighting changes and complex background scenes.

[0004] Smoke detection methods based on motion information: In order to overcome the challenges of illumination and texture changes, researchers began to use motion information for smoke detection. These methods identify the motion patterns of smoke by analyzing pixel changes or motion vectors in video sequences. Typical methods include optical flow estimation, background modeling, and target tracking. This method can better handle complex scenes and illumination changes, and improve the accuracy of smoke detection.

[0005] Smoke detection methods based on deep learning: With the rise of deep learning, researchers have begun to apply deep learning models to smoke detection tasks. Deep learning methods can learn higher-level feature representation and pattern recognition capabilities by training with large-scale annotated image or video data. Typical deep learning methods in smoke detection include convolutional neural networks (CNN), recurrent neural networks (RNN), and convolution-recurrent neural networks (CRNN). These methods have made significant progress in the accuracy and robustness of smoke detection. Patent CN116310757A discloses a multi-task real-time smoke detection method that combines the detection results of a real-time smoke segmentation network and a lightweight smoke detection network to determine whether smoke exists. Patent CN116152720A discloses a smoke detection method that uses an improved YOLOv5 model and combines inter-frame differences to predict the probability and location of smoke occurrence.

[0006] The above-mentioned smoke detection methods based on color and texture features, motion information, and deep learning generally use images or videos taken by a single / multiple pinhole cameras as input / training data. A single pinhole camera usually cannot cover the entire scene; multiple pinhole cameras have high installation requirements, low integration, and are difficult to maintain, and need to process images / videos from multiple perspectives. In addition, the above-mentioned smoke detection methods do not fully and effectively utilize image semantic information and optical flow motion information. With the development of camera technology, panoramic cameras and related technologies have become increasingly mature. Representative panoramic cameras such as Instax 360One X, Ricoh Theta Z1, and Samsung Gear 360 have broadened the scope of visual applications.

[0007] Compared with the existing technology, the smoke detection algorithm based on panoramic optical flow makes full use of the characteristics of fisheye images to achieve coverage of the entire scene. It makes full use of the motion information of the object and fuses the image semantic information with the optical flow motion information, further improving the performance of the target detection network. Summary of the invention

[0008] In order to overcome the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a visual smoke detection device based on panoramic optical flow. The present invention fully utilizes the advantages of fisheye cameras and combines optical flow motion information to achieve detection of smoke targets.

[0009] In order to achieve the above and other purposes, the present invention proposes a visual smoke detection device based on panoramic optical flow, comprising the following units: A fisheye camera unit, wherein the fisheye camera unit acquires fisheye image data, performs image correction, color correction and image stitching to obtain a fisheye image, and projects the fisheye image into an equirectangular image coordinate system through equirectangular projection to obtain an ERP image; An optical flow analysis unit, wherein the optical flow analysis unit calculates the optical flow field of the fisheye image and performs post-processing to obtain the fisheye optical flow, which represents the 2D projection of the 3D displacement vector of the object in the space in the fisheye pixel coordinate system, and contains the motion information of the object; An optical flow transformation unit, wherein the optical flow transformation unit projects the fisheye optical flow into an equidistant cylindrical image coordinate system to obtain an ERP optical flow; The smoke detection unit performs feature extraction and feature fusion on the ERP image and the ERP optical flow, and finally calculates the position information and confidence of the smoke. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flow chart of a smoke detection algorithm based on panoramic optical flow of the present invention; Figure 2This is a schematic diagram of the installation of a visual smoke detection device based on panoramic optical flow according to the present invention; Figure 3 This is a structural diagram of a visual smoke detection device based on panoramic optical flow of the present invention; Figure 4 It is a schematic diagram of an equidistant cylindrical projection method of fisheye optical flow according to the present invention. DETAILED DESCRIPTION

[0011] The following describes the implementation of the present invention through specific examples and in conjunction with the accompanying drawings. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific examples, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0012] Instructions attached Figure 1~2 This is a diagram showing one embodiment of the present invention. Figure 1 This is a flow chart of a smoke detection algorithm based on panoramic optical flow in the present invention. Figure 2 This is a schematic diagram of the installation of a visual smoke detection device based on panoramic optical flow in the present invention. Figure 3 This is a structural diagram of a visual smoke detection device based on panoramic optical flow in the present invention.

[0013] like Figure 2 As shown, the fire smoke 10 is located in the monitoring area, and the visual smoke detection device 11 is suspended on the top of the monitoring area, with its optical axis perpendicular to the lower surface of the monitoring area, looking down at the entire monitoring area, covering the entire scene, and collecting a continuous fisheye image sequence.

[0014] The internal structure of the visual smoke detection device is as follows Figure 3 As shown, it includes a fisheye camera unit 12, an optical flow analysis unit 13, an optical flow transformation unit 14 and a smoke detection unit 15. The fisheye camera unit 12 collects continuous fisheye images and generates an ERP image 7. The optical flow analysis unit 13 calculates the fisheye optical flow. The optical flow transformation unit 14 generates the ERP optical flow 5. The smoke detection unit 15 takes the ERP image 7 and the ERP optical flow 5 as input and outputs the location information and confidence of the smoke.

[0015] like Figure 1 As shown, the smoke detection algorithm based on panoramic optical flow includes five steps: optical flow calculation 2, optical flow projection 4, RGB projection 6, feature fusion 8 and smoke detection 9: Optical flow calculation 2 uses a high-precision optical flow estimation method to calculate the optical flow field of the fisheye image , fisheye optical flow Only valid within the central circular area.

[0016] The RGB projection 6 projects the fisheye image 1 into an ERP image 7 by equirectangular projection.

[0017] The optical flow projection 4 includes three steps: ERP-fisheye inverse projection 16, correlation point calculation 17 and fisheye-ERP projection 18. Figure 4 shown.

[0018] ERP-Fisheye Reverse Projection 16 converts the points on the pixel coordinate system of the equidistant cylindrical image Projected into the fisheye image pixel coordinate system, we get point : First, the points on the pixel coordinate system of the equirectangular image Projected onto the fisheye spherical coordinate system (unit sphere), we get point :

[0019]

[0020] in is the width of the ERP image, is the height of the ERP image, .

[0021] Next, the points on the fisheye spherical coordinate system Projected to the fisheye image physical coordinate system, we get point .set up is the fisheye image physical coordinate system, focal length , For point The extension line in the opposite direction of the origin The intersection of the planes (i.e. the image point of the pinhole camera model), is the image point of the fisheye camera, is the angle corrected according to the Kannala-Brandt model.

[0022]

[0023] According to the fisheye camera isometric projection model, The distance to the origin is , calculated The coordinates are:

[0024] Finally, find the pixel coordinates of the fisheye image :

[0025] in , , , It is the internal reference of fisheye camera.

[0026] Correlation point calculation calculates fisheye image pixels through optical flow warping algorithm Correlation points under the corresponding fisheye optical flow , that is :

[0027] in Represents fisheye optical flow along The weight of the axis, Represents fisheye optical flow along The weight of the axis.

[0028] Fisheye-ERP projection will associate points Project to The pixel coordinate system of the equidistant cylindrical image is obtained Matching points in equirectangular image pixel coordinates : First, the points in the equirectangular image pixel coordinate system are Projected into the fisheye image physical coordinate system, we get point :

[0029] Next, the point in the fisheye image physical coordinate system Projected into the fisheye spherical coordinate system, we get point , calculated according to (2.4) for:

[0030] According to (1.3) (2.4) (2.3) we can get and :

[0031] Finally, the point in the fisheye spherical coordinate system Project it into the equidistant cylindrical image pixel coordinate system to get the matching points : According to (1.1), we can get:

[0032]

[0033] Substituting (5)(6.1)(6.2)(6.3) into (7.1)(7.2) we obtain:

[0034] Substitute (4) into (7.3) (7.4) and and Differentiation The optical flow vector at for:

[0035] in It can be calculated by (1.1)(1.2)(1.3)(2.1)(2.3)(2.4)(3.1):

[0036] Substituting (8.3) (8.4) into formula (8.1) (8.2) we obtain:

Claims

1. A smoke detection device based on video analysis, used to determine whether there is fire smoke, includes the following units: A fisheye camera unit, wherein the fisheye camera unit acquires fisheye image data, performs image correction, color correction and image stitching to obtain a fisheye image, and projects the fisheye image into an equirectangular image coordinate system through equirectangular projection to obtain an ERP image; An optical flow analysis unit, wherein the optical flow analysis unit calculates the optical flow field of the fisheye image and performs post-processing to obtain the fisheye optical flow, which represents the 2D projection of the 3D displacement vector of the object in the space in the fisheye pixel coordinate system, and contains the motion information of the object; An optical flow transformation unit, wherein the optical flow transformation unit projects the fisheye optical flow into an equidistant cylindrical image coordinate system to obtain an ERP optical flow; The smoke detection unit performs feature extraction and feature fusion on the ERP image and the ERP optical flow, and finally calculates the position information and confidence of the smoke.

2. An equirectangular projection method for fisheye optical flow, characterized in that the optical flow vector in the fisheye image pixel coordinate system is projected into the equirectangular image pixel coordinate system to obtain a dense ERP optical flow, including ERP-fisheye inverse projection, correlation point calculation and fisheye-ERP projection: The ERP-fisheye inverse projection calculates the corresponding points of the points in the equidistant cylindrical image pixel coordinate system in the fisheye image pixel coordinate system through continuous coordinate system transformation; The calculation of the associated points is to calculate the associated points of the corresponding points in the fisheye image pixel coordinate system in the next frame by using a fisheye optical flow warping algorithm; The fisheye-ERP projection calculates the corresponding points of the associated points in the equidistant cylindrical image pixel coordinate system through continuous coordinate system transformation, and performs differential calculation with the initial points in the equidistant cylindrical image pixel coordinate system to obtain the equidistant cylindrical projection of the fisheye optical flow.

3. A smoke detection device according to claim 1, characterized in that: The smoke detection device includes at least one fisheye camera, and the fisheye camera is suspended on the top of the monitoring area, with its optical axis perpendicular to the lower surface of the monitoring area, looking down at the entire monitoring area, covering the entire scene, and collecting continuous fisheye image sequences.

Images