Depth estimation method for flame 3D reconstruction using single camera based on optical flow

Abstract

The present invention discloses a single-camera flame three-dimensional reconstruction depth estimation method based on optical flow, which can perform three-dimensional depth estimation of flames under complex working conditions and detect their combustion state; it has certain optimizations for the problems of high computational cost, expensive experimental equipment, and complex experimental operation in traditional flame three-dimensional reconstruction algorithms, and provides a new idea for flame monitoring under complex working conditions. The flame combustion environment constructed by this design uses a black baffle as the background to ensure that the background of the captured flame image is single, and the video is collected by a camera and then transmitted to a computer for frame extraction. The extracted flame image is first denoised, and then semantic segmentation is performed using a pre-trained network, and then optical flow estimation is performed, and finally the obtained optical flow map is converted into a three-dimensional depth map.

Description

Technical Field

[0001] The invention belongs to the technical field of flame video image segmentation, and relates to a single-camera flame three-dimensional reconstruction depth estimation method based on optical flow. Background Art

[0002] The three-dimensional model of flame has extremely important applications in the industrial field. The three-dimensional reconstruction of flame can help people understand the temperature distribution of flame and the degree of combustion more intuitively. It is of great significance for studying the propagation mechanism of flame and the dynamic simulation of flame model.

[0003] In fossil fuel power plants, burner combustion performance analysis is required to improve the efficiency of fuel combustion, ensure the safety of boilers, and reduce the emission of polluting waste. Therefore, it is necessary to establish a three-dimensional model of the combustion flame and then analyze the combustion characteristics. It can provide a decision-making basis for flame combustion optimization and accurately understand the combustion conditions of the boiler flame. However, due to the characteristics of flame flashing and no significant features, the three-dimensional reconstruction of the flame is very difficult and there are few research results. Although existing studies have shown that the three-dimensional structure of the flame can be reconstructed to a certain extent through tomography, matching dense point clouds, and pasting textures, there are problems such as expensive equipment and complicated processes.

[0004] In 2015, Ren Limao used the method of multi-eye stereo vision and texture pasting to reconstruct flames in three-dimensional form in "Research on Flame Three-Dimensional Reconstruction Algorithm Based on Computer Vision". However, due to the high instantaneous speed of flames, multi-eye stereo vision cannot guarantee that multiple cameras can capture images at the same time, so the accuracy of this method is low. In 2020, Shang Jinghui published "Three-Dimensional Flame Real Field Extraction Based on Deep Learning". The article first reconstructed the temperature field of the flame, and then used the edge detection algorithm to obtain 256 section results and fused these section results to obtain the final real field of the flame. This algorithm has high accuracy, but the implementation process is complicated. Summary of the invention

[0005] The purpose of the present invention is to provide a single-camera flame 3D reconstruction depth estimation method based on optical flow, which solves the problems of complex operation and high cost in existing flame 3D modeling methods.

[0006] The technical solution adopted by the present invention is a single-camera flame 3D reconstruction depth estimation method based on optical flow, which specifically includes the following steps:

[0007] Step 1, build a single-camera flame 3D reconstruction scene;

[0008] Step 2, obtaining a flame image through the scene constructed in step 1;

[0009] Step 3, performing denoising on the flame image obtained in step 2;

[0010] Step 4: train the semantic segmentation network;

[0011] Step 5, input the flame image into the semantic segmentation network trained in step 4 to obtain an image with only the flame part;

[0012] Step 6: Build an optical flow estimation network, input the image obtained in step 5 into the optical flow estimation network, and obtain an optical flow map;

[0013] Step 7: Based on the optical flow map obtained in step 6, the depth of the flame is calculated according to the relationship between the optical flow and the depth of the flame.

[0014] The present invention is also characterized in that:

[0015] In step 1: the single-camera flame 3D reconstruction scene includes three modules, namely: module 1: including a computer, module 2: including a bracket, on which a camera is installed, module 3: including a stage, on which a candle is arranged, and behind which a black baffle is arranged.

[0016] The bracket is a liftable bracket, which ensures that the camera aperture center and the candle flame burning center are placed on the same horizontal line and ensures that the camera's field of view does not exceed the black baffle.

[0017] The specific process of step 2 is:

[0018] Light a candle, turn on the camera, capture video, use a computer to receive the video, and use Ubuntu 18.04+python3.7+opencv to extract frames from the video to get a flame image.

[0019] In step 3, the flame image is denoised using the denoising method of the Ubuntu 18.04+Python3.7+Opencv module.

[0020] The specific process of step 4 is:

[0021] Step 5.1: Use labelme to label the dataset and mark the flame part as the sample set. The sample set has two types of pixels: flame as the target pixel and background pixel. The sample set has more than 1000 images.

[0022] Step 5.2, use Ubuntu 18.04+Python3.7+Tensorflow1.15.0 for model training.

[0023] In step 6, use Ubuntu 18.04+Python3.6+Pytorch0.4.1 to build the optical flow estimation network Flownet2.0.

[0024] In step 7, the relationship between optical flow and depth is as follows:

[0025]

[0026] In the formula, t(t x ,t y ,t z ) is the speed of the pixel in the three coordinates of x, y, and z, k is the correspondence between the actual distance and the pixel distance, and Z is the depth information of the corresponding pixel.

[0027] The beneficial effect of the present invention is that, compared with the traditional flame 3D reconstruction method, the present method is characterized by simple operation and low cost. The calculation of the flame 3D model under complex working conditions is very difficult. The existing methods such as tomography, matching dense point clouds, and pasting textures can reconstruct the flame 3D structure to a certain extent, but there are problems such as expensive equipment and complicated processes. The present invention can estimate the depth of the flame using only one camera. Compared with the traditional flame depth estimation, 3D reconstruction, and stereo matching methods, it has the advantages of low cost and simple operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the experimental scene of the single-camera flame 3D reconstruction depth estimation method based on optical flow of the present invention;

[0029] Figure 2 It is an algorithm flow chart of the single-camera flame 3D reconstruction depth estimation method based on optical flow of the present invention;

[0030] Figure 3 It is a semantic segmentation flow chart of the single-camera flame 3D reconstruction depth estimation method based on optical flow of the present invention;

[0031] Figure 4 This is an effect diagram of the semantic segmentation part of the single-camera flame 3D reconstruction depth estimation method based on optical flow of the present invention;

[0032] Figure 5 This is an effect diagram of optical flow estimation of the single-camera flame 3D reconstruction depth estimation method based on optical flow of the present invention;

[0033] Figure 6 It is the flame depth map finally obtained by the single-camera flame 3D reconstruction depth estimation method based on optical flow of the present invention;

[0034] Figure 7 This is the flame surface reconstruction result diagram finally obtained by the single-camera flame 3D reconstruction depth estimation method based on optical flow of the present invention.

[0035] In the figure, 1. computer, 2. camera, 3. candle, 4. stage, 5. bracket, 6. black baffle. DETAILED DESCRIPTION

[0036] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] The present invention provides a single-camera flame 3D reconstruction depth estimation method based on optical flow, such as Figure 1 As shown, it includes three modules; Module 1: Computer processing module, including computer 1, used to process images and calculate results; Module 2: Visual perception module, including bracket 5, on which camera 2 is installed, module 2 is mainly used to collect flame video images; Module 3: Combustion module, including stage 4, on which candle 3 is arranged, and behind which black baffle 6 is arranged, the function of module 3 is mainly to build a combustion environment, and a black baffle is used as the background to ensure that the background of the captured image is single and there is no interference from other factors.

[0038] The bracket 5 is used to ensure that the aperture center of the camera 2 and the burning center of the candle 3 are placed on the same horizontal line, and the field of view of the camera 2 does not exceed the black baffle 6, and the video is collected, and the collected video signal is transmitted to the computer 1 for calculation.

[0039] The single-camera flame 3D reconstruction depth estimation method based on optical flow of the present invention first uses module three to build a combustion environment, places a candle 3 on a flat stage 4, places a black baffle 6 on the back and lights the candle; module two uses a bracket 5 to ensure that the aperture center of the camera 2 and the flame burning center of the candle 3 are placed near the same horizontal line, and ensures that the field of view of the camera 2 does not exceed the black baffle 6, performs video acquisition, and transmits the acquired video signal to module one for calculation; finally, module one obtains the flame combustion video stream transmitted by module two, such as Figure 2 As shown, first, frame extraction is performed. To ensure that the extracted image has obvious optical flow changes, one frame is extracted every two frames. Finally, two frames of images are obtained, and bilateral filtering is performed on them to achieve the effect of maintaining edges, reducing noise and smoothing. Then the flame image is input into the trained semantic segmentation network to obtain an image with only the flame part, and then it is input into the optical flow estimation network to obtain an optical flow map. Finally, the optical flow is converted into depth to obtain the result.

[0040] The specific steps are as follows:

[0041] Step 1, light candle 3;

[0042] Step 2, turn on camera 2 and collect video;

[0043] Step 3, use computer 1 to receive the video, and use Ubuntu 18.04+python3.7+opencv to extract frames from the video to obtain a flame image;

[0044] Step 4: Use the denoising method of Ubuntu 18.04+Python3.7+Opencv module to denoise the flame image;

[0045] Step 5, use Ubuntu 18.04+Python3.7+Tensorflow1.15.0 to train the semantic segmentation network deeplabV3+ network, such as Figure 3 shown, including:

[0046] Step 5.1: Use labelme to label the dataset and mark the flame part as the sample set. The sample set has two types of pixels: flame as the target pixel and background pixel. The sample set has more than 1000 images.

[0047] Step 5.2, use Ubuntu 18.04+Python3.7+Tensorflow1.15.0 for model training;

[0048] Use the trained network for testing to ensure the best results, such as Figure 4 shown.

[0049] Step 6: Input the flame image into the semantic segmentation network trained in step 5 to obtain an image with only the flame part, such as Figure 5 As shown;

[0050] Step 7, use Ubuntu 18.04+Python3.6+Pytorch0.4.1 to build the optical flow estimation network Flownet2.0, input the image obtained in step 6 into the optical flow estimation network to obtain the optical flow map;

[0051] Step 8: After obtaining the optical flow map, the depth of the flame is calculated based on the relationship between the optical flow and the depth, and the flame depth map is obtained. Figure 6 As shown, the relationship between optical flow and depth is as follows:

[0052]

[0053] In the formula, t(t x ,t y ,t z ) is the speed of the pixel point in the three coordinates of x, y, and z, k is the corresponding relationship between the actual distance and the pixel distance, and Z is the depth information of the corresponding pixel point. The reconstructed image obtained by the present invention is as follows Figure 7 shown.

[0054] The single-camera flame 3D reconstruction depth estimation method based on optical flow in the present invention can estimate the depth of flames under complex working conditions and detect their combustion status; with respect to the traditional multi-camera stereoscopic vision 3D reconstruction algorithm, the present invention uses only a single camera and does not need to perform a series of complex operations such as camera parameter calibration to obtain the depth information of the target object, thereby solving the problem of complex calibration of the traditional multi-camera algorithm; the problems existing in the traditional flame 3D reconstruction algorithm, such as high computational cost, expensive experimental equipment, and complex experimental operation, are optimized to a certain extent; with respect to the matching-based optical flow method used in optical flow estimation in traditional literature and the method of estimating optical flow by minimizing the joint energy function composed of optical flow constraints and three-dimensional constraints, the present invention uses Flownet2.0 to perform optical flow estimation, which can achieve better results while also improving the generalization of the algorithm.

Claims

1. A single-camera flame 3D reconstruction depth estimation method based on optical flow, characterized by: The specific steps include: Step 1, build a single-camera flame 3D reconstruction scene; In the step 1, the single-camera flame 3D reconstruction scene includes three modules, namely: module 1: including a computer, module 2: including a bracket, on which a camera is installed, module 3: including a stage, on which a candle is arranged, and behind which a black baffle is arranged; The bracket is a liftable bracket, which ensures that the camera aperture center and the candle flame burning center are placed on the same horizontal line, and ensures that the camera's field of view does not exceed the black baffle; Step 2, obtaining a flame image through the scene constructed in step 1; The specific process of step 2 is: Light a candle, turn on the camera, collect video, use a computer to receive the video, and use Ubuntu 18.04+python3.7+opencv to extract frames from the video to get a flame image; Step 3, performing denoising on the flame image obtained in step 2; Step 4: train the semantic segmentation network; Step 5, input the flame image into the semantic segmentation network trained in step 4 to obtain an image with only the flame part; Step 6: Build an optical flow estimation network, input the image obtained in step 5 into the optical flow estimation network, and obtain an optical flow map; Step 7: Based on the optical flow map obtained in step 6, the depth of the flame is calculated according to the relationship between the optical flow and the depth of the flame.

2. The single-camera flame 3D reconstruction depth estimation method based on optical flow according to claim 1 is characterized by: In step 3, the flame image is denoised using the denoising method of the Ubuntu 18.04+Python3.7+Opencv module.

3. The single-camera flame 3D reconstruction depth estimation method based on optical flow according to claim 1 is characterized by: The specific process of step 4 is as follows: Step 5.1: Use labelme to label the dataset and mark the flame part as the sample set. The sample set has two types of pixels: one is the flame as the target pixel and the other is the background pixel. The sample set has more than 1000 images. Step 5.2, use Ubuntu 18.04+Python3.7+Tensorflow1.15.0 for model training.

4. The single-camera flame 3D reconstruction depth estimation method based on optical flow according to claim 1 is characterized by: In step 6, Ubuntu 18.04+Python3.6+Pytorch0.4.1 is used to build the optical flow estimation network Flownet2.

0.

5. The single-camera flame 3D reconstruction depth estimation method based on optical flow according to claim 4 is characterized in that: In step 7, the relationship between optical flow and depth is as follows: （1）； In the formula, It is the speed of the pixel in the three coordinates of x, y, and z, k is the corresponding relationship between the actual distance and the pixel distance, and Z is the depth information of the corresponding pixel.

Images