Ignition point positioning method under smoke shielding

Abstract

The invention provides an ignition point positioning method under smog shielding. The ignition point can be positioned when only visible light images exist and flames are completely or partially shielded by smog. The method comprises the following steps: acquiring each frame of image of a video, and performing image preprocessing on each frame of image, including improving the image contrast by using bilateral filtering denoising and histogram equalization; smooth filtering is carried out on the image sequence, and the subsequent feature matching complexity is reduced; determining an ROI (Region of Interest) in the image by utilizing the semantic segmentation result of the multi-frame image; determining the smoke flow direction in the ROI by using an optical flow method, and eliminating smoke offset caused by environmental factors such as wind power according to priori knowledge; and according to the segmented smoke region and the detected smoke flow direction, reversely positioning the position of the fire point. According to the method, the problem that the flame cannot be positioned due to smoke shielding in the visible light image is solved, and the fire point can be accurately positioned under the condition of less priori knowledge.

Description

Technical Field

[0001] The present invention relates to the technical field of fire-fighting image processing, and particularly to a method for locating a fire point under smoke occlusion. Background Art

[0002] With the continuous development of intelligent fire-fighting technology, the automatic positioning technology of fire points is increasingly used in various fire-fighting scenarios. The existing fire point positioning methods mainly include fire point positioning based on traditional image algorithms and fire point positioning based on visual neural networks. Fire point positioning based on traditional image algorithms relies on manual feature extraction, is sensitive to RGB features, but has poor robustness; for the fire point positioning method based on visual neural networks, its features depend on a large amount of training data and training methods, and the fire point positioning effect is greatly affected by the data annotation effect. For the situation where there is interference in a single-band image, such as smoke obscuring the flame, it will lead to inaccurate positioning or inability to position.

[0003] Multi-source image information fusion is a fire point positioning method that can reduce smoke interference. However, when it is used, at least two or more images of different bands are required, and the processing of multi-source images is relatively complex, with high requirements for computing power. Fire point positioning algorithms are mainly applied to intelligent fire-fighting equipment, and the hardware conditions are greatly restricted. Multi-source images rely on multiple sensors, and not all intelligent fire-fighting equipment can meet the requirements. The computing power limitation of the core processor will also affect the real-time performance of the multi-source image algorithm. Therefore, it cannot be applied to most occasions. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for locating a fire point under smoke occlusion in view of the limitations in the background art, which solves the problem of high hardware requirements in the prior art. The method of the present invention includes the following steps:

[0005] A method for locating a fire point under smoke occlusion includes the following steps:

[0006] Step 1: Obtain a video of a visible light camera, process it into an image stream using OpenCV, and process it with histogram equalization and bilateral filtering methods to obtain an input image;

[0007] Step 2: Input the input image into an improved DeepLabv3+ neural network for semantic segmentation. The neural network includes an encoder and a decoder, classifies each pixel in the image to identify the smoke area, and outputs it as a mask image;

[0008] Step 3: Create a window of size n to store the most recent n frames of images. When a new frame enters, the oldest frame is automatically removed, and at the same time, the weighted average value of all pixels in these n frames of images is calculated to generate a frame of smoothed image;

[0009] Step 4: Use the segmentation result of the image in the buffer to determine the union of all smoke regions, and use the minimum bounding rectangle of the union as the ROI region;

[0010] Step 5: Perform optical flow analysis within the ROI region. By comparing the dense optical flow within the ROI region obtained after applying the optical flow method to the current frame and the previous frame, identify the optical flow belonging to the smoke region; further judge whether there is an influence of wind and its direction based on the wind interference judgment, and determine the location of the fire source by backtracking the direction of smoke flow.

[0011] Preferably, in Step 1, the histogram equalization is as follows: Calculate the grayscale histogram of each frame of the image, calculate the cumulative distribution function of the image according to the histogram, and convert the grayscale value of the original image into a new grayscale value through a mapping function to achieve histogram equalization.

[0012] Preferably, in Step 1, the bilateral filtering method is as follows: Use the Gaussian function to calculate the spatial proximity and generate the spatial distance weight representing the spatial relationship between pixels; calculate the range weight based on the neighborhood difference between the original image and the guidance map to reflect the similarity of pixel values; multiply the spatial distance weight template and the range weight template, and perform normalization processing to obtain the final joint weight template; apply the joint weight template to weighted average all pixel values within the neighborhood of a specific position in the original image to obtain the new output value at this position, and process the entire image accordingly.

[0013] Preferably, in Step 2, for the encoder part of the neural network, the backbone feature extraction network uses depthwise separable convolution, and the enhanced feature extraction network part uses spatial pyramid pooling with dilated convolution; an improved DeepLabv3+ network adds a channel attention mechanism between the encoder and the decoder.

[0014] Preferably, the channel attention mechanism is as follows: Extract global spatial information through global average pooling and compress the spatial dimension; then generate the weight of each channel through two fully connected networks; finally, apply the learned weight to each channel of the input feature to enhance important channels and suppress unimportant channels.

[0015] Preferably, in Step 3, the pixel weighted average value is expressed as:

[0016]

[0017] where F smoothed is the smoothed frame, N is the number of frames in the buffer, F i is the pixel value of the i-th frame, w i is the weight of the i-th frame, satisfying The weights use exponential decay, giving higher weights to the most recent frames.

[0018] Preferably, in step 5, the wind interference judgment is as follows: in the case of no wind, the smoke optical flow distribution is between 0 and 180°, and the distribution in the first quadrant and the second quadrant is relatively uniform; according to the quadrant where the optical flow is located and the distribution ratio in the first quadrant and the second quadrant, it is judged whether the smoke is interfered by wind and the wind direction when the smoke is interfered by wind.

[0019] Preferably, when it is judged that there is no wind interference, the intersection area of the reverse extension line of the optical flow direction is used as the fire point; when it is judged that there is wind interference, the intersection area of the reverse extension line of the optical flow direction and the horizontal axis is used as the fire point.

[0020] Preferably, the horizontal axis is the dividing line between the optical flow from 0 to 180° and the optical flow from 180° to 360° in the optical flow map.

[0021] Preferably, in step 5, the search process is as follows: starting from the optical flow that meets the conditions at the uppermost part of the ROI area, search for the optical flow below in the reverse direction of the optical flow direction. After searching for the next optical flow point, the search direction becomes the reverse extension line direction of the new optical flow, and search downward in turn until the lowermost optical flow is searched, and the intersection of the reverse extension line of the lowermost optical flow is located as the fire point.

[0022] Compared with the prior art, the present invention adopts the above technical solutions and has the following beneficial effects:

[0023] (1) The present invention indirectly locates the flame by using the smoke flow direction, and solves the problem that the flame is blocked by smoke in the monocular visible light image, resulting in the inability to obtain the flame characteristics and thus unable to locate the fire point.

[0024] (2) Compared with the method of locating the flame by multi-source images, the present method has lower requirements for hardware, and only a visible light camera is required to realize the flame location.

[0025] (3) Compared with the existing fire point location algorithms under smoke occlusion, the present method has a smaller number of parameters, does not need to extract redundant features, and can realize the fire point location only by using image segmentation and the optical flow method, and is particularly easy to deploy on the mobile application side. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The following further specifically describes the present invention in conjunction with the drawings and specific embodiments, and the above and / or other advantages of the present invention will become clearer.

[0027] Figure 1 is the overall algorithm flow chart of an embodiment of the present invention;

[0028] Figure 2 is the schematic diagram of the model structure of the improved DeepLabv3+ network of an embodiment of the present invention;

[0029] Figure 3 Schematic diagram of locating a fire point by the reverse extension line of the optical flow direction in an embodiment of the present invention. Detailed implementation manners

[0030] The technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings.

[0031] The present invention can be implemented in many different forms and should not be considered limited to the embodiments described herein. On the contrary, these embodiments are provided so that the present invention will be thorough and complete, and will fully convey the scope of the present invention to those skilled in the art.

[0032] The present invention discloses a method for locating a fire point under smoke occlusion, which does not require training a diffusion model, does not need to know the fire point position in advance, and only uses optical flow to determine the fire point, solving the problem that the flame cannot be located due to smoke occlusion in visible light images, and can accurately locate the fire point under the condition of less prior knowledge.

[0033] As Figures 1 to 3 shown,

[0034] Step 1: Obtain a video of a visible light camera, process it into an image stream using OpenCV, and process it by combining histogram equalization and bilateral filtering methods to obtain an input image.

[0035] Specifically,

[0036] Step 1.1: Calculate the grayscale histogram of each frame of the image, calculate the cumulative distribution function of the image according to the histogram, and convert the grayscale value of the original image into a new grayscale value through a mapping function to achieve histogram equalization, thereby improving the image contrast. The equalized image obtained is used as a guidance map.

[0037] Step 1.2: Use a Gaussian function to calculate the spatial proximity to generate a spatial distance weight representing the spatial relationship between pixels; calculate the range weight based on the neighborhood difference between the original image and the guidance map to reflect the similarity of pixel values; multiply the spatial distance weight template and the range weight template, and perform normalization processing to obtain a final combined weight template; finally, apply the combined weight template to perform weighted averaging on all pixel values in the neighborhood of a specific position of the original image to obtain a new output value at this position, and process the entire image in this way to obtain the input image. The image processing method of the present application combines the ideas of histogram equalization and bilateral filtering, enhancing both the overall contrast of the image and retaining local details.

[0038] Step 2: Input the obtained image into an improved DeepLabv3+ neural network for semantic segmentation. The neural network includes an encoder and a decoder. The neural network extracts image features through the convolutional layers of the encoder, classifies pixel points in the decoder part to identify the smoke area, and outputs it as a mask image.

[0039] The neural network can be any segmentation network with multi-scale adaptability. To improve the overall response speed of the algorithm and obtain a better segmentation effect, an improved DeepLabv3+ network is used in this embodiment.

[0040] The DeepLabv3+ network uses an encoder-decoder structure. As Figure 2 shown, the encoder can accept a sequence of variable length as input and convert it into an encoded state with a fixed shape, and the decoder can map the encoded state with a fixed shape into a sequence of variable length to generate the required output.

[0041] In the encoder part, the backbone feature extraction network uses depthwise separable convolutions to reduce the number of convolutional kernel parameters, and the spatial pyramid pooling with dilated convolutions is used in the enhanced feature extraction network, which can enhance the scale adaptability while increasing the receptive field.

[0042] To increase the weight ratio of important features, a channel attention mechanism is used between the encoder and the decoder. First, global spatial information is extracted through global average pooling to compress the spatial dimension; second, a two-layer fully connected network is used to generate the weight of each channel; finally, the learned weights are applied to each channel of the input features to enhance important channels and suppress unimportant channels.

[0043] In this embodiment, the neural network adopted is a semantic segmentation network, and the segmentation area is more accurate, without the need to use other algorithms to extract the contour based on the detection box.

[0044] Step 3, create a window (i.e., buffer) of size n to store the most recent n frames of images. In this embodiment, a data structure with a fixed size is used, n = 3. Whenever a new frame arrives, the oldest frame is removed, and at the same time, the weighted average of all pixels in these three frames of images is calculated to generate a frame of smoothed image, which helps to reduce noise and stabilize subsequent analysis.

[0045] The calculation formula for the pixel weighted average is:

[0046]

[0047] where F smoothed is the smoothed frame, N is the number of frames in the buffer, set to 3, F i is the pixel value of the i-th frame, w i is the weight of the i-th frame, satisfying The weights use exponential decay, giving higher weights to the most recent frames.

[0048] Step 4: Using the segmentation results of the images in the buffer, find the union of all smoke regions, and use the minimum bounding rectangle of the union as the ROI (region of interest) to focus on the possible range of smoke activities.

[0049] Specifically,

[0050] Obtain the mask images of three frames of images from the buffer. The mask image is usually a binary image, where the regions of interest (such as smoke) are marked as non-zero values (usually 255), and other regions are zero. Perform a logical "OR" operation on the mask images of these three frames of images, that is, take the union, to obtain the common smoke region of the three frames of images, and include the common smoke region in all frames in a mask image. Use an image processing library such as OpenCV to calculate the bounding rectangle of the merged mask image. Determine the ROI region in the image according to the bounding rectangle. The ROI region is the key region for subsequent image processing operations. Finally, use the copyTo function of OpenCV or a similar method to crop the original image to the ROI region.

[0051] Step 5: Perform optical flow analysis and wind interference judgment within the ROI region, and use the smoke flow map after excluding wind interference to extend backward step by step to determine the fire point location.

[0052] Specifically,

[0053] As Figure 3 shown, from prior knowledge, it can be known that in the case of no wind, the optical flow of smoke is usually distributed between 0 and 180°, and the distribution in the first and second quadrants is relatively uniform. Based on this, it can be judged whether the smoke is affected by wind. Therefore, by comparing the dense optical flow within the ROI region obtained after applying the optical flow method for the current frame and the previous frame, the optical flow belonging to the smoke region can be identified, and further judge the distribution ratio of the directions of these optical flows in the first and second quadrants, so as to determine whether there is the influence of wind and its general direction.

[0054] Specifically, under windless conditions, the smoke will naturally disperse, and the direction of its optical flow may be diverse. For example, it may spread to the upper left and upper right. In this case, the reverse extension lines of the optical flow on the left and right sides may intersect at a point, and this intersection area is defined as the fire point. Under windy conditions, for example, when the wind blows to the left, the smoke will mainly disperse to the left. Assuming that the optical flow is in an ideal state at this time, the optical flow flows parallelly without a focus, and all the optical flows uniformly point to the upper left. To find an "intersection point" in this situation, a horizontal axis (i.e., the dividing line between the optical flow from 0° to 180° and the optical flow from 180° to 360°) needs to be determined, and the intersection area between the reverse extension line of the optical flow direction and the horizontal axis is defined as the fire point. The search process is as follows: Starting from the optical flow that meets the conditions (distributed between 0 and 180°) at the top of the ROI area, search for the optical flow below in the reverse direction of the optical flow direction. After searching for the next optical flow point, the search direction becomes the reverse extension line direction of the new optical flow, and so on, searching downwards until the bottom optical flow is reached. The intersection point of the reverse extension line of the bottom optical flow (the intersection point of the reverse extension lines of each optical flow under windless conditions, and the intersection point of the reverse extension line of the optical flow and the horizontal axis under windy conditions) is located as the fire point.

[0055] The present invention provides a method for locating a fire point under smoke occlusion. There are many methods and ways to specifically implement this technical solution. The above description is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention. Each component not clearly defined in this embodiment can be implemented by using the prior art.

Claims

1. A method for locating a fire point under smoke cover, characterized in that: The following steps are involved: Step 1, obtain the visible light camera video, process it into an image stream using OpenCV, and obtain the input image by combining histogram equalization and bilateral filtering methods; Step 2, inputting the input image into an improved DeepLabv3+ neural network for semantic segmentation, wherein the neural network includes an encoder and a decoder, classifying each pixel in the image to identify the smoke area, and outputting it as a mask image; Step 3: Create a window of size n to store the most recent n frames of images. When a new frame enters, the oldest frame is automatically removed. At the same time, the weighted average of all pixels in these n frames of images is calculated to generate a smoothed image. Step 4, using the segmentation result of the image in the buffer, determine the union of all smoke areas, and use the minimum circumscribed rectangle of the union as the ROI area; Step 5: Perform optical flow analysis in the ROI area. By comparing the dense optical flows in the ROI area obtained by applying the optical flow method to the current frame and the previous frame, identify the optical flow belonging to the smoke area. Based on the wind interference judgment, further determine whether there is wind influence and its direction, and determine the location of the fire source by reversely tracing the direction of the smoke flow.

2. The method according to claim 1, characterized in that In step 1, the histogram equalization is as follows: the grayscale histogram of each frame image is calculated, the cumulative distribution function of the image is calculated according to the histogram, and the grayscale value of the original image is converted into a new grayscale value through a mapping function to achieve histogram equalization.

3. The method according to claim 2, characterized in that In step 1, the bilateral filtering method is: using a Gaussian function to calculate spatial proximity and generate a spatial distance weight representing the spatial relationship between pixels; calculating a range weight based on the neighborhood difference between the original image and the guided image to reflect the similarity of pixel values; The spatial distance weight template and the range weight template are point-multiplied and normalized to obtain the final joint weight template; the joint weight template is applied to perform weighted averaging on all pixel values ​​in the neighborhood of a specific position of the original image to obtain a new output value for that position, and the entire image is processed accordingly.

4. The method according to claim 1, characterized in that In step 2, in the encoder part of the neural network, the backbone feature extraction network adopts depthwise separable convolution, and the enhanced feature extraction network part adopts spatial pyramid pooling with dilated convolution; The improved DeepLabv3+ network adds a channel attention mechanism between the encoder and decoder.

5. The method according to claim 4, characterized in that The channel attention mechanism is as follows: global spatial information is extracted through global average pooling to compress the spatial dimension; the weight of each channel is then generated through a two-layer fully connected network; finally, the learned weight is applied to each channel of the input feature to enhance important channels and suppress unimportant channels.

6. The method according to claim 1, characterized in that In step 3, the pixel weighted average is expressed as: Among them, F smoothed is the smoothed frame, N is the number of frames in the buffer, F i is the pixel value of the i-th frame, w i The weight of the i-th frame satisfies The weights use exponential decay, giving higher weight to recent frames.

7. The method according to any one of claims 1 to 6, characterized in that: In step 5, the wind interference is judged as follows: in the case of no wind, the smoke optical flow is distributed between 0 and 180°, and the distribution in the first quadrant and the second quadrant is relatively uniform; based on the quadrant where the optical flow is located and the distribution ratio in the first quadrant and the second quadrant, it is judged whether the smoke is disturbed by the wind and the wind direction when it is disturbed by the wind.

8. The method according to claim 7, characterized in that When it is judged that there is no wind interference, the intersection area of ​​the reverse extension line of the optical flow direction is taken as the fire point; when it is judged that there is wind interference, the intersection area of ​​the reverse extension line of the optical flow direction and the horizontal axis is taken as the fire point.

9. The method according to claim 8, characterized in that The horizontal axis is 0 to 180 degrees in the optical flow map The dividing line between optical flow and 180° to 360° optical flow.

10. The method according to claim 8, characterized in that In step 5, the search process is: starting from the optical flow that meets the conditions at the top of the ROI area, search the optical flow below in the opposite direction of the optical flow direction. After searching the next optical flow point, the search direction is changed to the direction of the reverse extension line of the new optical flow, and the search is continued downward until the bottom optical flow is found. The intersection of the reverse extension line of the bottom optical flow is located as the fire point.

Images