Real-time extraction method for live wire of forest grass

Abstract

The invention discloses a forest grass live wire real-time extraction method, relates to the technical field of deep neural networks, and solves the problems of low live wire real-time extraction efficiency and low processing speed when an unmanned aerial vehicle monitors a forest grass fire. The method comprises the steps that 1, a control terminal obtains an unmanned aerial vehicle forest grass fire scene real-time return image; 2, performing data preprocessing on the unmanned aerial vehicle forest grass fire scene real-time returned image to obtain preprocessed data; the preprocessing comprises frame splitting processing and image size normalization; 3, performing real-time reasoning on the preprocessed data by using an improved lightweight and efficient forest and grass live wire segmentation model LiFE-BONet to obtain a binary segmentation image containing a live wire and a background; and step 4, carrying out frame combination processing on the binary segmented images containing the live line and the background, and carrying out combination in sequence to obtain segmented images corresponding to the forest grass fire scene images which are input and returned by the unmanned aerial vehicle in real time. The scheme has a good application prospect in the field of forest steppe fire monitoring and emergency response.

Description

Technical Field

[0001] The present invention relates to the field of deep neural network technology, and in particular to a real-time extraction method for forest and grass fire lines. Background Art

[0002] In recent years, drone technology has rapidly developed. The various sensors and cameras it carries can capture high-resolution, localized images, which offer richer data features than macroscopic satellite remote sensing images. When combating the spread of forest and grassland fires, rescue technicians use the real-time fire scene images transmitted by drones to identify the location and status of the fire front. This allows for a comprehensive assessment of the fire's spread and the development of scientific solutions to effectively contain the spread of the fire and minimize the loss of forest resources.

[0003] Currently, segmentation models based on drone imagery are widely used in the field of forest fires. This provides a solution for extracting forest and grass fire lines within a small area. Modern drones can be equipped with small CPUs and GPUs, as well as pre-trained onboard deep neural network models, to identify and segment fire points in real time. This is due to the model's small number of parameters, making it more flexible for embedding in mobile devices. Furthermore, leveraging the powerful computing power of CPUs and GPUs, frame-by-frame recognition and segmentation of drone-transmitted images can meet real-time performance requirements. This provides scientific guidance for forest firefighting technicians to quickly understand the status of fire lines during rescue operations.

[0004] The purpose of this paper is to use the improved lightweight and efficient forest and grass fire line segmentation model (LiFE-BONet) to perform real-time segmentation and extraction of forest fire lines during the spread of forest fires, in order to quickly obtain the characterization features of forest fire lines (such as shape, size, etc.) and provide scientific guidance for forest fire rescue. Summary of the Invention

[0005] In order to solve the problems of low efficiency and slow processing speed in real-time fire line extraction when unmanned aerial vehicles monitor forest and grass fires, the present invention provides a real-time forest and grass fire line extraction method, comprising the following steps:

[0006] Step 1: The control terminal obtains the real-time image sent back by the UAV at the forest and grass fire scene;

[0007] Step 2: Preprocess the real-time image transmitted back from the UAV to obtain preprocessed data; the preprocessing includes frame removal and image resizing and normalization;

[0008] Step 3: Use the lightweight and efficient forest and grass fire line segmentation model LiFE-BONet to perform real-time inference on the preprocessed data to obtain a binary segmentation map containing the fire line and background;

[0009] Step 4: The binary segmentation images containing the fire line and the background are frame-merged and combined in sequence to obtain a segmented image corresponding to the forest and grass fire scene image transmitted back by the input drone in real time.

[0010] Furthermore, in step 1, the real-time image transmitted back from the forest and grass fire scene by the drone is visible light that completely covers the fire scene and clearly shows the outline of the fire line.

[0011] Furthermore, in step 2, the frame removal process specifically includes:

[0012] Step 2.1: Initialize the video stream. Based on the Python OpenCV library, access the real-time video stream of the drone through the cv2.VideoCapture function.

[0013] Step 2.2, read the frame by frame loop, use the cap.read() method in the loop to capture the current frame, and return the result ret and frame data frame;

[0014] Step 2.3, frame data validity verification, in each loop, by judging the ret status value to detect whether the frame reading is successful, if ret is True, then enter the data processing flow to obtain valid frame data; if it is False, then exit the loop and end the video processing;

[0015] Step 2.4: Color space conversion: convert the successfully read frame from BGR color space to RGB color space;

[0016] Step 2.5: Standardize the image format and use Image.fromarray() to convert the RGB format frame to the PIL image format.

[0017] Furthermore, in step 2, the image resizing and normalization specifically includes:

[0018] Step 2.6: Resize the input image frame to a uniform resolution of 512×512 using transforms.Resize((512,512));

[0019] Step 2.7: Convert the tensor by using transforms.ToTensor() to convert the PIL image format to a tensor format and automatically normalize the pixel values to the range [0, 1].

[0020] Step 2.8: Normalization. Call transforms.Normalize to normalize each channel of the tensor according to the predefined mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] to obtain the preprocessed data.

[0021] Furthermore, step 3 specifically includes:

[0022] Step 3.1, the preprocessed data is passed to the improved lightweight and efficient forest-grass fire line segmentation model LiFE-BONet for forward reasoning to generate output results;

[0023] Step 3.2, use torch.max(output,1) to extract the category label corresponding to the maximum value of each pixel from the output result output;

[0024] Step 3.3, convert the output result from a tensor to a NumPy array through predicted.squeeze(0).cpu().numpy();

[0025] Step 3.4: Get the height and width of the predicted result to ensure that the segmentation map is consistent with the original image size;

[0026] Step 3.5: Input the height and width of the NumPy array and the prediction result, initialize the three-channel array segmentation_map, set the background to black, mark the fire line area to red, and finally obtain the binary segmentation map containing the fire line and background through color mapping filling.

[0027] Furthermore, step 4 specifically includes:

[0028] Step 4.1. Use cv2.resize to adjust the segmentation map to the same size as the original image frame to obtain the adjusted segmentation map and achieve spatial dimension alignment.

[0029] Step 4.2: Convert the adjusted segmentation map from BGR to RGB format through cv2.cvtColor to obtain the segmentation image corresponding to the forest and grass fire scene image transmitted back in real time by the input drone.

[0030] The beneficial effects of the present invention are:

[0031] In the method of the present invention, various control terminals process the forest and grass fire line spread images sent back by drones, and then process them through the improved lightweight and efficient forest and grass fire line segmentation model LiFE-BONet, highlighting the flexibility advantage of this method.

[0032] In the method of the present invention, during the data preprocessing process, the images sent back by drones of various resolutions can be uniformly resized to 512×512, and then processed by the model to obtain a 512×512 line of fire segmentation map, and finally framed to the original size, highlighting the standardization advantage of this method.

[0033] In the method of the present invention, the improved lightweight and efficient forest and grass fire line segmentation model LiFE-BONet can efficiently extract fire lines in forest fire spread in various scenarios, and because of its small number of parameters, it can be well embedded in various control terminals for deployment, highlighting the strong applicability of the method.

[0034] In the method of the present invention, the images sent back by the UAV are processed by a frame splitting and frame grouping processing method to obtain a real-time segmentation map of the forest and grass fire line, which highlights the real-time nature of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of a method for real-time extraction of forest and grass fire lines according to the present invention;

[0036] Figure 2 Extract fire line effects in different scenarios for the real-time extraction method of forest and grassland fire lines. DETAILED DESCRIPTION

[0037] To make the technical solutions and advantages of the embodiments of the present invention more clearly understood, exemplary embodiments of the present invention are further described in detail below with reference to the accompanying drawings. It should be noted that the embodiments described are only a portion of the embodiments of the present invention, and are not an exhaustive list of all embodiments. It should be noted that the embodiments of the present invention and the features thereof may be combined with each other unless they conflict.

[0038] Example 1, combined Figure 1 This embodiment describes a method for real-time extraction of forest and grass fire lines, comprising the following steps:

[0039] Step 1: The control terminal obtains the real-time image sent back by the UAV at the forest and grass fire scene;

[0040] Step 2: Preprocess the real-time image transmitted back from the UAV to obtain preprocessed data; the preprocessing includes frame removal and image resizing and normalization;

[0041] Step 3: Use the lightweight and efficient forest and grass fire line segmentation model LiFE-BONet to perform real-time inference on the preprocessed data to obtain a binary segmentation map containing the fire line and background;

[0042] Step 4: The binary segmentation images containing the fire line and the background are frame-merged and combined in sequence to obtain a segmented image corresponding to the forest and grass fire scene image transmitted back by the input drone in real time.

[0043] Specifically, the control terminal includes a PC terminal capable of carrying a GPU or a mobile embedded terminal of an NPU;

[0044] The improved lightweight and efficient forest-grass fire line segmentation model first selects the segmentation model U-Net as the baseline framework, then performs lightweight optimization on it, introduces the designed deep feature enhancement edge and boundary enhancement modules to obtain the final lightweight and efficient forest-grass fire line segmentation model LiFE-BONet.

[0045] The baseline model (U-Net) encoder is replaced with the lightweight Mobilevit2-Net. The feature extraction of the backbone feature extraction network Mobilevit2-Net has 6 stages, where each stage represents a different feature extraction layer and each stage contains several MV2Blocks and MobileViTBlocks. In order to adapt to the input channel and size of the backbone extraction network, the original U-Net first layer is retained and named stem_conv. Its function is to retain lower-dimensional boundary contour information while organizing the channel, so that the model can learn more low-dimensional features and reduce the number of model parameters to achieve model lightweight. Since the introduced feature extraction network has one more stage than the original convolutional feature extraction network and retains the stage1 of the original network (that is, the stem_conv of this network), the encoding part of the final model has 7 stages (1 stem_conv and 6 stages).

[0046] The depth feature enhancement edge (En-Feature) is mainly used to make up for the information loss problem caused by network deepening, especially in the complex background of forest fires. It aims to use auxiliary edges to retain the features of each depth stage, and then enhance the main features to improve the segmentation effect. The feature enhancement edge starts from stage2, retains the low-dimensional boundary contour information of the fire line, and then continuously accumulates the feature information of the next stage, and then fuses the features with each depth stage to obtain the depth enhancement feature. During the fusion process, due to the influence of the image size, the depth separable convolution (DW_Conv) is used to process the size of the feature map in order to adapt to the feature size of each depth stage.

[0047] The boundary enhancement module (SL) is a cascade of MaxPooling, Sobel and Laplace operators. The Sobel operator is a first-order partial derivative, which can effectively detect the edges of the image and provide structural information in the image. This helps the model identify the contours of objects, thereby improving the accuracy of segmentation. The Laplace operator is a second-order partial derivative, which can capture higher-order features and emphasize regional changes, which is particularly important for distinguishing objects of different categories. It can better handle noise and details and help the model learn more complex boundaries; the boundary contour information extracted from the shallow features is highlighted by MaxPooling and sent to the subsequent two operators for operation, which can provide richer edge and structural information in the boundary feature extraction stage, thereby improving the performance of the semantic segmentation model. This module first performs grayscale processing on the input feature map, then convolves the Gx and Gy masks to obtain the horizontal boundary feature S_X and the vertical boundary feature S_Y, and finally obtains Gs through a fixed formula. This is the operation process of the Sobel operator. The subsequent Laplace operator operation is similar. It only needs to replace the mask. Finally, the original boundary contour is accumulated to its output through the residual to obtain the final enhanced boundary contour feature map.

[0048] In step 1, the real-time image transmitted back from the forest and grass fire scene by the drone is visible light that completely covers the fire scene and clearly shows the outline of the fire line.

[0049] In step 2, the frame removal process specifically includes:

[0050] Step 2.1: Initialize the video stream. Based on the Python OpenCV library, access the real-time video stream of the drone through the cv2.VideoCapture function.

[0051] Step 2.2, read the frame by frame loop, use the cap.read() method in the loop to capture the current frame, and return the result ret and frame data frame;

[0052] Step 2.3, frame data validity verification, in each loop, by judging the ret status value to detect whether the frame reading is successful, if ret is True, then enter the data processing flow to obtain valid frame data; if it is False, then exit the loop and end the video processing;

[0053] Step 2.4: Color space conversion: convert the successfully read frame from BGR color space to RGB color space;

[0054] Step 2.5: Standardize the image format and use Image.fromarray() to convert the RGB format frame to the PIL image format.

[0055] In step 2, the image resize normalization specifically includes:

[0056] Step 2.6: Resize the input image frame to a uniform resolution of 512×512 using transforms.Resize((512,512));

[0057] Step 2.7: Convert the tensor by using transforms.ToTensor() to convert the PIL image format to a tensor format and automatically normalize the pixel values to the range [0, 1].

[0058] Step 2.8: Normalization. Call transforms.Normalize to normalize each channel of the tensor according to the predefined mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] to obtain the preprocessed data.

[0059] Step 3 specifically includes:

[0060] Step 3.1, the preprocessed data is passed to the improved lightweight and efficient forest-grass fire line segmentation model LiFE-BONet for forward reasoning to generate output results;

[0061] Step 3.2, use torch.max(output,1) to extract the category label corresponding to the maximum value of each pixel from the output result output;

[0062] Step 3.3, convert the output result from a tensor to a NumPy array through predicted.squeeze(0).cpu().numpy();

[0063] Step 3.4: Get the height and width of the predicted result to ensure that the segmentation map is consistent with the original image size;

[0064] Step 3.5: Input the height and width of the NumPy array and the prediction result, initialize the three-channel array segmentation_map, set the background to black, mark the fire line area to red, and finally obtain the binary segmentation map containing the fire line and background through color mapping filling.

[0065] Step 4 specifically includes:

[0066] Step 4.1. Use cv2.resize to adjust the segmentation map to the same size as the original image frame to obtain the adjusted segmentation map and achieve spatial dimension alignment.

[0067] Step 4.2: Convert the adjusted segmentation map from BGR to RGB format through cv2.cvtColor to obtain the segmentation image corresponding to the forest and grass fire scene image transmitted back in real time by the input drone.

[0068] Specifically, through Figure 2 According to the segmentation results of video frames at different times, we can clearly observe the dynamic spread trend of the fire line, the accuracy of the boundary contour and the environmental adaptability of the model, and verify the generalization ability of the algorithm for complex scenes.

[0069] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0070] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A method for real-time extraction of forest and grass fire lines, characterized in that: The following steps are involved: Step 1: The control terminal obtains the real-time image sent back by the UAV at the forest and grass fire scene; Step 2: Preprocess the real-time image transmitted back from the UAV to obtain preprocessed data; the preprocessing includes frame removal and image resizing and normalization; Step 3: Use the lightweight and efficient forest and grass fire line segmentation model LiFE-BONet to perform real-time inference on the preprocessed data to obtain a binary segmentation map containing the fire line and background; Step 4: The binary segmentation images containing the fire line and the background are frame-merged and combined in sequence to obtain a segmented image corresponding to the forest and grass fire scene image transmitted back by the input drone in real time.

2. The method for real-time extraction of forest and grass fire lines according to claim 1, characterized in that: In step 1, the real-time image transmitted back from the forest and grass fire scene by the drone is visible light that completely covers the fire scene and clearly shows the outline of the fire line.

3. The method for real-time extraction of forest and grass fire lines according to claim 1, characterized in that: In step 2, the frame removal process specifically includes: Step 2.1: Initialize the video stream. Based on the Python OpenCV library, access the real-time video stream of the drone through the cv2.VideoCapture function. Step 2.2, read the frame by frame loop, use the cap.read() method in the loop to capture the current frame, and return the result ret and frame data frame; Step 2.3, frame data validity verification, in each loop, by judging the ret status value to detect whether the frame reading is successful, if ret is True, then enter the data processing flow to obtain valid frame data; if it is False, then exit the loop and end the video processing; Step 2.4: Color space conversion: convert the successfully read frame from BGR color space to RGB color space; Step 2.5: Standardize the image format and use Image.fromarray() to convert the RGB format frame to the PIL image format.

4. The method for real-time extraction of forest and grass fire lines according to claim 1, characterized in that: In step 2, the image resize normalization specifically includes: Step 2.6: Resize the input image frame to a uniform resolution of 512×512 using transforms.Resize((512,512)); Step 2.7: Convert the tensor by using transforms.ToTensor() to convert the PIL image format to a tensor format and automatically normalize the pixel values to the range [0, 1]. Step 2.8: Normalization. Call transforms.Normalize to normalize each channel of the tensor according to the predefined mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] to obtain the preprocessed data.

5. The method for real-time extraction of forest and grass fire lines according to claim 1, characterized in that: Step 3 specifically includes: Step 3.1, the preprocessed data is passed to the improved lightweight and efficient forest-grass fire line segmentation model LiFE-BONet for forward reasoning to generate output results; Step 3.2, use torch.max(output,1) to extract the category label corresponding to the maximum value of each pixel from the output result output; Step 3.3, convert the output result from a tensor to a NumPy array through predicted.squeeze(0).cpu().numpy(); Step 3.4: Get the height and width of the predicted result to ensure that the segmentation map is consistent with the original image size; Step 3.5: Input the height and width of the NumPy array and the prediction result, initialize the three-channel array segmentation_map, set the background to black, mark the fire line area to red, and finally obtain the binary segmentation map containing the fire line and background through color mapping filling.

6. The method for real-time extraction of forest and grass fire lines according to claim 1, characterized in that: Step 4 specifically includes: Step 4.

1. Use cv2.resize to adjust the segmentation map to the same size as the original image frame to obtain the adjusted segmentation map and achieve spatial dimension alignment. Step 4.2: Convert the adjusted segmentation map from BGR to RGB format through cv2.cvtColor to obtain the segmentation image corresponding to the forest and grass fire scene image transmitted back in real time by the input drone.

Images