Unmanned aerial vehicle monocular vision forest fire positioning method

Abstract

The invention discloses an unmanned aerial vehicle monocular vision forest fire positioning method, and aims to realize rapid detection, accurate positioning and three-dimensional reconstruction of an early fire. According to the method, a comprehensive fire monitoring system is constructed by integrating various sensors such as an unmanned aerial vehicle platform, a visible light camera, an infrared camera and an inertial measurement unit (IMU). Through monocular vision simultaneous localization and mapping (SLAM) technology and IMU-GNSS information fusion, the attitude of the camera is recovered in real time. And carrying out three-dimensional positioning on a fire point by utilizing a triangulation method and combining semantic information of visible light and infrared images. According to the method, the SfM (Structural From Motion) technology is adopted, the attitude of a camera and image feature points are combined, and the attitude of the camera and the three-dimensional point cloud are optimized through GBA (Global Bundle Administration), so that the positioning precision is improved. The method has the characteristics of high efficiency, accuracy, real-time performance and remarkable cost effectiveness.

Description

Technical Field

[0001] The invention belongs to the field of aviation technology, and in particular relates to a method for an unmanned aerial vehicle to realize monocular vision forest fire positioning. Background Art

[0002] In recent years, drone technology has experienced rapid development. It has not only been widely used in the military field, but has also gradually expanded to the civilian field, including environmental monitoring, disaster assessment and other aspects. Drones have the characteristics of strong flexibility, wide coverage and high operating efficiency, making them an important tool for forest fire monitoring and positioning. The high-resolution camera carried by drones can capture clear forest images, which provide an important data source for fire positioning. Through image analysis technology, fire features such as fire points and smoke can be identified, so as to accurately locate the fire. With the continuous advancement of computer vision and image processing technology, the images captured by drones can be efficiently processed and analyzed. Fire features such as fire points and smoke can be identified through algorithms, and combined with the positioning information of drones, accurate positioning of fires can be achieved. Drones are usually equipped with positioning systems such as GPS, which can obtain their own location information in real time. At the same time, combined with sensors such as inertial measurement units (IMUs), the accuracy and stability of positioning can be further improved. This provides reliable location information support for fire positioning. Existing positioning algorithms generally rely on binocular cameras or depth cameras to obtain the depth information of the target, resulting in high equipment costs. In addition, the cameras equipped on drone systems are generally fixed-angle, making it difficult to locate fires at abnormal angles. They are less versatile and difficult to be widely used. However, drone systems equipped with gimbal cameras can rotate the camera angle at will, and can accurately locate forest fires at abnormal angles.

[0003] However, existing pure visible light detection technology is difficult to accurately detect strong light and complex forest environments. Traditional methods often have difficulty in timely detecting fires in the early stages of a fire, especially when the fire is small and the smoke is light. Dual-light technology, especially infrared thermal imaging technology, can detect tiny temperature changes on the surface of an object, thereby detecting signs of a fire earlier. The forest environment is complex, and traditional positioning methods may reduce positioning accuracy due to factors such as tree obstruction and terrain undulations. The dual-light system carried by the drone combined with GPS, inertial navigation system (INS) and image processing technology can significantly improve the positioning accuracy of the fire location. The present invention proposes an airborne forest fire positioning algorithm based on dual light. By introducing infrared images to assist visible light detection to accurately locate the fire, and for more remote fire points in the picture, the rotation of the gimbal is controlled to achieve the positioning of the drone in the early forest fire while hovering. Summary of the invention

[0004] The purpose of the present invention is to provide a dual-light based UAV-mounted early forest fire positioning algorithm, which solves the problems of difficulty in early fire detection, insufficient positioning accuracy, and large depth information error in the prior art.

[0005] The specific implementation steps of the present invention are as follows:

[0006] Step 1: preprocess the infrared image and the visible light image, establish a matching relationship between the two-light images, and establish a spatial transformation model so that the transformed infrared image and the visible light image are aligned as much as possible in space.

[0007] Step 2: Use the triangulation calibration method to calculate the camera parameters, use the ORB-SLAM2 algorithm to restore the camera posture when shooting the fire, and perform corresponding depth information registration.

[0008] Step 3: Use the fire semantic information provided by the improved Deeplabv3+ segmentation algorithm and the fire center point information segmented from the infrared image to match the image features of the 3D points to obtain the corresponding depth information.

[0009] Step 4: Use triangulation to calculate the depth information error calculated in step 3 and make corrections.

[0010] Step 5: Calculate the relative distance between the fire point and the UAV and the specific global geographic location coordinates of the fire point based on the corresponding depth information combined with the rotation angle of the gimbal camera and the IMU data of the camera to complete the positioning of the fire point.

[0011] The present invention is also characterized in that:

[0012] Step 1 is implemented according to the following steps:

[0013] Step 1.1: Preprocess the infrared image and visible light image, including denoising, contrast enhancement, grayscale, etc., to improve the accuracy and efficiency of subsequent processing, and then extract ORB feature points with high distinguishability and stability from the preprocessed images for matching between different images.

[0014] Step 1.2: Based on the extracted feature points or feature areas, a matching relationship is established between the infrared image and the visible light image. Based on the matching point pairs, a spatial transformation model is established, and the model parameters are solved through an optimization algorithm so that the transformed infrared image and the visible light image are aligned as much as possible in space.

[0015] Step 2 is implemented as follows:

[0016] Step 2.1: Camera calibration: Use a calibration plate of known size (such as a chessboard), take multiple images of the calibration plate at different angles and positions, then use image processing technology to detect feature points (such as corner points) on the calibration plate, and use mathematical optimization methods to solve the internal and external parameters of the camera.

[0017] Step 2.2, ORB-SLAM2 extracts ORB feature points from each frame of the image, performs feature matching between consecutive frames, and establishes the correspondence between frames. Based on the feature matching results, ORB-SLAM2 estimates the initial pose of the camera using epipolar geometry and PnP (Perspective-n-Point) algorithm. Then, the camera's pose and map points are optimized through local BA (Bundle Adjustment). Some key frames are retained, and local maps are built on these key frames. The pose of the key frames and the observed map points are optimized through local BA. In order to eliminate the accumulated error, it is detected whether the camera has returned to the previously visited position, and the global pose is corrected through pose graph optimization.

[0018] Step 2.3, depth information registration, using triangulation technology for each pair of matching feature points, solve the coordinates of the point in three-dimensional space through the epipolar geometry equation. The three-dimensional coordinates of the feature points are iteratively optimized by minimizing the reprojection error (i.e., the difference between the observed image coordinates and the image coordinates calculated based on the three-dimensional coordinates and camera parameters). For each frame of the image, all the three-dimensional coordinates of the feature points obtained by triangulation can be converted into the depth map corresponding to the image. Each pixel value in the depth map represents the depth of the scene point corresponding to the pixel.

[0019] Step 3 is implemented as follows:

[0020] Step 3.1, semantic segmentation of fire in visible light images. The images acquired by the drone are input into the improved Deeplabv3+ segmentation network. The Deeplabv3+ network with a codec structure and a dilated spatial convolutional pooling pyramid has good segmentation accuracy for multi-scale targets, but the detection accuracy for small targets is low. Therefore, the cross attention mechanism and the ECA attention mechanism are used to enhance the feature expression ability of the model, and the lightweight upsampling operator is used to restore the network to a clearer target boundary, so that the segmentation accuracy of the improved network is higher.

[0021] Step 3.2: Perform fire threshold segmentation on the infrared camera image, and perform morphological operations on the processed result image to identify the high-temperature point, that is, the center of the fire.

[0022] Step 3.3: After detecting a fire, plan a reasonable flight trajectory (such as flying sideways to generate sufficient parallax), and use a feature matching algorithm to match the feature points of the fire area in the visible light image with the feature points of the fire center in the infrared image. Map the semantic segmentation results and feature matching results into 3D space, and calculate the depth information between the drone and the fire point.

[0023] Step 4 is implemented according to the following steps:

[0024] Step 4.1: Reconstruct the points in the three-dimensional space using the triangulation principle based on the camera parameters and the matched feature point pairs.

[0025] Step 4.2: Reproject the reconstructed 3D points back to the image plane using the camera parameters, and calculate the pixel distance between the projected points and the original feature points, i.e., the reprojection error.

[0026] Step 4.3: Minimize the reprojection error through an iterative optimization algorithm (Bundle Adjustment) to further improve the accuracy of the 3D point cloud.

[0027] Step 5 is implemented according to the following steps:

[0028] Step 5.1: Perform camera pose estimation: Combine the camera's motion trajectory and image feature points to restore the camera's pose (position and orientation).

[0029] Step 5.2: Combine the calculated depth information with the camera posture to calculate the relative position relationship between the fire point and the drone, and use GNSS technology to calculate the specific geographic location information of the fire point.

[0030] Step 5.3: After completing the fire point location, save the fire point information. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is an overall flow chart of the unmanned aerial vehicle monocular vision forest fire positioning method of the present invention;

[0032] Figure 2 It is a schematic diagram of triangulation between two figures of the present invention;

[0033] Figure 3 This is a schematic diagram of the reprojection error of the present invention;

[0034] Figure 4 It is a schematic diagram of the geometric relationship of the dual-light observation of the same hot spot in the present invention;

[0035] Figure 5 This is a schematic diagram of the area of ​​interest for the fire point of the present invention;

[0036] Figure 6The present invention is directed to a semantic segmentation network diagram based on improved Deeplabv3+;

[0037] Figure 7 A schematic diagram of the cross attention calculation of the present invention;

[0038] Figure 8 A schematic diagram of effective channel attention calculation for the present invention;

[0039] Fig. 9 This is a schematic diagram of the structure of a lightweight upsampling operator according to the present invention;

[0040] Fig.10 It is a schematic diagram of the depth experiment scene of the present invention;

[0041] Fig.11 This is a diagram of the dual light segmentation result of the present invention; DETAILED DESCRIPTION

[0042] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] The present invention UAV monocular vision forest fire positioning method, the flow chart is as follows Figure 1 As shown, the specific implementation steps are as follows:

[0044] Step 1.1, preprocess the infrared image and visible light image, including denoising, contrast enhancement, grayscale, etc., to improve the accuracy and efficiency of subsequent processing, and then extract ORB feature points with high distinguishability and stability from the preprocessed images for matching between different images;

[0045] Step 1.2: Based on the extracted feature points or feature areas, a matching relationship is established between the infrared image and the visible light image. The spatial relationship between the infrared camera and the visible light camera can be described by the camera extrinsics (rotation matrix R and translation vector t), that is, a point i on the infrared image is projected to the visible light camera, where its coordinates in the camera are set to p ir and p vi , and the pixel value in the image coordinates is recorded as q ir and q vi , the projection process can be expressed as:

[0046]

[0047] Where K is the camera intrinsic matrix (visible light or infrared camera), and d is the depth of the three-dimensional point i. The projection of the same point in the infrared camera and visible light camera coordinate systems also satisfies the following formula:

[0048] p ir =Rp vi +t (2)

[0049] In actual use, p ir and p vi The ratio is about 1, and because R, t, p ir and p vi are all constants based on the camera, so equation (2) can be rewritten as:

[0050]

[0051] In the formula and t′=K ir t is the unknown parameter of different cameras, which can be estimated by the least square method. When the number of measurements at different depths is greater than 6, a linear least square problem can be constructed to solve it. Therefore, a model can be obtained to describe the transformation relationship between infrared images and visible light images.

[0052] Step 2: Use the triangulation calibration method to calculate the camera parameters, use the ORB-SLAM2 algorithm to restore the camera posture when shooting the fire, and perform the corresponding depth information registration;

[0053] Combination Figures 2 to 5 , step 2 is implemented according to the following steps:

[0054] Step 2.1: Camera calibration: Use a 1m*1m calibration plate, take multiple images of the calibration plate at different angles and positions, and then use image processing technology to detect feature points (such as corner points) on the calibration plate to solve the internal and external parameters of the camera.

[0055] Step 2.2: Initialization of monocular camera pose recovery. At the beginning of monocular camera pose recovery, both the 3D points and the camera pose are unknown. The isomorphic matrix H or the essential matrix E is used to estimate the initial motion. Considering that the points around the fire are on an approximate plane, the isomorphic matrix can estimate the motion more accurately than the essential matrix, such as Figure 2 As shown, a 3D point P is projected onto two image planes, and the projection points on the image planes are p1 and p2 respectively. According to the definition of the isomorphic matrix, there is a translation relationship between the two points:

[0056]

[0057] In the formula To make them equal in scale, we can expand the above equation to get:

[0058]

[0059] Removing the constant from the formula and re-expanding it yields the following formula:

[0060] lh1v1+h2v1+h3-h7u1u2-h8v1u2=u2 (5)

[0061] h4u1+h5v1+h6-h7u1v2-h8v1v2=v2 (6)

[0062] A pair of matching points can construct 2 constraints. Therefore, at least 4 pairs of matching are required to solve 8 unknown parameters. Stack all 8 equations in a row to form a linear equation:

[0063]

[0064] Then calculate the rotation matrix and translation matrix by decomposing H.

[0065] As for the essential matrix E, according to the extreme outer constraint, the essential matrix E = t ∧ R, which is a 3×3 matrix containing 9 unknown variables. In order to estimate the essential matrix from the feature matching, the eight-point algorithm is used. Consider a pair of matching points, whose standardized coordinates are: x1 = [U1, v1, 1] T , x2=[u2,v2,1] T , and satisfy the following

[0066]

[0067] Write the matrix E in vector form:

[0068] e=[e1, e2, e3, e4, e5, e6, e7, e8, e9] T (9)

[0069] Then the polar constant can be written in linear form:

[0070] [u2u1,u2v1,u2,v2u1,v2u1,u2,u1,v1,1]·e=0 (10) All points are superimposed into an equation to obtain a linear equation system:

[0071]

[0072] Perform singular value decomposition on E, and the rotation matrix R and translation vector t can be obtained as follows:

[0073]

[0074] In order to avoid the influence of degradation and noise, the homomorphic matrix H and the essential matrix E are usually estimated simultaneously. Then the original motion of the camera is restored using the camera with smaller reprojection error.

[0075] Although ORB-SLAM2 estimates the camera pose for each frame, only those potential fire points with appropriate parallax are selected to participate in the fire point depth estimation. This is to suppress the noise caused by insufficient baseline. During each depth estimation process, the gimbal angle is fixed, so the translation vector of the fixed body coordinate system is equal to the translation vector of the fixed camera coordinate system. The scale estimate of ORBSLAM2 is then corrected using the translation amount provided by the drone platform, as shown in the formula.

[0076]

[0077] In the formula, s i is the true scale of the i-th frame, T is the translation vector, the superscript ref,i and the subscripts b,c are the coordinate system index and the coordinate system respectively, is the scale of the i-th frame in the camera-fixed coordinate system.

[0078] Step 3: Use the fire semantic information provided by the improved Deeplabv3+ segmentation algorithm and the fire center point information segmented from the infrared image to match the image features of the 3D points to obtain the corresponding depth information.

[0079] Combination Figures 6 to 9 , step 3 is implemented according to the following steps:

[0080] Step 3.1, stacking multiple hole convolutions will cause the network to have problems such as "grid effect" and loss of relevance of long-distance information. The Criss-Cross Attention Module can model the feature map context with extremely high efficiency, thereby obtaining long-distance context information. This module decomposes the self-attention structure into row attention and column attention, and approximates a dense self-attention calculation through two consecutive calculations, thus reducing the time and space complexity of the self-attention structure from O(N^2) to O(N), which not only achieves the same effect as the self-attention structure, but also significantly reduces the computational cost. Use the lightweight network ShuffleNetv2 to replace the Xception network for feature extraction, reduce the computational complexity and parameter amount of the network, and increase the network's running speed;

[0081] Step 3.2. In the semantic segmentation network, the shallow features of the image obtained by the decoder are crucial for the segmentation results after reconstructing the resolution. However, due to the large size of the shallow feature map obtained by the decoder, it is difficult for the network to learn the boundary information of important targets. Therefore, it is very important to embed a lightweight attention module to help the decoder better learn the shallow important features. Effective Channel Attention is a lightweight attention. However, studies have shown that dimensionality reduction operations reduce the effect of channel attention, and obtaining the relationship between all channels brings too much computation. ECA attention does not require dimensionality reduction operations and realizes local cross-channel interaction of features through one-dimensional convolution; thereby assigning different weights to different channels, that is, emphasizing important channel information and suppressing unimportant channel information. Embedding a recurrent cross attention module after the ASPP module and combining the cross attention mechanism with multi-scale information can further enhance the model's ability to capture global contextual information and improve the network's ability to extract pixel-level features;

[0082] Step 3.3, upsampling operation plays an irreplaceable role in the process of reconstructing the resolution of the semantic segmentation network. In the semantic segmentation task, it is usually necessary to reconstruct the low-resolution feature map into a high-resolution feature map, and then use the feature map for pixel-level prediction results to more accurately locate and identify the target area in the image. The upsampling operation can restore the detail information lost in the image during the downsampling process by increasing the number of pixels in the image and expanding the width and height of the image. This enables the network to understand the image content more finely and improve the accuracy and performance of segmentation. Traditional upsampling methods generally use the spatial position of the current pixel to determine the upsampling kernel without considering the correlation of adjacent pixels, resulting in blur and distortion of the reconstructed image. Although the transposed convolution can adaptively generate high-resolution output according to the size of the input feature map, it will make it difficult for the network to converge, and the chessboard effect will occur, affecting the quality of the generated image. The ECA attention module is embedded in the front end of the decoder structure, and the shallow feature map output by the deep convolutional network is input into the attention module to enhance the perception of important features of the deep neural network;

[0083] Step 3.4, the Content-Aware ReAssembly of Features (CARAFE) operator is a general, lightweight, and efficient upsampling operator. This operator does not have a fixed kernel and can dynamically generate kernels according to specific tasks; in addition, the receptive field obtained by this operator is much larger than that of traditional methods, and CARAFE has little computational overhead and is easy to integrate into the existing network architecture. Using the lightweight upsampling operator, CARAFE, instead of the traditional bilinear interpolation, adaptive upsampling is performed according to the image content to help improve the accuracy of image segmentation.

[0084] The visible light image aligned with the infrared image is input into the visible light forest fire segmentation network based on the improved Deeplabv3+, and the network will output the semantic segmentation result map after segmentation.

[0085] Step 3.5: After acquiring the infrared image, considering that it is difficult to directly separate the chromaticity and brightness information in the RGB color space, the image is converted from the RGB color space to the HSV color space. The conversion formula is as shown below:

[0086]

[0087]

[0088] In the HSV color space, H stands for hue, S stands for saturation, and V stands for value (also called brightness). In the Arctic isothermal color adjustment mode, the higher the temperature, the brighter the target of the infrared image. Therefore, the infrared image can be thresholded using the brightness channel of the converted image to segment the fire point and the background of the infrared image. The target image is segmented using the color space and threshold segmentation algorithm, and then the morphological operation is used to suppress the noise, and finally the sliding window method is used to find the location of the fire point.

[0089] Among them, g(x,y) is the grayscale value of the pixel at (x,y) in the image after segmentation, T is the segmentation threshold, which is obtained by experiment, and f(x,y) is the Y channel value of the image before segmentation.

[0090] After the image is thresholded, discrete pixels will appear around the segmented target. The opening operation in morphology can act as a filter to eliminate the interfering noise in the image. The operation of first erosion and then expansion can smooth the contour of the segmented target, disconnect narrow discontinuities and eliminate small protrusions.

[0091] The sliding window method is used to traverse the image after the thresholding operation. By traversing the local area (i.e., window) in the image, the pixel values ​​in each window are counted, and then the non-maximum suppression algorithm (Non-Maximum Suppression) is used to eliminate the remaining windows.

[0092] Step 3.6. After detecting a fire, plan a reasonable flight trajectory (such as flying sideways to produce sufficient parallax), and use a feature matching algorithm to match the feature points of the fire area in the visible light image with the feature points of the fire center in the infrared image. Map the semantic segmentation results and feature matching results into 3D space, and calculate the depth information between the drone and the fire point. Use the depth value around the suspected fire point to represent the depth of the fire point. First, calculate the outline of the potential fire point, and then fit a circle containing the minimum area outline. The selection strategy of the region of interest (ROI) is designed as follows: 1. The center of the region of interest is the same as the minimum area circle, and 2. The area of ​​the circular patch is 1.5 times the minimum area circle within a given threshold. Adjust the threshold to limit the ROI area, and then evenly extract ORB features in the patch. Use feature matching algorithm and histogram filtering to find corresponding reliable points between the reference frame and each additional frame. Finally, these matching points are triangulated into three-dimensional points with the intrinsic parameters of the camera and the frame pose.

[0093] Step 4: Use triangulation to calculate the depth information error calculated in step 3 and make corrections.

[0094] Step 4 is implemented according to the following steps:

[0095] Step 4.1. Once the monocular camera SLAM is initialized, the existing 3D points can track the new frame pose. Only at least 3 points are needed to estimate the camera pose. There are many ways to solve the PnP problem. This work uses nonlinear optimization to construct and iteratively solve the least squares problem, usually called the bundle adjustment (BA) method. Consider that there are n 3D points P in space, and their projected position on the image plane is P. The goal is to calculate the camera pose R and t by optimization. Let the coordinates of the point be P i =[X i ,Y i ,Z i ] T , whose projected pixel coordinate is p i =[u i ,v i ] T According to the pinhole projection model, the relationship between the 2D pixel position and the 3D pixel position is:

[0096] s i p i =KTP i (16)

[0097] Where K is the intrinsic parameter matrix of the camera, T is the position of the camera, and its form is linear algebra:

[0098]

[0099] This equation contains an implicit conversion from homogeneous coordinates to non-homogeneous coordinates. Due to the noise in the camera pose and the observation points, there is always a residual in the equation. Therefore, summing the residuals allows us to construct a least squares problem, which is then minimized to find the most likely camera pose and coordinates of the 3D point:

[0100]

[0101] This residual term is the error between the projected position and the observed position, called the reprojection error. The dimension of this residual is 2. After bundle adjustment optimization, both the camera pose and the 3D points are refined.

[0102] Considering that the feature points of the region are extracted uniformly, when processing the estimated values ​​to remove outliers, the effective depth range and 3-sigma limit are usually used in combination to ensure the accuracy and reliability of the data. This process, including depth estimation and filtering, is mainly filtered according to the following process:

[0103] 1: Calculate the overall median as the initial depth d0.

[0104] 2: Calculate the upper limit d of the effective depth value according to equation (32) tu and the lower limit d tl .

[0105] 3: Calculate the standard deviation σ and calculate the 3σ boundary d σu and d σl .

[0106] 4: Update the upper and lower limits: d u =max(d tu ,d σu ), d l =min(d tl ,d σl ). Estimates outside the bounds are then discarded.

[0107] 5: Determine whether the depth converges. If it converges, stop and return the result; otherwise, return to step 1.

[0108] 6: Return the estimated depth value

[0109] Step 5: Calculate the relative distance between the fire point and the UAV and the specific global geographic location coordinates of the fire point based on the corresponding depth information combined with the rotation angle of the gimbal camera and the IMU data of the camera to complete the positioning of the fire point.

[0110] Step 5 is implemented according to the following steps:

[0111] Step 5.1: After confirming the occurrence of a fire, the drone flies horizontally from left to right to collect visible light and infrared images near the fire point and complete the triangulation of the wildfire point. The movement from left to right is to generate sufficient parallax to improve the reliability of depth estimation.

[0112] Through sensor fusion, the camera pose is recovered in real time using monocular SLAM and IMU-GNSS information. Once the correctly scaled camera pose is recovered, the 3D points near the wildfire can be triangulated. Once the depth is estimated, the geographic location of the wildfire can be derived based on the geometric relationship with the gimbal angle.

[0113] By combining the gimbal with the camera angle, the depth can be directly used to generate the GPS position through equations (9) and (11). When locating the fire point, the offset between the drone body coordinate system and the H20T camera coordinate system relative to the fire point is ignored by default, that is, the drone's global positioning system (GPS) information can be directly applied to the camera.

[0114] In the navigation coordinate system (NED), the relative position of the fire point and the UAV can be defined as:

[0115]

[0116] Where h is the height of the drone in the navigation coordinate system, α and β are the yaw angle and pitch angle of the gimbal camera, respectively. These two angles and the navigation information of the drone can be obtained through DJI's OSDK software development kit. After calculating the relative position information of the fire point and the drone platform, the global coordinate information of the fire point can be expressed as:

[0117]

[0118] Where, L heart , B heart and H heart are the longitude, latitude and altitude of the fire point respectively, and R is the average radius of the earth.

[0119] Step 5.2: After the fire point is located, the fire point information can be saved. According to the task requirements, the processed camera posture and sparse point cloud can be saved by time. The MVS algorithm is used at the ground station to reconstruct the 3D information of the saved data and generate a local environment scene map.

[0120] Example 1

[0121] The present invention UAV monocular vision forest fire positioning method, the flow chart is as follows Figure 1 As shown, the specific implementation steps are as follows:

[0122] Step 1: preprocess the infrared image and the visible light image, establish a matching relationship between the two-light images, and establish a spatial transformation model so that the transformed infrared image and the visible light image are aligned as much as possible in space.

[0123] Step 2: Use the triangulation calibration method to calculate the camera parameters, use the ORB-SLAM2 algorithm to restore the camera posture when shooting the fire, and perform corresponding depth information registration.

[0124] Step 3: Use the fire semantic information provided by the improved Deeplabv3+ segmentation algorithm and the fire center point information segmented from the infrared image to match the image features of the 3D points to obtain the corresponding depth information.

[0125] Step 4: Use triangulation to calculate the depth information error calculated in step 3 and make corrections.

[0126] Step 5: Calculate the relative distance between the fire point and the UAV and the specific global geographic location coordinates of the fire point based on the corresponding depth information combined with the rotation angle of the gimbal camera and the IMU data of the camera to complete the positioning of the fire point.

[0127] Example 2

[0128] In order to better verify the application of the present invention in the field of UAV fire positioning, a typical example was selected for simulation to verify the effect of the present invention.

[0129] Experiment 1 is to apply the present invention to measure the depth of the fire point in an environment with a fire point to verify the effectiveness of the depth calculation. The actual depth distance data between the UAV and the fire point is obtained by a laser rangefinder, and the actual measurement data is compared with the relative depth distance calculated by the algorithm proposed in this paper to evaluate the effectiveness and accuracy of the proposed fire point positioning algorithm. During the experiment, this paper sets different altitudes for the UAV to simulate various actual scenarios. At each altitude, the UAV platform will measure and calculate the depth information of the fire point, and the resolution of the laser rangefinder is 0.1m. Fig.10 This is the specific scene diagram in the experiment.

[0130] Table 1 Statistics of fire depth test results

[0131]

[0132] Experiment 2 is to use the present invention to measure the geographical location of a fire point in an environment with a fire point. In the experiment, the UAV height is 15m, and the initial angle of the gimbal is -60°. Fig.11The original visible light image, the segmentation map of the visible light image based on the fire segmentation algorithm of the improved Deeplabv3+, and the result map after infrared segmentation feature matching. It can be seen in the figure that the result map of the fire segmentation algorithm based on the improved Deeplabv3+ can be highly coincident with the image feature matching of the infrared threshold segmentation, which is helpful for the precise positioning of the fire point in the later stage. The global coordinates of the fire point calculated in the end are (108.9867234°, 34.25725196°), and the actual GPS coordinates are (108.9867283°, 34.25725207°), with an error of about 0.450m.

Claims

1. The purpose of the present invention is to provide an early forest fire positioning algorithm based on dual light for unmanned aerial vehicles, which solves the problems of difficulty in early fire detection, insufficient positioning accuracy, and large depth information error in the prior art; The specific implementation steps of the present invention are as follows: Step 1: preprocess the infrared image and the visible light image, establish a matching relationship between the two-light images, and establish a spatial transformation model so that the transformed infrared image and the visible light image are aligned as much as possible in space; Step 2: Use the triangulation calibration method to calculate the camera parameters, use the ORB-SLAM2 algorithm to restore the camera posture when shooting the fire, and perform the corresponding depth information registration; Step 3: Use the fire semantic information provided by the improved Deeplabv3+ segmentation algorithm and the fire center point information segmented from the infrared image to perform image feature matching of 3D points to obtain the corresponding depth information; Step 4: Use triangulation to calculate the depth information error calculated in step 3 and correct it; Step 5: Calculate the relative distance between the fire point and the UAV and the specific global geographic location coordinates of the fire point based on the corresponding depth information combined with the rotation angle of the gimbal camera and the IMU data of the camera to complete the positioning of the fire point.

2. The present invention is also characterized in that: Step 1 is implemented according to the following steps: Step 1.1, preprocess the infrared image and visible light image, including denoising, contrast enhancement, grayscale, etc., to improve the accuracy and efficiency of subsequent processing, and then extract ORB feature points with high distinguishability and stability from the preprocessed images for matching between different images; Step 1.2: Based on the extracted feature points or feature areas, a matching relationship is established between the infrared image and the visible light image. Based on the matching point pairs, a spatial transformation model is established, and the model parameters are solved by an optimization algorithm so that the transformed infrared image and the visible light image are aligned as much as possible in space. Step 2 is implemented as follows: Step 2.1, camera calibration. Use a calibration plate of known size (such as a chessboard), take multiple images of the calibration plate at different angles and positions, then use image processing technology to detect feature points (such as corner points) on the calibration plate, and use mathematical optimization methods to solve the internal and external parameters of the camera; Step 2.2, ORB-SLAM2 extracts ORB feature points from each frame of the image, performs feature matching between consecutive frames, and establishes correspondence between frames. Based on the feature matching results, ORB-SLAM2 uses epipolar geometry and PnP (Perspective-n-Point) algorithm to estimate the initial pose of the camera. Then, the camera's pose and map points are optimized through local BA (Bundle Adjustment). Some key frames are retained, and local maps are built on these key frames. The pose of the key frames and the observed map points are optimized through local BA. In order to eliminate cumulative errors, detect whether the camera has returned to the previously visited position, and correct the global pose through pose graph optimization; Step 2.3, depth information registration, using triangulation technology for each pair of matching feature points, solve the coordinates of the point in three-dimensional space through the epipolar geometry equation. The three-dimensional coordinates of the feature points are iteratively optimized by minimizing the reprojection error (i.e., the difference between the observed image coordinates and the image coordinates calculated based on the three-dimensional coordinates and camera parameters). For each frame of the image, all the three-dimensional coordinates of the feature points obtained by triangulation can be converted into the depth map corresponding to the image. Each pixel value in the depth map represents the depth of the scene point corresponding to the pixel; Step 3 is implemented as follows: Step 3.1, use the lightweight network ShuffleNetv2 to replace the Xception network for feature extraction, reduce the computational complexity and parameter amount of the network, and improve the network's running speed; Step 3.2: Embed a recurrent cross attention module after the ASPP module. Combining the cross attention mechanism with multi-scale information can further enhance the model's ability to capture global context information and improve the network's ability to extract pixel-level features. Step 3.3, embed the ECA attention module at the front end of the decoder structure, input the shallow feature map output by the deep convolutional network into the attention module, and improve the perception ability of the deep neural network for important features; Step 3.4: Input the visible light image aligned with the infrared image into the visible light forest fire segmentation network based on the improved Deeplabv3+. The network will output the semantic segmentation result map after segmentation. Step 3.5: After acquiring the infrared image, the target image is segmented using the YCrCb color space and threshold segmentation algorithm, and then the morphological operation is used to suppress the noise, and finally the sliding window method is used to find the location of the fire point. Considering that it is difficult to directly separate the chromaticity and brightness information in the RGB color space, the image is converted from the RGB color space to the HSV color space; Step 3.6: After detecting a fire, plan a reasonable flight trajectory (such as flying sideways to generate sufficient parallax), and use a feature matching algorithm to match the feature points of the fire area in the visible light image with the feature points of the fire center in the infrared image. Map the semantic segmentation results and feature matching results into 3D space, and calculate the depth information between the drone and the fire point; Step 4 is implemented according to the following steps: Step 4.1, reconstruct the points in the three-dimensional space using the triangulation principle according to the camera parameters and the matched feature point pairs; Step 4.2, reproject the reconstructed 3D points back to the image plane through the camera parameters, and calculate the pixel distance between the projected points and the original feature points, i.e., the reprojection error; Step 4.3: Minimize the reprojection error through an iterative optimization algorithm (Bundle Adjustment) to further improve the accuracy of the 3D point cloud. Step 5 is implemented according to the following steps: Step 5.1: Perform camera pose estimation: Combine the camera's motion trajectory and image feature points to restore the camera's pose (position and orientation); Step 5.2: Combine the calculated depth information with the camera posture to calculate the relative position relationship between the fire point and the drone, and use GNSS technology to calculate the specific geographical location information of the fire point; Step 5.3: After completing the fire point location, save the fire point information.

Images