Forest live wire tracking and positioning method and system based on individual soldier equipment

Abstract

The invention discloses a forest fire line tracking and positioning method and system based on individual soldier equipment, relates to the technical field of forest fire positioning, and aims to solve the problem that a forest fire line cannot be accurately positioned in the prior art. According to the technical key points, the method comprises the steps: carrying out the data collection through employing designed individual equipment, and obtaining the pose data, radar point cloud data and image data; processing the radar point cloud data based on the pose data to obtain dense point clouds of the live wire area; processing the image data to obtain an unshielded live wire image; and based on calibrated data, carrying out fusion processing on the dense point cloud of the live wire region and the unshielded live wire image to obtain a three-dimensional coordinate of the live wire. According to the method, the problem that the positioning precision is reduced due to crown and smoke shielding in traditional forest fire positioning is solved, the problem that the sensor shakes in the advancing process of the individual soldier equipment is solved, and the problem that the live wire spreads or breaks away from the visual field of the sensor in the advancing process of the individual soldier equipment is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of forest fire positioning, and in particular to a method and system for tracking and positioning a forest fire line based on individual equipment. Background Art

[0002] After a forest fire occurs, quickly and accurately obtaining the location of the fire line and fire dynamics information is the key to efficient firefighting. However, although traditional fire line positioning methods such as satellite remote sensing, watchtowers and drone inspections can monitor large areas of forests, their accuracy and reliability are significantly reduced in the canopy, smoke obstruction and complex forest environments. Satellite monitoring has low resolution and is affected by weather and vegetation, making it difficult to accurately obtain surface fire line information under canopy cover. Although watchtowers and drones are flexible, the integrity and accuracy of fire line data collection are severely limited when faced with complex terrain, making it difficult to effectively locate forest fires. Summary of the invention

[0003] In view of the above problems, the present invention proposes a forest fire line tracking and positioning method and system based on individual equipment, in an effort to solve or alleviate one or more of the above technical problems.

[0004] According to one aspect of the present invention, a method for tracking and locating a fire line in the forest based on a single-soldier device is proposed, the method comprising:

[0005] S1. Use the designed individual equipment to collect data and obtain posture data, radar point cloud data and image data;

[0006] S2. Processing the radar point cloud data based on the posture data to obtain a dense point cloud of the firing line area;

[0007] S3, processing the image data to obtain an unobstructed fire line image;

[0008] S4. Based on the calibrated data, the dense point cloud of the fire line area and the unobstructed fire line image are fused to obtain the three-dimensional coordinates of the fire line.

[0009] Furthermore, the individual equipment described in S1 includes: a mechanical laser radar, a solid-state laser radar, a combined inertial navigation system, an antenna, a DTU, a gimbal, an infrared camera, and an IMU; wherein the mechanical laser radar is used to collect global point cloud data; the solid-state laser radar is used to collect local dense point cloud data on the front line; the antenna is used to receive and obtain GPS signals; the DTU is used to obtain base station differential data and transmit the differential data to the combined inertial navigation system; the combined inertial navigation system is used to collect the posture data of the main body of the individual equipment, and obtain RTK positioning information by combining the GPS signal and the differential data; the infrared camera is used to collect infrared image data on the front line; the IMU is used to record the posture data of the gimbal and the sensors integrated thereon; the gimbal is used to adjust the collection angle of the infrared camera and the solid-state laser radar.

[0010] Further, in S2, the radar point cloud data is processed based on the posture data to obtain a dense point cloud of the firing line area, including:

[0011] S21, generate a SLAM global point cloud map using a SLAM algorithm;

[0012] S22, calculating the RTK signal quality score according to the output data of the combined inertial navigation system;

[0013] S23, according to the RTK signal quality score, the RTK signal quality is divided into three intervals of high, medium and low. When the RTK signal is in the low quality interval, only the SLAM algorithm is used for positioning; when the RTK signal is in the medium quality interval, the RTK signal positioning and the SLAM algorithm are used for dynamic fusion positioning, and the dynamic fusion formula is:

[0014] x fusion (t) = (1-α(t))·x S (t)+α(t)·x R (t)+η·(x fusion (t-1)-x fusion (t))

[0015] Among them, x fusion (t) is the position coordinate obtained by fusing the RTK absolute position and the SLAM algorithm relative position at the current time t; x S (t) is the relative position coordinate generated by the SLAM algorithm, x R (t) is the absolute position coordinate provided by RTK positioning information, η is the time smoothing coefficient, x fusion (t-1) is the position coordinate after fusion at the previous moment; α(t) represents the dynamic weight parameter;

[0016] When the RTK signal is in the high-quality range, only the RTK signal is used for positioning;

[0017] S24, using the posture data and calibrated data provided by the IMU and the combined inertial navigation system, convert each frame of the local point cloud data of the fire line into the global coordinate system, so that the local point cloud data and the SLAM global point cloud are initially aligned;

[0018] S25. Divide the fire line area and non-fire line area in the aligned local point cloud data based on the dynamic threshold method:

[0019] M fire = {p∈P lidar ∣ρ(p)>ρ threshold}

[0020] M non-fire =P lidar \M fire

[0021] Among them, ρ(p) is the reflectivity value of point p; ρ threshold represents the dynamic reflectivity threshold; P lidar Represents the local point cloud data set of the fire line; M fire Represents the point cloud set of the fire line area, M non-fire Represents a point cloud set in the non-fireline area;

[0022] S26. Use the reflectivity threshold method to filter out the abnormal points in the point cloud set of the fire line area, and the retained fire area density point cloud is P filtered , the outlier filtering formula is:

[0023] P filtered ={p∈M fire ∣ρ min ≤ρ(p)≤ρ max}

[0024] Among them, ρ min , max are the upper and lower limits of reflectivity;

[0025] S27, through voxel filtering, the point cloud M of the non-fire line area non-fire Downsample to generate sparse point cloud;

[0026] S28, using the NDT matching algorithm to roughly match the sparse point cloud of the non-fire line area and the SLAM global point cloud to generate a preliminary global alignment result;

[0027] S29. Use the ICP algorithm to perform fine matching on the retained fire line area density point cloud and the SLAM global point cloud to obtain the point cloud coordinates of the fire line area dense point cloud in the world coordinate system.

[0028] Furthermore, the RTK signal quality score calculation formula in S22 is as follows:

[0029]

[0030]

[0031] Among them, the weight parameter α1+α2+α3+α4+α5=1; n sV represents the number of satellites; PDOP represents the geometric precision factor; SNR represents the signal-to-noise ratio; RSRP represents the signal strength; t delay Indicates latency; max indicates the maximum value corresponding to each preset indicator; min indicates the minimum value corresponding to each preset indicator.

[0032] Furthermore, the calculation formula of the dynamic weight parameter α(t) in S23 is as follows:

[0033]

[0034] Among them, S RTK Indicates the RTK signal quality score; is the inverse covariance matrix of RTK positioning information, is the inverse covariance matrix of the SLAM algorithm;

[0035] The dynamic reflectivity threshold calculation formula in S25 is as follows:

[0036] ρ threshold =μ ρ +k·σ ρ

[0037] Among them, μ ρ represents the mean value of the point cloud reflectivity; k represents the sensitivity control parameter; σ ρ Represents the standard deviation of the point cloud reflectivity.

[0038] Furthermore, in S3, the image data is processed to obtain an unobstructed fire line image, including:

[0039] S31, perform fire line recognition on the collected infrared image, perform contrast stretching and histogram equalization on the infrared image; use the Otsu method to automatically calculate the optimal segmentation threshold and perform binarization processing on the image; use the Canny edge detection algorithm to extract the fire line edge in the image and attach color to the fire line outline;

[0040] S32, based on the fire line image data sequence corresponding to the historically collected infrared images and the fire scene terrain, weather and combustible information, using a long short-term memory neural network to generate a fire line expansion morphology prediction frame;

[0041] S33, using the camera motion posture information recorded by the IMU, and using the rotation matrix and translation vector, the fire line image data sequence corresponding to the historically collected infrared image is converted and projected onto the current frame image plane to complete the rough alignment of the fire line images between frames;

[0042] S34, combining the current frame, the fire line image data corresponding to the historically collected infrared image, and the extended morphological prediction frame generated by S32, by extracting the key feature points of the fire line area, using the feature point matching method to establish the correspondence between frames, to achieve precise geometric alignment of the fire line segments, and obtain the unobstructed fire line image of the current frame.

[0043] Furthermore, the following steps are used during the data collection process of S1 to achieve continuous stabilization of the solid-state laser radar and infrared camera images:

[0044] A real-time attitude anti-compensation algorithm is used to convert the data into an angle offset value, which is then used to provide real-time anti-compensation to the gimbal control system. The attitude anti-compensation formula is as follows:

[0045] φ y (t) = φ y (t-1)-Δθ′ y

[0046] φ p (t) = φ p (t-1)-Δθ′ p

[0047] φ r (t) = φ r (t-1)-Δθ′ r

[0048] Among them, φ y ,φ p ,φ r They represent the yaw, pitch and roll angles of the 3D gimbal respectively, t represents the time; Δθ′ y , Δθ′ p , Δθ′ r It is the value of the IMU attitude change after coordinate conversion;

[0049] The three-dimensional gimbal attitude control formula is as follows:

[0050]

[0051] Among them, k y , k p , k r is the feedback gain parameter, e y , e p , e rIt represents the error of the gimbal attitude angle, and R represents the rotation matrix of the combined inertial navigation system coordinate system converted to the gimbal coordinate system.

[0052] Furthermore, during the data collection process of S1, the following steps are used to dynamically adjust the PTZ in real time to achieve the coincidence of the center of gravity of the line of fire and the center of the infrared image:

[0053] For the future fire line expansion morphology prediction image obtained by S32, traverse each pixel in the image and count the coordinate positions (x i ,y i ), calculate the center of gravity position C of the line of fire f (x c ,y c ), used to indicate the center point of the fire line, the calculation formula is as follows:

[0054]

[0055] in, is the weight of the fire line pixel; N is the total number of fire line pixels; a(i,j) is the gray value of the pixel (i,j);

[0056] Calculate the center position C of the infrared image m (x m ,y m ) and the center of gravity of the line of fire C f (x c ,y c ) between the two images (Δx, Δy);

[0057] The gimbal angle is adjusted in real time based on the offset. The angle adjustment formula is as follows:

[0058]

[0059] Among them, θ pitch ,θ yaw Respectively represent the pitch angle and yaw angle of the gimbal; K X ,K y Represent the proportional gain of pitch angle and yaw angle respectively; D X ,D y Represent the differential gains of pitch angle and yaw angle respectively; They respectively represent the rate of change of the offset.

[0060] Furthermore, in S4, the fusion processing of the dense point cloud of the fire line area and the unobstructed fire line image based on the calibrated data includes:

[0061] S41, combining the internal and external calibration data of the solid-state laser radar and the infrared camera, projecting the dense point cloud of the fire line area of ​​the current frame to the unobstructed fire line image of the current frame, realizing the conversion of the three-dimensional point cloud coordinates to the two-dimensional image pixel coordinates, and obtaining the corresponding relationship between the point cloud of the fire line area and the infrared image;

[0062] S42, using the fire line contour color information and pixel position information in the infrared image, associating and assigning values ​​to the fire line contour with color information as the three-dimensional points in the point cloud, extracting the point cloud carrying the color information, filtering out redundant point clouds, and obtaining the three-dimensional point cloud coordinates of the fire line in the three-dimensional space;

[0063] S43. Through the adaptive ICP algorithm and combined with the posture data provided by the IMU, the multi-frame three-dimensional point cloud of the fire line obtained in S42 is spliced ​​to achieve accurate positioning of the fire line in the world coordinate system.

[0064] According to another aspect of the present invention, a forest fire line tracking and positioning system based on individual soldier equipment is proposed, the system comprising:

[0065] A data acquisition module, which is configured to collect data using the designed individual equipment to acquire posture data, radar point cloud data and image data;

[0066] a point cloud processing module, configured to process the radar point cloud data based on the posture data to obtain a dense point cloud of the firing line area;

[0067] An image processing module configured to process the image data to obtain an unobstructed fire line image;

[0068] A fusion processing module is configured to perform fusion processing on the dense point cloud of the fire line area and the unobstructed fire line image based on the calibrated data to obtain the three-dimensional coordinates of the fire line.

[0069] The beneficial technical effects of the present invention are:

[0070] The present invention proposes a method and system for tracking and locating forest fire lines based on individual equipment. When a surface fire occurs under the forest, the individual equipment arrives at the designated position, moves along the fire line and uses multiple sensors to collect fire line data, thereby realizing accurate positioning of the fire line under the forest, solving the problem of reduced positioning accuracy caused by tree crowns and smoke obstruction in traditional forest fire positioning. Better fire line tracking and positioning are achieved through the complementary performance of mechanical laser radar and solid-state laser radar. Although the mechanical laser radar has a short measurement distance and sparse point cloud, it has the advantages of 360° rotation acquisition, fast point cloud acquisition speed, and is suitable for SLAM to build a global point cloud map; although the solid-state laser radar has a small field of view and a slow acquisition speed, its effective detection distance can reach 500 meters, and the point cloud is dense, which is suitable for collecting dense point clouds in local fire areas, solving the safety problem when individual equipment needs to collect fire lines at close range, while ensuring high accuracy and processing speed of data acquisition. Positioning and mapping are achieved through the SLAM algorithm, and an RTK signal quality detection module and an automatic calibration module are added. SLAM positioning is used when there is no RTK signal. When there is an RTK signal, the weight fusion algorithm of the RTK absolute position and the SLAM relative position is dynamically adjusted to achieve accurate repositioning and reduce SLAM cumulative errors. This solves the problem of being unable to accurately position the fire line when there is no GPS signal and 4G differential data transmission signal under the forest.

[0071] The present invention adopts a three-dimensional pan-tilt head with a shock-absorbing bed, which integrates IMU, solid-state laser radar and infrared camera. The shock-absorbing bed offsets high-frequency jitter during movement. The three-dimensional pan-tilt head achieves dynamic self-stabilization through posture data back-compensation, which solves the problem of sensor jitter of individual equipment during movement. In addition, in response to the dynamic changes of individual equipment movement and fire line spread, the present invention adopts a deep learning fire line morphology prediction algorithm to predict the future position of the fire line in advance, and dynamically adjusts the pan-tilt head angle in real time to ensure continuous tracking of the fire line, which solves the problem of the fire line leaving the sensor field of view due to the spread of the fire line or the movement of individual equipment.

[0072] The present invention extracts multiple fire lines blocked by tree trunks in infrared images, uses IMU to align the current frame, historical frames and the fire lines in step four to predict the fire line contours of future frames, and adopts a stitching algorithm based on multi-view fusion and feature matching to obtain the unobstructed fire line of the current frame; thus solving the problem of being unable to collect the complete fire line due to occlusion by tree trunks. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The present invention can be better understood by referring to the description given below in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification, and are used to further illustrate the preferred embodiments of the present invention and explain the principles and advantages of the present invention.

[0074] Figure 1The present invention is a flowchart of a method for tracking and locating a fire line in the forest based on a single-soldier device according to an embodiment of the present invention.

[0075] Figure 2 The present invention is a flowchart of a method for tracking and locating a line of fire in the forest based on a single-soldier device according to an embodiment of the present invention.

[0076] Figure 3 Schematic diagram of the structure of individual soldier equipment in an embodiment of the present invention.

[0077] Figure 4 Schematic diagram of another structure of the individual soldier equipment in the embodiment of the present invention.

[0078] Figure 5 1 is a diagram showing the conversion relationship between multiple sensor coordinate systems and the world coordinate system in an embodiment of the present invention.

[0079] Figure 6 This is a diagram showing an example of the process of tracking and locating a fire line in a forest in an embodiment of the present invention. DETAILED DESCRIPTION

[0080] In order to enable those skilled in the art to better understand the scheme of the present invention, exemplary implementations or embodiments of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described implementations or embodiments are only implementations or embodiments of a part of the present invention, not all of them. Based on the implementations or embodiments of the present invention, all other implementations or embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0081] In order to solve the problem that traditional fire line positioning methods are difficult to achieve real-time and accurate positioning in forest surface fires, and the current forest fire line positioning technologies such as satellite remote sensing and drone inspections have significantly reduced fire line positioning accuracy due to canopy occlusion and smoke interference, and forest positioning technologies face technical problems such as no GPS signal, complex terrain and difficulty in real-time communication, the present invention proposes a forest fire line tracking and positioning method and system based on individual equipment. By designing individual equipment equipped with multimodal sensors, laser radar point cloud data, infrared image data and posture data are collected in the forest, and data extraction and fusion are performed to achieve accurate positioning of the forest fire line.

[0082] The embodiment of the present invention proposes a forest fire line tracking and positioning method based on a single-soldier device, such as Figures 1-2 As shown, the method includes:

[0083] S1. Use the designed individual equipment to collect data and obtain posture data, radar point cloud data and image data;

[0084] S2. Processing the radar point cloud data based on the posture data to obtain a dense point cloud of the firing line area;

[0085] S3, processing the image data to obtain an unobstructed fire line image;

[0086] S4. Based on the calibrated data, the dense point cloud of the fire line area and the unobstructed fire line image are fused to obtain the three-dimensional coordinates of the fire line.

[0087] The method starts from S1. In S1, the designed individual equipment is used to collect data to obtain posture data, radar point cloud data and image data; the individual equipment includes: mechanical laser radar, solid-state laser radar, combined inertial navigation system, antenna, DTU, pan-tilt, infrared camera, IMU; wherein the mechanical laser radar is used to collect global point cloud data; the solid-state laser radar is used to collect local dense point cloud data of the front line; the antenna is used to receive and obtain GPS signals; the DTU is used to obtain base station differential data and transmit the differential data to the combined inertial navigation system; the combined inertial navigation system is used to collect the posture data of the main body of the individual equipment, and obtain RTK positioning information by combining GPS signals and the differential data; the infrared camera is used to collect infrared image data of the front line; the IMU is used to record the posture data of the pan-tilt and the sensors integrated thereon; the pan-tilt is used to adjust the collection angle of the infrared camera and the solid-state laser radar.

[0088] According to an embodiment of the present invention, a hardware structure of a single-soldier device is designed to integrate a solid-state laser radar, an infrared camera and an IMU, and install them on a three-dimensional gimbal equipped with a shock-absorbing bed to form an automatic tracking and acquisition module for the fire area. A mechanical laser radar and a combined inertial navigation are integrated to form a forest area mapping and positioning module, and a power supply module and a communication module are built.

[0089] The single-soldier equipment is mainly used for fire line monitoring and positioning under the forest. Considering the complex environment in the forest, rugged mountain roads, and dense vegetation, the backpack design was finally chosen. The staff carried the single-soldier equipment on their backs and walked along the fire line, and a reasonable safety distance was required. By integrating solid-state laser radar, infrared camera and IMU into one, and installing it on a three-dimensional pan-tilt with a shock-absorbing bed, the integrated design facilitates the calibration, installation and removal of the sensor. The single-soldier equipment inevitably shakes during the movement. The shock-absorbing bed reduces high-frequency vibrations, thereby reducing the impact on sensor accuracy. Since the fire line and the equipment are in dynamic change, the design of the three-dimensional pan-tilt provides flexible angle control capabilities, thereby realizing accurate data collection of local fire areas. By integrating mechanical laser radar and combined inertial navigation into one, it is used to realize forest area mapping and positioning. It is installed on the single-soldier equipment. The combined inertial navigation can record the posture changes of the entire single-soldier equipment during movement, and realize global mapping of the forest area and real-time positioning of the single-soldier equipment through the SLAM algorithm. The design of the power supply module takes into account the power supply requirements of each sensor, adds a voltage regulator and isolator, and is equipped with a protective shell and a heat dissipation device. The design of the shutter ventilation system ensures the stability and lightness of the equipment during long-term operation, and ensures that the equipment can continue to operate for a long time and has sufficient power. The design of the communication module is aimed at the problem that forest communications are affected by the canopy and trunk obstruction. The equipment transmits data to the drone relay platform through a wireless bridge, and then the drone platform transmits it to the ground end to achieve long-distance communication. The drone relay platform is not only used for data transmission, but also can assist individual equipment in path planning to ensure the safest and fastest collection of fire data.

[0090] Specifically, Figures 3-4 As shown, antenna 1 is used to receive GPS signals; the effective distance of mechanical laser radar 2 is 100M, and point cloud of the surrounding environment is quickly collected by 360° scanning, and it also includes mechanical laser radar base 3 and mechanical laser radar control box 15; combined inertial navigation system 4 is used to obtain quaternion, Euler angle, acceleration, angular velocity and other data, and obtain RTK positioning information by combining the obtained GPS signal and differential data; DTU14 is used to access the server through 4G communication of SIM card to obtain differential data of base station, and transmit the data back to combined inertial navigation; solid-state laser radar 12 is used to collect The dense point cloud within the effective distance of 500 meters obtains the local dense point cloud data of the fire area through scanning, in preparation for the subsequent fusion, and also includes a solid-state laser radar control box 13; the gimbal 9 is used for automatic dynamic tracking of the fire line, and also includes a gimbal base 5 and a gimbal control box 8; the infrared camera 10 is used to identify and shoot the fire line, obtain the infrared image data of the fire line, and is used for image stitching, fusion, etc.; the IMU 11 is used to record the posture changes of the gimbal and the sensors integrated thereon, for the preliminary matching of the local point cloud and the global point cloud, and for partial stitching of the solid-state laser radar point cloud and the infrared camera image data.

[0091] In this embodiment, optionally, the individual soldier equipment may also include a wireless bridge 7 and a portable display 6, wherein the wireless bridge 7 is used to transmit the position coordinates and image video data of the positioned fire line point cloud to the drone relay platform, and then transmitted to the ground end by the drone relay platform; the portable display 6 is used to display the effect image of fire line extraction and positioning and the position of the fire line in the image in real time, which is convenient for staff to monitor.

[0092] As an example, the antenna 1 can be installed at the top of the individual equipment and above the mechanical laser radar 2 to prevent other sensors from blocking it and affecting signal reception; the mechanical laser radar 2 can be installed below the antenna 1 and above the fixed base, and its height must be ensured to be higher than the height above the staff's head to avoid obstruction and achieve 360° all-round point cloud data collection; the solid-state laser radar 12 can be installed above the infrared camera 10 and integrated with it, with the same field of view direction, which is convenient for synchronous data collection; the infrared camera 10 can be installed below the solid-state laser radar 12 and above the gimbal 9, and the integration is convenient for synchronous data collection and overall rotation; the IMU 11 can be installed between the infrared camera 10 and the solid-state laser radar The device can be mounted on the side of the gimbal 9, the infrared camera 10 and the solid-state laser radar 12 to record the posture changes of the gimbal 9, the infrared camera 10 and the solid-state laser radar 12 during the line of fire tracking process; the gimbal 9 can be installed below the infrared camera 10 and above the gimbal support platform to ensure that the gimbal 9 is firmly fixed to the device body, which is convenient for dynamic adjustments such as gimbal tracking; the combined inertial navigation system 4 can be installed on the gimbal support platform to record the posture changes of the main body in real time, and it is directly below the antenna 1, which is also convenient for wiring and signal transmission; the wireless bridge 7 can be installed on the device support rod group and below the gimbal support platform, with an installation angle of 45° upward, which can effectively avoid the gimbal support platform from blocking the signal and transmit the data quickly and accurately to the UAV relay platform.

[0093] The multimodal sensors carried on the above-mentioned individual equipment are used to collect infrared images, point clouds, posture and other data. The mechanical lidar collects global environmental data in the forest, and the solid-state lidar collects local data on the front line.

[0094] When collecting data from multiple sensors, there are time differences and sensor internal clock drift, which lead to timestamp asynchronization. By introducing a global time synchronization node, a unified time base is assigned to each sensor system. At the same time, the standard time signal of the combined inertial navigation is assigned to the lidar, so that it can synchronize the internal timer and count in seconds. This ensures that during the data collection process, the data of all sensors can be synchronized under the same time base, eliminating the impact of time drift on data alignment.

[0095] Then, S2 is executed. In S2, the radar point cloud data is processed based on the posture data to obtain a dense point cloud of the firing line area. Specifically, the process includes:

[0096] S21, generate a SLAM global point cloud map using a SLAM algorithm;

[0097] S22, calculating the RTK signal quality score based on the data output by the combined inertial navigation system and the DTU;

[0098] S23. According to the RTK signal quality score, the signal quality is divided into three intervals: high, medium and low. When the RTK signal is in the low quality interval, only the SLAM algorithm is used for positioning; when the RTK signal is in the medium quality interval, RTK positioning and SLAM algorithm are used for dynamic fusion positioning. The dynamic fusion formula is:

[0099] x fusion (t) = (1-α(t))·x S (t)+α(t)·x R (t)+η·(x fusion (t-1)-x fusion (t))

[0100] Among them, x fusion 9t) is the position coordinate obtained by fusing the RTK absolute position and the SLAM algorithm relative position at the current time t; x S (t) is the relative position coordinate generated by the SLAM algorithm, x R (t) is the absolute position coordinate provided by RTK positioning information, η is the time smoothing coefficient, x fusion (t-1) is the position coordinate after fusion at the previous moment; α(t) represents the dynamic weight parameter;

[0101] When the RTK signal is in the high-quality range, only the RTK signal is used for positioning;

[0102] S24, using the posture data and calibrated data provided by the IMU and the combined inertial navigation system, convert each frame of the local point cloud data of the fire line into the global coordinate system, so that the local point cloud data and the SLAM global point cloud are initially aligned;

[0103] S25. Divide the fire line area and non-fire line area in the aligned local point cloud data based on the dynamic threshold method:

[0104] M fire = {p∈P lidar ∣ρ(p)>ρ threshold}

[0105] M non-fire =P lidar \M fire

[0106] Among them, ρ(p) is the reflectivity value of point p; ρ threshold represents the dynamic reflectivity threshold; Plidar Represents the local point cloud data set of the fire line; M fire Represents the point cloud set of the fire line area, M non-fire Represents a point cloud set in the non-fireline area;

[0107] S26. Use the reflectivity threshold method to filter out the abnormal points in the point cloud set of the fire line area, and the retained fire area density point cloud is P filtered , the outlier filtering formula is:

[0108] P filtered ={p∈M fire ∣ρ min ≤ρ(p)≤ρ max}

[0109] Among them, ρ min , max are the upper and lower limits of reflectivity;

[0110] S27, through voxel filtering, the point cloud M of the non-fire line area non-fire Downsample to generate sparse point cloud;

[0111] S28, using the NDT matching algorithm to roughly match the sparse point cloud of the non-fire line area and the SLAM global point cloud to generate a preliminary global alignment result;

[0112] S29. Use the ICP algorithm to perform fine matching on the retained fire line area density point cloud and the SLAM global point cloud to obtain the point cloud coordinates of the fire line area dense point cloud in the world coordinate system.

[0113] According to an embodiment of the present invention, first, S21, positioning and mapping are achieved through a SLAM algorithm.

[0114] Then, in order to address the problem of GPS signal and 4G differential data transmission signal being blocked under the forest, the individual equipment can achieve RTK centimeter-level positioning accuracy before entering the forest area, providing accurate initial position information for the SLAM algorithm. After entering the forest area, it will automatically switch to SLAM mode, add RTK signal quality detection module and automatic calibration module, use SLAM positioning when there is no RTK signal, and dynamically adjust the weight fusion algorithm of RTK absolute position and SLAM relative position when there is RTK signal, to achieve accurate repositioning and reduce SLAM cumulative error.

[0115] Specifically, S22, RTK signal quality detection module: output the number of satellites n through the combined inertial navigation system sV , geometric precision factor PDOP, signal-to-noise ratio SNR, signal strength RSRP output by DTU sensor, delay t delay , calculate the RTK signal quality score S RTK , the specific formula is as follows:

[0116]

[0117] Among them, α1+α2+α3+α4+α5=1, which needs to be adjusted dynamically according to the signal quality score S RTK The signal quality is divided into three intervals. The segmentation threshold needs to be assigned according to the actual measurement data. As an example, it can be: low quality interval (0≤S RTK ≤0.4), medium quality range (0.4 RTK< 0.9), high quality interval (0.9≤S RTK ≤1).

[0118] S23. In the low-quality range, the system switches to SLAM mode only for positioning; in the medium-quality range, the system enters the dynamic fusion positioning mode of RTK and SLAM; in the high-quality range, the system switches the mode for high-precision RTK positioning, and calibrates and repositions the SLAM data at the same time.

[0119] Among them, the RTK signal quality score is medium quality 0.4 RTK <0.9, the system uses the dynamic fusion mode of RTK and SLAM data to balance the weights of the two, which can not only alleviate the problem of RTK signal instability, but also reduce the cumulative error of SLAM, and achieve smooth transition and high-precision positioning. The dynamic weight α(t) calculation formula is:

[0120]

[0121] in, It is the inverse covariance matrix of RTK data. The reliability of RTK data is reflected by the combined inertial navigation output. It is the inverse covariance matrix of the SLAM data, and the output of the algorithm reflects the reliability of the SLAM data.

[0122] The dynamic fusion formula of RTK and SLAM positioning data is:

[0123] x fusion (t) = (1-α(t))·x S (t)+α(t)·x R (t)+η·(x fusion (t-1)-x fusion (t))

[0124] Among them, x fusion (t) is the position coordinate obtained by fusing the RTK absolute position and the SLAM relative position at the current moment, x S (t) is the relative position coordinate generated by SLAM, x R (t) is the absolute position coordinate provided by RTK, η is the time smoothing coefficient, x​​fusion (t-1) is the position coordinate obtained by fusing the RTK absolute position and SLAM relative position at the previous moment.

[0125] When the RTK signal quality score is 0.9≤S RTK ≤1, the RTK signal quality is reliable, and the system only relies on RTK data for positioning x fision (t) = x RTK (t).

[0126] Furthermore, the positioning data x obtained in multiple intervals fusion (t) The SLAM algorithm is introduced to correct the accumulated error of SLAM and update the SLAM global point cloud map P in the world coordinate system in real time. SLAM (t), the formula is:

[0127] P SLAM (t) = P SLAM (t-1)∪(T(x fusion (t))·P local (t))

[0128] Among them, P SLAM (t) is the global point cloud map of SLAM at the current moment, P SLAM (t-1) The SLAM global point cloud map at the previous moment, P local (t) The local point cloud collected by the mechanical laser radar at the current moment, T(x fusion (t)) The local point cloud P local (t) Transformation matrix to the world coordinate system.

[0129] Then, through the posture information provided by IMU and combined inertial navigation, the local point cloud collected by the solid-state laser radar is gradually converted from the laser radar coordinate system to the SLAM global point cloud coordinate system to complete the preliminary alignment. Based on the reflectivity characteristics of the solid-state laser radar point cloud, the fire line area and non-fire line area are dynamically identified and divided. The point cloud in the non-fire line area is downsampled to reduce the point cloud density, and is roughly matched with the SLAM global point cloud to further narrow the matching range. The point cloud in the fire line area is affected by the thermal effect of the flame. The abnormal points are filtered out by the reflectivity threshold method. After retaining the high-quality point cloud, it is finely matched with the SLAM global point cloud to obtain the precise coordinates of the dense point cloud in the fire line area in the world coordinate system.

[0130] Specifically, S24 uses the high-precision pose information and coordinate system conversion relationship provided by the IMU and the combined inertial navigation to convert each frame of the solid-state lidar coordinate system into the SLAM global coordinate system to achieve preliminary alignment of the point cloud, thereby reducing the search space for subsequent point cloud matching and providing an initial position estimate. The formula is:

[0131] PS =R INSS (R IMUINS (R IMULS P lidar +T IMULS )+T IMUINS )+T INSS

[0132] Among them, P S is the point cloud in the SLAM global coordinate system, R INSS , T INSS is the rotation matrix and translation vector from the combined inertial navigation coordinate system to the SLAM global coordinate system, R IMUINS , T IMUINS is the rotation matrix and translation vector from the IMU coordinate system to the combined inertial navigation coordinate system, R IMULS , T IMULS is the rotation matrix and translation vector from the solid-state laser radar coordinate system to the IMU coordinate system, P lidar It is a point cloud collected by a solid-state laser radar in one frame.

[0133] like Figure 5 As shown in the figure, the coordinate system transformation relationship is determined as follows: by using the relative fixation of the IMU and the solid-state laser radar, the first coordinate transformation relationship between the IMU coordinate system and the solid-state laser radar coordinate system is obtained through external parameter calibration, and the rotation matrix R is used. IMULS and the translation vector T IMULS Complete the alignment of the coordinate system to ensure that the point cloud data of the solid-state laser radar is accurately converted to the IMU coordinate system; obtain the second coordinate transformation relationship between the IMU coordinate system and the combined inertial navigation coordinate system through calibration, and use the rotation matrix R IMUINS and the translation vector T IMUINS Complete the conversion of the coordinate system and realize the accurate conversion of the points in the IMU coordinate system to the combined inertial navigation coordinate system; obtain the third coordinate conversion relationship between the combined inertial navigation coordinate system and the SLAM coordinate system through dynamic calibration. The combined inertial navigation coordinate system can be converted into the third coordinate system through the rotation matrix R INSS and the translation vector T INSS Convert the SLAM global coordinate system to achieve global point cloud positioning and fusion; use the first, second and third coordinate conversion relationships to obtain the conversion relationship between the solid-state laser radar coordinate system and the SLAM global coordinate system, and obtain the fourth coordinate conversion relationship between the SLAM global coordinate system and the world coordinate system through dynamic calibration. The point cloud of the SLAM global coordinate system can be obtained through the rotation matrix R SW and the translation vector T SW Convert to world coordinate system.

[0134] S25. Use the point cloud reflectivity characteristics of the solid-state laser radar to dynamically identify the fire line area and the non-fire line area. The fire line area has a high reflectivity value due to flame interference. By analyzing the reflectivity distribution of the point cloud, the fire line area and the non-fire line area are divided based on the dynamic threshold method:

[0135] M fire = {p∈P lidar ∣ρ(p)>ρ threshold}

[0136] M non-fire =P lidar \M fire

[0137] Among them, ρ threshold is the reflectivity threshold, M fire is the point cloud set of the firing line area, M non-fire is the point cloud set of the non-fire line area, ρ(p) is the reflectivity value of point p; dynamic threshold ρ threshold The formula for dynamic adjustment based on the reflectivity distribution of the point cloud is:

[0138] ρ threshold =μ ρ +k·σ ρ

[0139] Among them, μ ρ is the mean value of the reflectivity of the point cloud, k is the sensitivity control parameter, which needs to be adjusted according to the scene, σ ρ is the standard deviation of the point cloud reflectivity.

[0140] S26. Based on the preliminary alignment, the fire line area point cloud is divided, and due to the influence of the flame thermal effect, there are abnormal reflectivity. The abnormal points are filtered out by the reflectivity threshold method, and the retained fire area density point cloud is P filtered , the outlier filtering formula is:

[0141] P filtered ={p∈M fire ∣ρ min ≤ρ(p)≤ρ max}

[0142] Among them, P filtered is the filtered fire line area point cloud, ρ min , max are the upper and lower limits of reflectivity.

[0143] S27, based on the preliminary alignment, the non-fire line area point cloud M is divided by voxel filtering non-fire Downsampling is performed to generate a sparse point cloud and reduce the amount of subsequent matching calculations. The formula is:

[0144] P spase=VoxclFilter(M non-fire )

[0145] Among them, P spase It is the point cloud of the non-fire line area after sparseness.

[0146] S28. For the obtained sparse point cloud P in the non-fire area spase And SLAM global point cloud P SLAM , the fast NDT algorithm is used to achieve rough matching of the sparse point cloud in the non-fire line area and generate a preliminary global alignment result (R coarse ,T coarse ), the specific objective function is:

[0147]

[0148] S29, in the obtained non-fire line area rough matching result (R coarse ,T coarse ), the dense point cloud P of the reserved fire line area is fire Perform fine matching of ICP algorithm with SLAM global point cloud and optimize the final transformation matrix (R fine ,T fine ), to achieve accurate matching of each frame point cloud collected by solid-state laser radar with the mechanical laser radar SLAM point cloud. The specific objective function is:

[0149]

[0150] Dense point cloud of the firing line area after precise matching The point cloud coordinates in the world coordinate system are:

[0151]

[0152] Among them, R SW , T SW It is the rotation matrix and translation vector from the SLAM global coordinate system to the world coordinate system.

[0153] Then execute S3, in which the image data is processed to obtain an unobstructed fire line image; specifically including: S31, fire line identification of the collected infrared image, contrast stretching and histogram equalization of the infrared image; using the Otsu method to automatically calculate the optimal segmentation threshold and binarize the image; using the Canny edge detection algorithm to extract the fire line edge in the image and attach color to the fire line outline; S32, based on the fire line image data sequence corresponding to the historically collected infrared image and the fire scene terrain, meteorology and combustible information, using the long short-term memory neural network to generate the extended morphology prediction frame of the fire line;

[0154] S33. Using the camera motion posture information recorded by the IMU, and utilizing the rotation matrix and translation vector, the fire line image data sequence corresponding to the historically collected infrared image is converted and projected onto the current frame image plane to complete the coarse alignment of the fire line images between frames; S34. Combining the current frame, the fire line image data corresponding to the historically collected infrared image, and the extended morphological prediction frame generated by S32, the key feature points of the fire line area are extracted, and the corresponding relationship between frames is established using the feature point matching method to achieve accurate geometric alignment of the fire line segments and obtain the unobstructed fire line image of the current frame.

[0155] According to an embodiment of the present invention, firstly, the fire line is identified by using an infrared image collected by an infrared camera. According to the infrared imaging principle, the pixels close to 255 are the fire line area, and the pixels close to 0 are the unburned area. The infrared image is subjected to contrast stretching and histogram equalization to improve the overall contrast of the image, making the fire line area more obvious. The Otsu method is used to automatically calculate the optimal segmentation threshold thr, and the image is binarized to further enhance the contrast between the fire line and the background. The Canny edge detection algorithm is used to extract the edge of the fire line in the image, and the outline of the fire line is colored.

[0156] Then, due to the dynamic complexity of the fire line, based on the historical fire line image data sequence {I t-1 ,I t-2 ,...,I t-n}, introduce deep learning algorithms to predict the dynamic changes of the fire line in real time, use convolutional long short-term memory networks to model the spatial and temporal characteristics of the fire line, and combine environmental information as input to generate the future fire line's extended morphology prediction frame I predict,t+1 .

[0157] Then, the infrared camera is integrated with the IMU, and the position information of each frame of infrared image recorded by the IMU is used to calculate the rotation matrix R IMU and the translation vector T IMU , convert and project the historical frame fire line image to the current frame image plane to complete the rough alignment of the fire line images between frames;

[0158] P trans =R IMU ·P history +T IMU

[0159] Among them, P trans is the projection coordinate of the firing line area of ​​the historical frame in the image plane of the current frame, P history is the pixel coordinate in the firing line area of ​​the historical frame, R IMU is the rotation matrix provided by the IMU, T IMU is the translation vector provided by the IMU;

[0160] Finally, based on the coarse matching results, the infrared fire line images of the current frame, historical frames and the generated predicted frame are combined, the key feature points of the fire line area are extracted, the correspondence between frames is established using the feature point matching method, and the homography matrix is ​​constructed to achieve accurate geometric alignment of the fire line segments, complete the seamless splicing of the current frame fire line image, and obtain the complete fire line infrared image of the current frame;

[0161] Then, S4 is executed. In S4, based on the calibrated data, the dense point cloud of the fire line area and the unobstructed fire line image are fused to obtain the three-dimensional coordinates of the fire line; specifically, the following steps are performed:

[0162] S41. Combine the internal and external calibration data of the solid-state laser radar and the infrared camera, project the dense point cloud of the fire line area of ​​the current frame to the unobstructed fire line image of the current frame, realize the conversion of the three-dimensional point cloud coordinates to the two-dimensional image pixel coordinates, and obtain the accurate correspondence between the point cloud of the fire line area and the infrared image;

[0163] S42, using the fire line contour color information and pixel position information in the infrared image, associating and assigning values ​​to the fire line contour with color information as a three-dimensional point in the point cloud, extracting the point cloud carrying the color information, filtering out redundant point clouds, and obtaining the precise three-dimensional point cloud coordinates of the fire line in the three-dimensional space;

[0164] S43. Through the adaptive ICP algorithm and combined with the posture data provided by the IMU, the multi-frame fire line three-dimensional point cloud obtained in S42 is spliced ​​to achieve accurate positioning of the complete fire line in the world coordinate system.

[0165] According to an embodiment of the present invention, the purpose of the present invention is to obtain the three-dimensional point cloud coordinates of the fire line under the forest in the world coordinate system, fuse the current frame fire line image without tree trunk occlusion with the local fire area point cloud of the solid-state laser radar in the world coordinate system, and extract the three-dimensional coordinates of the fire line in the world coordinate system. The specific process is as follows:

[0166] The integrated solid-state laser radar and infrared camera are used to perform external parameter calibration and establish the conversion relationship between the solid-state laser radar coordinate system and the infrared coordinate system;

[0167] Through the obtained external parameter calibration results and the internal parameter matrix of the infrared camera, the 3D point cloud collected by the solid-state laser radar is projected onto the infrared image plane, the 3D point cloud coordinates are converted into 2D pixel coordinates, and the 3D point cloud P in the laser radar coordinate system is converted into 2D pixel coordinates. lidar =(X lidar ,Y lidar ,Z lidar ) is converted into a three-dimensional point cloud P in the infrared camera coordinate system camera =(X camera ,Y camera ,Z camera), transform the infrared camera coordinate system to the pixel coordinates (u, v) of the infrared image, the formula is:

[0168]

[0169] Among them, R is the rotation matrix, t is the translation vector, and K is the intrinsic parameter matrix of the infrared camera;

[0170] Using the fire line outline color information and pixel position information in the infrared image, the fire line outline with color information is associated and assigned to the three-dimensional points in the point cloud, the point cloud carrying color information in the solid-state laser radar point cloud is extracted, and the redundant point cloud is filtered out to obtain the precise three-dimensional point cloud coordinates of the fire line in three-dimensional space;

[0171] Through the adaptive ICP algorithm and combined with the posture data provided by the IMU, the acquired multi-frame fire line three-dimensional point cloud is spliced ​​to achieve precise positioning of the complete fire line in the world coordinate system.

[0172] The three-dimensional gimbal is used to achieve dynamic self-stabilization. The infrared camera is used to identify and extract the fire line. The fire line shape prediction algorithm based on deep learning is used to predict the future position of the fire line in advance, and the gimbal angle is adjusted dynamically in real time to ensure dynamic tracking of the fire line.

[0173] Furthermore, since individual equipment is always in motion during the march, the sensors on the equipment are frequently shaken, which in turn affects the acquisition accuracy of data such as infrared images and solid-state lidar point clouds. To solve this problem, the sensor is installed on a three-dimensional gimbal, and its position data is recorded through the combined inertial navigation on the individual equipment. The data is compensated to the three-dimensional gimbal in real time to achieve self-stabilizing control to maintain the stability of the equipment. In addition, the individual equipment moves along the line of fire, and during the dynamic spread of the line of fire, it is necessary to ensure that the line of fire is always within the field of view of the sensor. Deep learning is used to generate a predicted frame of the extended shape of the line of fire, and the gimbal angle is dynamically adjusted in advance to ensure continuous tracking of the line of fire and reduce monitoring delays caused by equipment lag.

[0174] In this embodiment, preferably, the following steps are used to achieve continuous stabilization of the solid-state laser radar and infrared camera images during the data collection process of S1:

[0175] A real-time attitude anti-compensation algorithm is used to convert the data into an angle offset value, which is then used to provide real-time anti-compensation to the gimbal control system. The attitude anti-compensation formula is as follows:

[0176] φ y (t) = φ y (t-1)-Δθ′ y

[0177] φ p (t) = φ p(t-1)-Δθ′ p

[0178] φ r (t) = φ r (t-1)-Δθ′ r

[0179] where φ y ,φ p ,φ r They represent the yaw, pitch and roll angles of the 3D gimbal respectively, t represents the time; Δθ′ y , Δθ′ p , Δθ′ r It is the value of the IMU attitude change after coordinate conversion;

[0180] The three-dimensional gimbal attitude control formula is as follows:

[0181]

[0182] where k y , k p , k r is the feedback gain parameter, e y , e p , e r It represents the error of the gimbal attitude angle, and R represents the rotation matrix of the combined inertial navigation system coordinate system converted to the gimbal coordinate system.

[0183] In this embodiment, preferably, during the data collection process of S1, the following steps are used to dynamically adjust the pan / tilt in real time to achieve the coincidence of the center of gravity position of the firing line and the center position of the infrared image:

[0184] For the future fire line expansion morphology prediction image obtained by S32, traverse each pixel in the image and count the coordinate positions (x i ,y i ), calculate the center of gravity position C of the line of fire f (x c ,y c ), used to indicate the center point of the fire line, the calculation formula is as follows:

[0185]

[0186] in, is the weight of the fire pixel. The higher the brightness, the greater the weight. a(i,j) is the gray value of the pixel (i,j). i ,y i is the horizontal and vertical coordinates of the i-th fire line pixel, and N is the total number of fire line pixels.

[0187] Calculate the center position C of the infrared image m (xm ,y m ) and the center of gravity of the line of fire C f (x c ,y c ) between the center position C of the infrared image. m (x m ,y m ) is a fixed value determined by the size of the infrared image, W×H, where W is the width of the image and H is the height of the image.

[0188]

[0189] Based on the offset, the gimbal is controlled so that the center of gravity of the line of fire coincides with the center of the infrared image, which can ensure that the line of fire is always in the center of the infrared image; due to the motion characteristics and inertia of the gimbal, the PD controller is introduced to adjust the gimbal angle:

[0190]

[0191] Among them, θ pitch ,θ yaw Respectively represent the pitch angle and yaw angle of the gimbal; K X ,K y Represent the proportional gain of pitch angle and yaw angle respectively; D X ,D y Represent the differential gains of pitch angle and yaw angle respectively; They respectively represent the rate of change of the offset.

[0192] In order to fully verify the technical effect of the present invention, the following experimental verification was carried out on the technical solution proposed in the present invention.

[0193] The experiment was carried out in the selected forest area. In order to ensure the fire safety in the forest, a 5×5 meter site was selected in the forest area for testing. Three stainless steel basins filled with boiling water were set up as heat sources in the experiment. The three heat sources were located in front of the tree trunk to simulate the fire line position, verify the accuracy of the fire line data collected by the individual equipment, and the centers of the three heat sources P1, P2, and P3 were used as reference points. The experiment includes two control groups: the first group uses the method of self-assembled RTK base station to collect the precise coordinates of the three reference points P′1, P′2, and P′3 as absolute reference values, as shown in Table 1; the second group starts the sensor of the individual equipment when the RTK signal outside the forest is good, and receives the GPS signal through the combined inertial navigation for centimeter-level RTK positioning, providing high-precision initial position coordinates for SLAM. This group of experiments adds RTK signal quality detection module and automatic calibration module. When there is no RTK signal, SLAM positioning is used. When there is an RTK signal, the RTK absolute position and SLAM relative position are dynamically integrated. The weight adjustment algorithm is used to achieve accurate repositioning and reduce the SLAM cumulative error. Figure 6 The process of tracking and locating the fire line in the forest is shown. During the experiment, the simulated fire line was moved in the direction to complete the mapping and positioning of the global forest area. The equipment uses a three-dimensional gimbal to achieve dynamic self-stabilization, and a solid-state laser radar to collect dense point clouds in the local fire area. The infrared camera identifies and extracts the contour of the heat source, and controls the infrared gimbal for tracking. The solid-state laser radar is integrated with the IMU, and the mechanical laser radar is integrated with the combined inertial navigation. The two obtain the rotation matrix and translation vector through external parameter calibration, thereby converting the dense point cloud of the local fire area collected by the solid-state laser radar into the world coordinate system, completing the preliminary alignment with the global forest area point cloud generated by SLAM. Dense point clouds are collected in the heat source area, and sparse point clouds are collected in the non-heat source area. Rough matching is performed with the help of sparse point clouds, and fine matching is achieved through the dense point clouds in the heat source area, so as to dynamically obtain accurate point cloud matching, so that the point cloud data collected by the solid-state laser radar can obtain accurate GPS coordinates in the world coordinate system. Finally, the infrared image of the heat source identified by the infrared camera is fused with the dense point cloud of the heat source area in the world coordinate system to realize the projection of the three-dimensional point cloud on the two-dimensional infrared image, and the three-dimensional point cloud of the heat source contour is extracted. The three extracted circular heat source contour point clouds are fitted respectively, and the GPS coordinates of the center of each circular contour are calculated as the reference points P′1′, P′2′, and P′3′ in the control group 2, as shown in Table 2.

[0194] Table 1 Control group 1

[0195]

[0196] Table 2 Control group II

[0197]

[0198] Another embodiment of the present invention provides a forest fire line tracking and positioning system based on individual soldier equipment, the system comprising:

[0199] A data acquisition module, which is configured to collect data using the designed individual equipment to acquire posture data, radar point cloud data and image data;

[0200] a point cloud processing module, configured to process the radar point cloud data based on the posture data to obtain a dense point cloud of the firing line area;

[0201] An image processing module configured to process the image data to obtain an unobstructed fire line image;

[0202] A fusion processing module is configured to perform fusion processing on the dense point cloud of the fire line area and the unobstructed fire line image based on the calibrated data to obtain the three-dimensional coordinates of the fire line.

[0203] The functions of the forest fire line tracking and positioning system based on individual equipment described in an embodiment of the present invention can be explained by the aforementioned forest fire line tracking and positioning method based on individual equipment. Therefore, for the parts not described in detail in the system embodiment, please refer to the above method embodiment and will not be repeated here.

[0204] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments are contemplated within the scope of the invention thus described. The disclosure of the present invention is intended to be illustrative rather than restrictive of the scope of the invention, which is defined by the appended claims.

Claims

1. A method for tracking and locating the line of fire in the forest based on individual equipment, characterized in that: include: S1. Use the designed individual equipment to collect data and obtain posture data, radar point cloud data and image data; S2. Processing the radar point cloud data based on the posture data to obtain a dense point cloud of the firing line area; S3, processing the image data to obtain an unobstructed fire line image; S4. Based on the calibrated data, the dense point cloud of the fire line area and the unobstructed fire line image are fused to obtain the three-dimensional coordinates of the fire line.

2. According to claim 1, a method for tracking and locating the line of fire in the forest based on individual equipment is characterized in that: The individual equipment described in S1 includes: a mechanical laser radar, a solid-state laser radar, a combined inertial navigation system, an antenna, a DTU, a gimbal, an infrared camera, and an IMU; wherein the mechanical laser radar is used to collect global point cloud data; the solid-state laser radar is used to collect local dense point cloud data on the front line; the antenna is used to receive and obtain GPS signals; the DTU is used to obtain base station differential data and transmit the differential data to the combined inertial navigation system; the combined inertial navigation system is used to collect the posture data of the main body of the individual equipment, and obtain RTK positioning information by combining the GPS signal and the differential data; the infrared camera is used to collect infrared image data on the front line; the IMU is used to record the posture data of the gimbal and the sensors integrated thereon; the gimbal is used to adjust the collection angle of the infrared camera and the solid-state laser radar.

3. A method for tracking and locating the line of fire in the forest based on individual equipment according to claim 2, characterized in that: In S2, the radar point cloud data is processed based on the posture data to obtain a dense point cloud of the firing line area, including: S21, generate a SLAM global point cloud map using a SLAM algorithm; S22, calculating the RTK signal quality score according to the output data of the combined inertial navigation system; S23, according to the RTK signal quality score, the RTK signal quality is divided into three intervals of high, medium and low. When the RTK signal is in the low quality interval, only the SLAM algorithm is used for positioning; when the RTK signal is in the medium quality interval, the RTK signal positioning and the SLAM algorithm are used for dynamic fusion positioning, and the dynamic fusion formula is: x fusion (t)=(1-α(t))·x S (t)+α(t)·x R (t)+η·(x fusion (t-1)-x fusion (t)) Among them, x fusion (t) is the position coordinate obtained by fusing the RTK absolute position and the SLAM algorithm relative position at the current time t; x S (t) is the relative position coordinate generated by the SLAM algorithm, x R (t) is the absolute position coordinate provided by RTK positioning information, η is the time smoothing coefficient, x fusion (t-1) is the position coordinate after fusion at the previous moment; α(t) represents the dynamic weight parameter; When the RTK signal is in the high-quality range, only the RTK signal is used for positioning; S24, using the posture data and calibrated data provided by the IMU and the combined inertial navigation system, convert each frame of the local point cloud data of the fire line into the global coordinate system, so that the local point cloud data and the SLAM global point cloud are initially aligned; S25. Divide the fire line area and non-fire line area in the aligned local point cloud data based on the dynamic threshold method: M fire ={p∈P lidar ∣ρ(p)>ρ threshold } M non-fire =P lidar \M fire Among them, ρ(p) is the reflectivity value of point p; ρ threshold represents the dynamic reflectivity threshold; P lidar Represents the local point cloud data set of the fire line; M fire Represents the point cloud set of the fire line area, M non-fire Represents a point cloud set in the non-fireline area; S26. Use the reflectivity threshold method to filter out the abnormal points in the point cloud set of the fire line area, and the retained fire area density point cloud is P filtered , the outlier filtering formula is: P filtered ={p∈M fire ∣ρ min ≤ρ(p)≤ρ max } Among them, ρ min , max are the upper and lower limits of reflectivity; S27, through voxel filtering, the point cloud M of the non-fire line area non-fire Downsample to generate sparse point cloud; S28, using the NDT matching algorithm to roughly match the sparse point cloud of the non-fire line area and the SLAM global point cloud to generate a preliminary global alignment result; S29. Use the ICP algorithm to perform fine matching on the retained fire line area density point cloud and the SLAM global point cloud to obtain the point cloud coordinates of the fire line area dense point cloud in the world coordinate system.

4. The method for tracking and locating the line of fire in the forest based on individual equipment according to claim 3 is characterized in that: The calculation formula of RTK signal quality score in S22 is as follows: Among them, the weight parameter α1+α2+α3+α4+α5=1; n sV represents the number of satellites; PDOP represents the geometric precision factor; SNR represents the signal-to-noise ratio; RSRP represents the signal strength; t delay Indicates latency; max indicates the maximum value corresponding to each preset indicator; min indicates the minimum value corresponding to each preset indicator.

5. The method for tracking and locating the line of fire in the forest based on individual equipment according to claim 4 is characterized in that: The calculation formula of the dynamic weight parameter α(t) in S23 is as follows: Among them, S RTK Indicates the RTK signal quality score; is the inverse covariance matrix of RTK positioning information, is the inverse covariance matrix of the SLAM algorithm; The dynamic reflectivity threshold calculation formula in S25 is as follows: r threshold =μ ρ +k·s ρ Among them, μ ρ represents the mean value of the point cloud reflectivity; k represents the sensitivity control parameter; σ ρ Represents the standard deviation of the point cloud reflectivity.

6. The method for tracking and locating the line of fire in the forest based on individual equipment according to claim 5 is characterized in that: Processing the image data in S3 to obtain an unobstructed fire line image includes: S31, perform fire line recognition on the collected infrared image, perform contrast stretching and histogram equalization on the infrared image; use the Otsu method to automatically calculate the optimal segmentation threshold and perform binarization processing on the image; use the Canny edge detection algorithm to extract the fire line edge in the image and attach color to the fire line outline; S32, based on the fire line image data sequence corresponding to the historically collected infrared images and the fire scene terrain, weather and combustible information, using a long short-term memory neural network to generate a fire line expansion morphology prediction frame; S33, using the camera motion posture information recorded by the IMU, and using the rotation matrix and translation vector, the fire line image data sequence corresponding to the historically collected infrared image is converted and projected onto the current frame image plane to complete the rough alignment of the fire line images between frames; S34, combining the current frame, the fire line image data corresponding to the historically collected infrared image, and the extended morphological prediction frame generated by S32, by extracting the key feature points of the fire line area, using the feature point matching method to establish the correspondence between frames, to achieve precise geometric alignment of the fire line segments, and obtain the unobstructed fire line image of the current frame.

7. The method for tracking and locating the line of fire in the forest based on individual equipment according to claim 6 is characterized in that: The following steps are used during the S1 data collection process to achieve continuous stabilization of the solid-state LiDAR and infrared camera images: A real-time attitude anti-compensation algorithm is used to convert the data into an angle offset value, which is then used to provide real-time anti-compensation to the gimbal control system. The attitude anti-compensation formula is as follows: f y (t)=φ y (t-1)-Δθ' y f p (t)=φ p (t-1)-Δθ' p f r (t)=φ r (t-1)-Δθ' r Among them, φ y ,φ p ,φ r Respectively represent the yaw, pitch and roll angles of the 3D gimbal, t represents the time; Δθ' y , Δθ' p , Δθ' r It is the value of the IMU attitude change after coordinate conversion; The three-dimensional gimbal attitude control formula is as follows: Among them, k y , k p , d r is the feedback gain parameter, e y , e p , e r It represents the error of the gimbal attitude angle, and R represents the rotation matrix of the combined inertial navigation system coordinate system converted to the gimbal coordinate system.

8. The method for tracking and locating the line of fire in the forest based on individual equipment according to claim 7 is characterized in that: During the data collection process of S1, the following steps are used to dynamically adjust the PTZ in real time to achieve the coincidence of the center of gravity of the line of fire and the center of the infrared image: For the future fire line expansion morphology prediction image obtained by S32, traverse each pixel in the image and count the coordinate positions (x i ,y i ), calculate the center of gravity position C of the line of fire f (x c ,y c ), used to indicate the center point of the fire line, the calculation formula is as follows: in, is the weight of the fire line pixel; N is the total number of fire line pixels; a(i,j) is the gray value of the pixel (i,j); Calculate the center position C of the infrared image m (x m ,y m ) and the center of gravity of the line of fire C f (x c ,y c ) between the two images (Δx, Δy); The gimbal angle is adjusted in real time based on the offset. The angle adjustment formula is as follows: Among them, θ pitch ,θ yaw Respectively represent the pitch angle and yaw angle of the gimbal; K X ,K y Represent the proportional gain of pitch angle and yaw angle respectively; D X ,D y Represent the differential gains of pitch angle and yaw angle respectively; They respectively represent the rate of change of the offset.

9. The method for tracking and locating the line of fire in the forest based on individual equipment according to claim 8, characterized in that: In S4, the fusion processing of the dense point cloud of the fire line area and the unobstructed fire line image based on the calibrated data includes: S41, combining the internal and external calibration data of the solid-state laser radar and the infrared camera, projecting the dense point cloud of the fire line area of ​​the current frame to the unobstructed fire line image of the current frame, realizing the conversion of the three-dimensional point cloud coordinates to the two-dimensional image pixel coordinates, and obtaining the corresponding relationship between the point cloud of the fire line area and the infrared image; S42, using the fire line contour color information and pixel position information in the infrared image, associating and assigning values ​​to the fire line contour with color information as the three-dimensional points in the point cloud, extracting the point cloud carrying the color information, filtering out redundant point clouds, and obtaining the three-dimensional point cloud coordinates of the fire line in the three-dimensional space; S43. Through the adaptive ICP algorithm and combined with the posture data provided by the IMU, the multi-frame three-dimensional point cloud of the fire line obtained in S42 is spliced ​​to achieve accurate positioning of the fire line in the world coordinate system.

10. A forest fire line tracking and positioning system based on individual soldier equipment, characterized in that: include: A data acquisition module, which is configured to collect data using the designed individual equipment to acquire posture data, radar point cloud data and image data; a point cloud processing module, configured to process the radar point cloud data based on the posture data to obtain a dense point cloud of the firing line area; An image processing module configured to process the image data to obtain an unobstructed fire line image; A fusion processing module is configured to perform fusion processing on the dense point cloud of the fire line area and the unobstructed fire line image based on the calibrated data to obtain the three-dimensional coordinates of the fire line.

Images