A forest fire line tracking positioning method and system based on individual equipment

Abstract

The application discloses a forest fire line tracking positioning method and system based on a single soldier device, relates to the technical field of forest fire positioning, and aims to solve the problem that the prior art cannot accurately position a forest fire line. Technical points of the application include the following: a designed single soldier device is used to collect data, and pose data, radar point cloud data and image data are obtained; the radar point cloud data is processed based on the pose data, and dense point cloud of a fire line area is obtained; the image data is processed, and an unobstructed fire line image is obtained; the dense point cloud of the fire line area and the unobstructed fire line image are fused based on calibrated data, and three-dimensional coordinates of the fire line are obtained. The application solves the problem of decreased positioning accuracy caused by tree crown and smoke obstruction in traditional forest fire positioning, solves the problem of sensor shaking of the single soldier device during movement, and solves the problem of the fire line separating from the field of view of the sensor due to the spread of the fire line or the movement of the single soldier device.

Description

Technical Field

[0001] This invention relates to the field of forest fire location technology, specifically to a method and system for tracking and locating forest fire lines based on individual soldier equipment. Background Technology

[0002] After a forest fire breaks out, rapidly and accurately obtaining information on the location and dynamics of the fire line is crucial for efficient firefighting. However, traditional fire line location methods, such as satellite remote sensing, lookout towers, and drone patrols, while capable of monitoring large areas of forest, suffer significant drops in accuracy and reliability due to canopy cover, smoke obstruction, and complex forest environments. Satellite monitoring, due to its low resolution and susceptibility to weather and vegetation effects, struggles to accurately acquire surface fire information under canopy cover. While lookout towers and drones offer flexibility, the completeness and accuracy of fire line data acquisition are severely limited in complex terrain, making effective location of forest fires difficult. Summary of the Invention

[0003] In view of the above problems, the present invention proposes a forest fire tracking and positioning method and system based on individual soldier equipment, in an attempt to solve or alleviate one or more of the above technical problems.

[0004] According to one aspect of the present invention, a method for tracking and locating forest fire lines based on individual soldier equipment is proposed, the method comprising:

[0005] S1. Use the designed individual soldier equipment to collect data, including pose data, radar point cloud data, and image data.

[0006] S2. Process the radar point cloud data based on the pose data to obtain a dense point cloud in the fire line area;

[0007] S3. Process the image data to obtain an unobstructed fire line image;

[0008] S4. Based on the calibrated data, perform fusion processing on the dense point cloud of the fire line area and the unobstructed fire line image to obtain the three-dimensional coordinates of the fire line.

[0009] Further, the individual soldier equipment mentioned in S1 includes: a mechanical lidar, a solid-state lidar, a combined inertial navigation system, an antenna, a DTU, a gimbal, an infrared camera, and an IMU; wherein, the mechanical lidar is used to collect global point cloud data; the solid-state lidar is used to collect local dense point cloud data of the fire line; the antenna is used to receive and acquire GPS signals; the DTU is used to acquire base station differential data and transmit the differential data to the combined inertial navigation system; the combined inertial navigation system is used to collect the pose data of the main body of the individual soldier equipment and obtain RTK positioning information by combining the GPS signal and the differential data; the infrared camera is used to collect infrared image data of the fire line; the IMU is used to record the pose data of the gimbal and the sensors integrated on it; the gimbal is used to adjust the acquisition angle of the infrared camera and the solid-state lidar.

[0010] Further, in S2, processing the radar point cloud data based on the pose data to obtain a dense point cloud of the fire line area includes:

[0011] S21. Generate a SLAM global point cloud map using the SLAM algorithm;

[0012] S22. Calculate the RTK signal quality score based on the output data of the integrated inertial navigation system;

[0013] S23. Based on the RTK signal quality score, the RTK signal quality is divided into three intervals: high, medium, and low. When the RTK signal is in the low quality interval, only the SLAM algorithm is used for localization. When the RTK signal is in the medium quality interval, RTK signal localization and SLAM algorithm are dynamically fused for localization. The dynamic fusion formula is as follows:

[0014] x fusion (t)=(1-α(t))·x S (t)+α(t·x R (t)+η·(x fusion (t-1)-x fusion (t))

[0015] Where, x fusion (t) represents the position coordinates obtained at time t after fusing the absolute position from the RTK algorithm and the relative position from the SLAM algorithm; x S (t) represents the relative position coordinates generated by the SLAM algorithm, x R (t) represents the absolute position coordinates provided by RTK positioning information, η is the time smoothing coefficient, and x fusion (t-1) represents the position coordinates after fusion at the previous time step; α(t) represents the dynamic weight parameter;

[0016] When the RTK signal is in the high-quality range, only the RTK signal is used for positioning;

[0017] S24. Using the pose data and calibrated data provided by the IMU and the combined inertial navigation system, the local point cloud data of each frame of the fire line is transformed into the global coordinate system, so that the local point cloud data and the SLAM global point cloud are initially aligned.

[0018] S25. Divide the fire-line region and non-fire-line region in the aligned local point cloud data based on the dynamic threshold method:

[0019] M fire ={p∈P lidar |ρ(p)>ρ threshold}

[0020] M non-fire =P lidar \M fire

[0021] Where ρ(p) is the reflectance value at point p; ρ threshold Indicates the dynamic reflectivity threshold; P lidar M represents the local point cloud data set of the fireline; fire M represents the point cloud set of the fire zone. non-fire This represents a collection of point clouds in non-fire-area regions;

[0022] S26. Use the reflectivity threshold method to filter out outliers in the fire zone point cloud set, retaining the fire zone density point cloud as P. filtered The outlier filtering formula is:

[0023] P filtered ={p∈M fire ∣ρ min ≤ρ(p)≤ρ max}

[0024] Where, ρ min ρ max These are the upper and lower limits of reflectivity;

[0025] S27. Apply voxel filtering to the point cloud M in the non-fireline region. non-fire Downsampling is performed to generate a sparse point cloud;

[0026] S28. Use the NDT matching algorithm to perform coarse matching between the sparse point cloud in the non-fireline area and the global point cloud of SLAM to generate preliminary global alignment results.

[0027] S29. Use the ICP algorithm to perform fine matching between the preserved fire-line area density point cloud and the SLAM global point cloud to obtain the point cloud coordinates of the dense point cloud in the fire-line area in the world coordinate system.

[0028] Furthermore, the formula for calculating the RTK signal quality score in S22 is as follows:

[0029]

[0030]

[0031] Wherein, the weight parameters α1+α2+α3+α4+α5=1; n sV Indicates the number of satellites; PDOP represents the geometric precision factor; SNR represents the signal-to-noise ratio; RSRP represents the signal strength; t delay represents latency; max represents the maximum value corresponding to each preset indicator; min represents the minimum value corresponding to each preset indicator.

[0032] Furthermore, the formula for calculating the dynamic weight parameter α(t) described in S23 is as follows:

[0033]

[0034] Among them, S RTK Indicates the RTK signal quality score; It is the inverse of the covariance matrix of RTK positioning information. It is the inverse of the covariance matrix of the SLAM algorithm;

[0035] The formula for calculating the dynamic reflectivity threshold in S25 is as follows:

[0036] ρ threshold =μ ρ +k·σ ρ

[0037] Where, μ ρ σ represents the mean reflectance of the point cloud; k represents the sensitivity control parameter; σ ρ The standard deviation of point cloud reflectance.

[0038] Furthermore, in S3, the image data is processed to obtain an unobstructed fireline image, including:

[0039] S31. Perform fire line identification on the acquired infrared image, and perform contrast stretching and histogram equalization on the infrared image; use the Otsu method to automatically calculate the optimal segmentation threshold, and perform binarization processing on the image; use the Canny edge detection algorithm to extract the fire line edges in the image, and add color to the fire line contours.

[0040] S32. Based on the fire line image data sequence corresponding to the historically acquired infrared images and the fire site topography, meteorology and combustible material information, use a long short-term memory neural network to generate extended morphology prediction frames of the fire line.

[0041] S33. Using the camera motion pose information recorded by the IMU, the fire line image data sequence corresponding to the historically acquired infrared images is converted and projected onto the current frame image plane using the rotation matrix and translation vector to complete the coarse alignment of the inter-frame fire line images.

[0042] S34. Combining the current frame, the fire line image data corresponding to the historically acquired infrared images, and the extended morphology prediction frame generated in S32, key feature points of the fire line area are extracted, and the inter-frame correspondence is established using the feature point matching method to achieve precise geometric alignment of the fire line segments and obtain an unobstructed fire line image of the current frame.

[0043] Furthermore, during the data acquisition process of S1, the following steps are used to achieve continuous stabilization of the solid-state lidar and infrared camera images:

[0044] A real-time attitude inverse compensation algorithm is used to convert the data into angular offset values, which are then used to provide real-time inverse compensation to the gimbal's control system. The formula for attitude inverse compensation is as follows:

[0045] φ y (t)=φ y (t-1)-Δθ′ y

[0046] φ p (t)=φ p (t-1)-Δθ′ p

[0047] φ r (t)=φ r (t-1)-Δθ′ r

[0048] Where, φ y φ p φ r Δθ′ represents the yaw, pitch, and roll angles of the 3D gimbal, respectively, and t represents time. y ,Δθ′ p ,Δθ′ r It is the value of the IMU attitude change after coordinate transformation;

[0049] The formula for 3D gimbal attitude control is as follows:

[0050]

[0051] Where, k y k p k r It is the feedback gain parameter, e y e p e rR represents the error of the gimbal attitude angle, and R represents the rotation matrix for transforming the coordinate system of the combined inertial navigation system to the coordinate system of the gimbal.

[0052] Furthermore, during the data acquisition process of S1, the pan-tilt unit is dynamically adjusted in real time using the following steps to achieve alignment between the fire line center of gravity and the infrared image center:

[0053] For the predicted extended form of the future fireline obtained by S32, traverse each pixel in the image and count the coordinates (x, y, y) of the pixels belonging to the fireline. i ,y i ), calculate the center of gravity C of the fire line. f (x c ,y c The center point of the fire line is represented by the following formula:

[0054]

[0055] in, is the weight of the fireline pixel; N is the total number of fireline pixels; a(i,j) is the gray value of pixel (i,j);

[0056] Calculate the center position C of the infrared image m (x m ,y m The center of gravity C of the fire line f (x c ,y c The offset (Δx, Δy) between them;

[0057] The gimbal angle is adjusted in real time based on the aforementioned offset, and the angle adjustment formula is as follows:

[0058]

[0059] Where, θ pitch ,θ yaw These represent the pitch angle and yaw angle of the gimbal, respectively; K X ,K y D represents the proportional gain of pitch and yaw angles, respectively; X D y These represent the differential gains of the pitch and yaw angles, respectively. These represent the rate of change of the offset.

[0060] Furthermore, the fusion processing in S4 based on the calibrated data of the dense point cloud of the fire line area and the unobstructed fire line image includes:

[0061] S41. Combining the internal and external parameter calibration data of solid-state lidar and infrared camera, the dense point cloud of the fire line area in the current frame is projected onto the unobstructed fire line image of the current frame, realizing the conversion of three-dimensional point cloud coordinates to two-dimensional image pixel coordinates, and obtaining the correspondence between the point cloud of the fire line area and the infrared image.

[0062] S42. Using the color information and pixel position information of the fire line outline in the infrared image, the fire line outline with color information is associated with and assigned to the three-dimensional points in the point cloud. The point cloud carrying color information is extracted, and redundant point clouds are filtered out to obtain the three-dimensional point cloud coordinates of the fire line in three-dimensional space.

[0063] S43. Using an adaptive ICP algorithm and combining the pose data provided by the IMU, the multi-frame fireline 3D point cloud obtained in S42 is stitched together to achieve precise positioning of the fireline in the world coordinate system.

[0064] According to another aspect of the present invention, a forest fire tracking and positioning system based on individual soldier equipment is proposed, the system comprising:

[0065] The data acquisition module is configured to collect data using the designed individual soldier equipment, acquiring pose data, radar point cloud data, and image data.

[0066] A point cloud processing module is configured to process the radar point cloud data based on the pose data to obtain a dense point cloud in the fire line area.

[0067] An image processing module is configured to process the image data to obtain an unobstructed fire line image;

[0068] The fusion processing module is configured to perform fusion processing on the dense point cloud of the fire line area and the unobstructed fire line image based on the calibrated data to obtain the three-dimensional coordinates of the fire line.

[0069] The beneficial technical effects of this invention are:

[0070] This invention proposes a method and system for tracking and locating forest fire lines based on individual soldier equipment. When a forest fire occurs, the individual soldier equipment reaches a designated location and, by moving along the fire line and using multiple sensors to collect fire line data, achieves precise location of the forest fire line, solving the problem of decreased positioning accuracy caused by tree canopy and smoke obstruction in traditional forest fire location. By complementing the performance of mechanical lidar and solid-state lidar, superior fire line tracking and location are achieved. Although mechanical lidar has a shorter measurement distance and sparser point cloud, it has the advantage of 360° rotating acquisition and fast point cloud acquisition speed, making it suitable for SLAM to build a global point cloud map. Although solid-state lidar has a smaller field of view and slower acquisition speed, its effective detection range can reach 500 meters, and its point cloud is dense, making it suitable for collecting dense point clouds in local fire areas. This solves the safety problem when individual soldier equipment needs to collect fire line data at close range, while ensuring high data acquisition accuracy and processing speed. The SLAM algorithm is used for localization and mapping. An RTK signal quality detection module and an automatic calibration module are added. When there is no RTK signal, SLAM positioning is used. When there is an RTK signal, the weight fusion algorithm of RTK absolute position and SLAM relative position is dynamically adjusted to achieve accurate relocation, reduce SLAM cumulative error, and solve the problem of not being able to accurately locate the fire line in the forest when there is no GPS signal and no 4G differential data transmission signal.

[0071] This invention employs a 3D gimbal with a vibration damping bed, integrating an IMU, solid-state lidar, and infrared camera. The vibration damping bed counteracts high-frequency jitter during movement, and the 3D gimbal achieves dynamic self-stabilization through pose data inverse compensation, solving the problem of sensor jitter in individual soldier equipment during movement. Furthermore, addressing the dynamic changes in individual soldier equipment movement and fire line spread, this invention uses a deep learning-based fire line pattern prediction algorithm to predict the future position of the fire line in advance and dynamically adjust the gimbal angle in real time, ensuring continuous tracking of the fire line and solving the problem of the fire line leaving the sensor's field of view during fire line spread or individual soldier equipment movement.

[0072] This invention extracts multiple fire lines obscured by tree trunks from infrared images, uses an IMU to align the fire line contours of the current frame, historical frames, and the fire line predicted in step four for future frames, and employs a stitching algorithm based on multi-view fusion and feature matching to obtain the unobstructed fire line in the current frame; thus solving the problem of not being able to collect complete fire lines due to tree trunk obstruction. Attached Figure Description

[0073] The present invention can be better understood by referring to the description given below in conjunction with the accompanying drawings, which together with the following detailed description are included in and form part of this specification, and are used to further illustrate preferred embodiments of the invention and explain the principles and advantages of the invention.

[0074] Figure 1This is a flowchart illustrating a forest fire tracking and positioning method based on individual soldier equipment, as described in an embodiment of the present invention.

[0075] Figure 2 This is a flowchart illustrating a forest fire tracking and positioning method based on individual soldier equipment, as described in an embodiment of the present invention.

[0076] Figure 3 This is a schematic diagram of the individual soldier equipment structure in an embodiment of the present invention.

[0077] Figure 4 This is another structural schematic diagram of the individual soldier equipment in an embodiment of the present invention.

[0078] Figure 5 This is a diagram showing the transformation relationship between multiple sensor coordinate systems and the world coordinate system in an embodiment of the present invention.

[0079] Figure 6 This is an example diagram illustrating the process of tracking and locating forest fire lines in an embodiment of the present invention. Detailed Implementation

[0080] To enable those skilled in the art to better understand the present invention, exemplary embodiments or examples of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments or examples are merely some, not all, of the embodiments or examples of the present invention. All other embodiments or examples obtained by those skilled in the art based on the embodiments or examples of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0081] To address the challenge of achieving real-time and accurate fireline location in forest understory fires using traditional methods, and considering the significant decrease in accuracy caused by canopy obstruction and smoke interference in current forest fireline location technologies such as satellite remote sensing and drone patrols, as well as the technical difficulties of lacking GPS signals, navigating complex terrain, and hindering real-time communication, this invention proposes a forest fireline tracking and location method and system based on a single-soldier device. By designing a single-soldier device equipped with multimodal sensors, it collects lidar point cloud data, infrared image data, and pose data in the forest, and then extracts and fuses this data to achieve precise location of the forest fireline.

[0082] This invention proposes a method for tracking and locating forest fire lines based on individual soldier equipment, such as... Figures 1-2 As shown, the method includes:

[0083] S1. Use the designed individual soldier equipment to collect data, including pose data, radar point cloud data, and image data.

[0084] S2. Process the radar point cloud data based on the pose data to obtain a dense point cloud in the fire line area;

[0085] S3. Process the image data to obtain an unobstructed fire line image;

[0086] S4. Based on the calibrated data, perform fusion processing on the dense point cloud of the fire line area and the unobstructed fire line image to obtain the three-dimensional coordinates of the fire line.

[0087] The method begins in S1. In S1, data acquisition is performed using a designed individual soldier device to obtain pose data, radar point cloud data, and image data. This individual soldier device includes: a mechanical lidar, a solid-state lidar, a combined inertial navigation system (INS), an antenna, a DTU, a gimbal, an infrared camera, and an IMU. Specifically, the mechanical lidar is used to acquire global point cloud data; the solid-state lidar is used to acquire localized dense point cloud data along the fire line; the antenna is used to receive and acquire GPS signals; the DTU is used to acquire differential data from the base station and transmit this differential data to the combined INS; the combined INS is used to acquire pose data of the individual soldier device's main body and, combined with the GPS signal and the differential data, obtain RTK positioning information; the infrared camera is used to acquire infrared image data along the fire line; the IMU is used to record the pose data of the gimbal and its integrated sensors; and the gimbal is used to adjust the acquisition angles of the infrared camera and the solid-state lidar.

[0088] According to an embodiment of the present invention, a hardware structure for individual soldier equipment is designed, integrating a solid-state laser radar, an infrared camera, and an IMU into one unit, and installing it on a three-dimensional gimbal equipped with a shock-absorbing bed to form an automatic fire zone tracking and acquisition module. A mechanical laser radar and a combined inertial navigation system are integrated to form a forest area mapping and positioning module, and a power supply module and a communication module are built.

[0089] The individual soldier equipment is primarily designed for monitoring and locating fire lines in forests. Considering the complex forest environment, rugged mountain paths, and dense vegetation, a backpack-style design was chosen. Personnel carry the equipment along the fire line while maintaining a safe distance. A solid-state lidar, infrared camera, and IMU are integrated into a single unit, mounted on a 3D pan-tilt platform with a shock-absorbing bed. This integrated design facilitates sensor calibration, installation, and disassembly. The equipment inevitably experiences shaking during movement; the shock-absorbing bed reduces high-frequency vibrations, minimizing their impact on sensor accuracy. Since both the fire line and the equipment are dynamically changing, the 3D pan-tilt platform provides flexible angle adjustment capabilities, enabling precise data acquisition of localized fire zones. A mechanical lidar and combined inertial navigation system (INS) are integrated for forest area mapping and positioning. Mounted on the individual soldier equipment, the INS records the equipment's pose changes throughout its movement. SLAM algorithms are used for global mapping of the forest area and real-time positioning of the individual soldier equipment. The power supply module design considers the power requirements of each sensor, incorporating voltage regulators and isolators, and is equipped with a protective housing and heat dissipation device. By designing a louvered ventilation system, the stability and lightweight design of the equipment are ensured during long-term operation, guaranteeing continuous operation and sufficient power. The communication module addresses the issue of communication being affected by tree canopy and trunk obstruction within the forest. The equipment transmits data to a UAV relay platform via a wireless bridge, and then the UAV platform transmits it to the ground station, enabling long-distance communication. The UAV relay platform not only handles data transmission but also assists individual soldier equipment in path planning, ensuring the safest and fastest acquisition of fire-fighting data.

[0090] Specifically, such as Figures 3-4 As shown, antenna 1 is used to receive GPS signals; the mechanical lidar 2 has an effective range of 100M and quickly acquires point clouds of the surrounding environment through 360° scanning, and also includes a mechanical lidar base 3 and a mechanical lidar control box 15; the integrated inertial navigation system 4 is used to obtain data such as quaternions, Euler angles, acceleration, and angular velocity, and combines the acquired GPS signals and differential data to obtain RTK positioning information; DTU 14 is used to access the server through 4G communication of the SIM card to obtain differential data from the base station and transmit the data back to the integrated inertial navigation system; solid-state lidar 12 is used to collect... The system includes a dense point cloud with an effective range of 500 meters, which is scanned to obtain local dense point cloud data of the fire zone to prepare for subsequent fusion. It also includes a solid-state lidar control box 13; a gimbal 9 is used for automatic dynamic tracking of the fire line, and includes a gimbal base 5 and a gimbal control box 8; an infrared camera 10 is used to identify and photograph the fire line to obtain infrared image data of the fire line for image stitching, fusion, etc.; and an IMU 11 is used to record the pose changes of the gimbal and the sensors integrated on it for preliminary matching of local and global point clouds, as well as partial stitching of solid-state lidar point cloud and infrared camera image data.

[0091] In this embodiment, optionally, the individual soldier equipment may also include a wireless bridge 7 and a portable display 6. The wireless bridge 7 is used to transmit the location coordinates and image / video data of the fire line point cloud after positioning to the UAV relay platform, and then the UAV relay platform transmits it to the ground end. The portable display 6 is used to display the effect image of fire line extraction and positioning and the position of the fire line in the image in real time, which is convenient for the staff to monitor.

[0092] As an example, antenna 1 can be installed at the very top of the individual soldier's equipment, above the mechanical lidar 2, to prevent other sensors from obstructing it and affecting signal reception; the mechanical lidar 2 can be installed below antenna 1 and above the fixed base, ensuring its height is higher than the operator's head to avoid obstruction and achieve 360° omnidirectional point cloud data acquisition; the solid-state lidar 12 can be installed above the infrared camera 10, integrated with it, with the same field of view for easy synchronous data acquisition; the infrared camera 10 can be installed below the solid-state lidar 12 and above the gimbal 9, integrated for easy synchronous data acquisition and overall rotation; the IMU 11 can be installed on both the infrared camera 10 and the solid-state lidar. The system records the pose changes of the integrated gimbal 9, infrared camera 10, and solid-state lidar 12 during fire-tracking from 12 sides. The gimbal 9 can be installed below the infrared camera 10 and above the gimbal support platform, ensuring that the gimbal 9 is firmly fixed to the main body of the device and facilitating dynamic adjustments such as gimbal tracking. The combined inertial navigation system 4 can be installed on the gimbal support platform to record the pose changes of the main body in real time. It is located directly below the antenna 1, which also facilitates wiring and signal transmission. The wireless bridge 7 can be installed on the device support rod assembly and below the gimbal support platform at an upward angle of 45°, which can effectively avoid the gimbal support platform from blocking the signal and transmit the data quickly and accurately to the UAV relay platform.

[0093] Using the multimodal sensors mounted on the aforementioned individual soldier equipment, infrared images, point clouds, pose data, etc. are collected; mechanical lidar collects global environmental data in the forest; and solid-state lidar collects local data of the fire line.

[0094] When multiple sensors acquire data, time differences and internal clock drift within the sensors can cause timestamp asynchrony. To address this, a global time synchronization node is introduced to uniformly assign a time reference to all sensor systems. Simultaneously, the standard time signal from the combined inertial navigation system is provided to the lidar, enabling it to synchronize its internal timer in seconds. This ensures that data from all sensors is synchronized under the same time reference during acquisition, eliminating the impact of time drift on data alignment.

[0095] Then, step S2 is executed. In step S2, the radar point cloud data is processed based on the pose data to obtain a dense point cloud of the fire line area; specifically, this includes:

[0096] S21. Generate a SLAM global point cloud map using the SLAM algorithm;

[0097] S22. Calculate the RTK signal quality score based on the data output from the integrated inertial navigation system and the DTU.

[0098] S23. Based on the RTK signal quality score, the signal quality is divided into three intervals: high, medium, and low. When the RTK signal is in the low quality interval, only the SLAM algorithm is used for positioning. When the RTK signal is in the medium quality interval, RTK positioning and SLAM algorithm are dynamically fused for positioning. The dynamic fusion formula is as follows:

[0099] x fusion (t)=(1-α(t))·x S (t)+α(t·x R (t)+η·(x fusion (t-1)-x fusion (t))

[0100] Where, x fusion (9t) represents the position coordinates obtained at time t after fusing the absolute position from the RTK algorithm and the relative position from the SLAM algorithm; x S (t) represents the relative position coordinates generated by the SLAM algorithm, x R (t) represents the absolute position coordinates provided by RTK positioning information, η is the time smoothing coefficient, and x fusion (t-1) represents the position coordinates after fusion at the previous time step; α(t) represents the dynamic weight parameter;

[0101] When the RTK signal is in the high-quality range, only the RTK signal is used for positioning;

[0102] S24. Using the pose data and calibrated data provided by the IMU and the combined inertial navigation system, the local point cloud data of each frame of the fire line is transformed into the global coordinate system, so that the local point cloud data and the SLAM global point cloud are initially aligned.

[0103] S25. Divide the fire-line region and non-fire-line region in the aligned local point cloud data based on the dynamic threshold method:

[0104] M fire ={p∈P lidar |ρ(p)>ρ threshold}

[0105] M non-fire =P lidar \M fire

[0106] Where ρ(p) is the reflectance value at point p; ρ threshold Indicates the dynamic reflectivity threshold; Plidar M represents the local point cloud data set of the fireline; fire M represents the point cloud set of the fire zone. non-fire This represents a collection of point clouds in non-fire-area regions;

[0107] S26. Use the reflectivity threshold method to filter out outliers in the fire zone point cloud set, retaining the fire zone density point cloud as P. filtered The outlier filtering formula is:

[0108] P filtered ={p∈M fire ∣ρ min ≤ρ(p)≤ρ max}

[0109] Where, ρ min ρ max These are the upper and lower limits of reflectivity;

[0110] S27. Apply voxel filtering to the point cloud M in the non-fireline region. non-fire Downsampling is performed to generate a sparse point cloud;

[0111] S28. Use the NDT matching algorithm to perform coarse matching between the sparse point cloud in the non-fireline area and the global point cloud of SLAM to generate preliminary global alignment results.

[0112] S29. Use the ICP algorithm to perform fine matching between the preserved fire-line area density point cloud and the SLAM global point cloud to obtain the point cloud coordinates of the dense point cloud in the fire-line area in the world coordinate system.

[0113] According to an embodiment of the present invention, firstly, S21, localization and mapping are achieved through the SLAM algorithm.

[0114] Then, to address the issues of lack of GPS signal and 4G differential data transmission signal due to forest cover, the individual soldier's equipment can achieve RTK centimeter-level positioning accuracy before entering the forest area, providing accurate initial position information for the SLAM algorithm. After entering the forest area, it automatically switches to SLAM mode, adding an RTK signal quality detection module and an automatic calibration module. When there is no RTK signal, SLAM positioning is used. When there is an RTK signal, the weight fusion algorithm of RTK absolute position and SLAM relative position is dynamically adjusted to achieve accurate repositioning and reduce SLAM cumulative error.

[0115] Specifically, the S22 and RTK signal quality detection modules output the number of satellites n through the integrated inertial navigation system. sV Geometric precision factor (PDOP), signal-to-noise ratio (SNR), signal strength (RSRP) of the DTU sensor output, and time delay (t). delay Calculate the RTK signal quality score S RTK The specific formula is as follows:

[0116]

[0117] Where α1+α2+α3+α4+α5=1, it needs to be dynamically adjusted according to the signal quality score S. RTK The value of is used to divide the signal quality into three intervals. The segmentation threshold needs to be assigned based on the actual measurement data. As an example, it can be: low quality interval (0≤S) RTK ≤0.4), medium mass range (0.4) RTK< 0.9), high-quality range (0.9≤S) RTK ≤1).

[0118] S23. In the low-quality range, the system switches to SLAM-only mode for positioning; in the medium-quality range, the system enters the dynamic fusion positioning mode of RTK and SLAM; in the high-quality range, the system switches modes to perform high-precision RTK positioning, while calibrating and repositioning the SLAM data.

[0119] Among them, the RTK signal quality score is 0.4. RTK Within the range <0.9, the system employs a dynamic fusion mode of RTK and SLAM data, balancing their weights. This alleviates the instability of RTK signals and reduces the cumulative error of SLAM, achieving smooth transition and high-precision positioning. The dynamic weight α(t) is calculated using the following formula:

[0120]

[0121] in, It is the inverse of the covariance matrix of RTK data, which reflects the reliability of RTK data from the output of the combined inertial navigation system. It is the inverse of the covariance matrix of SLAM data, and its output reflects the reliability of the SLAM data.

[0122] The formula for dynamic fusion of RTK and SLAM positioning data is as follows:

[0123] x fusion (t)=(1-α(t))·x S (t)+α(t·x R (t)+η·(x fusion (t-1)-x fusion (t))

[0124] Where, x fusion (t) represents the position coordinates obtained at the current moment after fusing the absolute position of RTK and the relative position of SLAM, x S (t) represents the relative position coordinates generated by SLAM, x R (t) is the absolute position coordinate provided by RTK, η is the time smoothing coefficient, and x​​fusion (t-1) is the position coordinate obtained after fusing the absolute position of RTK and the relative position of SLAM at the previous time step.

[0125] RTK signal quality fraction 0.9≤S RTK Within the ≤1 range, the RTK signal quality is reliable, and the system relies solely on RTK data for positioning. fision (t)=x RTK (t).

[0126] Furthermore, the positioning data x obtained from multiple intervals fusion (t) The SLAM algorithm is introduced to correct the accumulated error of SLAM and to update the SLAM global point cloud map P in the world coordinate system in real time. SLAM (t), the formula is:

[0127] P SLAM (t)=P SLAM (t-1)∪(T(x fusion (t))·P local (t))

[0128] Among them, P SLAM (t) is the global point cloud map of SLAM at the current time, P SLAM (t-1) The SLAM global point cloud map of the previous time step, P local (t) The local point cloud acquired by the mechanical lidar at the current moment, T(x) fusion (t) will local point cloud P local (t) is the transformation matrix for converting to the world coordinate system.

[0129] Then, using the pose information provided by the IMU and the combined inertial navigation system, the local point cloud collected by the solid-state lidar is gradually transformed from the lidar coordinate system to the SLAM global point cloud coordinate system to complete the initial alignment. Based on the reflectivity characteristics of the solid-state lidar point cloud, the fire line area and non-fire line area are dynamically identified and divided. The point cloud in the non-fire line area is downsampled to reduce the point cloud density and is coarsely matched with the SLAM global point cloud to further narrow the matching range. The point cloud in the fire line area is affected by the thermal effect of the flame. Abnormal points are filtered out by the reflectivity threshold method. After retaining the high-quality point cloud, it is finely matched with the SLAM global point cloud to obtain the precise coordinates of the dense point cloud in the fire line area in the world coordinate system.

[0130] Specifically, S24, using the high-precision pose information and coordinate system transformation relationship provided by the IMU and the combined inertial navigation system, transforms the coordinate system of each frame of the solid-state lidar to the SLAM global coordinate system, achieving initial alignment of the point cloud. This reduces the search space and provides initial position estimation for subsequent point cloud matching. The formula is as follows:

[0131] PS =R INSS (R IMUINS (R IMULS P lidar +T IMULS )+T IMUINS )+T INSS

[0132] Among them, P S It is a point cloud in the SLAM global coordinate system, R INSS T INSS R is the rotation matrix and translation vector from the combined inertial navigation coordinate system to the SLAM global coordinate system. IMUINS T IMUINS R is the rotation matrix and translation vector from the IMU coordinate system to the combined inertial navigation coordinate system. IMULS T IMULS P is the rotation matrix and translation vector from the solid-state lidar coordinate system to the IMU coordinate system. lidar It is a point cloud captured by a solid-state lidar in one frame.

[0133] like Figure 5 As shown, the coordinate system transformation relationship is determined as follows: Utilizing the relative fixation between the IMU and the solid-state lidar, the first coordinate transformation relationship between the IMU coordinate system and the solid-state lidar coordinate system is obtained through extrinsic parameter calibration, using the rotation matrix R. IMULS Translation vector T IMULS Complete coordinate system alignment to ensure accurate transformation of solid-state lidar point cloud data into the IMU coordinate system; obtain the second coordinate transformation relationship between the IMU coordinate system and the combined inertial navigation coordinate system through calibration, and use the rotation matrix R. IMUINS Translation vector T IMUINS The coordinate system transformation is completed, accurately converting points from the IMU coordinate system to the combined inertial navigation coordinate system. A third coordinate transformation relationship between the combined inertial navigation coordinate system and the SLAM coordinate system is obtained through dynamic calibration. The combined inertial navigation coordinate system can be transformed using the rotation matrix R. INSS Translation vector T INSS Transform the SLAM global coordinate system to achieve global point cloud localization and fusion; utilize the first, second, and third coordinate transformation relationships to obtain the transformation relationship between the solid-state lidar coordinate system and the SLAM global coordinate system; obtain the fourth coordinate transformation relationship between the SLAM global coordinate system and the world coordinate system through dynamic calibration; the point cloud in the SLAM global coordinate system can be transformed using the rotation matrix R. SW Translation vector T SW Transform the world coordinate system.

[0134] S25. Utilize the point cloud reflectivity characteristics of solid-state lidar to dynamically identify fire-line and non-fire-line regions. Fire-line regions have higher reflectivity values ​​due to flame interference. By analyzing the reflectivity distribution of the point cloud, fire-line and non-fire-line regions are divided based on a dynamic threshold method.

[0135] M fire ={p∈P lidar |ρ(p)>ρ threshold}

[0136] M non-fire =P lidar \M fire

[0137] Where, ρ threshold It is the reflectivity threshold, M fire It is a collection of point clouds in the fire zone, M non-fire It is a set of point clouds in non-fire zone, where ρ(p) is the reflectance value of point p; dynamic threshold ρ threshold The formula for dynamically adjusting the reflectance distribution of point clouds is as follows:

[0138] ρ threshold =μ ρ +k·σ ρ

[0139] Where, μ ρ This represents the mean reflectance of the point cloud, k is a sensitivity control parameter that needs to be adjusted according to the scene, and σ... ρ It is the standard deviation of point cloud reflectance.

[0140] S26. Based on the initial alignment, the fire line area point cloud exhibits reflectivity anomalies due to the thermal effect of the flame. Anomalies are filtered out using the reflectivity threshold method, and the remaining fire zone density point cloud is P. filtered The outlier filtering formula is:

[0141] P filtered ={p∈M fire ∣ρ min ≤ρ(p)≤ρ max}

[0142] Among them, P filtered It is the filtered fireline area point cloud, ρ min ρ max These are the upper and lower limits of reflectivity.

[0143] S27. Based on the initial alignment, voxel filtering is used to refine the point cloud M of the non-fireline region. non-fire Downsampling is performed to generate a sparse point cloud, reducing the computational cost of subsequent matching. The formula is as follows:

[0144] P spase=VoxclFilter(M non-fire )

[0145] Among them, P spase It is a point cloud of non-fireline regions after sparsification.

[0146] S28. For the obtained sparse point cloud P in the non-fire zone spase and SLAM global point cloud P SLAM A coarse matching of sparse point clouds in non-fire-line regions is achieved using the fast NDT algorithm, generating preliminary global alignment results (R). coarse ,T coarse The specific objective function is:

[0147]

[0148] S29. Coarse matching results in the obtained non-fireline area (R) coarse ,T coarse Based on this, the dense point cloud P in the preserved fire line area was analyzed. fire Fine-grained matching with the SLAM global point cloud using the ICP algorithm is performed to optimize the final transformation matrix (R). fine ,T fine This achieves precise matching between each frame of point cloud acquired by the solid-state lidar and the SLAM point cloud acquired by the mechanical lidar. The specific objective function is:

[0149]

[0150] Dense point cloud of the fire zone after precise matching The point cloud coordinates in the world coordinate system are:

[0151]

[0152] Among them, R SW T SW It is the rotation matrix and translation vector from the SLAM global coordinate system to the world coordinate system.

[0153] Then, S3 is executed, in which the image data is processed to obtain an unobstructed fireline image; specifically including: S31, fireline identification is performed on the acquired infrared image, contrast stretching and histogram equalization are performed on the infrared image; the optimal segmentation threshold is automatically calculated using the Otsu method, and the image is binarized; the fireline edges in the image are extracted using the Canny edge detection algorithm, and the fireline contour is colored; S32, based on the fireline image data sequence corresponding to the historically acquired infrared images and the fire site terrain, meteorological and combustible material information, an extended morphology prediction frame of the fireline is generated using a long short-term memory neural network;

[0154] S33. Using the camera motion pose information recorded by the IMU, and employing rotation matrices and translation vectors, the fire line image data sequence corresponding to historically acquired infrared images is transformed and projected onto the current frame image plane to complete the coarse alignment of the fire line images between frames. S34. Combining the fire line image data corresponding to the current frame and historically acquired infrared images, as well as the extended morphology prediction frame generated in S32, key feature points of the fire line region are extracted, and a feature point matching method is used to establish the inter-frame correspondence to achieve precise geometric alignment of the fire line segments and obtain the unobstructed fire line image of the current frame.

[0155] According to an embodiment of the present invention, firstly, fire lines are identified by infrared images acquired by an infrared camera. Based on the imaging principle of infrared, areas with a pixel count close to 255 are fire lines, and areas with a pixel count close to 0 are unburned areas. Contrast stretching and histogram equalization are performed on the infrared images to improve the overall contrast of the images, making the fire lines more obvious. The optimal segmentation threshold thr is automatically calculated using the Otsu method, and the images are binarized to further enhance the contrast between the fire lines and the background. The Canny edge detection algorithm is used to extract the fire line edges in the images, and the fire line contours are colored.

[0156] Then, due to the dynamic complexity of the fire line, based on the historical fire line image data sequence {I t-1 ,I t-2 ,...,I t-n This paper introduces deep learning algorithms to predict the dynamic changes of the fire line in real time. A convolutional long short-term memory network is used to model the spatial and temporal features of the fire line, and environmental information is used as input to generate a prediction frame I of the extended morphology of the future fire line. predict,t+1 .

[0157] Then, the infrared camera and IMU are integrated into one unit. The pose information of each frame of infrared image recorded by the IMU is used to generate a rotation matrix R. IMU Translation vector T IMU The historical frame fireline images are converted and projected onto the current frame image plane to complete the coarse alignment of the inter-frame fireline images.

[0158] P trans =R IMU ·P history +T IMU

[0159] Among them, P trans P is the projection coordinate of the historical frame fire line region onto the current frame image plane. history These are the pixel coordinates in the historical frame's fireline region, R IMU It is a rotation matrix provided by the IMU, T IMU It is a translation vector provided by the IMU;

[0160] Finally, based on the coarse matching results, and combining the infrared fire line images of the current frame, historical frames, and generated predicted frames, key feature points of the fire line area are extracted, and the inter-frame correspondence is established using the feature point matching method. A homography matrix is ​​constructed to achieve precise geometric alignment of the fire line segments, and seamless stitching of the fire line image of the current frame is completed to obtain the complete infrared fire line image of the current frame.

[0161] Then, step S4 is executed. In step S4, the dense point cloud of the fire line area and the unobstructed fire line image are fused based on the calibrated data to obtain the three-dimensional coordinates of the fire line; specifically, this includes:

[0162] S41. By combining the internal and external parameter calibration data of solid-state lidar and infrared camera, the dense point cloud of the fire line area in the current frame is projected onto the unobstructed fire line image of the current frame, realizing the conversion of three-dimensional point cloud coordinates to two-dimensional image pixel coordinates, and obtaining the accurate correspondence between the point cloud of the fire line area and the infrared image.

[0163] S42. Using the color information and pixel position information of the fire line outline in the infrared image, the fire line outline with color information is associated with and assigned to the three-dimensional points in the point cloud, the point cloud carrying color information is extracted, the redundant point cloud is filtered out, and the accurate three-dimensional point cloud coordinates of the fire line in three-dimensional space are obtained.

[0164] S43. Using an adaptive ICP algorithm and combining pose data provided by the IMU, the multi-frame fireline 3D point cloud obtained in S42 is stitched together to achieve accurate positioning of the complete fireline in the world coordinate system.

[0165] According to an embodiment of the present invention, the purpose of the present invention is to obtain the three-dimensional point cloud coordinates of the forest fire line in the world coordinate system. The present invention fuses the current frame fire line image without tree trunk obstruction with the local fire area point cloud of the solid-state lidar in the world coordinate system to extract the three-dimensional coordinates of the fire line in the world coordinate system. The specific process is as follows:

[0166] External parameter calibration is performed using an integrated solid-state lidar and infrared camera to establish the transformation relationship between the solid-state lidar coordinate system and the infrared coordinate system;

[0167] Using the obtained extrinsic parameter calibration results and the intrinsic parameter matrix of the infrared camera, the 3D point cloud acquired by the solid-state lidar is projected onto the infrared image plane, converting the 3D point cloud coordinates into 2D pixel coordinates, and then converting the 3D point cloud P in the lidar coordinate system... lidar =(X lidar ,Y lidar Z lidar Convert P to a 3D point cloud in the infrared camera coordinate system camera =(X camera ,Y camera Z cameraThe formula for transforming the infrared camera coordinate system to the pixel coordinates (u,v) of the infrared image is as follows:

[0168]

[0169] Where R is the rotation matrix, t is the translation vector, and K is the intrinsic parameter matrix of the infrared camera;

[0170] By utilizing the color information and pixel position information of the fire line outline in the infrared image, the fire line outline with color information is associated with and assigned to three-dimensional points in the point cloud. The point cloud carrying color information is extracted from the solid-state lidar point cloud, and redundant point clouds are filtered out to obtain the accurate three-dimensional point cloud coordinates of the fire line in three-dimensional space.

[0171] By using an adaptive ICP algorithm and combining pose data provided by the IMU, the acquired multi-frame 3D point cloud of the fireline is stitched together to achieve precise positioning of the complete fireline in the world coordinate system.

[0172] Dynamic self-stabilization is achieved using a 3D gimbal. The fire line is identified and extracted by an infrared camera. A fire line shape prediction algorithm based on deep learning predicts the future position of the fire line in advance and dynamically adjusts the gimbal angle in real time to ensure dynamic tracking of the fire line.

[0173] Furthermore, since the individual soldier equipment is constantly in motion during movement, the sensors on the equipment are subject to frequent jitter, which affects the acquisition accuracy of data such as infrared images and solid-state lidar point clouds. To solve this problem, the sensors are mounted on a 3D gimbal, and their pose data is recorded by the combined inertial navigation system on the individual soldier equipment. The data is then fed back to the 3D gimbal in real time to achieve self-stabilizing control and maintain the stability of the equipment. In addition, as the individual soldier equipment moves along the line of fire and the line of fire dynamically spreads, it is necessary to ensure that the line of fire is always within the field of view of the sensors. Deep learning is used to generate prediction frames of the extended shape of the line of fire, and the gimbal angle is dynamically adjusted in advance to ensure continuous tracking of the line of fire and reduce monitoring delays caused by equipment lag.

[0174] In this embodiment, preferably, the following steps are used during the data acquisition process of S1 to achieve continuous stabilization of the solid-state lidar and infrared camera images:

[0175] A real-time attitude inverse compensation algorithm is used to convert the data into angular offset values, which are then used to provide real-time inverse compensation to the gimbal's control system. The formula for attitude inverse compensation is as follows:

[0176] φ y (t)=φ y (t-1)-Δθ′ y

[0177] φ p (t)=φ p(t-1)-Δθ′ p

[0178] φ r (t)=φ r (t-1)-Δθ′ r

[0179] Where φ y φ p φ r Δθ′ represents the yaw, pitch, and roll angles of the 3D gimbal, respectively, and t represents time. y ,Δθ′ p ,Δθ′ r It is the value of the IMU attitude change after coordinate transformation;

[0180] The formula for 3D gimbal attitude control is as follows:

[0181]

[0182] Where k y k p k r It is the feedback gain parameter, e y e p e r R represents the error of the gimbal attitude angle, and R represents the rotation matrix for transforming the coordinate system of the combined inertial navigation system to the coordinate system of the gimbal.

[0183] In this embodiment, preferably, the pan-tilt unit is dynamically adjusted in real time during the data acquisition process of S1 using the following steps to achieve the overlap between the center of gravity of the fire line and the center of the infrared image:

[0184] For the predicted extended form of the future fireline obtained by S32, traverse each pixel in the image and count the coordinates (x, y, y) of the pixels belonging to the fireline. i ,y i ), calculate the center of gravity C of the fire line. f (x c ,y c The center point of the fire line is represented by the following formula:

[0185]

[0186] in, This refers to the weight of the pixel in FireWire; the higher the brightness, the greater the weight. a(i,j) is the grayscale value of pixel (i,j), and x... i ,y i is the x-coordinate and y-coordinate of the i-th fireline pixel, and N is the total number of fireline pixels.

[0187] Calculate the center position C of the infrared image m (xm ,y m The center of gravity C of the fire line f (x c ,y c The offset (Δx, Δy) between the centers of the infrared image and the center position C. m (x m ,y m ) is a fixed value determined by the size W×H of the infrared image, where W is the width of the image and H is the height of the image.

[0188]

[0189] The gimbal is controlled based on the offset to ensure that the center of gravity of the fire line coincides with the center of the infrared image, guaranteeing that the fire line remains centered in the infrared image. Due to the motion characteristics and inertia of the gimbal, the PD controller is used to adjust the gimbal angle.

[0190]

[0191] Where, θ pitch ,θ yaw These represent the pitch angle and yaw angle of the gimbal, respectively; K X ,K y D represents the proportional gain of pitch and yaw angles, respectively; X D y These represent the differential gains of the pitch and yaw angles, respectively. These represent the rate of change of the offset.

[0192] To fully verify the technical effects of the present invention, the following experiments were conducted on the technical solution proposed in this invention.

[0193] The experiment was conducted in a selected forest area. To ensure fire safety within the forest, a 5×5 meter testing site was chosen. Three stainless steel basins filled with boiling water were used as heat sources, all positioned in front of tree trunks to simulate fire lines and verify the accuracy of fire line data collected by the individual soldier's equipment. The centers P1, P2, and P3 of the three heat sources were used as reference points. The experiment included two control groups: the first group used a self-assembled RTK base station method to collect the precise coordinates of the three reference points P′1, P′2, and P′3 as absolute baseline values ​​(see Table 1); the second group activated the individual soldier's equipment sensors when RTK signals were good outside the forest and used a combined inertial navigation system to receive GPS signals for centimeter-level RTK positioning, providing high-precision initial position coordinates for SLAM. This group of experiments added an RTK signal quality detection module and an automatic calibration module. When there was no RTK signal, SLAM positioning was used; when there was an RTK signal, the absolute RTK position and the relative SLAM position were dynamically fused, and a weighted adjustment algorithm was used to achieve accurate repositioning and reduce SLAM cumulative error. Figure 6 This demonstrates the process of tracking and locating forest fire lines. During the experiment, the device moves along the simulated fire line to complete the mapping and location of the entire forest area. The equipment utilizes a 3D gimbal for dynamic self-stabilization, a solid-state lidar to collect dense point clouds of local fire areas, an infrared camera to identify and extract the contours of heat sources, and controls the infrared gimbal for tracking. The solid-state lidar is integrated with an IMU, and the mechanical lidar is integrated with a combined inertial navigation system. Both are calibrated using extrinsic parameters to obtain rotation matrices and translation vectors, thereby transforming the dense point clouds of local fire areas collected by the solid-state lidar into the world coordinate system, completing the initial alignment with the global forest area point cloud generated by SLAM. Dense point clouds are collected in the heat source area and sparse point clouds are collected in the non-heat source area. Coarse matching is performed using the sparse point cloud, and fine matching is achieved using the dense point cloud in the heat source area. This dynamically obtains accurate point cloud matching, enabling the point cloud data collected by the solid-state lidar to obtain accurate GPS coordinates in the world coordinate system. Finally, the infrared image of the heat source identified by the infrared camera is fused with the dense point cloud of the heat source area in the world coordinate system to realize the projection of the three-dimensional point cloud onto the two-dimensional infrared image. The three-dimensional point cloud of the heat source contour is extracted, and the extracted three circular heat source contour point clouds are fitted respectively to calculate the GPS coordinates of the center of each circular contour. These coordinates are used as reference points P′1′, P′2′, and P′3′ in control group 2, as shown in Table 2.

[0194] Table 1 Control Group 1

[0195]

[0196] Table 2 Control Group II

[0197]

[0198] Another embodiment of the present invention proposes a forest fire tracking and positioning system based on individual soldier equipment, the system comprising:

[0199] The data acquisition module is configured to collect data using the designed individual soldier equipment, acquiring pose data, radar point cloud data, and image data.

[0200] A point cloud processing module is configured to process the radar point cloud data based on the pose data to obtain a dense point cloud in the fire line area.

[0201] An image processing module is configured to process the image data to obtain an unobstructed fire line image;

[0202] The fusion processing module is configured to perform fusion processing on the dense point cloud of the fire line area and the unobstructed fire line image based on the calibrated data to obtain the three-dimensional coordinates of the fire line.

[0203] The function of the forest fire line tracking and positioning system based on individual soldier equipment described in this embodiment of the invention can be explained by the aforementioned forest fire line tracking and positioning method based on individual soldier equipment. Therefore, for the parts not described in detail in the system embodiment, please refer to the above method embodiment, and they will not be repeated here.

[0204] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. The disclosure of the invention is illustrative and not restrictive, and the scope of the invention is defined by the appended claims.

Claims

1. A method for tracking and locating forest fire lines based on individual soldier equipment, characterized in that, include: S1. Data acquisition is performed using the designed individual soldier equipment to obtain pose data, radar point cloud data, and image data; the individual soldier equipment includes mechanical lidar, solid-state lidar, integrated inertial navigation system, antenna, DTU, gimbal, infrared camera, and IMU. S2. Process the radar point cloud data based on the pose data to obtain a dense point cloud of the fire line area; including: S21. Generate a SLAM global point cloud map using the SLAM algorithm; S22. Calculate the RTK signal quality score based on the output data of the integrated inertial navigation system; S23. Based on the RTK signal quality score, the RTK signal quality is divided into three intervals: high, medium, and low. When the RTK signal is in the low quality interval, only the SLAM algorithm is used for localization. When the RTK signal is in the medium quality interval, RTK signal localization and SLAM algorithm are dynamically fused for localization. The dynamic fusion formula is as follows: ； in, It is the position coordinate obtained by fusing the absolute position of RTK and the relative position of SLAM algorithm at the current time t; These are the relative position coordinates generated by the SLAM algorithm. These are the absolute position coordinates provided by RTK positioning information. It is the time smoothing coefficient. These are the position coordinates after merging from the previous moment; Indicates dynamic weight parameters; When the RTK signal is in the high-quality range, only the RTK signal is used for positioning; S24. Using the pose data and calibrated data provided by the IMU and the combined inertial navigation system, the local point cloud data of each frame of the fire line is transformed into the global coordinate system, so that the local point cloud data and the SLAM global point cloud are initially aligned. S25. Divide the fire-line region and non-fire-line region in the aligned local point cloud data based on the dynamic threshold method: ； ； in, It is a point The reflectivity value; Indicates the dynamic reflectivity threshold; This represents a local point cloud data set in the fire zone; This represents the point cloud set in the fire zone. This represents a collection of point clouds in non-fire-area regions; S26. Using the reflectivity threshold method, outliers in the fire zone point cloud set are filtered out, and the remaining fire zone density point cloud is: The outlier filtering formula is: ； in, , These are the upper and lower limits of reflectivity; S27. Point cloud analysis of non-fire-line areas using voxel filtering. Downsampling is performed to generate a sparse point cloud; S28. Use the NDT matching algorithm to perform coarse matching between the sparse point cloud in the non-fireline area and the global point cloud of SLAM to generate preliminary global alignment results. S29. Use the ICP algorithm to perform fine matching between the preserved fire zone density point cloud and the SLAM global point cloud to obtain the point cloud coordinates of the dense point cloud in the fire zone in the world coordinate system. S3. Process the image data to obtain an unobstructed fire line image; S4. Based on the calibrated data, perform fusion processing on the dense point cloud of the fire line area and the unobstructed fire line image to obtain the three-dimensional coordinates of the fire line.

2. The forest fire tracking and positioning method based on individual soldier equipment according to claim 1, characterized in that, The mechanical lidar in S1 is used to collect global point cloud data; the solid-state lidar is used to collect local dense point cloud data of the fire line; the antenna is used to receive and acquire GPS signals; the DTU is used to acquire base station differential data and transmit the differential data to the integrated inertial navigation system; the integrated inertial navigation system is used to collect the pose data of the main body of the individual soldier's equipment and obtain RTK positioning information by combining the GPS signal and the differential data; the infrared camera is used to collect infrared image data of the fire line; the IMU is used to record the pose data of the gimbal and the sensors integrated on it; the gimbal is used to adjust the acquisition angle of the infrared camera and the solid-state lidar.

3. The forest fire tracking and positioning method based on individual soldier equipment according to claim 1, characterized in that, The formula for calculating the RTK signal quality score in S22 is as follows: ； Among them, weight parameters + + + + =1; Indicates the number of satellites; PDOP represents the geometric precision factor; SNR represents the signal-to-noise ratio; RSRP represents the signal strength. represents latency; max represents the maximum value corresponding to each preset indicator; min represents the minimum value corresponding to each preset indicator.

4. The forest fire tracking and positioning method based on individual soldier equipment according to claim 3, characterized in that, The dynamic weight parameters described in S23 The calculation formula is as follows: ； in, Indicates the RTK signal quality score; It is the inverse of the covariance matrix of RTK positioning information. It is the inverse of the covariance matrix of the SLAM algorithm; The formula for calculating the dynamic reflectivity threshold in S25 is as follows: ； in, This represents the mean reflectance of the point cloud; Indicates the sensitivity control parameters; The standard deviation of point cloud reflectance.

5. A forest fire tracking and positioning method based on individual soldier equipment according to claim 4, characterized in that, S3 involves processing the image data to obtain an unobstructed fire line image, including: S31. Perform fire line identification on the acquired infrared image, and perform contrast stretching and histogram equalization on the infrared image; use the Otsu method to automatically calculate the optimal segmentation threshold, and perform binarization processing on the image; use the Canny edge detection algorithm to extract the fire line edges in the image, and add color to the fire line contours. S32. Based on the fire line image data sequence corresponding to the historically acquired infrared images and the fire site topography, meteorology and combustible material information, use a long short-term memory neural network to generate extended morphology prediction frames of the fire line. S33. Using the camera motion pose information recorded by the IMU, the fire line image data sequence corresponding to the historically acquired infrared images is converted and projected onto the current frame image plane using the rotation matrix and translation vector to complete the coarse alignment of the inter-frame fire line images. S34. Combining the current frame, the fire line image data corresponding to the historically acquired infrared images, and the extended morphology prediction frame generated in S32, key feature points of the fire line area are extracted, and the inter-frame correspondence is established using the feature point matching method to achieve precise geometric alignment of the fire line segments and obtain an unobstructed fire line image of the current frame.

6. The forest fire tracking and positioning method based on individual soldier equipment according to claim 5, characterized in that, The following steps are used to achieve continuous stabilization of images from the solid-state lidar and infrared camera during the S1 data acquisition process: A real-time attitude inverse compensation algorithm is used to convert the data into angular offset values, which are then used to provide real-time inverse compensation to the gimbal's control system. The formula for attitude inverse compensation is as follows: ； ； ； in, , , These represent the yaw, pitch, and roll angles of the 3D gimbal, respectively, and t represents the time. , , It is the value of the IMU attitude change after coordinate transformation; The formula for 3D gimbal attitude control is as follows: ； in, , , It is the feedback gain parameter. R represents the error of the gimbal attitude angle, and R represents the rotation matrix for transforming the coordinate system of the combined inertial navigation system to the coordinate system of the gimbal.

7. A forest fire tracking and positioning method based on individual soldier equipment according to claim 6, characterized in that, During the data acquisition process of S1, the pan-tilt unit is dynamically adjusted in real time using the following steps to achieve alignment between the fire line center of gravity and the infrared image center: For the predicted extended form of the future fireline obtained by S32, traverse each pixel in the image and count the coordinates of the pixels belonging to the fireline. Calculate the center of gravity of the fire line , used to represent the center point of the fire line, is calculated using the following formula: ； in, It represents the weight of the fireline pixels; N is the total number of fireline pixels. It is a pixel. grayscale value; Calculate the center position of the infrared image Center of gravity of the fire line offset between ( , ); The gimbal angle is adjusted in real time based on the aforementioned offset, and the angle adjustment formula is as follows: ； ； in, These represent the pitch angle and yaw angle of the gimbal, respectively. These represent the proportional gains for pitch and yaw angles, respectively. These represent the differential gains of the pitch and yaw angles, respectively. These represent the rate of change of the offset.

8. A forest fire line tracking and positioning method based on individual soldier equipment according to claim 7, characterized in that, S4 involves fusing the dense point cloud of the fire line region and the unobstructed fire line image based on the calibrated data, including: S41. Combining the internal and external parameter calibration data of solid-state lidar and infrared camera, the dense point cloud of the fire line area in the current frame is projected onto the unobstructed fire line image of the current frame, realizing the conversion of three-dimensional point cloud coordinates to two-dimensional image pixel coordinates, and obtaining the correspondence between the point cloud of the fire line area and the infrared image. S42. Using the color information and pixel position information of the fire line outline in the infrared image, the fire line outline with color information is associated with and assigned to the three-dimensional points in the point cloud. The point cloud carrying color information is extracted, and redundant point clouds are filtered out to obtain the three-dimensional point cloud coordinates of the fire line in three-dimensional space. S43. Using an adaptive ICP algorithm and combining the pose data provided by the IMU, the multi-frame fireline 3D point cloud obtained in S42 is stitched together to achieve precise positioning of the fireline in the world coordinate system.

9. A forest fire tracking and positioning system based on individual soldier equipment, characterized in that, The system is implemented based on the forest fire line tracking and positioning method based on individual soldier equipment according to any one of claims 1-8; the system includes: The data acquisition module is configured to collect data using the designed individual soldier equipment, acquiring pose data, radar point cloud data, and image data. A point cloud processing module is configured to process the radar point cloud data based on the pose data to obtain a dense point cloud in the fire line area. An image processing module is configured to process the image data to obtain an unobstructed fire line image; The fusion processing module is configured to perform fusion processing on the dense point cloud of the fire line area and the unobstructed fire line image based on the calibrated data to obtain the three-dimensional coordinates of the fire line.

Images