A method and system for automatic adjustment of laser pointing based on unmanned aerial vehicles

Abstract

The application discloses a kind of based on unmanned plane's laser shooting automatic adjustment method and system, it is related to unmanned plane technical field, including, real-time acquisition IMU, GPS, laser radar and visible light camera data, preliminary processing is carried out;The data after processing is fused by Kalman filtering algorithm, the attitude of unmanned plane is estimated in real time;Deep learning model is constructed, and the contour and direction of target object are automatically identified;According to the contour and direction of target object, the offset distance of image center point to target object is calculated;Establish environment model, calculate the compensation parameter of laser shooting direction;According to attitude estimation result, the offset distance of image center point to target object and the compensation parameter of laser shooting direction, dynamically adjust laser shooting direction;By according to the contour and direction information of target object, the center point coordinates of target object in image are determined, real-time adjustment laser shooting direction, even in the case where unmanned plane moves or target object moves also can maintain the accurate pointing of laser.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a method and system for automatic laser direction adjustment based on UAVs. Background Technology

[0002] In recent years, with the rapid development and widespread application of drone technology, drones have been increasingly used in various fields such as agriculture, surveying and mapping, security, and logistics. Especially in fields such as precision agriculture, environmental monitoring, and emergency rescue, the autonomous navigation and target recognition capabilities of drones have become a research hotspot. Traditional drone navigation and target recognition mainly rely on single sensors. These sensors perform relatively stably in static or simple environments, but in complex dynamic environments, due to signal interference, data delays, and other issues, it is difficult to achieve high-precision navigation and target recognition.

[0003] Despite significant advancements in multi-sensor fusion technology and deep learning for UAV navigation and target recognition, existing solutions still have shortcomings in certain aspects. First, traditional Kalman filtering algorithms often assume a Gaussian distribution of sensor errors when processing multi-sensor data. This assumption is not always valid in practical applications, especially in complex dynamic environments where the nonlinearity and uncertainty of sensor data can lead to biases in the filtering results. Second, while existing target recognition methods can achieve high-precision object identification, their real-time performance in dynamic environments still needs improvement. Furthermore, existing laser direction adjustment methods primarily rely on static environment models, lacking adaptability to dynamic environments and failing to maintain precise laser pointing even when the UAV or target object is moving. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a method and system for automatic laser direction adjustment based on unmanned aerial vehicles (UAVs) to solve the problems of low laser direction accuracy and insufficient real-time performance of UAVs in complex dynamic environments in the prior art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, embodiments of the present invention provide a method for automatic laser direction adjustment based on a drone, which includes real-time acquisition of data from an IMU, GPS, lidar, and visible light camera, and preliminary processing thereof;

[0008] The processed data is fused using the Kalman filter algorithm to estimate the UAV's attitude in real time.

[0009] Build deep learning models to automatically identify the outline and orientation of target objects;

[0010] Calculate the offset distance from the image center point to the target object based on the object's outline and orientation.

[0011] Establish an environmental model and calculate the compensation parameters for the laser trajectory;

[0012] The laser direction is dynamically adjusted based on the attitude estimation results, the offset distance from the image center point to the target object, and the compensation parameters of the laser direction.

[0013] As a preferred embodiment of the UAV-based automatic laser direction adjustment method of the present invention, the real-time acquisition and preliminary processing of data from IMU, GPS, lidar, and visible light camera includes the following steps.

[0014] Plan the flight path of the drone and set the latitude, longitude coordinates and altitude of the target point on the ground station;

[0015] Launch the drone and perform initial positioning using the inertial navigation system and GPS;

[0016] When the drone reaches the target area, the IMU is activated to collect the drone's angular velocity and acceleration data, the GPS is used to collect the drone's latitude and longitude coordinates and altitude data, the lidar is used to collect the three-dimensional point cloud data around the target point, and the visible light camera is used to collect the image data of the target point.

[0017] The collected data undergoes data cleaning, data filtering, noise suppression, and data alignment.

[0018] As a preferred embodiment of the UAV-based automatic laser direction adjustment method of the present invention, the real-time estimation of the UAV's attitude by fusing the processed data using a Kalman filter algorithm includes the following steps.

[0019] Initialize the state vector and covariance matrix;

[0020] The state vector and covariance matrix are updated using GPS data and IMU data to obtain a preliminary attitude estimate of the UAV.

[0021] The ICP algorithm is used to register the point cloud data collected by the lidar with the map point cloud to obtain the relative pose change.

[0022] The relative pose change obtained from point cloud registration is applied to the updated state vector of the Kalman filter to obtain the corrected attitude estimate of the UAV.

[0023] As a preferred embodiment of the UAV-based automatic laser direction adjustment method of the present invention, the following steps are included in constructing a deep learning model to automatically identify the contour and orientation of the target object:

[0024] Collect a large amount of image data containing the target object, annotate the outline and orientation information of the target object, and construct a training dataset;

[0025] A convolutional neural network model is trained using a labeled dataset, and a loss function is used to optimize the bounding box regression and orientation angle classification of the target object.

[0026] During the drone's flight, image data is captured in real time and input into a trained convolutional neural network model for target recognition, extracting the contour and orientation information of the target object. The expression is as follows:

[0027]

[0028] Where O represents the contour and orientation information of the target object, C represents the contour of the target object, I represents the input image, α represents the orientation angle of the target object, i represents the contour index of the target object, N represents the total number of points on the contour of the target object, and x represents the x-axis. i and y i Let x and y be the x and y coordinates of the i-th point on the contour of the target object, respectively. and These are the average x-coordinate and y-coordinate of the contour points, respectively.

[0029] As a preferred embodiment of the UAV-based automatic laser direction adjustment method of the present invention, the method for calculating the offset distance from the image center point to the target object based on the outline and orientation of the target object includes the following steps:

[0030] Based on the outline and orientation information of the target object, determine the coordinates of the center point of the target object in the image;

[0031] Using the camera's intrinsic parameters, the center point coordinates of the target object in the image are converted to their distance in the physical world. The offset of the target object in the physical world is then calculated using the following expression:

[0032]

[0033] Where ΔX is the horizontal offset of the target object in the physical world, ΔY is the vertical offset of the target object in the physical world, w is the width of the image, h is the height of the image, Z is the distance from the target object to the camera, and f x f is the horizontal focal length of the camera. y This is the camera's vertical focal length.

[0034] As a preferred embodiment of the UAV-based automatic laser direction adjustment method of the present invention, the steps of establishing an environmental model and calculating the compensation parameters of the laser direction include the following:

[0035] The attitude and position information of the UAV estimated by the Kalman filter is used to transform the point cloud data collected by the lidar into the global coordinate system to form a three-dimensional point cloud map.

[0036] The point cloud data is transformed into a 3D model of the surrounding environment using a meshing method.

[0037] Determine the location of the target point of the laser beam in the 3D model;

[0038] Based on the position of the laser beam aimed at the target point, the position and distance of obstacles along the laser beam's path are calculated using the following expression:

[0039]

[0040] Where d is the distance to the nearest obstacle on the laser's path, and P global Let p be the coordinates of the point cloud data in the global coordinate system, and p be a point in the point cloud data. tatget The coordinates of the target point in the global coordinate system;

[0041] Based on the position and distance of obstacles along the laser's path, compensation parameters for the laser trajectory are designed, expressed as follows:

[0042]

[0043] Where Δθ is the pitch angle compensation parameter for the laser beam direction, Δφ is the yaw angle compensation parameter for the laser beam direction, and k d is a scaling factor, and D is the maximum effective distance of the laser beam.

[0044] As a preferred embodiment of the UAV-based automatic laser direction adjustment method of the present invention, the dynamic adjustment of the laser direction based on attitude estimation results, the offset distance from the image center point to the target object, and the compensation parameters of the laser direction includes the following steps:

[0045] The control algorithm is designed, taking the attitude estimation result, the offset distance from the image center point to the target object, and the compensation parameters for the laser direction as input, and outputting the laser angle adjustment command to ensure that the laser always points to the target object. The expression is:

[0046]

[0047] Where, θ cmd For adjusting the pitch angle of the laser, φ cmd This is the yaw angle adjustment command for the laser, where φ is the pitch angle of the UAV, θ is the roll angle of the UAV, and k is the yaw angle. X k is the scaling factor for the horizontal offset. Y This is the scaling factor for the vertical offset.

[0048] Secondly, the present invention provides an automatic laser direction adjustment system based on a drone, comprising,

[0049] The data acquisition module collects data from IMU, GPS, LiDAR, and visible light camera in real time and performs preliminary processing.

[0050] The data fusion module uses the Kalman filter algorithm to fuse the processed data and estimate the attitude of the UAV in real time.

[0051] The target recognition module builds a deep learning model to automatically identify the outline and orientation of target objects;

[0052] The offset calculation module calculates the offset distance from the image center point to the target object based on the object's outline and orientation.

[0053] The environmental modeling module builds an environmental model and calculates compensation parameters for the laser beam direction.

[0054] The laser trajectory adjustment module dynamically adjusts the laser trajectory based on the attitude estimation results, the offset distance from the image center point to the target object, and the compensation parameters of the laser trajectory.

[0055] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements any step of the automatic laser direction adjustment method based on a UAV as described in the first aspect of the present invention.

[0056] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the automatic laser direction adjustment method based on a UAV as described in the first aspect of the present invention.

[0057] The beneficial effects of this invention are as follows: real-time acquisition of data from IMU, GPS, LiDAR, and visible light camera for preliminary processing; fusion of the processed data using a Kalman filter algorithm to estimate the UAV's attitude in real time; construction of a deep learning model to automatically identify the contour and orientation of the target object; calculation of the offset distance from the image center point to the target object based on the object's contour and orientation; establishment of an environmental model to calculate compensation parameters for the laser trajectory; dynamic adjustment of the laser trajectory based on the attitude estimation results, the offset distance from the image center point to the target object, and the compensation parameters; and determination of the target object's center point coordinates in the image based on its contour and orientation information, providing an accurate reference point for subsequent offset distance calculations. The system can adjust the laser trajectory in real time, maintaining precise laser pointing even when the UAV or the target object is moving. Attached Figure Description

[0058] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart of the automatic laser direction adjustment method based on UAV in Example 1.

[0060] Figure 2 This is a flowchart of the automatic laser direction adjustment system for the UAV in Example 1.

[0061] Figure 3 A computer device diagram illustrating a method for obtaining the sequence of multi-resource inertia and primary frequency modulation auxiliary services. Detailed Implementation

[0062] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0063] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0064] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0065] Example 1, referring to Figures 1-3 This is the first embodiment of the present invention, which provides a method for automatic adjustment of laser direction based on a drone, including the following steps:

[0066] The S100 acquires data from IMU, GPS, LiDAR, and visible light camera in real time, and performs preliminary processing including the following steps.

[0067] Plan the UAV's flight path and set the latitude, longitude, and altitude of the target point on the ground station; start the UAV and perform initial positioning using the inertial navigation system and GPS; when the UAV reaches the target area, activate the IMU to collect the UAV's angular velocity and acceleration data, the GPS to collect the UAV's latitude, longitude, and altitude data, the LiDAR to collect 3D point cloud data around the target point, and the visible light camera to collect image data of the target point; perform data cleaning, data filtering, noise suppression, and data alignment processing on the collected data.

[0068] It should be noted that by planning the UAV's flight path and setting the latitude, longitude, and altitude of the target point on the ground station, precise planning of the UAV mission was achieved, ensuring that the UAV could fly along the predetermined path and improving the efficiency and safety of mission execution. By launching the UAV and performing initial positioning using the inertial navigation system and GPS, the UAV's initial positioning was achieved, laying the foundation for subsequent precise navigation and control and ensuring the stability and reliability of the UAV during takeoff. When the UAV reached the target area, the IMU was activated to collect the UAV's angular velocity and acceleration data, the GPS collected the UAV's latitude, longitude, and altitude data, the LiDAR collected 3D point cloud data around the target point, and the visible light camera collected image data of the target point. This enabled real-time acquisition of multi-sensor data, providing rich environmental information and basic data support for subsequent data processing and fusion. Data cleaning, filtering, noise suppression, and alignment were performed on the collected data, achieving data preprocessing, removing noise and outliers, improving data quality and reliability, and providing high-quality data input for subsequent data fusion and target recognition.

[0069] S200: The processed data is fused using the Kalman filter algorithm to estimate the UAV's attitude in real time, including the following steps.

[0070] The initial state vector includes the UAV's attitude (pitch, roll, and yaw angles) and covariance matrix; the state vector and covariance matrix are updated using GPS and IMU data to obtain a preliminary UAV attitude estimate; the ICP algorithm (Iterative Closest Point Algorithm) is used to register the point cloud data collected by the LiDAR with the map point cloud to obtain the relative pose change; the relative pose change obtained from the point cloud registration is applied to the updated state vector using the Kalman filter to obtain the corrected UAV attitude estimate.

[0071] It should be noted that the Kalman filter was initialized by initializing the state vector and covariance matrix, providing initial conditions for subsequent state estimation and ensuring the stability and convergence of the filter. By updating the state vector and covariance matrix using GPS and IMU data, a preliminary attitude estimate of the UAV was obtained, achieving preliminary fusion of multi-sensor data, improving the accuracy and robustness of attitude estimation, and reducing the impact of errors from individual sensors. The ICP algorithm was used to register the point cloud data collected by the LiDAR with the map point cloud to obtain relative pose changes, achieving accurate registration of LiDAR data and providing high-precision relative pose information, further improving the accuracy of attitude estimation. By applying the relative pose changes obtained from point cloud registration to the updated state vector of the Kalman filter, a corrected attitude estimate of the UAV was obtained, achieving deep fusion of multi-sensor data, further improving the accuracy and robustness of attitude estimation, and ensuring the stability and reliability of the UAV in complex dynamic environments. S300. Constructing a deep learning model to automatically identify the contour and orientation of target objects includes the following steps:

[0072] A large amount of image data containing target objects was collected, and the contour and orientation information of the target objects were labeled to construct a training dataset. A convolutional neural network model was trained using the labeled dataset, and a loss function was used to optimize the bounding box regression and orientation angle classification of the target objects. During the drone's flight, image data was captured in real time and input into the trained convolutional neural network model for target recognition, extracting the contour and orientation information of the target objects, expressed as:

[0073]

[0074] Where O represents the contour and orientation information of the target object, C represents the contour of the target object, I represents the input image, α represents the orientation angle of the target object, i represents the contour index of the target object, N represents the total number of points on the contour of the target object, and x represents the x-axis. i and y i Let x and y be the x and y coordinates of the i-th point on the contour of the target object, respectively. and These are the average x-coordinate and y-coordinate of the contour points, respectively.

[0075] It should be noted that by collecting a large amount of image data containing target objects, annotating the contours and orientation information of the target objects, and constructing a training dataset, a high-quality training dataset was built, providing sufficient data support for the training of the deep learning model and improving the model's generalization ability and recognition accuracy. By training the convolutional neural network model using the labeled dataset and optimizing the bounding box regression and orientation angle classification of the target objects using a loss function, the training and optimization of the deep learning model were achieved, improving the recognition accuracy and orientation estimation accuracy of the target objects and providing reliable recognition results for subsequent target tracking and laser direction adjustment. By capturing image data in real time during the drone's flight and inputting it into the trained convolutional neural network model for target recognition, extracting the contours and orientation information of the target objects, real-time recognition and orientation estimation of the target objects were achieved, improving the system's real-time performance and ensuring accurate operation in dynamic environments.

[0076] S400. Calculating the offset distance from the image center point to the target object based on the object's contour and orientation includes the following steps:

[0077] Based on the outline and orientation information of the target object, determine the coordinates of its center point in the image; use the camera's intrinsic parameters to convert the center point coordinates of the target object in the image into a distance in the physical world, and calculate the offset of the target object in the physical world, expressed as:

[0078]

[0079] Where ΔX is the horizontal offset of the target object in the physical world, ΔY is the vertical offset of the target object in the physical world, w is the width of the image, h is the height of the image, Z is the distance from the target object to the camera, and f x f is the horizontal focal length of the camera. y This is the camera's vertical focal length.

[0080] It should be noted that by determining the center point coordinates of the target object in the image based on its contour and orientation information, the target object is accurately located, providing an accurate reference point for subsequent offset distance calculation. By using the camera's intrinsic parameters to convert the center point coordinates of the target object in the image into distance in the physical world, the offset of the target object in the physical world is calculated, achieving an accurate conversion from image coordinates to the physical world. This provides high-precision offset distance information and an accurate basis for adjusting the laser direction.

[0081] S500. Establishing an environmental model and calculating the compensation parameters for the laser beam direction includes the following steps:

[0082] Using Kalman filter-estimated UAV attitude and position information, the point cloud data collected by LiDAR is transformed into a global coordinate system to form a 3D point cloud map. A meshing method is then used to convert the point cloud data into a 3D model of the surrounding environment. The position of the target point towards the laser beam is determined within the 3D model. Based on the position of the target point, the positions and distances of obstacles along the laser beam path are calculated, expressed as follows:

[0083]

[0084] Where d is the distance to the nearest obstacle on the laser's path, and P global Let p be the coordinates of the point cloud data in the global coordinate system, and p be a point in the point cloud data. target The coordinates of the target point in the global coordinate system;

[0085] Based on the position and distance of obstacles along the laser's path, compensation parameters for the laser trajectory are designed, expressed as follows:

[0086]

[0087] Where Δθ is the pitch angle compensation parameter for the laser beam direction, Δφ is the yaw angle compensation parameter for the laser beam direction, and k d is a scaling factor, and D is the maximum effective distance of the laser beam.

[0088] It should be noted that by using the UAV attitude and position information estimated by the Kalman filter, the point cloud data collected by the LiDAR is transformed into a global coordinate system to form a 3D point cloud map, realizing the global coordinate transformation of the point cloud data and providing accurate 3D data support for environmental modeling. By using a meshing method, the point cloud data is transformed into a 3D model of the surrounding environment, realizing the construction of the environmental model and providing detailed environmental information, which provides a basis for laser trajectory path planning and obstacle detection. By determining the position of the target point of the laser trajectory in the 3D model, the position and distance of obstacles on the laser trajectory path are calculated, realizing the accurate planning of the laser trajectory path, providing a basis for obstacle detection and obstacle avoidance, and ensuring the safety and effectiveness of the laser trajectory. By designing compensation parameters for the laser trajectory based on the position and distance of obstacles on the laser trajectory path, the dynamic adjustment of the laser trajectory is realized, improving the accuracy and robustness of the laser trajectory, and ensuring precise operation in complex environments.

[0089] S600. Based on the attitude estimation results, the offset distance from the image center point to the target object, and the compensation parameters for the laser trajectory, dynamically adjust the laser trajectory, including the following steps:

[0090] The control algorithm is designed, taking the attitude estimation result, the offset distance from the image center point to the target object, and the compensation parameters for the laser direction as input, and outputting the laser angle adjustment command to ensure that the laser always points to the target object. The expression is:

[0091]

[0092] Where, θ cmd For adjusting the pitch angle of the laser, φ cmd This is the yaw angle adjustment command for the laser, where φ is the pitch angle of the UAV, θ is the roll angle of the UAV, and k is the yaw angle. X k is the scaling factor for the horizontal offset. Y This is the scaling factor for the vertical offset.

[0093] Furthermore, θ cmd and φ cmd The value range is [-π, π], representing the pitch and yaw angle adjustment commands of the laser;

[0094] When θ cmd A positive value indicates an upward adjustment, and a negative value indicates a downward adjustment; when φ cmd A positive value indicates adjustment to the right, while a negative value indicates adjustment to the left.

[0095] It should be noted that by designing a control algorithm, the attitude estimation result, the offset distance from the image center point to the target object, and the compensation parameters of the laser direction are taken as inputs, and the laser angle adjustment command is output, realizing the dynamic adjustment of the laser direction. This ensures that the laser always points to the target object, maintaining accuracy even when the UAV or the target object is moving. By taking the attitude estimation result, the offset distance from the image center point to the target object, and the compensation parameters of the laser direction as inputs, and outputting the laser angle adjustment command, multiple factors are comprehensively considered, improving the accuracy and robustness of the laser direction and ensuring precise operation in complex dynamic environments.

[0096] This embodiment also provides an automatic laser direction adjustment system based on a drone, including:

[0097] The system comprises the following modules: a data acquisition module, which acquires data in real time from IMU (Inertial Measurement Unit), GPS, LiDAR, and visible light camera, and performs preliminary processing; a data fusion module, which fuses the processed data using a Kalman filter algorithm to estimate the UAV's attitude in real time; a target recognition module, which builds a deep learning model to automatically identify the contour and orientation of target objects; an offset calculation module, which calculates the offset distance from the image center point to the target object based on the object's contour and orientation; an environment modeling module, which establishes an environment model and calculates compensation parameters for the laser beam; and a laser beam adjustment module, which dynamically adjusts the laser beam direction based on the attitude estimation results, the offset distance from the image center point to the target object, and the compensation parameters for the laser beam direction.

[0098] This embodiment also provides a computer device applicable to the automatic laser direction adjustment method based on UAVs, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the automatic laser direction adjustment method based on UAVs as proposed in the above embodiment.

[0099] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0100] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the automatic laser direction adjustment method based on a UAV as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0101] In summary, this invention achieves the following: real-time acquisition and preliminary processing of data from IMU, GPS, LiDAR, and visible light cameras; fusion of the processed data using a Kalman filter algorithm to estimate the UAV's attitude in real time; construction of a deep learning model to automatically identify the contour and orientation of target objects; calculation of the offset distance from the image center point to the target object based on the object's contour and orientation; establishment of an environmental model to calculate compensation parameters for the laser trajectory; dynamic adjustment of the laser trajectory based on the attitude estimation results, the offset distance from the image center point to the target object, and the compensation parameters; and determination of the target object's center point coordinates in the image based on its contour and orientation information, providing an accurate reference point for subsequent offset distance calculations. The system can adjust the laser trajectory in real time, maintaining precise laser pointing even when the UAV or the target object is moving.

[0102] Example 2, referring to Table 1, is the second embodiment of the present invention. To further verify the technical solution of the present invention, experimental simulation data of the automatic laser direction adjustment method based on UAV are given.

[0103] The target point was set at latitude and longitude coordinates (116.4074, 39.9042) and altitude of 100 meters at the ground station. A straight flight path was planned from the starting point (116.4070, 39.9040, altitude 50 meters) to the target point.

[0104] Launch the drone and perform initial positioning using the Inertial Navigation System (INS) and Global Positioning System (GPS). After takeoff, the drone uses GPS data for initial positioning to ensure it is at the predetermined starting point.

[0105] When the UAV reaches the target area, the inertial measurement unit (IMU) is activated to collect the UAV's angular velocity and acceleration data, the GPS collects the UAV's latitude, longitude, and altitude data, the lidar collects 3D point cloud data around the target point, and the visible light camera collects image data of the target point. The collected data undergoes data cleaning, filtering, noise suppression, and alignment to ensure accuracy and integrity.

[0106] Initialize the state vector and covariance matrix, and update the state vector and covariance matrix using GPS data and IMU data to obtain a preliminary UAV attitude estimate.

[0107] The Iterative Closest Point (ICP) algorithm is used to register the point cloud data acquired by the LiDAR with the map point cloud to obtain the relative pose change.

[0108] The relative pose change obtained from point cloud registration is applied to the updated state vector of the Kalman filter to obtain the corrected UAV attitude estimate.

[0109] Collect a large amount of image data containing the target object, annotate the outline and orientation information of the target object, and construct a training dataset.

[0110] A convolutional neural network (CNN) model is trained using a labeled dataset, and a loss function is used to optimize the bounding box regression and orientation angle classification of the target object.

[0111] During the drone's flight, image data is captured in real time and input into a trained CNN model for target recognition, extracting the contour and orientation information of the target object.

[0112] Based on the outline and orientation information of the target object, determine the coordinates of the center point of the target object in the image.

[0113] The camera's intrinsic parameters are used to convert the center point coordinates of the target object in the image into distance in the physical world, and the horizontal and vertical offsets of the target object in the physical world are calculated.

[0114] The attitude and position information of the UAV estimated by the Kalman filter is used to transform the point cloud data collected by the lidar into the global coordinate system to form a three-dimensional point cloud map.

[0115] The point cloud data is transformed into a 3D model of the surrounding environment using a meshing method.

[0116] Determine the position of the target point of the laser beam in the 3D model, and calculate the position and distance of obstacles along the laser beam path.

[0117] Based on the position and distance of obstacles in the laser beam path, design compensation parameters for the laser beam direction.

[0118] The control algorithm is designed to take the attitude estimation result, the offset distance from the image center point to the target object, and the compensation parameters of the laser direction as inputs, and output the laser angle adjustment command to ensure that the laser always points to the target object, maintaining accuracy even when the drone or the target object is moving.

[0119] The details are shown in Table 1 below:

[0120] Table 1 Experimental Record Data Table

[0121]

[0122]

[0123] Comparative analysis of the data in the table above clearly shows that the attitude estimation accuracies of existing technologies 1 and 2 are 0.5 meters and 0.6 meters, respectively, while the attitude estimation accuracies of solutions 1 to 4 of this invention are all below 0.2 meters, with the highest reaching 0.1 meters. This indicates that this invention, through the Kalman filtering algorithm and multi-sensor data fusion, significantly improves the attitude estimation accuracy of the UAV and reduces the impact of errors from individual sensors.

[0124] The target recognition accuracies of existing technologies 1 and 2 are 10 pixels and 12 pixels, respectively, while the target recognition accuracies of solutions 1 to 4 of this invention are all below 6 pixels, with the lowest reaching 4 pixels. This indicates that this invention, by constructing a deep learning model, automatically identifies the contour and orientation of target objects, significantly improving the accuracy and robustness of target recognition.

[0125] The offset distance calculation errors of existing technologies 1 and 2 are 0.3 meters and 0.4 meters, respectively, while the offset distance calculation errors of solutions 1 to 4 of this invention are all below 0.12 meters, with the lowest reaching 0.05 meters. This indicates that this invention significantly improves the accuracy of offset distance calculation through precise image processing and camera intrinsic parameter correction.

[0126] The laser aiming accuracies of existing technologies 1 and 2 are 0.4 meters and 0.5 meters, respectively, while the laser aiming accuracies of solutions 1 to 4 of this invention are all below 0.15 meters, with the lowest reaching 0.05 meters. This indicates that this invention, by dynamically adjusting the laser aiming, ensures that the laser always points to the target object, maintaining accuracy even when the drone or the target object is moving.

[0127] The environmental modeling errors of existing technologies 1 and 2 are 0.2 meters and 0.3 meters, respectively, while the environmental modeling errors of solutions 1 to 4 of this invention are all below 0.12 meters, with the lowest reaching 0.05 meters. This indicates that this invention significantly improves the accuracy of the environmental model through precise registration of lidar data and environmental modeling.

[0128] The overall performance scores of existing technologies 1 and 2 are 6 and 5, respectively, while the overall performance scores of solutions 1 to 4 of this invention are all above 8, with the highest reaching 10. This indicates that the present invention exhibits higher overall performance and reliability in practical applications and can better adapt to the task requirements in complex and dynamic environments.

[0129] In summary, this invention significantly improves the autonomous navigation and target recognition capabilities of UAVs through multi-sensor data fusion, deep learning models, and environmental modeling techniques, particularly demonstrating higher accuracy in the automatic adjustment of laser direction.

[0130] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for automatic laser direction adjustment based on unmanned aerial vehicles (UAVs), characterized in that: include, Real-time acquisition of data from IMU, GPS, LiDAR, and visible light cameras, followed by preliminary processing; The processed data is fused using the Kalman filter algorithm to estimate the UAV's attitude in real time. Build deep learning models to automatically identify the outline and orientation of target objects; Calculate the offset distance from the image center point to the target object based on the object's outline and orientation. The steps to establish an environmental model and calculate the compensation parameters for the laser beam direction are as follows: The attitude and position information of the UAV estimated by the Kalman filter is used to transform the point cloud data collected by the lidar into the global coordinate system to form a three-dimensional point cloud map. The point cloud data is transformed into a 3D model of the surrounding environment using a meshing method. Determine the location of the target point of the laser beam in the 3D model; Based on the position of the laser beam aimed at the target point, the position and distance of obstacles along the laser beam's path are calculated using the following expression: ； in, The distance to the nearest obstacle on the laser's path. The coordinates of the point cloud data in the global coordinate system. For a point in point cloud data, The coordinates of the target point in the global coordinate system; Based on the position and distance of obstacles along the laser's path, compensation parameters for the laser trajectory are designed, expressed as follows: ； in, The elevation angle compensation parameter for the laser beam direction. The yaw angle compensation parameter for the laser beam direction. This is the proportionality coefficient. This represents the maximum effective distance of the laser beam. The laser direction is dynamically adjusted based on the attitude estimation results, the offset distance from the image center point to the target object, and the compensation parameters of the laser direction. Dynamically adjusting the laser direction includes the following steps: The control algorithm is designed, taking the pose estimation result, the offset distance from the image center point to the target object, and the compensation parameters of the laser direction as input, and outputting the laser angle adjustment command, expressed as: ； in, This is a command to adjust the laser's pitch angle. This is the command to adjust the yaw angle of the laser. The pitch angle of the drone. For the roll angle of the drone, This is the scaling factor for the horizontal offset. This is the scaling factor for the vertical offset; This represents the horizontal offset of the target object in the physical world. This represents the vertical offset of the target object in the physical world.

2. The automatic laser direction adjustment method based on UAV as described in claim 1, characterized in that: The real-time acquisition and preliminary processing of data from IMU, GPS, LiDAR, and visible light cameras includes the following steps. Plan the flight path of the drone and set the latitude, longitude coordinates and altitude of the target point on the ground station; Launch the drone and perform initial positioning using the inertial navigation system and GPS; When the drone reaches the target area, the IMU is activated to collect the drone's angular velocity and acceleration data, the GPS is used to collect the drone's latitude and longitude coordinates and altitude data, the lidar is used to collect the three-dimensional point cloud data around the target point, and the visible light camera is used to collect the image data of the target point. The collected data undergoes data cleaning, data filtering, noise suppression, and data alignment.

3. The automatic laser direction adjustment method based on UAV as described in claim 2, characterized in that: The process of fusing the processed data using the Kalman filter algorithm to estimate the UAV's attitude in real time includes the following steps. Initialize the state vector and covariance matrix; The state vector and covariance matrix are updated using GPS data and IMU data to obtain a preliminary attitude estimate of the UAV. The ICP algorithm is used to register the point cloud data collected by the lidar with the map point cloud to obtain the relative pose change. The relative pose change obtained from point cloud registration is applied to the updated state vector of the Kalman filter to obtain the corrected attitude estimate of the UAV.

4. The automatic laser direction adjustment method based on UAV as described in claim 3, characterized in that: Building a deep learning model to automatically identify the contour and orientation of a target object includes the following steps: Collect a large amount of image data containing the target object, annotate the outline and orientation information of the target object, and construct a training dataset; A convolutional neural network model is trained using a labeled dataset, and a loss function is used to optimize the bounding box regression and orientation angle classification of the target object. During the drone's flight, image data is captured in real time and input into a trained convolutional neural network model for target recognition, extracting the contour and orientation information of the target object. The expression is as follows: ； in, This provides the outline and orientation information of the target object. The outline of the target object. For the input image, The direction angle of the target object. Index the outline of the target object. The total number of points on the outline of the target object. and The first one on the outline of the target object The x and y coordinates of each point, and These are the average x-coordinate and y-coordinate of the contour points, respectively.

5. The automatic laser direction adjustment method based on UAV as described in claim 4, characterized in that: Calculating the offset distance from the image center point to the target object based on the object's outline and orientation includes the following steps: Based on the outline and orientation information of the target object, determine the coordinates of the center point of the target object in the image; Using the camera's intrinsic parameters, the center point coordinates of the target object in the image are converted to their distance in the physical world. The offset of the target object in the physical world is then calculated using the following expression: ； in, This represents the horizontal offset of the target object in the physical world. This represents the vertical offset of the target object in the physical world. The width of the image. The height of the image. The distance from the target object to the camera. The horizontal focal length of the camera. This is the camera's vertical focal length.

6. A UAV-based automatic laser direction adjustment system, based on the UAV-based automatic laser direction adjustment method according to any one of claims 1 to 5, characterized in that: include, The data acquisition module collects data from IMU, GPS, LiDAR, and visible light camera in real time and performs preliminary processing. The data fusion module uses the Kalman filter algorithm to fuse the processed data and estimate the attitude of the UAV in real time. The target recognition module builds a deep learning model to automatically identify the outline and orientation of target objects; The offset calculation module calculates the offset distance from the image center point to the target object based on the object's outline and orientation. The environmental modeling module builds an environmental model and calculates compensation parameters for the laser beam direction. The laser trajectory adjustment module dynamically adjusts the laser trajectory based on the attitude estimation results, the offset distance from the image center point to the target object, and the compensation parameters of the laser trajectory.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the automatic laser direction adjustment method based on any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the automatic laser direction adjustment method based on any one of claims 1 to 5.

Images